* Merge the two echo functions echo and linkedEcho into a single templated
  Client member function _run(). This reduces code duplication but the resulting
  echo loop is less readable because of more if constexpr switches.
* Move more code out of the echo loop into separate functions:
  * unexpected echo message
  * on ECONNRESET handling
  * error handling
  * termination condition

* Shutdown the client connections after the echo phase has ended.
* Send "quit\n" on the last client connection (Though it is not guarantied
  that there are no more open connection because we parallelize the termination)
* Free memory after the Runtime has terminated

The break was not in the right scope and thus exiting the echo loop
immediately in the first iteration.

Since stats are printed in the Runtime destructor we can not simply call
exit() and have to properly terminate the Runtime.
When the echo server receives a "quit" message all active connections will
return and the Runtime will terminate if there is no work left.

Histograms can only be collected when using a fixed amount of iterations.

When the '--histogram <file>' argument is passed each Client
collects 4 time stamps (each 8 byte):

1. Before requesting the send operation
2. After requesting the send operation
3. After getting unblocked and dispatched because the send operation finished
4. After getting unblocked and dispatched because the recv operation finished

Taking the timestamps is enabled using a template and thus does not introduce
any runtime cost if they are not used except binary size.

Before termination three latencies are calculated and written to the histogram
file as csv data for each client and each echo.

1. total_latency := (T4 - T1)
2. after_send_latency := (T4 - T2)
3. after_send_dispatch_latency := (T4 - T3)

This makes looking at individual experiment results and identifying
outliers easier.

Split output between two ini sections: global, clients.
The global section includes results like:
Arguments, total time spent in each phase, ...

The clients section contains a sigle key 'csv' containing the csv data
from each client inclufing the header:
iterations, latency, reconnects, unexpected echos

This leaves the post processing of the data to another program
and does not hide experiment behavior (e.g. starvation of single
clients) behind averages.

Remove support for appending to an output file, because we do not
use it in our emper-io-evaluation and it does not work anymore with the
new output format.

* Unconditionally reconnect when the TCP connection was reset and count the reconnects
* Send Client id and iteration in each echo
* Use a Client class holding a clients state instead of a pthread style clientFunc
* Create the Client object in the client Fiber to use NUMA first-touch policy
* Reduce code duplication:
  * starting a new Client is now done with Client::startNew
  * latency measurement and error handling are moved from the clientIteration*
    functions in the Client::run function

My first test runs don't show unexpected echos. Therefore I think
unexpected echos are rare and their average will always be 0 even if
some sparse unexpected echos occur.

* Use emper::lib::math::RunningAverage instead of doing it manually
* Count the unexpected Echos, print a error to the log and go continue
* Return a duration and error message pair from our clientIteration* functions
* Print the current iteration on error
* Wait 10 seconds after an error before terminating

Build up output string and don't incrementally output it to stderr
to hopefully prevent error messages interleaving.
Only output the least significant byte of the unsigned int cast.
Always use two digits per byte and remove the whitespace.
Should have used printf in the first place.

Rename the macro used to build up the debug message, to better reflect
its generality.
It can be used anywhere to build a string with C++ stream formatting.
std::string s = EMPER_BUILD_STR("foo = " << 1 " , bar = " << bar(42));

Since WDBG was the only log macro without a terminating newline, which
was in fact broken according to flow, we can remove it and the new line
handling code in the LOG macro.

While we haven't figured out why the version using linked futures breaks
we can use a simpler implementation which calls submitAndWait on both futures
separately

First of all this replaces the call to sleep(execution_seconds)
which calls POSIX sleep(3) which is definitely not what we want here
blocking one worker thread.

Using a std::thread for termination was chosen because we assume it
will be more precise than the emper native emper::sleep.

Specifying execution time and iterations is not allowed.

* Open file only when the benchmark was successful
* Append to existing files without writing the header

Nagle's algorithm which tries to prevent small TCP frames is harmful
for our throughput if we send small echos.

Arithmetic means are inaccurate if the sample has extreme outliers
therefore we record and report the total execution times.

This is useful for my evaluation. Because with an output file I
just have to check if it exists to see if an echoclient execution was
successful.

Use getaddrinfo instead to get the socket information instead of
using always ipv4 sockets.
Pass host and port as strings for use with getaddrinfo.

Start client Fibers in batches to make sure we don't exceed the server's
TCP listen backlog.

Florian Fischer authored 4 years ago and Florian Schmaus committed 4 years ago

Modified-by: Florian Schmaus <flow@cs.fau.de>

The echo client established X connections and start the
echo phase after all sockets are connected.

Each client Fiber measures the time from sending the message until
receiving the echo.

We are unable to switch to C++20 until
https://github.com/mesonbuild/meson/issues/8084 is fixed.

This also solves a dependency declaration issue in WorkerSleepExample:
Prior to this change, a clean build could potentially result in

ninja  -C build
ninja: Entering directory `build'
[13/57] Compiling C++ object apps/worker_sleep_example.p/WorkerSleepExample.cpp.o
FAILED: apps/worker_sleep_example.p/WorkerSleepExample.cpp.o
ccache c++ -Iapps/worker_sleep_example.p -Iapps -I../apps -Iemper -I../emper -Iemper/include -I../emper/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std=c++17 -O2 -g -Wno-non-virtual-dtor -MD -MQ apps/worker_sleep_example.p/WorkerSleepExample.cpp.o -MF apps/worker_sleep_example.p/WorkerSleepExample.cpp.o.d -o apps/worker_sleep_example.p/WorkerSleepExample.cpp.o -c ../apps/WorkerSleepExample.cpp
../apps/WorkerSleepExample.cpp:12:10: fatal error: emper-version.h: No such file or directory
   12 | #include "emper-version.h"        // for EMPER_FULL_VERSION
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
[17/57] Generating emper-version.h with a custom command
ninja: build stopped: subcommand failed.
make: *** [Makefile:23: build] Error 1

because worker_sleep_example_exec should have also depend on
emper_version_h. However this is obviously error prone, as users
easily forget to add this dependency. Instead we add
emper::getFullVersion() which is part of the EMPER shared object (not
just of a single header).