emper merge requestshttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests2021-12-02T15:58:00Zhttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/280EchoClient: improve the help message2021-12-02T15:58:00ZFlorian FischerEchoClient: improve the help messagehttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/279add concurrent BPS test2021-11-24T17:36:10ZFlorian Fischeradd concurrent BPS testThe test introduces multiple cycles of Semaphores and
a Fiber for each semaphore blocking and signaling the next.
Through work-stealing the fibers from a cycle should be spread
across different workers and thus test concurrent use of
Bin...The test introduces multiple cycles of Semaphores and
a Fiber for each semaphore blocking and signaling the next.
Through work-stealing the fibers from a cycle should be spread
across different workers and thus test concurrent use of
BinaryPrivateSemaphores.
Cycle of length 3: Sem A -> Sem B -> Sem C -> Sem A -> ...
Algorithm:
if isFirstInCycle
signal next
wait
if not isFirstInCycle
signal nexthttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/278echoclient: add a state variable for debugging2021-11-24T15:00:11ZFlorian Fischerechoclient: add a state variable for debugginghttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/277[PipeSleepStrategy] fix notifyFromAnywhere2021-11-15T18:30:50ZFlorian Fischer[PipeSleepStrategy] fix notifyFromAnywhereDon't decrease the sleeper count in the CAS loop further than `-count`, which is the threshold we need to ensure that the notification
will be observed.
Decreasing it further than our threshold is not faulty it just results in
unnecessar...Don't decrease the sleeper count in the CAS loop further than `-count`, which is the threshold we need to ensure that the notification
will be observed.
Decreasing it further than our threshold is not faulty it just results in
unnecessary skipped sleeps.
Don't call `writeNotifications` with a negative count.
Which will be interpreted as an unsigned value and thus results
in writing way to much hints to the pipe and jamming it.
If the original value before a successfully CAS is already negative
we called `writeNotifications` with this negative value.
This is fixed by using the `max(toWakeup, 0)`.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/276make the victim count in work-stealing configurable2021-11-11T15:26:17ZFlorian Fischermake the victim count in work-stealing configurableAdd two new mutual exclusive meson_options:
* work_stealing_victim_count: Which sets an absolute number of victims
* work_stealing_victim_denominator: Set victim count to #workers/denominatorAdd two new mutual exclusive meson_options:
* work_stealing_victim_count: Which sets an absolute number of victims
* work_stealing_victim_denominator: Set victim count to #workers/denominatorhttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/275[tools] Update check-format (from Mazstab)2021-11-10T12:14:19ZFlorian Schmaus[tools] Update check-format (from Mazstab)Sync tools/check-format of EMPER and Mazstab by using the newer
Mazstab version of the script.Sync tools/check-format of EMPER and Mazstab by using the newer
Mazstab version of the script.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/274Fixes for clang-tidy 132021-11-10T12:13:52ZFlorian SchmausFixes for clang-tidy 13While we do not have yet LLVM 13 in the gitlab-ci, I use it on my
systems. So fix the new warnings found with clang-tidy 13.While we do not have yet LLVM 13 in the gitlab-ci, I use it on my
systems. So fix the new warnings found with clang-tidy 13.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/272Random computation echoserver2021-10-29T08:16:02ZFlorian FischerRandom computation echoserverTo expose the benefits of io-stealing or a completer thread it seams helpful to have a echo server with differnt computation times. So that outstanding work on workers currently executing long computations can be reintroduced to the syst...To expose the benefits of io-stealing or a completer thread it seams helpful to have a echo server with differnt computation times. So that outstanding work on workers currently executing long computations can be reintroduced to the system via io-stealing or the completer improving the overall throughput and reducing latencies.
This implementation is really simple and biased.
Possible improvements:
* Use thread_local random number generator to produce even distribution.
* Use only two fixed times and select one of both with a given probability.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/273Ci debian testing dev bump2021-10-28T17:10:59ZFlorian SchmausCi debian testing dev bumphttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/260[RFC] IO-stealing analogue to work-stealing2021-10-13T19:41:24ZFlorian Fischer[RFC] IO-stealing analogue to work-stealingIO stealing is analog to work-stealing and means that worker thread
without work will try to steal IO completions (CQEs) from other worker's
IoContexts. The work stealing algorithm is modified to check a victims
CQ after findig their wor...IO stealing is analog to work-stealing and means that worker thread
without work will try to steal IO completions (CQEs) from other worker's
IoContexts. The work stealing algorithm is modified to check a victims
CQ after findig their work queue empty.
This approach in combination with future additions (global notifications
on IO completions, and lock free CQE consumption) are a realistic candidate
to replace the completer thread without loosing its benefits.
To allow IO stealing the CQ must be synchronized which is already the
case with the `IoContext::cq_lock`.
Currently stealing workers always try to pop a single CQE (this could
be configurable).
Steal attempts are recorded in the IoContext's Stats object and
successfully stolen IO continuations in the `AbstractWorkStealingWorkerStats`.
I moved the code transforming CQEs into continuation Fibers from
reapCompletions into a seperate function to make the rather complicated
function more readable and thus easier to understand.
Remove the default CallerEnvironment template arguments to make
the code more explicit and prevent easy errors (not propagating
the caller environment or forgetting the function takes a caller environment).
`io::Stats` now need to use atomics because multiple thread may increment
them in parallel from EMPER and the OWNER.
And since using `std::atomic<T*>` in `std::map` is not easily possible we
use the compiler `__atomic_*` builtins.
Add, adjust and fix some comments.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/271[fsearch] remove obsolete offset tracking code2021-10-13T15:32:41ZFlorian Fischer[fsearch] remove obsolete offset tracking codehttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/269print runtime stats to the environment variable EMPER_STATS_FILE2021-10-11T11:30:29ZFlorian Fischerprint runtime stats to the environment variable EMPER_STATS_FILE* Make all stats print methods accept a `std::ostream` as output.
* Move the printing of runtime component stats into`Runtime::printStats`.
* Use `Runtime::printStats` instead of `Runtime::printLastRuntimeStats` in
`~Runtime`, because ...* Make all stats print methods accept a `std::ostream` as output.
* Move the printing of runtime component stats into`Runtime::printStats`.
* Use `Runtime::printStats` instead of `Runtime::printLastRuntimeStats` in
`~Runtime`, because we are already in a runtime which may differ from
`Runtime::currentRuntime`.
* Write the stats in `~Runtime` to a possible file passed in the
environment variable `EMPER_STATS_FILE`https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/270[sleep_strategy/Stats] remove old obsolete stats member2021-10-11T09:12:34ZFlorian Fischer[sleep_strategy/Stats] remove old obsolete stats memberBefore 713c0f047d25be1c1f5ccf7eaf3c18a0af6918d2 `emper::sleep_strategy::Stats`
had the actual stats member but with 713c0f047d25be1c1f5ccf7eaf3c18a0af6918d2
`WorkerStats` are introduced which have the actual stats member but I
apparently...Before 713c0f047d25be1c1f5ccf7eaf3c18a0af6918d2 `emper::sleep_strategy::Stats`
had the actual stats member but with 713c0f047d25be1c1f5ccf7eaf3c18a0af6918d2
`WorkerStats` are introduced which have the actual stats member but I
apparently forgot to remove the old now obsolete members.
Thanks clang for detecting this!
See: https://gitlab.cs.fau.de/aj46ezos/emper/-/jobs/456981https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/268[Common.hpp] fix CACHE_LINE_EXCLUSIVE macro2021-10-11T09:06:50ZFlorian Fischer[Common.hpp] fix CACHE_LINE_EXCLUSIVE macroThe macro did not replace symbol in __symbol_mem and always
created the literal symbol __symbol_mem which makes the
macro unusable in the same scope twice.The macro did not replace symbol in __symbol_mem and always
created the literal symbol __symbol_mem which makes the
macro unusable in the same scope twice.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/267[AbstractWorkStealingStats] actually use hint stats2021-10-11T09:06:07ZFlorian Fischer[AbstractWorkStealingStats] actually use hint statshttps://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/266[WakeupStrategy] fix the throttle algorithm for notifiaction from anywhere2021-10-11T09:04:34ZFlorian Fischer[WakeupStrategy] fix the throttle algorithm for notifiaction from anywhereThe throttle algorithm had the same problem like our sleep algorithms
where notifications from anywhere race with a worker going to
sleep resulting in lost wakeups.
In our sleep strategies we prevent those races by preventing sleep attem...The throttle algorithm had the same problem like our sleep algorithms
where notifications from anywhere race with a worker going to
sleep resulting in lost wakeups.
In our sleep strategies we prevent those races by preventing sleep attempts
when notifying from anywhere.
The throttle algorithm also does now exactly this. A notifier from anywhere
will now always set the WakeupStrategy state to notified.
If the state was previously pending this new approach does not differ from
the previous behavior and a sleeping worker will be notified.
If the state was waking the waking worker skips its sleep if it observes
the WakeupStrategy state as notified.
Fixes #26.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/265[log] improve timestamp scalability and increase LogBuffer size2021-09-27T12:09:17ZFlorian Fischer[log] improve timestamp scalability and increase LogBuffer sizestd::localtime takes a global lock and is therefore not scalable and
inapplicable for analyzing timing sensible bugs.
Introduce a new option to add UTC timestamps. This allows on my system
to double the CPU load while using mmapped loggi...std::localtime takes a global lock and is therefore not scalable and
inapplicable for analyzing timing sensible bugs.
Introduce a new option to add UTC timestamps. This allows on my system
to double the CPU load while using mmapped logging.
Also increase the LogBuffer size from 1MB to 1GB because I had some
crashes where a renewed buffer was still used.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/264[EchoServer] set SO_REUSEPORT on the listen socket2021-09-25T16:57:16ZFlorian Fischer[EchoServer] set SO_REUSEPORT on the listen socketThis is needed by emper-io-eval because apparently our startup/termination
times are not enough for the OS to allow the rebinding of the same tcp tuple.
Also make all global variables static because they don't have to be exported.This is needed by emper-io-eval because apparently our startup/termination
times are not enough for the OS to allow the rebinding of the same tcp tuple.
Also make all global variables static because they don't have to be exported.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/262[ConcurrentNetworkEchoTest] scale work with the available CPUs and log_level2021-09-24T12:02:43ZFlorian Fischer[ConcurrentNetworkEchoTest] scale work with the available CPUs and log_levelThe chosen amounts of echos take on my 16 core ryzen system ~5seconds.
This should hopefully reduce CI timeouts where we are not sure if they are
bugs or legit timeouts. Furthermore this should reduce the amount of logs
we write and have...The chosen amounts of echos take on my 16 core ryzen system ~5seconds.
This should hopefully reduce CI timeouts where we are not sure if they are
bugs or legit timeouts. Furthermore this should reduce the amount of logs
we write and have to store after each CI run.https://gitlab.cs.fau.de/i4/manycore/emper/-/merge_requests/261[SimpleDiskAndNetworkTest] don't terminate the Runtime during the test2021-09-24T11:14:34ZFlorian Fischer[SimpleDiskAndNetworkTest] don't terminate the Runtime during the testBefore 05dc56ed the two test cases testDiskAndNetwork, testIov
where combined in one function which was terminating the runtime after the client
closed its socketafter the test was done.
This was already broken before 05dc56ed but after ...Before 05dc56ed the two test cases testDiskAndNetwork, testIov
where combined in one function which was terminating the runtime after the client
closed its socketafter the test was done.
This was already broken before 05dc56ed but after the separation of the
two use cases the race between the Runtime terminating and the execution of
the last parts of the test-runner's alphaFiber became bigger.
When the Runtime terminates because the last worker running the testIoV
code has outstanding IO the test-runner's successSem is never notified.
Fixes #25. Makes !259 obsolete.