allocbench - benchmark tool for POSIX memory allocators
allocbench is a POSIX memory allocator benchmarking framework and tooling. It was used in Florian Fischer's BA thesis.
What can allocbench do for you ?
- Deterministically builds and patches allocators from git or source archives
- Couples allocators with various included benchmarks as well as your custom ones
- Supports you in analyzing your benchmark results by providing statistical and plot helper functions
- Comes with support for two different malloc tracers to help you understand benchmark and allocator behavior
- Contains numerous wildly used benchmarks in allocator research (espresso, cfrac, larson, ...)
- It is extended easily with our own allocators and or custom benchmarks
To obtain allocbench run
git clone https://muhq.space/software/allocbench.gitRequirements
- python >= 3.8
- make, find, gcc (build dependencies)
- perf (
perf stat -dis the default command to measure benchmark results) - util-linux (
whereisis used to find system installed allocators) - git, tar to handle external artifacts
- numpy, scipy and matplotlib to summarize results and generate plots
Usage
allocbench consists of three small utilities: bench.py, summarize.py and merge.py.
bench.py is used to prepare, analyze and run benchmarks.
usage: bench.py [-h] [--analyze] [-r RUNS] [-v]
[-b BENCHMARKS [BENCHMARKS ...]]
[-xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
[-a ALLOCATORS [ALLOCATORS ...]] [-rd RESULTDIR] [--license]
[--version]
benchmark memory allocators
optional arguments:
-h, --help show this help message and exit
--analyze analyze benchmark behavior using malt
-r RUNS, --runs RUNS how often the benchmarks run
-v, --verbose more output
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to run
-xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
explicitly excluded benchmarks
-a ALLOCATORS [ALLOCATORS ...], --allocators ALLOCATORS [ALLOCATORS ...]
allocators to test
-rd RESULTDIR, --resultdir RESULTDIR
directory where all results go
--license print license info and exit
--version print version info and exit./summarize.py is used to summarize results created with bench.py.
It groups the included allocators into categories to produce readable and not extremely noisy plots.
usage: summarize.py [-h] [-t FILE_EXT] [--license] [--version]
[-b BENCHMARKS [BENCHMARKS ...]]
[-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
results
Summarize allocbench results in allocator sets
positional arguments:
results path to results
optional arguments:
-h, --help show this help message and exit
-t FILE_EXT, --file-ext FILE_EXT
file extension used for plots
--license print license info and exit
--version print version info and exit
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to summarize
-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
benchmarks to exclude./merge.py can combine the results of different benchmark runs.
usage: merge.py [-h] [--license] [--version] [-b BENCHMARKS [BENCHMARKS ...]]
[-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
src dest
Merge to allocbench results
positional arguments:
src results which should be merged into dest
dest results in which src should be merged
optional arguments:
-h, --help show this help message and exit
--license print license info and exit
--version print version info and exit
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to summarize
-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
benchmarks to excludeExamples
./bench.py -b loopruns only the loop benchmark for all allocators found on the system and will put its
results in $PWD/results/$HOSTNAME/<time>/loop.
./bench.py -a "tcmalloc*"builds all tcmalloc variants shipped with allocbench and runs all benchmarks.
./summarize.py <path/to/saved/results>summarizes the previously created results.
Benchmarks
You want to compare allocators with your own software or add a new benchmark, have a look at doc/Benchmarks.md.
Allocators
By default all included allocators will be build and measured. For more precise control
about used allocators use the -a option.
For more details about what allocators are available, how they are used and how to include new once have a look at doc/Allocators.md.
License
This program is released under GPLv3. You can find a copy of the license in the LICENSE file in the programs root directory.