Readme.md 5.3 KB
Newer Older
Florian Fischer's avatar
Florian Fischer committed
1
# allocbench - benchmark tool for POSIX memory allocators
Florian Fischer's avatar
Florian Fischer committed
2

Florian Fischer's avatar
Florian Fischer committed
3
allocbench is a POSIX memory allocator benchmarking framework and tooling.
4
5
6
7
8
9
10
11
12
13
It was used in Florian Fischer's [BA thesis](https://muhq.space/ba.html).

What can allocbench do for you ?

* Deterministically builds and patches allocators from git or source archives
* Couples allocators with various included benchmarks as well as your custom ones
* Supports you in analyzing your benchmark results by providing statistical and plot helper functions
* Comes with support for two different malloc tracers to help you understand benchmark and allocator behavior
* Contains numerous wildly used benchmarks in allocator research (espresso, cfrac, larson, ...)
* It is extended easily with our own allocators and or custom benchmarks
Florian Fischer's avatar
Florian Fischer committed
14
15

To obtain allocbench run
16

Florian Fischer's avatar
Florian Fischer committed
17
18
19
20
```shell
git clone https://muhq.space/software/allocbench.git
```

Florian Fischer's avatar
Florian Fischer committed
21
22
## Requirements

23
* python >= 3.8
24
* make, find, gcc (build dependencies)
Florian Fischer's avatar
Florian Fischer committed
25
26
* perf (`perf stat -d` is the default command to measure benchmark results)
* util-linux (`whereis` is used to find system installed allocators)
Florian Fischer's avatar
Florian Fischer committed
27
* git, tar to handle external artifacts
28
* numpy, scipy and matplotlib to summarize results and generate plots
Florian Fischer's avatar
Florian Fischer committed
29
30


Florian Fischer's avatar
Florian Fischer committed
31
## Usage
Florian Fischer's avatar
Florian Fischer committed
32
33
allocbench consists of three small utilities: `bench.py`, `summarize.py` and `merge.py`.
`bench.py` is used to prepare, analyze and run benchmarks.
Florian Fischer's avatar
Florian Fischer committed
34

Florian Fischer's avatar
Florian Fischer committed
35
36
	usage: bench.py [-h] [--analyze] [-r RUNS] [-v]
	                [-b BENCHMARKS [BENCHMARKS ...]]
Florian Fischer's avatar
Florian Fischer committed
37
	                [-xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
Florian Fischer's avatar
Florian Fischer committed
38
39
	                [-a ALLOCATORS [ALLOCATORS ...]] [-rd RESULTDIR] [--license]
	                [--version]
Florian Fischer's avatar
Florian Fischer committed
40

Florian Fischer's avatar
Florian Fischer committed
41
42
43
44
	benchmark memory allocators

	optional arguments:
	  -h, --help            show this help message and exit
Florian Fischer's avatar
Florian Fischer committed
45
	  --analyze             analyze benchmark behavior using malt
Florian Fischer's avatar
Florian Fischer committed
46
47
48
49
50
51
52
53
54
55
56
	  -r RUNS, --runs RUNS  how often the benchmarks run
	  -v, --verbose         more output
	  -b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
	                        benchmarks to run
	  -xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
	                        explicitly excluded benchmarks
	  -a ALLOCATORS [ALLOCATORS ...], --allocators ALLOCATORS [ALLOCATORS ...]
	                        allocators to test
	  -rd RESULTDIR, --resultdir RESULTDIR
	                        directory where all results go
	  --license             print license info and exit
Florian Fischer's avatar
Florian Fischer committed
57
58
59
60
61
	  --version             print version info and exit

`./summarize.py` is used to summarize results created with bench.py.
It groups the included allocators into categories to produce readable and not extremely noisy plots.

62
	usage: summarize.py [-h] [-t FILE_EXT] [--license] [--version]
Florian Fischer's avatar
Florian Fischer committed
63
64
65
66
67
68
69
70
71
72
73
	                    [-b BENCHMARKS [BENCHMARKS ...]]
	                    [-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
	                    results

	Summarize allocbench results in allocator sets

	positional arguments:
	  results               path to results

	optional arguments:
	  -h, --help            show this help message and exit
74
75
	  -t FILE_EXT, --file-ext FILE_EXT
	                        file extension used for plots
Florian Fischer's avatar
Florian Fischer committed
76
77
78
79
80
81
82
	  --license             print license info and exit
	  --version             print version info and exit
	  -b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
	                        benchmarks to summarize
	  -x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
	                        benchmarks to exclude

83

Florian Fischer's avatar
Florian Fischer committed
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
`./merge.py` can combine the results of different benchmark runs.

	usage: merge.py [-h] [--license] [--version] [-b BENCHMARKS [BENCHMARKS ...]]
	                [-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
	                src dest

	Merge to allocbench results

	positional arguments:
	  src                   results which should be merged into dest
	  dest                  results in which src should be merged

	optional arguments:
	  -h, --help            show this help message and exit
	  --license             print license info and exit
	  --version             print version info and exit
	  -b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
	                        benchmarks to summarize
	  -x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
	                        benchmarks to exclude
Florian Fischer's avatar
Florian Fischer committed
104

Florian Fischer's avatar
Florian Fischer committed
105
106
107
108
### Examples

	./bench.py -b loop

109
runs only the loop benchmark for all allocators found on the system and will put its
Florian Fischer's avatar
Florian Fischer committed
110
results in `$PWD/results/$HOSTNAME/<time>/loop`.
Florian Fischer's avatar
Florian Fischer committed
111

112
	./bench.py -a "tcmalloc*"
Florian Fischer's avatar
Florian Fischer committed
113

114
builds all tcmalloc variants shipped with allocbench and runs all benchmarks.
Florian Fischer's avatar
Florian Fischer committed
115

Florian Fischer's avatar
Florian Fischer committed
116
	./summarize.py <path/to/saved/results>
Florian Fischer's avatar
Florian Fischer committed
117

Florian Fischer's avatar
Florian Fischer committed
118
summarizes the previously created results.
Florian Fischer's avatar
Florian Fischer committed
119
120
121
122

## Benchmarks

You want to compare allocators with your own software or add a new benchmark,
Florian Fischer's avatar
Florian Fischer committed
123
have a look at [doc/Benchmarks.md](doc/Benchmarks.md).
Florian Fischer's avatar
Florian Fischer committed
124
125
126

## Allocators

Florian Fischer's avatar
Florian Fischer committed
127
128
By default all included allocators will be build and measured. For more precise control
about used allocators use the `-a` option.
Florian Fischer's avatar
Florian Fischer committed
129

Florian Fischer's avatar
Florian Fischer committed
130
131
For more details about what allocators are available, how they are used and how to
include new once have a look at [doc/Allocators.md](doc/Allocators.md).
Florian Fischer's avatar
Florian Fischer committed
132

Florian Fischer's avatar
add GPL    
Florian Fischer committed
133
134
135
136
## License

This program is released under GPLv3. You can find a copy of the license
in the LICENSE file in the programs root directory.