Plot and compare benchmarks.

Plot benchmarks and compare them. An easy to use package to produce pretty graphs from the csv file generated by criterion or gauge in a few lines of code. It is high level yet pretty flexible in what you can do with it e.g. you can compare groups of benchmarks side by side showing full measurements or a delta, or plot the performance regression of one version of your package against another version.

See the README for comprehensive documentation.

bench-graph

An easy to use package to plot the benchmarking results data generated by gauge or criterion (with --csv or --csvraw options).

Easy and Flexible Charting

Usually we have a bunch of benchmarks in the benchmark measurement code and that is not how we may want to present the results in graph. We may want only a subset of the benchmarks and in a different order, with more palatable names to present. Maybe we want to plot multiple graphs from the results file. bench-graph hides the low level charting details but allows configuration of high level parameters, specifically:

• translate the names of the benchmarks before charting
• filter which benchmarks you want to include on the plot
• specify the order of benchmarks in the plot
• specify the scales
• plot multiple charts from the same measurement file
• plot comparison (full or difference) between groups of benchmarks

Plot Comparisons

We can compare the difference in benchmark results of different versions of the same package or the difference between similar benchmarks of different packages. Comparison between many benchmarks can be shown side by side as full results of both, or as the difference from a baseline.

See the test file for a complicated example. Simpler charts can be plotted with just a few lines of code. Here is a sample plot from the data in test/results.csv , showing the time taken by three packages:

Here is a plot of the same data but this time showing full time for the first package and the difference in time (i.e. the excess time taken) for the other two packages.

Just use stack test in this package and see all the charts generated in the charts directory.

Plot Performance Regressions

See the streamly repo for an example script using bench-graph to automatically plot perforformance changes across commits. In the repo if you run this command:

$ ./bench.sh --help # prints help
$ ./bench.sh --compare --base HEAD^

It generates a comparative graph of benchmark changes from the previous commit to the HEAD commit. The generated graph looks like this, the negative delta means there is performance improvement:

The bench-graph code for this can be found here.

The shell script functionality may be included in the bench-graph itself in future.

Plot Any Fields

Any of the fields available in gauge/criterion output can be plotted, for example:

• time
• cycles
• total allocations (allocated)
• bytes copied during GC (bytesCopied)
• CPU time taken by the mutator i.e. other than GC (mutatorCpuSeconds)
• CPU time taken by the garbage collector (gcCpuSeconds)

Contributions and Feedback

Contributions are welcome! This is just an early draft of the package, with maybe lots of bugs and lack of required functionality, any feedback on improvements is welcome. Please raise an issue for anything you want to suggest or discuss.