Measures and compares the performance of streaming libraries.
This package provides micro-benchmarks to measure and compare the performance of various streaming implementations in Haskell.
The following packages are supported:
base (Haskell lists)
streamly
streaming
pipes
machines
conduit
drinkery
Streaming Benchmarks
This package provides micro-benchmarks to measure and compare the performance of various streaming implementations in Haskell.
We have taken due to care to make sure that we are benchmarking correctly and fairly. See the notes on correct benchmarking.
DISCLAIMER: This package is a result of benchmarking effort done during the development of streamly by the authors of streamly.
Benchmarks
The benchmark names are obvious, some of them are described below. Single operation benchmarks:
| Name | Description |
|---|---|
drain | Just discards all the elements in the stream |
drop-all | drops all element using the drop operation |
last | extract the last element of the stream |
fold | sum all the numbers in the stream |
map | increments each number in the stream by 1 |
take-all | Use take to retain all the elements in the stream |
filter-even | Keep even numbers, discard odd |
scan | scan the stream using + operation |
mapM | transform the stream using a monadic action |
zip | combines corresponding elements of the two streams together |
Composite operation benchmarks:
| Name | Description |
|---|---|
map x 4 | perform map operation 4 times |
take-map | take followed by a map |
For more details on how each benchmark is implemented see this benchmark file.
Each benchmark is run in a separate process to avoid any effects of GC interference and sharing across benchmarks.
Benchmark Results
Below we present some results comparing streamly with other streaming implementations. Due care has been taken to keep the comparisons fair. We have optimized each library's code to the best of our knowledge, please point out if you find any measurement issues.
Reproducing benchmark results
Commands to reproduce the benchmark results are provided in each section below. But before you run those commands you need to build the reporting tool once using the following command. Note that this command works with only ghc-8.8.4 or lower. However, after building this tool you can run the benchmarks with any GHC version.
$ cabal install --flag dev --installdir charts --with-compiler ghc-8.8.4 bench-report
Nix users can use bench-report.nix. It uses an older version of nixpkgs that contains the required dependencies:
$ nix-shell bench-report.nix --run "cabal install --flag dev --installdir charts bench-report"
Streamly vs Haskell Lists
Streamly, when used with Identity monad, is almost the same as Haskell lists (in the base package). See this for more details.
The following table compares the timing of several operations for streamly with lists using a one million element stream. For brevity only those operations where the performance of the two packages differ by more than 10% are shown in the table below.
| Benchmark | streamly(μs) | list(μs) | list/streamly |
|---|---|---|---|
| drop-map x 4 | 375.09 | 76925.32 | 205.08 |
| filter-drop x 4 | 382.03 | 54848.54 | 143.57 |
| drop-scan x 4 | 795.81 | 76716.79 | 96.40 |
| filter-scan x 4 | 795.60 | 44559.15 | 56.01 |
| scan-map x 4 | 1192.19 | 48838.22 | 40.97 |
| take-map x 4 | 1500.99 | 60126.58 | 40.06 |
| filter-take x 4 | 1502.01 | 48766.87 | 32.47 |
| take-drop x 4 | 1499.62 | 41720.03 | 27.82 |
| take-scan x 4 | 1874.94 | 51283.30 | 27.35 |
| drop-one x 4 | 375.33 | 8993.87 | 23.96 |
| dropWhile-false x 4 | 374.61 | 8957.79 | 23.91 |
| dropWhile-false | 374.83 | 8670.05 | 23.13 |
| drop-one | 390.77 | 8681.85 | 22.22 |
| dropWhile-true | 571.60 | 12237.48 | 21.41 |
| drop-all | 562.94 | 8262.38 | 14.68 |
| take-all | 624.83 | 564.34 | 1/1.11 |
| scan x 4 | 795.83 | 385.85 | 1/2.06 |
| appendR[10000] | 360.75 | 126.95 | 1/2.84 |
| concatMap | 34957.71 | 1124.85 | 1/31.08 |
- streamly-0.8.0, base-4.14.1.0, ghc-8.10.4, Linux
To reproduce these results use the following commands:
$ ./bench.sh --benchmarks "pure-streamly,list" --measure
$ ./bench.sh --benchmarks "pure-streamly,list" --diff multiples
Streamly vs Streaming
The following table compares the timing of several operations for streamly with streaming using a million element stream.
| Benchmark | streamly(μs) | streaming(μs) | streaming/streamly |
|---|---|---|---|
| appendR[10000] | 326.56 | 1301176.69 | 3984.54 |
| mapM x 4 | 374.42 | 223591.08 | 597.17 |
| filter-map x 4 | 381.07 | 194903.88 | 511.47 |
| filter-scan x 4 | 795.66 | 233527.90 | 293.50 |
| filter-all-in x 4 | 375.40 | 102629.64 | 273.38 |
| filter-drop x 4 | 387.15 | 99096.98 | 255.96 |
| map x 4 | 386.49 | 94944.87 | 245.66 |
| drop-map x 4 | 375.62 | 89669.37 | 238.73 |
| scan x 4 | 797.00 | 166332.40 | 208.70 |
| scan-map x 4 | 1194.30 | 238804.48 | 199.95 |
| filter-even x 4 | 396.37 | 77865.47 | 196.45 |
| drop-scan x 4 | 796.98 | 156063.52 | 195.82 |
| takeWhile-true x 4 | 562.49 | 90183.53 | 160.33 |
| scan | 375.24 | 47520.57 | 126.64 |
| filter-take x 4 | 1498.55 | 189635.34 | 126.55 |
| mapM | 388.10 | 46689.61 | 120.30 |
| take-map x 4 | 1500.71 | 178954.50 | 119.25 |
| zip | 656.65 | 66689.73 | 101.56 |
| take-scan x 4 | 2380.35 | 241675.75 | 101.53 |
| filter-all-in | 375.97 | 33590.14 | 89.34 |
| map | 375.02 | 33081.13 | 88.21 |
| filter-even | 393.26 | 30458.46 | 77.45 |
| filter-all-out | 382.87 | 26826.21 | 70.07 |
| take-all x 4 | 1499.71 | 101332.53 | 67.57 |
| take-drop x 4 | 1498.53 | 98281.99 | 65.59 |
| takeWhile-true | 562.62 | 31863.25 | 56.63 |
| foldl' | 388.22 | 18503.15 | 47.66 |
| drop-all | 562.08 | 25200.32 | 44.83 |
| take-all | 768.65 | 33247.97 | 43.26 |
| dropWhile-true | 564.87 | 24431.50 | 43.25 |
| last | 385.53 | 15240.85 | 39.53 |
| dropWhile-false | 374.83 | 14566.70 | 38.86 |
| drop-one | 374.80 | 14565.01 | 38.86 |
| drop-one x 4 | 375.88 | 14448.67 | 38.44 |
| dropWhile-false x 4 | 390.12 | 14619.42 | 37.47 |
| drain | 375.06 | 13702.29 | 36.53 |
| toList | 117708.83 | 201444.81 | 1.71 |
- streamly-0.8.0, streaming-0.2.3.0, ghc-8.10.4, Linux
To reproduce these results use the following commands:
$ ./bench.sh --benchmarks "streamly,streaming" --measure
$ ./bench.sh --benchmarks "streamly,streaming" --diff multiples
Streamly vs Pipes
The following table compares the timing of several operations for streamly with pipes using a million element stream.
| Benchmark | streamly(μs) | pipes(μs) | pipes/streamly |
|---|---|---|---|
| appendR[10000] | 327.90 | 901135.92 | 2748.21 |
| mapM x 4 | 375.20 | 407184.39 | 1085.23 |
| filter-map x 4 | 381.52 | 366759.70 | 961.31 |
| drop-map x 4 | 375.48 | 281296.82 | 749.16 |
| filter-all-in x 4 | 375.60 | 222331.68 | 591.93 |
| filter-drop x 4 | 387.44 | 222830.71 | 575.14 |
| drop-scan x 4 | 797.23 | 336737.89 | 422.39 |
| filter-even x 4 | 389.87 | 152688.91 | 391.64 |
| filter-scan x 4 | 797.38 | 309733.91 | 388.44 |
| drop-one x 4 | 375.48 | 139851.13 | 372.46 |
| map x 4 | 386.56 | 136289.32 | 352.57 |
| dropWhile-false x 4 | 390.72 | 137395.44 | 351.65 |
| scan-map x 4 | 1194.38 | 381286.88 | 319.23 |
| takeWhile-true x 4 | 562.86 | 165143.23 | 293.40 |
| scan x 4 | 796.68 | 222986.17 | 279.90 |
| mapM | 388.19 | 95576.97 | 246.21 |
| filter-all-in | 375.21 | 71297.42 | 190.02 |
| take-map x 4 | 1502.76 | 275887.24 | 183.59 |
| scan | 374.81 | 65549.13 | 174.89 |
| take-drop x 4 | 1503.43 | 256448.45 | 170.58 |
| filter-even | 390.29 | 66183.72 | 169.57 |
| filter-all-out | 376.99 | 59074.54 | 156.70 |
| drop-one | 375.19 | 58395.24 | 155.64 |
| dropWhile-false | 375.35 | 58223.03 | 155.12 |
| map | 375.05 | 57736.43 | 153.94 |
| filter-take x 4 | 1503.00 | 227925.71 | 151.65 |
| take-scan x 4 | 2455.91 | 354284.33 | 144.26 |
| zip | 657.07 | 86011.93 | 130.90 |
| takeWhile-true | 564.14 | 61390.21 | 108.82 |
| take-all x 4 | 1502.32 | 139730.70 | 93.01 |
| dropWhile-true | 564.03 | 49227.19 | 87.28 |
| drop-all | 562.05 | 46505.37 | 82.74 |
| take-all | 824.09 | 60511.34 | 73.43 |
| drain | 375.29 | 26390.59 | 70.32 |
| foldl' | 397.34 | 19064.05 | 47.98 |
| last | 387.11 | 17364.44 | 44.86 |
| toList | 117257.09 | 207405.94 | 1.77 |
- streamly-0.8.0, pipes-4.3.16, ghc-8.10.4, Linux
To reproduce these results use the following commands:
$ ./bench.sh --benchmarks "streamly,pipes" --measure
$ ./bench.sh --benchmarks "streamly,pipes" --diff multiples
Streamly vs Conduit
The following table compares the timing of several operations for streamly with conduit using a million element stream.
| Benchmark | streamly(μs) | conduit(μs) | conduit/streamly |
|---|---|---|---|
| mapM x 4 | 375.46 | 297002.31 | 791.04 |
| filter-map x 4 | 380.79 | 267543.81 | 702.60 |
| drop-map x 4 | 375.66 | 232307.84 | 618.39 |
| filter-drop x 4 | 386.05 | 235029.15 | 608.81 |
| filter-scan x 4 | 796.56 | 306556.67 | 384.85 |
| drop-scan x 4 | 797.19 | 300789.06 | 377.31 |
| zip | 657.29 | 210069.05 | 319.60 |
| filter-all-in x 4 | 375.24 | 118506.68 | 315.82 |
| scan-map x 4 | 1194.67 | 360671.18 | 301.90 |
| map x 4 | 387.00 | 113497.14 | 293.27 |
| drop-one x 4 | 375.49 | 101842.95 | 271.23 |
| dropWhile-false x 4 | 389.44 | 102051.22 | 262.04 |
| scan x 4 | 796.72 | 190479.35 | 239.08 |
| takeWhile-true x 4 | 564.58 | 114459.57 | 202.73 |
| filter-even x 4 | 391.76 | 72369.30 | 184.73 |
| filter-take x 4 | 1502.04 | 267921.27 | 178.37 |
| take-map x 4 | 1502.88 | 238875.95 | 158.95 |
| take-drop x 4 | 1500.34 | 232606.19 | 155.04 |
| take-scan x 4 | 2443.83 | 309738.86 | 126.74 |
| mapM | 389.15 | 41897.48 | 107.66 |
| scan | 375.40 | 38137.85 | 101.59 |
| take-all x 4 | 1502.32 | 110682.74 | 73.67 |
| filter-all-in | 375.31 | 26024.21 | 69.34 |
| dropWhile-false | 375.10 | 25307.13 | 67.47 |
| map | 375.18 | 23088.09 | 61.54 |
| drop-one | 375.43 | 22020.65 | 58.65 |
| filter-even | 392.28 | 21504.28 | 54.82 |
| takeWhile-true | 562.79 | 29012.68 | 51.55 |
| filter-all-out | 378.76 | 15736.05 | 41.55 |
| drop-all | 562.89 | 19916.48 | 35.38 |
| foldl' | 388.88 | 12499.03 | 32.14 |
| dropWhile-true | 564.43 | 17983.35 | 31.86 |
| take-all | 784.67 | 24425.36 | 31.13 |
| last | 385.75 | 10974.84 | 28.45 |
| drain | 375.18 | 4272.15 | 11.39 |
| appendR[10000] | 326.93 | 1207.88 | 3.69 |
| toList | 116441.26 | 199138.09 | 1.71 |
- streamly-0.8.0, conduit-1.3.4.1, ghc-8.10.4, Linux
To reproduce these results use the following commands:
$ ./bench.sh --benchmarks "streamly,conduit" --measure
$ ./bench.sh --benchmarks "streamly,conduit" --diff multiples
Comparing other libraries
This package supports many streaming libraries. Use the following command to see all available benchmarks:
$ ./bench.sh --help
You can then select the libraries you want to compare:
$ ./bench.sh --benchmarks "streaming,pipes" --measure
Adding New Libraries
It is trivial to add a new package. This is how a benchmark file for a streaming package looks like. Pull requests are welcome, we will be happy to help, just join the gitter chat and ask!