README.md in benchmark_driver-0.10.13 vs README.md in benchmark_driver-0.10.14
- old
+ new
@@ -143,18 +143,20 @@
str-interp
2.5.0: 3258489.7 i/s
2.4.1: 1889805.6 i/s - 1.72x slower
```
-## Output options
+## Output Options
By default, there are following output options.
-* compare: benchmark-ips's `compare!`-like output (default)
-* simple: ruby's original `benchmark/driver.rb`-like simple output
-* markdown: output in markdown table
-* record: serialize results in `benchmark_driver.record.yml`, to change outputs later as you like
+| Output | Description |
+|:-------|:------------|
+| compare | benchmark-ips's `compare!`-like output (default) |
+| simple | ruby's original `benchmark/driver.rb`-like simple output |
+| markdown | output in markdown table |
+| record | serialize results in `benchmark_driver.record.yml`, to change outputs later as you like |
With `benchmark-driver` CLI, you can specify it with `-o [output]` or `--output [output]`.
With Ruby interface, you can specify it like:
@@ -162,9 +164,202 @@
Benchmark.driver do |x|
x.prelude %{ array = [] }
x.report 'Array#empty?', %{ array.empty? }
x.output 'markdown'
end
+```
+
+### compare
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --output compare --rbenv '2.4.2;2.5.0'
+Calculating -------------------------------------
+ 2.4.2 2.5.0
+ empty 195.957M 129.970M i/s - 20.000M times in 0.102063s 0.153882s
+ blank 66.554M 55.630M i/s - 20.000M times in 0.300507s 0.359519s
+
+Comparison:
+ empty
+ 2.4.2: 195957398.5 i/s
+ 2.5.0: 129969716.9 i/s - 1.51x slower
+
+ blank
+ 2.4.2: 66554190.0 i/s
+ 2.5.0: 55629883.2 i/s - 1.20x slower
+```
+
+### simple
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --output simple --rbenv '2.4.2;2.5.0'
+benchmark results (i/s):
+ 2.4.2 2.5.0
+empty 184.084M 117.319M
+blank 65.843M 62.093M
+```
+
+### markdown
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --output markdown --rbenv '2.4.2;2.5.0'
+```
+
+benchmark results (i/s)
+
+| | 2.4.2| 2.5.0|
+|:------|:-------|:-------|
+|empty |187.296M|117.662M|
+|blank | 58.895M| 58.852M|
+
+### record
+
+Measure first, output with various formats later.
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --output record --rbenv '2.4.2;2.5.0'
+benchmarking....
+
+$ benchmark-driver benchmark_driver.record.yml --output compare
+Calculating -------------------------------------
+ empty 153.380M 114.228M i/s - 20.000M times in 0.130395s 0.175088s
+ blank 67.834M 64.328M i/s - 20.000M times in 0.294836s 0.310906s
+
+Comparison:
+ empty: 153380113.9 i/s
+ blank: 67834321.4 i/s - 2.26x slower
+
+$ benchmark-driver benchmark_driver.record.yml --output simple
+benchmark results (i/s):
+empty 153.380M 114.228M
+blank 67.834M 64.328M
+```
+
+### gruff
+
+There is [benchmark\_driver-output-gruff](https://github.com/benchmark-driver/benchmark_driver-output-gruff)
+plugin that renders a graph using gruff.gem.
+
+
+
+### benchmark-driver.github.io
+
+Benchmark results for https://benchmark-driver.github.io are also stored by [output plugin](https://github.com/benchmark-driver/skybench/tree/master/plugins/benchmark_driver-output-skybench).
+You can integrate any system as you like.
+
+## Runner Options
+
+Runner decides metrics to be collected. There are following default runner options.
+
+ips, time, memory, once
+
+| Runner | Description |
+|:-------|:------------|
+| ips | Iteration per second (default) |
+| time | Elapsed seconds |
+| memory | Max resident set. This is supported only on Linux for now. |
+| once | Forces `loop_count` to 1 for testing |
+| command\_stdout | Special runner to integrate existing benchmarks |
+
+### ips
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --runner ips --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+ 2.4.3 2.5.0
+ empty 228.802M 180.125M i/s - 20.000M times in 0.087412s 0.111034s
+ blank 90.012M 88.853M i/s - 20.000M times in 0.222193s 0.225090s
+
+Comparison:
+ empty
+ 2.4.3: 228801720.5 i/s
+ 2.5.0: 180124821.8 i/s - 1.27x slower
+
+ blank
+ 2.4.3: 90012021.7 i/s
+ 2.5.0: 88853269.4 i/s - 1.01x slower
+```
+
+### time
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --runner time --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+ 2.4.3 2.5.0
+ empty 0.087 0.110 s - 20.000M times
+ blank 0.217 0.219 s - 20.000M times
+
+Comparison:
+ empty
+ 2.5.0: 0.1 s
+ 2.4.3: 0.1 s - 1.26x slower
+
+ blank
+ 2.5.0: 0.2 s
+ 2.4.3: 0.2 s - 1.01x slower
+```
+
+### memory
+
+```
+$ be exe/benchmark-driver examples/yaml/blank_loop.yml --runner memory --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+ 2.4.3 2.5.0
+ empty 9.192M 9.364M bytes - 20.000M times
+ blank 9.080M 9.372M bytes - 20.000M times
+
+Comparison:
+ empty
+ 2.4.3: 9192000.0 bytes
+ 2.5.0: 9364000.0 bytes - 1.02x larger
+
+ blank
+ 2.4.3: 9080000.0 bytes
+ 2.5.0: 9372000.0 bytes - 1.03x larger
+```
+
+### once
+
+Only for testing purpose.
+
+```
+$ benchmark-driver examples/yaml/blank_loop.yml --runner once --rbenv '2.4.3;2.5.0'
+Calculating -------------------------------------
+ 2.4.3 2.5.0
+ empty 1.818M 2.681M i/s - 1.000 times in 0.000001s 0.000000s
+ blank 1.531M 2.421M i/s - 1.000 times in 0.000001s 0.000000s
+
+Comparison:
+ empty
+ 2.5.0: 2680965.1 i/s
+ 2.4.3: 1818181.8 i/s - 1.47x slower
+
+ blank
+ 2.5.0: 2421307.5 i/s
+ 2.4.3: 1531393.6 i/s - 1.58x slower
+```
+
+### command\_stdout
+
+See following examples:
+
+* https://github.com/benchmark-driver/optcarrot
+* https://github.com/benchmark-driver/fluentd-benchmark
+
+If you benchmark can run with `ruby foo bar`, specify `foo bar` to `command:`.
+Then write `stdout_to_metrics:` to convert stdout to metrics. This runner can be used only with YAML interface for now.
+
+```
+$ benchmark-driver benchmark.yml --verbose 1 --rbenv '2.6.0-dev;2.6.0-dev,--jit'
+2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
+2.6.0-dev,--jit: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
+Calculating -------------------------------------
+ 2.6.0-dev 2.6.0-dev,--jit
+ optcarrot 51.866 67.445 fps
+
+Comparison:
+ optcarrot
+ 2.6.0-dev,--jit: 67.4 fps
+ 2.6.0-dev: 51.9 fps - 1.30x slower
```
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/k0kubun/benchmark_driver.