README.md in benchmark_driver-0.1.0 vs README.md in benchmark_driver-0.2.0
- old
+ new
@@ -1,164 +1,167 @@
-# BenchmarkDriver
+# BenchmarkDriver [](https://travis-ci.org/k0kubun/benchmark_driver)
Benchmark driver for different Ruby executables
## Installation
$ gem install benchmark_driver
## Usage
```
-$ benchmark_driver -h
+$ exe/benchmark_driver -h
Usage: benchmark_driver [options] [YAML]
- -d, --duration [SECONDS] Duration seconds to run each benchmark (default: 1)
-e, --executables [EXECS] Ruby executables (e1::path1; e2::path2; e3::path3;...)
- -r, --result-format=FORMAT Result output format [time|ips] (default: time)
+ -i, --ips [SECONDS] Measure IPS in duration seconds (default: 1)
+ -l, --loop-count [COUNT] Measure execution time with loop count (default: 100000)
-v, --verbose
```
### Running single script
-With following `bm_app_erb.yml`,
+With following `example_single.yml`,
```yml
prelude: |
require 'erb'
-
- title = "hello world!"
- content = "hello world!\n" * 10
-
- data = <<EOS
- <html>
- <head> <%= title %> </head>
- <body>
- <h1> <%= title %> </h1>
- <p>
- <%= content %>
- </p>
- </body>
- </html>
- EOS
-
-benchmark: |
- ERB.new(data).result(binding)
+ erb = ERB.new(%q[Hello <%= 'World' %>])
+benchmark: erb.result
```
you can benchmark the script with multiple ruby executables.
```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby
benchmark results:
Execution time (sec)
name ruby1 ruby2
-bm_app_erb 1.028 1.010
+example_single 0.958 0.972
Speedup ratio: compare with the result of `ruby1' (greater is better)
name ruby2
-bm_app_erb 1.018
+example_single 0.986
```
-And you can change benchmark output by `-r` option.
+And you can change benchmark output to IPS (iteration per second) by `-i` option.
```
-$ benchmark_driver bm_app_erb.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_single.yml -e ruby1::ruby -e ruby2::ruby -i
Result -------------------------------------------
ruby1 ruby2
- bm_app_erb 16082.8 i/s 15541.7 i/s
+ example_single 99414.1 i/s 99723.3 i/s
-Comparison: bm_app_erb
- ruby1: 16082.8 i/s
- ruby2: 15541.7 i/s - 1.03x slower
+Comparison: example_single
+ ruby2: 99723.3 i/s
+ ruby1: 99414.1 i/s - 1.00x slower
```
### Running multiple scripts
One YAML file can contain multiple benchmark scripts.
-With following `erb_compile_render.yml`,
+With following `example_multi.yml`,
```yml
-- name: erb_compile
- prelude: |
- require 'erb'
-
- title = "hello world!"
- content = "hello world!\n" * 10
-
- data = <<EOS
- <html>
- <head> <%= title %> </head>
- <body>
- <h1> <%= title %> </h1>
- <p>
- <%= content %>
- </p>
- </body>
- </html>
- EOS
-
- benchmark: |
- ERB.new(data).src
-
-- name: erb_render
- prelude: |
- require 'erb'
-
- title = "hello world!"
- content = "hello world!\n" * 10
-
- data = <<EOS
- <html>
- <head> <%= title %> </head>
- <body>
- <h1> <%= title %> </h1>
- <p>
- <%= content %>
- </p>
- </body>
- </html>
- EOS
-
- src = "def self.render(title, content); #{ERB.new(data).src}; end"
- mod = Module.new
- mod.instance_eval(src, "(ERB)")
-
- benchmark: |
- mod.render(title, content)
+prelude: |
+ a = 'a' * 100
+ b = 'b' * 100
+benchmarks:
+ - name: join
+ benchmark: |
+ [a, b].join
+ - name: interpolation
+ benchmark: |
+ "#{a}#{b}"
```
you can benchmark the scripts with multiple ruby executables.
```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby
benchmark results:
Execution time (sec)
name ruby1 ruby2
-erb_compile 0.987 0.985
-erb_render 0.834 0.809
+join 0.022 0.022
+interpolation 0.026 0.026
Speedup ratio: compare with the result of `ruby1' (greater is better)
name ruby2
-erb_compile 1.002
-erb_render 1.031
+join 1.045
+interpolation 1.002
```
```
-$ benchmark_driver erb_compile_render.yml -e ruby1::ruby -e ruby2::ruby -r ips
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -e ruby1::ruby -e ruby2::ruby -i
Result -------------------------------------------
ruby1 ruby2
- erb_compile 30374.0 i/s 30832.1 i/s
- erb_render 628403.5 i/s 624588.0 i/s
+ join 4701954.3 i/s 4639520.3 i/s
+ interpolation 4263170.0 i/s 4044083.0 i/s
-Comparison: erb_compile
- ruby2: 30832.1 i/s
- ruby1: 30374.0 i/s - 1.02x slower
+Comparison: join
+ ruby1: 4701954.3 i/s
+ ruby2: 4639520.3 i/s - 1.01x slower
-Comparison: erb_render
- ruby1: 628403.5 i/s
- ruby2: 624588.0 i/s - 1.01x slower
+Comparison: interpolation
+ ruby1: 4263170.0 i/s
+ ruby2: 4044083.0 i/s - 1.05x slower
```
+### Configuring modes
+
+There are 2 modes:
+
+- Loop count: Enabled by `-l`. Optionally you can change count to loop by `-l COUNT`.
+- IPS: Enabled by `-i`. Optionally you can change duration by `-i DURATION`.
+
+Specifying both `-l` and `-i` is nonsense.
+
+### YAML syntax
+You can specify `benchmark:` or `benchmarks:`.
+
+#### Single
+```yml
+name: String # optional (default: file name)
+prelude: String # optional
+loop_count: Integer # optional
+benchmark: String # required
+```
+
+#### Multi
+
+```yml
+prelude: String # optional (shared)
+loop_count: Integer # optional (shared)
+benchmarks:
+ - name: String # required
+ prelude: String # optional (benchmark specific)
+ loop_count: Integer # optional (benchmark specific)
+ benchmark: String # required
+```
+
+### Debugging
+
+If you have a trouble like an unexpectedly fast result, you should check benchmark script by `-v`.
+
+```
+$ exe/benchmark_driver ruby_benchmark_set/example_multi.yml -v
+--- Running "join" with "ruby" 957780 times ---
+a = 'a' * 100
+b = 'b' * 100
+
+
+i = 0
+while i < 957780
+ i += 1
+[a, b].join
+
+end
+```
+
+## TODO
+
+- Measure multiple times and use minimum result
+- Retry and reject negative result in ips mode
+- Change not to take long time for iteration count estimation in ips mode
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/k0kubun/benchmark_driver.