README.md in benchmark_driver-0.8.6 vs README.md in benchmark_driver-0.9.0
- old
+ new
@@ -1,25 +1,26 @@
-# Benchmark::Driver [](https://travis-ci.org/k0kubun/benchmark_driver)
+# BenchmarkDriver [](https://travis-ci.org/k0kubun/benchmark_driver)
Fully-featured accurate benchmark driver for Ruby
## Project Status
-Beta. Features are almost implemented but interface may change in the future.
+**Under Construction**
## Features
### Accurate Measurement
- Low overhead benchmark by running generated script instead of calling Proc
+- Profiling memory, high-precision real time, user time and system time
- Running multiple times to minimize measurement errors
-- Profiling memory, high-precision real time
### Pluggable & Fully Featured
- Flexible and real-time output format in ips, execution time, markdown table, etc.
-- Runner and output are all pluggable
-- Bundler integration for benchmark that requires gems
+- Benchmark with various profiling/running options
+- Integrated benchmark support using external libraries
+- Runner, profiler and output format are all pluggable
### Flexible Interface
- Ruby interface similar to benchmark stdlib, benchmark-ips
- YAML input to easily manage structured benchmark set
@@ -31,65 +32,59 @@
$ gem install benchmark_driver
```
## Usage
-### Ruby Interface: Compatible Mode
+### Ruby Interface
-This interface is compatible with `Benchmark.bm` and `Benchmark.ips`, so it's good for migration.
-
-```rb
-require 'benchmark/driver'
-require 'active_support/all'
-
-Benchmark.driver do |x|
- array = []
- x.report('blank?') { array.blank? }
- x.report('empty?') { array.empty? }
- x.compare!
-end
-```
-
-### Ruby Interface: Low Overhead Mode
-
This interface generates code to profile with low overhead and executes it.
```rb
-require 'benchmark/driver'
-require 'active_support/all'
+require 'benchmark_driver'
-Benchmark.driver do |x|
- x.prelude %{ array = [] }
+Benchmark.drive do |x|
+ x.prelude = <<~RUBY
+ require 'active_support/all'
+ array = []
+ RUBY
+
x.report 'blank?', %{ array.blank? }
x.report 'empty?', %{ array.empty? }
end
```
or simply:
```rb
-require 'benchmark/driver'
-require 'active_support/all'
+require 'benchmark_driver'
-Benchmark.driver do |x|
- x.prelude %{ array = [] }
- x.report %{ array.blank?' }
- x.report %{ array.empty?' }
+Benchmark.drive do |x|
+ x.prelude = <<~RUBY
+ require 'active_support/all'
+ array = []
+ RUBY
+
+ x.report %{ array.blank? }
+ x.report %{ array.empty? }
end
```
### Structured YAML Input
With `benchmark-driver` command, you can describe benchmark with YAML input.
```
$ benchmark-driver -h
Usage: benchmark-driver [options] [YAML]
- -e, --executables [EXECS] Ruby executables (e1::path1; e2::path2; e3::path3;...)
- --rbenv [VERSIONS] Ruby executables in rbenv (2.3.5;2.4.2;...)
- -c, --compare Compare results (currently only supported in ips output)
- -r, --repeat-count [NUM] Try benchmark NUM times and use the fastest result
+ -r, --runner [TYPE] Specify runner type: ips, time, memory (default: ips)
+ -o, --output [TYPE] Specify output type: compare, simple, markdown (default: compare)
+ -e, --executables [EXECS] Ruby executables (e1::path1,arg1,...; e2::path2,arg2;...)
+ --rbenv [VERSIONS] Ruby executables in rbenv (x.x.x,arg1,...;y.y.y,arg2,...;...)
+ --repeat-count [NUM] Try benchmark NUM times and use the fastest result (TODO)
+ --bundler Install and use gems specified in Gemfile
+ --filter [REGEXP] Filter out benchmarks with given regexp
+ --run-duration [SECONDS] Warmup esitmates loop_count to run for this duration (default: 3)
```
#### Running single script
With following `example_single.yml`,
@@ -102,20 +97,21 @@
```
you can benchmark the script with multiple ruby executables.
```
-$ exe/benchmark-driver examples/yaml/example_single.yml --rbenv '2.4.2;trunk' --compare
+$ benchmark-driver example_single.yml --rbenv '2.4.1;2.5.0'
Warming up --------------------------------------
- erb.result 10.973k i/100ms
+ erb.result 71.683k i/s
Calculating -------------------------------------
- 2.4.2 trunk
- erb.result 109.268k 123.611k i/s - 548.675k in 4.017080s 4.438720s
+ 2.4.1 2.5.0
+ erb.result 72.387k 75.046k i/s - 215.049k times in 2.970833s 2.865581s
Comparison:
- erb.result (trunk): 123611.1 i/s
- erb.result (2.4.2): 109268.4 i/s - 1.13x slower
+ erb.result
+ 2.5.0: 75045.5 i/s
+ 2.4.1: 72386.8 i/s - 1.04x slower
```
#### Running multiple scripts
One YAML file can contain multiple benchmark scripts.
@@ -131,23 +127,26 @@
```
you can benchmark the scripts with multiple ruby executables.
```
-$ exe/benchmark-driver examples/yaml/example_multi.yml --rbenv '2.4.2;trunk' --compare
+$ benchmark-driver example_multi.yml --rbenv '2.4.1;2.5.0'
Warming up --------------------------------------
- join 515.787k i/100ms
- str-interp 438.646k i/100ms
+ join 2.509M i/s
+ str-interp 1.772M i/s
Calculating -------------------------------------
- 2.4.2 trunk
- join 5.200M 4.740M i/s - 20.631M in 3.967750s 4.352565s
- str-interp 4.306M 6.034M i/s - 21.932M in 4.075159s 3.634986s
+ 2.4.1 2.5.0
+ join 2.661M 2.863M i/s - 7.527M times in 2.828771s 2.629191s
+ str-interp 1.890M 3.258M i/s - 5.315M times in 2.812240s 1.630997s
Comparison:
- str-interp (trunk): 6033674.6 i/s
- join (2.4.2): 5199794.6 i/s - 1.16x slower
- join (trunk): 4740075.1 i/s - 1.27x slower
- str-interp (2.4.2): 4305563.1 i/s - 1.40x slower
+ join
+ 2.5.0: 2862755.1 i/s
+ 2.4.1: 2660777.4 i/s - 1.08x slower
+
+ str-interp
+ 2.5.0: 3258489.7 i/s
+ 2.4.1: 1889805.6 i/s - 1.72x slower
```
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/k0kubun/benchmark_driver.