BigBench

BigBench is a http penetration tool. It allows you to test the performance of any web server with very high loads. It:

Installation

gem install bigbench

Further Reading

Requirements

Test Receipts

How do the test receipts look like? As easy as possible. For example like this in example.rb:

BigBench.configure do |config|
  config.duration   = 2.minutes
  config.output     = "example.ljson"
  config.users      = 5
  config.basic_auth = ['username', 'password']
end

benchmark "default website pages" => "http://localhost:3000" do
    get "/"
    get "/blog"
    get "/imprint"
    get "/admin", :basic_auth => ['username', 'password']
end

benchmark "login and logout" => "http://localhost:3000" do
    post "/login",  :params => { :name => "test@user.com", :password => "secret" }
    post "/logout", :params => { :name => "test@user.com" }
end

post_process :statistics

Generator

You can have your test receipts generated!

bigbench generate sample

Single Host vs. Multiple Hosts Testing

You can either test with a single machine right from your local host, or with multiple machines using bots. No matter what, the test receipt will stay the same.

Single Host

BigBench allows you to run your tests against every host from your local machine. The command for this looks like this:

bigbench local example.rb

Multiple Hosts with Bots

BigBench uses a bot design pattern which means you can run your tests from multiple hosts. Everything you need for this is a redis that is reachable from all testing hosts. Every host simply starts a bot that is checking for a new test receipt every minute like this:

bigbench bot redis_url:port redis_password

Then to run the tests from all hosts simply use the same receipt as you would use for a local run and call it like this:

bigbench bots example.rb redis_url:port redis_password

This will upload the test receipt to all bots and make them run it. Every bot reports its results back to the redis and the local machine then combines, and writes them to the output file. So you test with the same receipts and get the same results, no matter if your testing from the local host or with multiple bots.

Output

How does the recorded output look like? It’s in the *.ljson format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:

{"elapsed":0.002233,"start":1333981203.542233,"stop":1333981203.54279,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00331,"start":1333981203.5434968,"stop":1333981203.5438669,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.004248,"start":1333981203.544449,"stop":1333981203.544805,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00521,"start":1333981203.545397,"stop":1333981203.5457668,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00615,"start":1333981203.546355,"stop":1333981203.546707,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.007127,"start":1333981203.547328,"stop":1333981203.5476842,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.008024,"start":1333981203.548226,"stop":1333981203.5485811,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.008904,"start":1333981203.549105,"stop":1333981203.549461,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.009803,"start":1333981203.550003,"stop":1333981203.55036,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.010678,"start":1333981203.550882,"stop":1333981203.551235,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.011549,"start":1333981203.5517519,"stop":1333981203.552106,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.012417,"start":1333981203.5526242,"stop":1333981203.552974,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.013294,"start":1333981203.553495,"stop":1333981203.553851,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.014166,"start":1333981203.5543702,"stop":1333981203.554723,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.015043,"start":1333981203.555247,"stop":1333981203.5556,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.01592,"start":1333981203.556119,"stop":1333981203.5564768,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
...

The advantage with this file format is, that it can be parsed and computed very efficiently because the JSON parser doesn’t have to parse a whole JSON array with with loads of objects but simply one objectline by line.

Post Processors

After the benchmark has finished you can create hooks and write plugins that do something with the collected data. To setup a hook simply use the post_process method to add a block or run a predefined plugin:

# Run BigBench::PostProcessor::Statistics
post_process :statistics

# Run a block that could do anything
post_process do

    total_trackings, total_errors = 0, 0                
    each_tracking do |tracking|
        total_trackings += 1
        total_errors    += 1 unless tracking[:status] == 200
    end

    Twitter.post "Just run BigBench with #{total_trackings} trackings and #{total_errors} errors."

end

It’s also very easy to write an own post processor. The basic structure is like this:

module BigBench
    module PostProcessor
        module SamplePostProcessor

            def self.run!(options)
                # Do whatever you want here
            end

        end
    end
end

You can hook it in with:

post_process :sample_post_processor
# or
post_process BigBench::PostProcessor::SamplePostProcessor

Post Processor Environment API

BigBench automatically supports a great load of functionality for every post processor it would need anyways. This functionality is offered through the

Included Post Processors

By default BigBench ships with a few very useful post processors that might already fit your needs perfectly. The full list of included post processors is shown in the wiki:

Running Post Processors separately

You can also re-run the currently defined post processors or run a separate post processor you never even defined in the first place without collecting the test data again like this:

# Re-run the postprocessors defined in example.rb
bigbench process example.rb

# Run a separate post processor independently - the already defined post processors are ignored
bigbench process example.rb statistics

Contribute, create great post processors and send me a pull request!

Load Comparison

BigBench is awfully good at creating high loads on web servers. A quick benchmark comparison to Apache's JMeter shows that BigBench is able to create 16% more load than JMeter.

Test Configuration: Apache’s JMeter vs. BigBench

<table>

<tr>
        <th>Parameter           </th>
        <th>Value                               </th>
</tr>
<tr>
        <td> Test Duration         </td>
        <td> 2 Minutes                             </td>
</tr>
<tr>
        <td> Concurrency(Threads)  </td>
        <td> 20                                    </td>
</tr>
<tr>
        <td> Rack Server           </td>
        <td> Thin                                  </td>
</tr>
<tr>
        <td> Rack Host             </td>
        <td> localhost                             </td>
</tr>
<tr>
        <td> Rack Request          </td>
        <td> GET: 200, Body: Test                </td>
</tr>
<tr>
        <td> Ruby Version          </td>
        <td> ruby 1.9.3p125 [x86_64-darwin11.3.0]  </td>
</tr>
<tr>
        <td> JMeter Version        </td>
        <td> 2.6 r1237317                          </td>
</tr>
<tr>
        <td> BigBench Version      </td>
        <td> 0.2                                   </td>
</tr>

</table>

Test Results

<table>

<tr>
        <th>Value               </th>
        <th>JMeter          </th>
        <th>BigBench        </th>
</tr>
<tr>
        <td> Total Requests        </td>
        <td> 48.014            </td>
        <td> 55.484            </td>
</tr>
<tr>
        <td> Requests/sec          </td>
        <td> 377               </td>
        <td> 462               </td>
</tr>
<tr>
        <td> Percentages           </td>
        <td> 100%%             </td>
        <td> 116%              </td>
</tr>

</table>

Version History

0.5

0.4

0.3

0.2

0.1