evdispatch

→ ‘evdispatch’

What

A library for making HTTP requests in parallel.

Installing

Download: curl-7.18.1.tar.gz

  tar -zxf curl-7.18.1.tar.gz
  cd curl-7.18.1/
  ./configure && make && sudo make install
  sudo gem install evdispatch

The basics

The library provides an interface to a background posix thread running a libev event loop. HTTP requests are processed using libcurl and it’s multi interface. From Ruby requests and responses are passed back and forth to the event loop thread using a synchronized queue. In the future, responses may be processed in ruby by monitoring a file handle, making it possible to use IO.select.

The intended use of this library was to provide a method for web applications to make multiple concurrent service requests to satisify a single web request.

For example in rails this might look like:

  class DashController < ApplicationController
    def index
      @blogs_id = $dispatcher.request("http://10.0.6.45/service/blogs")
      @news_id = $dispatcher.request("http://10.0.6.45/service/news")
      @messages_id = $dispatcher.request("http://10.0.6.45/service/messages")
    end
  end

  index.html.erb
  <% blogs = JSON.parse($dispatcher.response(@blogs_id)[:body]) %>
  <ul>
  <% for b in blogs %>
    <li><a href="<%= b['link'] %>" ><%= b['title'] %></a></li>
  <% end %>
  </ul>
  <% news = JSON.parse($dispatcher.response(@news_id)[:body]) %>
  <ul>
  <% for n in news %>
    <li><a href="<%= n['link'] %>" ><%= n['title'] %></a></li>
  <% end %>
  </ul>

What’s interesting is if the blogs response is not back and it blocks on $dispatcher.response(@blogs_id) chances are high that the news_id will have returned it’s response by the time it gets finished with the blogs. This is because the background thread does not block while ruby waits for the first response. One thing to keep in mind is the background thread will block if it has to resolve DNS names.

In my testing I’ve successfully run 3 million requests. Doing 10000 iterations of 300 concurrent requests, on a Linux FC7 duel P4 machine. Memory utilization stays flat at around 14 to 19 megs with about 2,000 – 12,000 objects.

Demonstration of usage

require 'rubygems'
require 'evdispatch'

# create a new dispatch loop
d = Evdispatch::Loop.new

# start the event loop thread
d.start

# send a dispatch http request and store a handle to the request
google_id = d.request_http("http://www.google.com/")

# do some processing and later on check for the response
response = d.response( google_id )

puts response[:response_time]
puts response[:body]

# sometime later you can stop the event loop
d.stop

Forum

http://groups.google.com/group/evdispatch

How to submit patches

Read the 8 steps for fixing other people’s code and for section 8b: Submit patch to Google Groups, use the Google Group above.

The trunk repository is http://evdispatch.rubyforge.org/svn/trunk for anonymous access.

F.A.Q.

• I get an error when installing the gem: “error: ‘curl_socket_t’ has not been declared”

  You need to have at least version 7.17.1 of libcurl. I recommend at least version: 7.18.1.

• Mac OSX (Darwin) Crashing on HTTP POST request

  There are some issues with crashes on darwin builds when making an HTTP POST. This doesn’t happen on Linux and I’m actively looking into the issue on my apple.

License

This code is free to use under the terms of the MIT license.

Contact

Comments are welcome. See the forum

Xullicious
Theme extended from Paul Battley