README.md in backburner-0.0.1 vs README.md in backburner-0.0.2

- old
+ new

@@ -1,19 +1,71 @@ # Backburner -Backburner is a beanstalkd-powered job queue designed to be as simple and easy to use as possible. -You can create background jobs, place those on specialized queues and process them later. +Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue which can handle a very high volume of jobs. +You create background jobs and place those on multiple work queues to be processed later. -Processing background jobs reliably has never been easier. Backburner works with any ruby-based -web framework but is well-suited for use with [Sinatra](http://sinatrarb.com) and [Padrino](http://padrinorb.com). +Processing background jobs reliably has never been easier then with beanstalkd and Backburner. This gem works with any ruby-based +web framework but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails. -If you want to use beanstalk for job processing, consider using Backburner. Backburner is heavily inspired by Resque and DelayedJob. -Backburner can be a persistent queue if the beanstalk persistence mode is enabled, supports priority, delays, and timeouts. -Backburner stores jobs as simple JSON payloads. +If you want to use beanstalk for your job processing, consider using Backburner. +Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads. +Backburner can be a persistent queue if the beanstalk persistence mode is enabled, supports multiple queues, priorities, delays, and timeouts. +## Why Backburner? + +Backburner is well tested and has a familiar, no-nonsense approach to job processing but that is of secondary importance. +Let's face it; there are a lot of options for background job processing. [DelayedJob](https://github.com/collectiveidea/delayed_job), +and [Resque](https://github.com/defunkt/resque) are the first that come to mind immediately. So, how do we make sense +of which one to use? And why use Backburner over other alternatives? + +The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue +libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of. +In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through +**ActiveRecord** and a database such as PostgreSQL. + +The work queue underlying these gems tells you infinitely more about the differences then anything else. +Beanstalk is probably the best solution for job queues available today for many reasons. +The real question then is... "Why Beanstalk?". + +## Why Beanstalk? + +Illya has an excellent blog post +[Scalable Work Queues with Beanstalk](http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/) and +Adam Wiggins posted [an excellent comparison](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/). + +You will quickly see that **beanstalkd** is an underrated but incredible project that is extremely well-suited as a job queue. +Significantly better suited for this task then Redis or a database. Beanstalk is a simple, +and a very fast work queue service rolled into a single binary - it is the memcached of work queues. +Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. +[PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day. + +A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) +because it is an in-memory, event-driven system. Powered by libevent under the hood, +it requires zero setup (launch and forget, ala memcached), optional log based persistence, an easily parsed ASCII protocol, +and a rich set of tools for job management that go well beyond a simple FIFO work queue. + +Beanstalk supports the following features natively, out of the box, without any questions asked: + + * **Parallel Queues** - Supports multiple work queues, which are created and deleted on demand. + * **Reliable** - Beanstalk’s reserve, work, delete cycle, with a timeout on a job, means bad clients basically can't lose a job. + * **Scheduling** - Delay enqueuing jobs by a specified interval to schedule processing later. + * **Fast** - Beanstalkd is **significantly** [faster then alternatives](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend). Easily processes thousands of jobs a second. + * **Priorities** - Specify a higher priority and those jobs will jump ahead to be processed first accordingly. + * **Persistence** - Jobs are stored in memory for speed (ala memcached), but also logged to disk for safe keeping. + * **Federation** - Fault-tolerance and horizontal scalability is provided the same way as Memcache - through federation by the client. + * **Buried jobs** - When a job causes an error, you can bury it which keeps it around for later debugging and inspection. + +Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support. +In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup. + ## Installation +First, you probably want to [install beanstalkd](http://kr.github.com/beanstalkd/download.html), which powers the job queues. +Depending on your platform, this should be as simple as (for Ubuntu): + + $ sudo apt-get install beanstalkd + Add this line to your application's Gemfile: gem 'backburner' And then execute: @@ -59,41 +111,48 @@ NewsletterMailer.deliver_text_to_email(email, body) end end ``` -Notice that you must include the `Backburner::Queue` module and that you can set a `queue` name within the job automatically. -Jobs can then be enqueued using: +Notice that you can include the optional `Backburner::Queue` module so you can specify a `queue` name for this job. +Jobs can be enqueued with: ```ruby -Backburner.enqueue NewsletterJob, 'lorem ipsum...', 5 +Backburner.enqueue NewsletterJob, 'foo@admin.com', 'lorem ipsum...' ``` `Backburner.enqueue` accepts first a ruby object that supports `perform` and then a series of parameters to that object's `perform` method. The queue name used by default is the normalized class name (i.e `{namespace}.newsletter-job`) if not otherwise specified. ### Simple Async Jobs ### In addition to defining custom jobs, a job can also be enqueued by invoking the `async` method on any object which -includes `Backburner::Performable`. +includes `Backburner::Performable`. Async enqueuing works for both instance and class methods on any _performable_ object. ```ruby class User include Backburner::Performable def activate(device_id) @device = Device.find(device_id) # ... end + + def self.reset_password(user_id) + # ... + end end +# Async works for instance methods on a persisted model @user = User.first @user.async(:pri => 1000, :ttr => 100, :queue => "user.activate").activate(@device.id) +# ..as well as for class methods +User.async(:pri => 100, :delay => 10.seconds).reset_password(@user.id) ``` -This will automatically enqueue a job that will run `activate` with the specified argument for that user record. +This will automatically enqueue a job for that user record that will run `activate` with the specified argument. The queue name used by default is the normalized class name (i.e `{namespace}.user`) if not otherwise specified. Note you are able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`. ### Working Jobs @@ -172,21 +231,34 @@ ### Logging Right now, all logging happens to standard out and can be piped to a file or any other output manually. More on logging coming later. -### Front-end Monitoring +### Front-end To be completed is an admin dashboard that provides insight into beanstalk jobs via a simple Sinatra front-end. Coming soon. -## Why Backburner? +### Workers in Production -To be filled in. DelayedJob, Resque, Stalker, et al. +Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html) +is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile: + # Rakefile + require 'backburner/tasks' + +and then you can start the rake task with: + + $ rake backburner:work + $ QUEUES=newsletter-sender,push-message rake backburner:work + +The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/) +which watches processes and ensures their stability. A simple God recipe for Backburner can be found in +[examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb). + ## Acknowledgements - * Nathan Esquenazi - Project maintainer + * [Nathan Esquenazi](https://github.com/nesquena) - Project maintainer * Kristen Tucker - Coming up with the gem name * [Tim Lee](https://github.com/timothy1ee), [Josh Hull](https://github.com/joshbuddy), [Nico Taing](https://github.com/Nico-Taing) - Helping me work through the idea * [Miso](http://gomiso.com) - Open-source friendly place to work ## Contributing @@ -197,9 +269,20 @@ 4. Push to the branch (`git push origin my-new-feature`) 5. Create new Pull Request ## References -The code in this project has been adapted from a few excellent projects: +The code in this project has been made in light of a few excellent projects: * [DelayedJob](https://github.com/collectiveidea/delayed_job) - * [Stalker](https://github.com/han/stalker) + * [Resque](https://github.com/defunkt/resque) + * [Stalker](https://github.com/han/stalker) + +Thanks to these projects for inspiration and certain design and implementation decisions. + +## Links + + * Code: `git clone git://github.com/nesquena/backburner.git` + * Home: <http://github.com/nesquena/backburner> + * Docs: <http://rdoc.info/github/nesquena/backburner/master/frames> + * Bugs: <http://github.com/nesquena/backburner/issues> + * Gems: <http://gemcutter.org/gems/backburner> \ No newline at end of file