README.md in backburner-0.0.3 vs README.md in backburner-0.1.0

- old
+ new

@@ -1,60 +1,61 @@ # Backburner -Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue which can handle a very high volume of jobs. -You create background jobs and place those on multiple work queues to be processed later. +Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue that can handle a very high volume of jobs. +You create background jobs and place them on multiple work queues to be processed later. -Processing background jobs reliably has never been easier then with beanstalkd and Backburner. This gem works with any ruby-based -web framework but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails. +Processing background jobs reliably has never been easier than with beanstalkd and Backburner. This gem works with any ruby-based +web framework, but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails. If you want to use beanstalk for your job processing, consider using Backburner. Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads. -Backburner can be a persistent queue if the beanstalk persistence mode is enabled, supports multiple queues, priorities, delays, and timeouts. +Backburner can be a persistent queue when the beanstalk persistence mode is enabled. +It supports multiple queues, priorities, delays, and timeouts. ## Why Backburner? -Backburner is well tested and has a familiar, no-nonsense approach to job processing but that is of secondary importance. -Let's face it; there are a lot of options for background job processing. [DelayedJob](https://github.com/collectiveidea/delayed_job), +Backburner is well tested and has a familiar, no-nonsense approach to job processing, but that is of secondary importance. +Let's face it, there are a lot of options for background job processing. [DelayedJob](https://github.com/collectiveidea/delayed_job), and [Resque](https://github.com/defunkt/resque) are the first that come to mind immediately. So, how do we make sense of which one to use? And why use Backburner over other alternatives? The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of. In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through **ActiveRecord** and a database such as PostgreSQL. -The work queue underlying these gems tells you infinitely more about the differences then anything else. +The work queue underlying these gems tells you infinitely more about the differences than anything else. Beanstalk is probably the best solution for job queues available today for many reasons. The real question then is... "Why Beanstalk?". ## Why Beanstalk? Illya has an excellent blog post [Scalable Work Queues with Beanstalk](http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/) and Adam Wiggins posted [an excellent comparison](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/). You will quickly see that **beanstalkd** is an underrated but incredible project that is extremely well-suited as a job queue. -Significantly better suited for this task then Redis or a database. Beanstalk is a simple, +Significantly better suited for this task than Redis or a database. Beanstalk is a simple, and a very fast work queue service rolled into a single binary - it is the memcached of work queues. Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. [PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day. A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) because it is an in-memory, event-driven system. Powered by libevent under the hood, -it requires zero setup (launch and forget, ala memcached), optional log based persistence, an easily parsed ASCII protocol, +it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol, and a rich set of tools for job management that go well beyond a simple FIFO work queue. Beanstalk supports the following features natively, out of the box, without any questions asked: - * **Parallel Queues** - Supports multiple work queues, which are created and deleted on demand. - * **Reliable** - Beanstalk’s reserve, work, delete cycle, with a timeout on a job, means bad clients basically can't lose a job. + * **Parallel Queues** - Supports multiple work queues created on demand. + * **Reliable** - Beanstalk’s reserve, work, delete cycle ensures reliable processing. * **Scheduling** - Delay enqueuing jobs by a specified interval to schedule processing later. - * **Fast** - Beanstalkd is **significantly** [faster then alternatives](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend). Easily processes thousands of jobs a second. - * **Priorities** - Specify a higher priority and those jobs will jump ahead to be processed first accordingly. - * **Persistence** - Jobs are stored in memory for speed (ala memcached), but also logged to disk for safe keeping. - * **Federation** - Fault-tolerance and horizontal scalability is provided the same way as Memcache - through federation by the client. - * **Buried jobs** - When a job causes an error, you can bury it which keeps it around for later debugging and inspection. + * **Fast** - Processes thousands of jobs per second; **significantly** [faster than alternatives](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend). + * **Priorities** - Specify priority so important jobs can be processed quickly. + * **Persistence** - Jobs are stored in memory for speed, but logged to disk for safe keeping. + * **Federation** - Horizontal scalability provided through federation by the client. + * **Error Handling** - Bury any job which causes an error for later debugging and inspection. Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support. In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup. ## Installation @@ -80,18 +81,22 @@ Backburner is extremely simple to setup. Just configure basic settings for backburner: ```ruby Backburner.configure do |config| - config.beanstalk_url = "beanstalk://127.0.0.1" + config.beanstalk_url = ["beanstalk://127.0.0.1", "beanstalk://127.0.0.1:11301"] config.tube_namespace = "some.app.production" config.on_error = lambda { |e| puts e } config.default_priority = 65536 config.respond_timeout = 120 end ``` +The `beanstalk_url` supports a string such as 'beanstalk://127.0.0.1' or an array of addresses. +The `tube_namespace` is the prefix used for all tubes related to this backburner queue. +The `on_error` is a callback that gets invoked with the error whenever a job fails + ## Usage Backburner allows you to create jobs and place them on a beanstalk queue, and later pull those jobs off the queue and process them asynchronously. @@ -131,12 +136,12 @@ includes `Backburner::Performable`. Async enqueuing works for both instance and class methods on any _performable_ object. ```ruby class User include Backburner::Performable - queue "newsletter" # defaults to 'newsletter-job' - queue_priority 1000 # most urgent priority is 0 + queue "user-jobs" # defaults to 'user' + queue_priority 500 # most urgent priority is 0 def activate(device_id) @device = Device.find(device_id) # ... end @@ -146,18 +151,19 @@ end end # Async works for instance methods on a persisted model @user = User.first -@user.async(:pri => 1000, :ttr => 100, :queue => "user.activate").activate(@device.id) +@user.async(:ttr => 100, :queue => "activate").activate(@device.id) # ..as well as for class methods User.async(:pri => 100, :delay => 10.seconds).reset_password(@user.id) ``` This will automatically enqueue a job for that user record that will run `activate` with the specified argument. +Note that you can set the queue name and queue priority at the class level and +you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`. The queue name used by default is the normalized class name (i.e `{namespace}.user`) if not otherwise specified. -Note you are able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`. ### Working Jobs Backburner workers are processes that run forever handling jobs that get reserved. Starting a worker in ruby code is simple: @@ -194,12 +200,12 @@ ### Default Queues Workers can be easily restricted to processing only a specific set of queues as shown above. However, if you want a worker to process **all** queues instead, then you can leave the queue list blank. -When you execute a worker without queues specified, any queue for a known job queue class with `include Backburner::Queue` will be processed. To access the list of known -queue classes, you can use: +When you execute a worker without queues specified, any queue for a known job queue class with `include Backburner::Queue` will be processed. +To access the list of known queue classes, you can use: ```ruby Backburner::Worker.known_queue_classes # => [NewsletterJob, SomeOtherJob] ``` @@ -234,25 +240,31 @@ ### Logging Right now, all logging happens to standard out and can be piped to a file or any other output manually. More on logging coming later. -### Front-end +### Web Front-end -To be completed is an admin dashboard that provides insight into beanstalk jobs via a simple Sinatra front-end. Coming soon. +Be sure to check out the Sinatra-powered project [beanstalkd_view](https://github.com/denniskuczynski/beanstalkd_view) +by [denniskuczynski](http://github.com/denniskuczynski) which provides an excellent overview of the tubes and +jobs processed by your beanstalk workers. An excellent addition to your Backburner setup. ### Workers in Production Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html) is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile: - # Rakefile - require 'backburner/tasks' +```ruby +# Rakefile +require 'backburner/tasks' +``` and then you can start the rake task with: - $ rake backburner:work - $ QUEUES=newsletter-sender,push-message rake backburner:work +```bash +$ rake backburner:work +$ QUEUES=newsletter-sender,push-message rake backburner:work +``` The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/) which watches processes and ensures their stability. A simple God recipe for Backburner can be found in [examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb). \ No newline at end of file