README.md in backburner-0.0.1 vs README.md in backburner-0.0.2
- old
+ new
@@ -1,19 +1,71 @@
# Backburner
-Backburner is a beanstalkd-powered job queue designed to be as simple and easy to use as possible.
-You can create background jobs, place those on specialized queues and process them later.
+Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue which can handle a very high volume of jobs.
+You create background jobs and place those on multiple work queues to be processed later.
-Processing background jobs reliably has never been easier. Backburner works with any ruby-based
-web framework but is well-suited for use with [Sinatra](http://sinatrarb.com) and [Padrino](http://padrinorb.com).
+Processing background jobs reliably has never been easier then with beanstalkd and Backburner. This gem works with any ruby-based
+web framework but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails.
-If you want to use beanstalk for job processing, consider using Backburner. Backburner is heavily inspired by Resque and DelayedJob.
-Backburner can be a persistent queue if the beanstalk persistence mode is enabled, supports priority, delays, and timeouts.
-Backburner stores jobs as simple JSON payloads.
+If you want to use beanstalk for your job processing, consider using Backburner.
+Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads.
+Backburner can be a persistent queue if the beanstalk persistence mode is enabled, supports multiple queues, priorities, delays, and timeouts.
+## Why Backburner?
+
+Backburner is well tested and has a familiar, no-nonsense approach to job processing but that is of secondary importance.
+Let's face it; there are a lot of options for background job processing. [DelayedJob](https://github.com/collectiveidea/delayed_job),
+and [Resque](https://github.com/defunkt/resque) are the first that come to mind immediately. So, how do we make sense
+of which one to use? And why use Backburner over other alternatives?
+
+The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue
+libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of.
+In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through
+**ActiveRecord** and a database such as PostgreSQL.
+
+The work queue underlying these gems tells you infinitely more about the differences then anything else.
+Beanstalk is probably the best solution for job queues available today for many reasons.
+The real question then is... "Why Beanstalk?".
+
+## Why Beanstalk?
+
+Illya has an excellent blog post
+[Scalable Work Queues with Beanstalk](http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/) and
+Adam Wiggins posted [an excellent comparison](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/).
+
+You will quickly see that **beanstalkd** is an underrated but incredible project that is extremely well-suited as a job queue.
+Significantly better suited for this task then Redis or a database. Beanstalk is a simple,
+and a very fast work queue service rolled into a single binary - it is the memcached of work queues.
+Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project.
+[PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day.
+
+A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
+because it is an in-memory, event-driven system. Powered by libevent under the hood,
+it requires zero setup (launch and forget, ala memcached), optional log based persistence, an easily parsed ASCII protocol,
+and a rich set of tools for job management that go well beyond a simple FIFO work queue.
+
+Beanstalk supports the following features natively, out of the box, without any questions asked:
+
+ * **Parallel Queues** - Supports multiple work queues, which are created and deleted on demand.
+ * **Reliable** - Beanstalk’s reserve, work, delete cycle, with a timeout on a job, means bad clients basically can't lose a job.
+ * **Scheduling** - Delay enqueuing jobs by a specified interval to schedule processing later.
+ * **Fast** - Beanstalkd is **significantly** [faster then alternatives](http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend). Easily processes thousands of jobs a second.
+ * **Priorities** - Specify a higher priority and those jobs will jump ahead to be processed first accordingly.
+ * **Persistence** - Jobs are stored in memory for speed (ala memcached), but also logged to disk for safe keeping.
+ * **Federation** - Fault-tolerance and horizontal scalability is provided the same way as Memcache - through federation by the client.
+ * **Buried jobs** - When a job causes an error, you can bury it which keeps it around for later debugging and inspection.
+
+Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
+In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup.
+
## Installation
+First, you probably want to [install beanstalkd](http://kr.github.com/beanstalkd/download.html), which powers the job queues.
+Depending on your platform, this should be as simple as (for Ubuntu):
+
+ $ sudo apt-get install beanstalkd
+
Add this line to your application's Gemfile:
gem 'backburner'
And then execute:
@@ -59,41 +111,48 @@
NewsletterMailer.deliver_text_to_email(email, body)
end
end
```
-Notice that you must include the `Backburner::Queue` module and that you can set a `queue` name within the job automatically.
-Jobs can then be enqueued using:
+Notice that you can include the optional `Backburner::Queue` module so you can specify a `queue` name for this job.
+Jobs can be enqueued with:
```ruby
-Backburner.enqueue NewsletterJob, 'lorem ipsum...', 5
+Backburner.enqueue NewsletterJob, 'foo@admin.com', 'lorem ipsum...'
```
`Backburner.enqueue` accepts first a ruby object that supports `perform` and then a series of parameters
to that object's `perform` method. The queue name used by default is the normalized class name (i.e `{namespace}.newsletter-job`)
if not otherwise specified.
### Simple Async Jobs ###
In addition to defining custom jobs, a job can also be enqueued by invoking the `async` method on any object which
-includes `Backburner::Performable`.
+includes `Backburner::Performable`. Async enqueuing works for both instance and class methods on any _performable_ object.
```ruby
class User
include Backburner::Performable
def activate(device_id)
@device = Device.find(device_id)
# ...
end
+
+ def self.reset_password(user_id)
+ # ...
+ end
end
+# Async works for instance methods on a persisted model
@user = User.first
@user.async(:pri => 1000, :ttr => 100, :queue => "user.activate").activate(@device.id)
+# ..as well as for class methods
+User.async(:pri => 100, :delay => 10.seconds).reset_password(@user.id)
```
-This will automatically enqueue a job that will run `activate` with the specified argument for that user record.
+This will automatically enqueue a job for that user record that will run `activate` with the specified argument.
The queue name used by default is the normalized class name (i.e `{namespace}.user`) if not otherwise specified.
Note you are able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`.
### Working Jobs
@@ -172,21 +231,34 @@
### Logging
Right now, all logging happens to standard out and can be piped to a file or any other output manually. More on logging coming later.
-### Front-end Monitoring
+### Front-end
To be completed is an admin dashboard that provides insight into beanstalk jobs via a simple Sinatra front-end. Coming soon.
-## Why Backburner?
+### Workers in Production
-To be filled in. DelayedJob, Resque, Stalker, et al.
+Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html)
+is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
+ # Rakefile
+ require 'backburner/tasks'
+
+and then you can start the rake task with:
+
+ $ rake backburner:work
+ $ QUEUES=newsletter-sender,push-message rake backburner:work
+
+The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/)
+which watches processes and ensures their stability. A simple God recipe for Backburner can be found in
+[examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb).
+
## Acknowledgements
- * Nathan Esquenazi - Project maintainer
+ * [Nathan Esquenazi](https://github.com/nesquena) - Project maintainer
* Kristen Tucker - Coming up with the gem name
* [Tim Lee](https://github.com/timothy1ee), [Josh Hull](https://github.com/joshbuddy), [Nico Taing](https://github.com/Nico-Taing) - Helping me work through the idea
* [Miso](http://gomiso.com) - Open-source friendly place to work
## Contributing
@@ -197,9 +269,20 @@
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request
## References
-The code in this project has been adapted from a few excellent projects:
+The code in this project has been made in light of a few excellent projects:
* [DelayedJob](https://github.com/collectiveidea/delayed_job)
- * [Stalker](https://github.com/han/stalker)
+ * [Resque](https://github.com/defunkt/resque)
+ * [Stalker](https://github.com/han/stalker)
+
+Thanks to these projects for inspiration and certain design and implementation decisions.
+
+## Links
+
+ * Code: `git clone git://github.com/nesquena/backburner.git`
+ * Home: <http://github.com/nesquena/backburner>
+ * Docs: <http://rdoc.info/github/nesquena/backburner/master/frames>
+ * Bugs: <http://github.com/nesquena/backburner/issues>
+ * Gems: <http://gemcutter.org/gems/backburner>
\ No newline at end of file