README.md in backburner-1.0.0 vs README.md in backburner-1.1.0

- old
+ new

@@ -86,39 +86,43 @@ Backburner is extremely simple to setup. Just configure basic settings for backburner: ```ruby Backburner.configure do |config| - config.beanstalk_url = ["beanstalk://127.0.0.1", "..."] - config.tube_namespace = "some.app.production" - config.on_error = lambda { |e| puts e } - config.max_job_retries = 3 # default 0 retries - config.retry_delay = 2 # default 5 seconds - config.default_priority = 65536 - config.respond_timeout = 120 - config.default_worker = Backburner::Workers::Simple - config.logger = Logger.new(STDOUT) - config.primary_queue = "backburner-jobs" - config.priority_labels = { :custom => 50, :useless => 1000 } - config.reserve_timeout = nil + config.beanstalk_url = ["beanstalk://127.0.0.1", "..."] + config.tube_namespace = "some.app.production" + config.namespace_separator = "." + config.on_error = lambda { |e| puts e } + config.max_job_retries = 3 # default 0 retries + config.retry_delay = 2 # default 5 seconds + config.retry_delay_proc = lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 3) } + config.default_priority = 65536 + config.respond_timeout = 120 + config.default_worker = Backburner::Workers::Simple + config.logger = Logger.new(STDOUT) + config.primary_queue = "backburner-jobs" + config.priority_labels = { :custom => 50, :useless => 1000 } + config.reserve_timeout = nil end ``` The key options available are: -| Option | Description | -| ----------------- | ------------------------------- | -| `beanstalk_url` | Address such as 'beanstalk://127.0.0.1' or an array of addresses. | -| `tube_namespace` | Prefix used for all tubes related to this backburner queue. | -| `on_error` | Lambda invoked with the error whenever any job in the system fails. | -| `default_worker` | Worker class that will be used if no other worker is specified. | -| `max_job_retries` | Integer defines how many times to retry a job before burying. | -| `retry_delay` | Integer defines the base time to wait (in secs) between job retries. | -| `logger` | Logger recorded to when backburner wants to report info or errors. | -| `primary_queue` | Primary queue used for a job when an alternate queue is not given. | -| `priority_labels` | Hash of named priority definitions for your app. | -| `reserve_timeout` | Duration to wait for work from a single server, or nil for forever. | +| Option | Description | +| ----------------- | ------------------------------- | +| `beanstalk_url` | Address such as 'beanstalk://127.0.0.1' or an array of addresses. | +| `tube_namespace` | Prefix used for all tubes related to this backburner queue. | +| `namespace_separator` | Separator used for namespace and queue name | +| `on_error` | Lambda invoked with the error whenever any job in the system fails. | +| `default_worker` | Worker class that will be used if no other worker is specified. | +| `max_job_retries` | Integer defines how many times to retry a job before burying. | +| `retry_delay` | Integer defines the base time to wait (in secs) between job retries. | +| `retry_delay_proc` | Lambda calculates the delay used, allowing for exponential back-off. | +| `logger` | Logger recorded to when backburner wants to report info or errors. | +| `primary_queue` | Primary queue used for a job when an alternate queue is not given. | +| `priority_labels` | Hash of named priority definitions for your app. | +| `reserve_timeout` | Duration to wait for work from a single server, or nil for forever. | ## Breaking Changes Since **v0.4.0**: Jobs used to be placed into default queues based on the name of the class enqueuing i.e NewsletterJob would be put into a 'newsletter-job' queue. After 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs" @@ -152,11 +156,11 @@ 1000 # most urgent priority is 0 end # optional, defaults to respond_timeout def self.queue_respond_timeout - 300 # number of seconds before job times out + 300 # number of seconds before job times out, 0 to avoid timeout end end ``` You can include the optional `Backburner::Queue` module so you can easily specify queue settings for this job: @@ -164,11 +168,11 @@ ```ruby class NewsletterJob include Backburner::Queue queue "newsletter-sender" # defaults to 'backburner-jobs' tube queue_priority 1000 # most urgent priority is 0 - queue_respond_timeout 300 # number of seconds before job times out + queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout def self.perform(email, body) NewsletterMailer.deliver_text_to_email(email, body) end end @@ -182,21 +186,25 @@ `Backburner.enqueue` accepts first a ruby object that supports `perform` and then a series of parameters to that object's `perform` method. The queue name used by default is `{namespace}.backburner-jobs` unless otherwise specified. +You may also pass a lambda as the queue name and it will be evaluated when enqueuing a +job (and passed the Job's class as an argument). This is especially useful when combined +with "Simple Async Jobs" (see below). + ### Simple Async Jobs ### In addition to defining custom jobs, a job can also be enqueued by invoking the `async` method on any object which includes `Backburner::Performable`. Async enqueuing works for both instance and class methods on any _performable_ object. ```ruby class User include Backburner::Performable queue "user-jobs" # defaults to 'user' queue_priority 500 # most urgent priority is 0 - queue_respond_timeout 300 # number of seconds before job times out + queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout def activate(device_id) @device = Device.find(device_id) # ... end @@ -214,12 +222,60 @@ ``` This automatically enqueues a job for that user record that will run `activate` with the specified argument. Note that you can set the queue name and queue priority at the class level and you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`. -The queue name used by default is `{namespace}.backburner-jobs` if not otherwise specified. +The queue name used by default is `{namespace}.backburner-jobs` if not otherwise +specified. + +If a lambda is given for `queue`, then it will be called and given the +_performable_ object's class as an argument: + +```ruby +# Given the User class above +User.async(:queue => lambda { |user_klass| ["queue1","queue2"].sample(1).first }).do_hard_work # would add the job to either queue1 or queue2 randomly +``` + +### Using Async Asynchronously ### + +It's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in [delayed_job](https://github.com/collectiveidea/delayed_job#queuing-jobs). To accomplish this, the `Backburner::Performable` module exposes two `handle_asynchronously` convenience methods +which accept the same options as the `async` method: + +```ruby +class User + include Backburner::Performable + + def send_welcome_email + # ... + end + + # ---> For instance methods + handle_asynchronously :send_welcome_email, queue: 'send-mail', pri: 5000, ttr: 60 + + def self.update_recent_visitors + # ... + end + + # ---> For class methods + handle_static_asynchronously :update_recent_visitors, queue: 'long-tasks', ttr: 300 +end +``` + +Now, all calls to `User.update_recent_visitors` or `User#send_welcome_email` will automatically be handled asynchronously when invoked. Similarly, you can call these methods directly on the `Backburner::Performable` module to apply async behavior outside the class: + +```ruby +# Given the User class above +Backburner::Performable.handle_asynchronously(User, :activate, ttr: 100, queue: 'activate') +``` + +Now all calls to the `activate` method on a `User` instance will be async with the provided options. + +#### A Note About Auto-Async + +Because an async proxy is injected and used in place of the original method, you must not rely on the return value of the method. Using the example `User` class above, if my `send_welcome_email` returned the status of an email submission and I relied on that to take some further action, I will be surprised after rewiring things with `handle_asynchronously` because the async proxy actually returns the (boolean) result of `Backburner::Worker.enqueue`. + ### Working Jobs Backburner workers are processes that run forever handling jobs that are reserved from the queue. Starting a worker in ruby code is simple: ```ruby @@ -364,11 +420,13 @@ ``` $ QUEUE=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work ``` For more information on the threads_on_fork worker, check out the -[ThreadsOnFork Worker](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) documentation. +[ThreadsOnFork Worker](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) documentation. Please note that the `ThreadsOnFork` worker does not work on Windows due to its lack of `fork`. + + Additional workers such as individual `threaded` and `forking` strategies will hopefully be contributed in the future. If you are interested in helping out, please let us know. ### Default Queues @@ -399,20 +457,33 @@ The `default_queues` stores the specific list of queues that should be processed by default by a worker. ### Failures When a job fails in backburner (usually because an exception was raised), the job will be released -and retried again (with progressive delays in between) until the `max_job_retries` configuration is reached. +and retried again until the `max_job_retries` configuration is reached. ```ruby Backburner.configure do |config| config.max_job_retries = 3 # retry jobs 3 times config.retry_delay = 2 # wait 2 seconds in between retries end ``` Note the default `max_job_retries` is 0, meaning that by default **jobs are not retried**. + +As jobs are retried, a progressively-increasing delay is added to give time for transient +problems to resolve themselves. This may be configured using `retry_delay_proc`. It expects +an object that responds to `#call` and receives the value of `retry_delay` and the number +of times the job has been retried already. The default is a cubic back-off, eg: + +```ruby +Backburner.configure do |config| + config.retry_delay = 2 # The minimum number of seconds a retry will be delayed + config.retry_delay_proc = lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 3) } +end +``` + If continued retry attempts fail, the job will be buried and can be 'kicked' later for inspection. You can also setup a custom error handler for jobs using configure: ```ruby @@ -466,10 +537,43 @@ The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/) which watches processes and ensures their stability. A simple God recipe for Backburner can be found in [examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb). +#### Command-Line Interface + +Instead of using the Rake tasks, you can use Backburner's command-line interface (CLI) – powered by the [Dante gem](https://github.com/nesquena/dante) – to launch daemonized workers. Several flags are available to control the process. Many of these are provided by Dante itself, such as flags for logging (`-l`), the process' PID (`-P`), whether to daemonize (`-d`) or kill a running process (`-k`). Backburner provides a few more: + + +##### Queues (`-q`) + +Control which queues the worker will watch with the `-q` flag. Comma-separate multiple queue names and, if you're using the `ThreadsOnFork` worker, colon-separate the settings for thread limit, garbage limit and retries limit (eg. `send_mail:4:10:3`). See its [wiki page](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) for some more details. + +```ruby +backburner -q send_mail,create_thumbnail # You may need to use `bundle exec` +``` + +##### Boot an app (`-r`) + +Load an app with the `-r` flag. Backburner supports automatic loading for both Rails and Padrino apps when started from the their root folder. However, you may point to a specific app's root using this flag, which is very useful when running workers from a service script. + +```ruby +path="/var/www/my-app/current" +backburner -r "$path" +``` + +##### Load an environment (`-e`) + +Use the `-e` flag to control which environment your app should use: + +```ruby +environment="production" +backburner -e $environment +``` + +#### Reconnecting + In Backburner, if the beanstalkd connection is temporarily severed, several retries to establish the connection will be attempted. After several retries, if the connection is still not able to be made, a `Beaneater::NotConnected` exception will be raised. You can manually catch this exception, and attempt another manual retry using `Backburner::Worker.retry_connection!`. ### Web Front-end @@ -479,9 +583,10 @@ jobs processed by your beanstalk workers. An excellent addition to your Backburner setup. ## Acknowledgements * [Nathan Esquenazi](https://github.com/nesquena) - Project maintainer + * [Dave Myron](https://github.com/contentfree) - Multiple features and doc improvements * Kristen Tucker - Coming up with the gem name * [Tim Lee](https://github.com/timothy1ee), [Josh Hull](https://github.com/joshbuddy), [Nico Taing](https://github.com/Nico-Taing) - Helping me work through the idea * [Miso](http://gomiso.com) - Open-source friendly place to work * [Evgeniy Denisov](https://github.com/silentshade) - Multiple fixes and cleanups * [Andy Bakun](https://github.com/thwarted) - Fixes to how multiple beanstalkd instances are processed