Qless In 50 words or less

Qless is a Redis-backed, robust, language-agnostic job queueing system. It is a collection of Lua scripts that run on a Redis server, as well as this web app). Qless is open-sourced under the MIT license and currently has bindings for Ruby, Python with C++ and Node.js support coming soon.

A job is a unit of work. A queue can contain several jobs that are scheduled to be run at a certain time, several jobs that are waiting to run, and jobs that are currently running. A worker is a process on a host, identified uniquely, that asks for jobs from the queue, performs some process associated with that job, and then marks it as complete. When it's completed, it can be put into another queue.

Jobs can only be in one queue at a time. That queue is whatever queue they were last put in. So if a worker is working on a job, and you move it, the worker's request to complete the job will be ignored.

A job can be canceled, which means it disappears into the ether, and we'll never pay it any mind ever again. A job can be dropped, which is when a worker fails to heartbeat or complete the job in a timely fashion, or a job can be failed, which is when a host recognizes some systematically problematic state about the job. A worker should only fail a job if the error is likely not a transient one; otherwise, that worker should just drop it and let the system reclaim it.

Jobs don't get dropped on the floor

even in the face of the flakiest workers

Jobs are stored temporarily

but automatically expire after a configurable amount of time

Tagging and Tracking

for keeping tabs on important jobs

High Performance

thanks to Redis and Lua

Scheduled Work

for delayed or periodic processing

Retry Logic

for jobs that have been dropped on the floor

Web App

for checking up on progress and monitoring

Priority

for jobs means you can put selected jobs before others

Stats

about how long jobs wait, how long they take to process

These benchmarks were performed on a m1.medium, which has 3.75GB of RAM and 2 ECUs. Actually, I'm going to wait until development's a little more complete before posting these.