= Simple background loops framework

<tt>loops</tt> is a small and lightweight framework for Ruby on Rails, Merb and other ruby
frameworks created to support simple background loops in your application which are usually
used to do some background data processing on your servers (queue workers, batch tasks
processors, etc).

_Warning_: If you use some pre-2.0 version of this plugin, read a dedicated paragraph below.


== What would you use it for?

Originally loops plugin was created to make our own loops code a bit more organized. We used
to have dozens of different modules with methods that were called with script/runner and then
used with nohup and other painful backgrounding techniques. When you have such a number of
loops/workers to run in background it becomes a nightmare to manage them on a regular basis
(restarts, code upgrades, status/health checking, etc).

After a few takes on writing our scripts in a more organized way we were able to generalize most
of the code so now our loops started looking like a classes with a single mandatory public method
called *run*. Everything else (spawning many workers, managing them, logging, backgrounding,
pid-files management, etc) is handled by the plugin itself.


== But there are dozens of libraries like this! Why do we need yet another one?

The major idea behind this small project was to create a deadly simple and yet robust framework to
be able to run some tasks in background and do not think about spawning many workers, restarting
them when they die, etc. So, if you need to be able to run either one or many copies of your worker
and you do not want to think about re-spawning your scripts when they die and you do not want to
spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your
loop as a separate process controlled by monit/god/etc), then you should try this framework --
you're going to like it.


== How to install?

There are two options when approaching db-charmer installation:

* using the gem (recommended)
* install as a Rails plugin

To install as a gem, add this to your environment.rb:

    config.gem 'loops'

And then run the command:

    sudo rake gems:install

To install loops as a Rails plugin you need to do rhw following:

    ./script/plugin install git://github.com/kovyrin/loops.git

This will install the whole package in your vendor/plugins directory.
For merb applications, just check out the code and place it to the vendor/plugins directory.


After you are done with the installation, you need to generate binary and configuration
files by running:

    ./script/generate loops

This will create the following list of files:
* <tt>./script/loops</tt> - binary file that will be used to manage your loops
* <tt>./config/loops.yml</tt> - example configuration file
* <tt>./app/loops/simple.rb</tt> - REALLY simple loop example
* <tt>./app/loops/queue_loop.rb</tt> - simple ActiveMQ queue worker


== How to use?

Here is a simple loop scaffold for you to start from (put this file to
<tt>app/loops/hello_world_loop.rb</tt>):

    class HelloWorldLoop < Loops::Base
      def run
        with_period_of(1) do # period is in seconds
          debug("Hello, debug log!")
          sleep(config['sleep_period']) # Do something "useful" and make it configurable
          debug("Hello, debug log (yes, once again)!")
        end
      end
    end

When you have your loop ready to use, add the following lines to your (maybe empty yet)
<tt>config/loops.yml</tt> file:

    hello_world:
      type: simple
      sleep_period: 10

This is it! To manage your loop, just run one of the following commands:

* To list all configured loops:

    $ ./script/loops list

* To run all enabled (actually non-disabled) loops in foreground:

    $ ./script/loops start

* To run all enabled loops in background:

    $ ./script/loops start -d

* To run specific loop in background:

    $ ./script/loops start hello_world -d

* To see all possible options:

    $ ./script/loops help


_Notice:_ If you use loops as a gem, you will get a system binary called <tt>loops</tt> which you
could use instead of your <tt>./script/loops<tt> binary:

    $ loops help


== How to run more than one worker?

If you want to have more than one copy of your worker running, that is as simple as adding one
option to your loop configuration:

    hello_world:
      type: simple
      sleep_period: 10
      workers_number: 1

This <tt>workers_number</tt> option would tell loops manager to spawn more than one copy of
your loop and run them in parallel. The only thing you'd need to do is to remember about
concurrent work of your loops. For example, if you have some kind of database table with elements
you need to process, you can create a simple database-based locks system or use any
memcache-based locks.


== How to run more than one loop using the same class?

You can run the same loop class with different configuration parameters by explicitly identifying
the loop class to execute:

    hello:
      type: simple
      loop_name: some_module/my_worker
      language: English

    salut:
      type: simple
      loop_name: some_module/my_worker
      language: French

Now two independent sets of loops are using the same class <tt>SomeModule::MyWorkerLoop</tt>
customized by the language parameter.


== I want to keep my loop running on machine reboots. How to do it?

We use monit to keep loop monitors runnings. You could use something like this in your configs:

    check process loop-slow_logs with pidfile /your/project/current/tmp/pids/loop-slow_logs.pid
      group loops
      start program "/your/project/current/script/loops start slow_logs -e loops -p tmp/pids/loop-slow_logs.pid -d"
      stop  program "/your/project/current/script/loops stop  slow_logs -e loops -p tmp/pids/loop-slow_logs.pid"


== ActiveMQ-based workers? What's that?

In some of our worker loops we poll ActiveMQ queue and process its items to perform some
asynchronous operations. So, to make it simpler for us to create such a workers, we've created
really simple loops class extension that wraps your code with basic queue polling/acknowledging
code and as the result, you can create a loops like this:

    class MyQueueLoop < Loops::Queue
      def process_message(message)
        debug "Received a message: #{message.body}"
        debug "sleeping..."
        sleep(0.5 + rand(10) / 10.0) # do something "useful" with the message :-)
        debug "done..."
      end
    end

With configs like this:

    # An example of a STOMP queue-based loop
    my_queue:
      type: queue
      host: 127.0.0.1
      port: 61613
      queue_name: blah

Of course, this solution scales perfectly and to make your queue processing faster you just
need to add more workers (by adding <tt>workers_number: N</tt> option).

_Warning_: This type of loops requires you to have the <tt>stomp</tt> gem installed in your
system.


== There is this <tt>workers_engine</tt> option in the config file. What do you use it for?

There are two so called "workers engines" in loops: <tt>fork</tt> and <tt>thread</tt>.
They're used to control the way process manager would spawn new loops workers:
with the <tt>fork</tt> engine we'll load all the loops classes and then fork ruby interpreter
as many times as many workers we need. With the <tt>thread</tt> engine we'd do Thread.new
instead of forking. Thread engine could be useful if you are sure your loop won't lock
ruby interpreter (it does not do native calls, etc) or if you use some interpreter that does
not support forks (like jruby).

The default engine is <tt>fork</tt>.


== What Ruby implementations does it work for?

We've tested and used the plugin on MRI 1.8.6/1.8.7 and on JRuby 1.4.0. At this point we do
not support demonization in JRuby and never tested the code on Ruby 1.9. Obviously because
of JVM limitations you won't be able to use <tt>fork</tt> workers engine in JRuby, but
threaded workers do pretty well.

Recommended version of ruby ro run loops is Ruby Enterprise Edition. This is because we
have a support for their Copy-On-Write friendly garbage collector that makes your loops
much smaller in memory (since they share the most of the code). Even with one loop process
you'd save some memory because your loop monitor process would share most of the memory with
the loop itself. When you run on RubyEE, you could use <tt>loops stats</tt> command to get
detailed loops memory stats:

        [root@analyics current]# ./script/loops stats

        --------- Loops processes ----------
        PID   PPID  VMSize    Private  Name
        ------------------------------------
        9062  1     199.3 MB  32.4 MB  loops monitor: activemq
        9234  9062  211.9 MB  37.5 MB  loop worker: activemq
        9251  9062  213.3 MB  38.4 MB  loop worker: activemq
        9267  9062  211.9 MB  37.1 MB  loop worker: activemq
        9268  9062  211.9 MB  38.0 MB  loop worker: activemq
        ### Processes: 5
        ### Total private dirty RSS: 183.33 MB



== Migrating from pre-2.0 releases

Before version 2.0 has been released, this code was developed as a Rails plugin only and
did not have any versions numbering system in place. So we call all those old versions
a pre-2.0 releases. If you use one of those relases (if your loops plugin does not have
the <tt>VERSION.yml</tt> file in the root directory), be careful when upgrading because
there are a few incompatible changes we have made in the loops command: <tt>-h</tt>,
<tt>-a</tt>, <tt>-s</tt>, <tt>-L</tt> and <tt>-l</tt> options were deprecated
and replaced with a friendlier word commands.
Use <tt>loops help</tt> to get help.


== Who are the authors?

This plugin has been created in Scribd.com for our internal use and then the sources were opened
for other people to use. All the code in this package has been developed by Alexey Kovyrin,
Dmytro Shteflyuk and Alexey Verkhovsky for Scribd.com and is released under the MIT license.
For more details, see LICENSE file.