*PUPPET QUEUEING

Puppet Queueing is a feature which is designed to take some load
off of the PuppetMaster by transferring the task of updating the
database to a separate program which is named puppetqd (Puppet
Queue Daemon).

Currently this is only supported for "Storeconfigs" which is
documented at:

http://puppetlabs.com/trac/puppet/wiki/UsingStoredConfiguration

In the future this feature can be extended to any new puppet
data which involves storage in a database.

*OPERATION

In a nutshell:

  puppetmasterd -> stomp -> service -> stomp -> puppetqd -> database

At the moment the only messaging protocol supported is "stomp". Although
others could be implemented, stomp is considered by many as the
default queueing mechanism for Ruby and Rails applications. It is
distributed as a Ruby gem and is easily installed.

(The queueing code inside Puppet has been written so that when other
interfaces and protocols are implemented they will be easy to use by
changing settings in puppet.conf).

The "service" in the diagram above is any queueing service that supports
the Stomp API. For details refer to:

  http://xircles.codehaus.org/projects/stomp

Both puppetmasterd and puppetqd subscribe to the same queueing service
using the stomp interface. As puppetmasterd posts data to the queue,
puppetqd receives it and stores it. The details of how to connect to
the service and the name of the queue to use are set in puppet.conf:

  [main]
    queue_type = stomp
    queue_source = stomp://localhost:61613
  [puppetmasterd]
    async_storeconfigs = true

Note: since puppetmasterd needs to recover the data being stored at a
later time, both puppetmasterd and puppetqd need to work with the same
database as defined in the STORECONFIGS setup.

*QUEUEING SERVICES

As mentioned previously any queueing service that supports the Stomp
protocol can be used. Which one you use depends on your needs. We have
tested with two of the most popular services - StompServer and ActiveMQ.

+ StompServer

    http://rubyforge.org/projects/stompserver/

StompServer is a lightweight queueing service written in Ruby which is
suitable for testing or low volume puppet usage. Works well when both
puppetmasterd and puppetd are running on the same machine that it's running
on but we encountered some problems when using it from multiple machines.

Just install the stompserver gem and run 'stompserver'.

+ Apache ActiveMQ

    http://activemq.apache.org

Considered by many to be the most popular message service in use today,
ActiveMQ has hundreds of features for scaling, persistence and so on.

Although installation is fairly simple, the configuration can seem quite
intimidating, but for our use a one line change to the  standard configuration
is all that is required and is explained at:

    http://activemq.apache.org/stomp.html

Other customization of the internal workings of ActiveMQ, if any, will depend
on your needs and deployment. A quick skimming of the ActiveMQ documentation
will give you enough info to decide.

Others

We have looked at but not tried some other queuing services which are
compatible with the Stomp API:

+ POE Component Message Queue
+ JBoss Messaging (with 3rd party support for Stomp)

*SCALING

For StoreConfigs you basically need to have the catalog for a node stored
in the database before the next time the node connects and asks for a
new catalog.

If the puppetd on your nodes is set to check every 30 minutes,
then it would seem that there is no problem. However if you have 3000
nodes you have a LOT of catalogs to store and it is possible you will
not get a catalog saved in time.

Running puppetmaster, your queueing service and puppetqd on the same
machine means that they are all competing for the same CPU cycles. Bumping
up the power of the server they are running on may be enough to handle
even fairly large deployments.

However since most queueing services (even StompServer) are designed to
deliver messages from a "queue" to whoever asks for the next message you
can split things up between machines:

  puppetmaster1 --\             /-- puppetqd1 -\
  puppetmaster2 ----> ActiveMQ ---> puppetqd2 ---> database
  puppetmaster3 --/             \-- puppetqd33 -/
                                 \- puppetqd4-/

This is, of course a totally contrived example, but it gets the point
across. As long as the data gets to the database, it doesn't matter
which machines or services it goes through.

Although for StoreConfigs absolute reliability is not a requirement as
a new catalog will be sent the next time a node connects, some amount
of persistence should some process crash may be desirable. Both ActiveMQ
and MySQL (and other databases) have these kind of features built in
which can be activated as needed.