# @title Ruby AMQP gem: Troubleshooting and debugging AMQP applications

h1. Troubleshooting and debugging AMQP applications

h2. This Documentation Has Moved to rubyamqp.info

amqp gem documentation guides are now hosted on "rubyamqp.info":http://rubyamqp.info.


h2. About this guide

This guide describes tools and strategies that help in troubleshooting and debugging applications that use AMQP in general and "Ruby AMQP gem":http://github.com/ruby-amqp/amqp
in particular.


h2. Covered versions

This guide covers "Ruby amqp gem":http://github.com/ruby-amqp/amqp v0.8.0 and later.



h2. First steps

Whenever something doesn't work, check the following things before asking on the mailing list:

 * AMQP broker log.
 * List of users in a particular vhost you are trying to connect.
 * Network connectivity. We know, it's obvious, yet even experienced developers and devops engineers struggle with network access misconfigurations every once in a while.
 * If EventMachine is started in a separate thread, make sure that isn't dead. If it is, this usually means there was an exception that caused it to terminate,
   or environment it is running in "mixes threads and fork(2) system call":http://bit.ly/fork-and-threads.



h2. Inspecting AMQP broker log file

In this section we will cover typical problems that can be tracked down by reading AMQP broker log. We will use RabbitMQ as an example, however, different AMQP brokers
often log most of the same issues.

RabbitMQ logs abrupt TCP connection failures, timeouts, protocol version mismatches and so on.
If you are running RabbitMQ, log locations for various operating systems and distributions is documented in the "RabbitMQ installation guide":http://www.rabbitmq.com/install.html

On Mac OS X, RabbitMQ installed via Homebrew logs to $HOMEBREW_HOME/var/log/rabbitmq/rabbit@$HOSTNAME.log. For example, if you have Homebrew installed at /usr/local and
your hostname is giove, log will be at /usr/local/var/log/rabbitmq/rabbit@giove.log.

Here is what authentication failure looks like in RabbitMQ log:

<pre>
=ERROR REPORT==== 17-May-2011::17:37:58 ===
exception on TCP connection <0.4770.0> from 127.0.0.1:46551
{channel0_error,starting,
                {amqp_error,access_refused,
                            "AMQPLAIN login refused: user 'pipeline_agent' - invalid credentials",
                            'connection.start_ok'}}
</pre>

This means that connection attempt with username pipeline_agent failed because credentials were invalid. If you are seeing this message, make sure username,
password *and vhost* are correct.


The following entry:

<pre>
=ERROR REPORT==== 17-May-2011::17:26:28 ===
exception on TCP connection <0.4201.62> from 10.8.0.30:57990
{bad_header,<<65,77,81,80,0,0,9,1>>}
</pre>

Means that client supports AMQP 0.9.1 but broker doesn't (RabbitMQ versions pre-2.0 only support AMQP 0.8, for example). If you are using amqp gem 0.8 or later
and seeing this entry in your broker log, you are connecting to AMQP broker that is too old to support this AMQP version. In case of RabbitMQ, make sure you run
version 2.0 or later.

TBD


h2. Handling channel-level exceptions

A broad range of problems result in AMQP channel exceptions: an indication by the broker that there was an issue application needs to be aware of.
Channel-level exceptions are typically not fatal and can be recovered from. Some examples are:

 * Exchange is re-declared with attributes different from the original declaration. For example, a non-durable exchange is being re-declared as durable.
 * Queue is re-declared with attributes different from the original declaration. For example, an autodeletable queue is being re-declared as non-autodeletable.
 * Queue is bound to an exchange that does not exist.

and so on. When troubleshooting AMQP applications, it is recommended that you detect and handle channel-level exceptions on all channels your
application may use. For that, use {AMQP::Channel#on_error} method as demonstrated below:

<pre>
<code>
events_channel.on_error do |ch, channel_close|
  puts "Channel-level exception on the events channel: #{channel_close.reply_text}"
end

commands_channel.on_error do |ch, channel_close|
  puts "Channel-level exception on the commands channel: #{channel_close.reply_text}"
end
</code>
</pre>

Defining channel-level exception handlers will reveal many issues that it might take more time to detect using other troubleshooting techniques.


h2. Testing network connection with AMQP broker using Telnet

One simple way to check network connection between a particular network node and a node running AMQP broker is to use `telnet`:

<pre>
telnet [host or ip] 5672
</pre>

then enter any random string of text and hit Enter. AMQP broker should immediately close down the connection. Here is an example
session:

<pre>
telnet localhost 5672
Connected to localhost.
Escape character is '^]'.
adjasd
AMQP    Connection closed by foreign host.
</pre>

If instead Telnet exits after printing

<pre>
telnet: connect to address [host or ip]: Connection refused
telnet: Unable to connect to remote host
</pre>

then connection from the machine you are running Telnet tests on and AMQP broker fails. This can be due to many different reasons, but
it is a good idea to check these two things first:

 * Firewall configuration for port 5672
 * DNS setup (if hostname is used)



h2. Broker Startup Issues

h3. Missing erlang-os-mon on Debian and Ubuntu

The following error on RabbitMQ startup on Debian or Ubuntu

<pre>
ERROR: failed to load application os_mon: {"no such file or directory","os_mon.app"}
</pre>

suggests that *erlang-os-mon* package is not installed.



h2. Authors

This guide was written by "Michael Klishin":http://twitter.com/michaelklishin and edited by "Chris Duncan":https://twitter.com/celldee.



h2. Tell us what you think!

Please take a moment and tell us what you think about this guide "on Twitter":http://twitter.com/rubyamqp or "Ruby AMQP mailing list":http://groups.google.com/group/ruby-amqp:
what was unclear? what wasn't covered? maybe you don't like guide style or grammar and spelling are incorrect? Readers feedback is
key to making documentation better.

If mailing list communication is not an option for you for some reason, you can "contact guides author directly":mailto:michael@novemberain.com?subject=amqp%20gem%20documentation


<div id="disqus_thread"></div>
<script type="text/javascript">
    /* * * CONFIGURATION VARIABLES * * */
    var disqus_shortname = 'rubyamqpdocs'; // required: replace example with your forum shortname

    var disqus_developer = 0; // set to 1 on local machine for testing comments
    var disqus_identifier = 'amqp_troubleshooting';
    var disqus_url = 'http://rdoc.info/github/ruby-amqp/amqp/master/file/docs/Troubleshooting.textile';

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
        var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
        dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>