# AllSeeingEye

A highly configurable Gem to collect specific data from requests, and view that data in a cute little Sinatra app.

AllSeeingEye writes information about each request it observes into Redis. What kind of information? Why, whatever
you want! From the relatively banal (IP address, URI, user agent) to complicated application-specific data
(user account, ordered product, money spent), anything can be recorded by AllSeeingEye, stored in Redis, and indexed to be
viewed quickly in Sinatra. With pretty Flot-based graphs, even.

It comes with a Rails2 integration by default, but I think with a little work it'd be pretty easy to get it into Rails3.

## Background

Our application receives about half a million requests every day. While we have Google Analytics, and it's totally awesome,
many of those requests come from our iPhone app, and those requests don't receive the level of tracking we need. We could load
reporting software onto a users' iOS device: but, ethical issues aside, why would we want to? Almost anything we'd be
interested in tracking requires server interaction anyway, so it's easier and faster to track it all on the server side.

The obvious solution was to mine the Rails logfiles, but we have multiple servers and weaving all the logs was a pain -- and
even when we did, some of the data we wanted wasn't exposed. My next thought was to log requests to our database, but we
experience enough database traffic as is without writing every request. A MemCached-backed write-through buffer temporarily
worked but when the writes happened the database still got very unhappy... and the table got enormous very quickly.

Redis was a good compromise though. We use it for the truly excellent Resque, so it's already part of our stack. It's fast
and, while it's persistent, it's not exactly a tragedy if the Redis DB gets hosed, as none of the request data is so important
we can't just start collecting it again. Combined with some code to store arbitrary data, we quickly had a winner on our hands.

So, AllSeeingEye was born out of our need to track incoming requests accurately, quickly, and in a way that is easily
displayable to our business users.

## Installation

Get Redis via the amazing Homebrew: `brew install redis`

Add AllSeeingEye to your project's Gemfile: `gem 'all\_seeing\_eye'`

Install it with Bundler: `bundle install`

## Setup

bundle exec ruby script/generate all\_seeing\_eye

This puts the AllSeeingEye config file at config/all\_seeing\_eye.yml. The file is annotated with all the information you
need to get started, but some quick explanation:

### Timeout
AllSeeingEye's data collection is wrapped by a timeout to ensure speediness. I keep this value at 30 milliseconds (0.03), but
feel free to change it to suit your application's needs.

### Round To Seconds
One data point per request is too many to sensibly view on a graph. This rounds the created\_at time of requests
to whatever (I choose 10 minutes), so that all requests within 10 minutes are collected and displayed together.

### Redis
Environment and then location of the redis server. If you do not include this key, AllSeeingEye tries to look for a redis.yml
or a resque.yml file instead to find the server.

### Models
You can define a number of different models to collect data with. Each has its own separate namespace, so data won't be
shared between them. I only use one right now (I call it 'request'), but if you want more then go for it.

Each model definition should be structured like this:

model\_name:
first\_field\_name:
object: object_name
method: .method_name
second\_field\_name:
object: hash
method: "['key']"

Model names and field names are completely arbitrary, and I try to humanize them as best as I can.

Object and method are the literal object and the actual method you want executed, with one exception: the object
'controller' is translated into 'self'. The result of this is stored by AllSeeingEye in redis. So, for example, an
object could be "request" and the method could be "request\_uri". I'm trying to do a little secure sanity-checking here
rather than just directly eval'ing a string, but you can still mess things up pretty badly if you choose a bad object
and/or bad method... so try not to do that.

As a helpful hint, recording "request\_uri" isn't nearly as helpful as you might think. Try to save requests without any params
and without a leading slash by defining a method in ApplicationController instead:

protected
def clean_request
request.request_uri.split('?').first[1..-1]
end

Then, in your all\_seeing\_eye.yml:

request:
uri:
object: controller
method: .clean_request

## Running

When your Rails application starts, you should see each request contain "+++ Request watched by AllSeeingEye." That means
AllSeeingEye is running successfully. Good job!

Starting the server is simple. Just type `all-seeing-eye` from the commandline and Sinatra should start automatically.

## Extras

I included the Nginx config we use for AllSeeingEye as a small bonus. It's in examples/all\_seeing\_eye-nginx.conf.

## Contributing to AllSeeingEye

* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
* Fork the project
* Start a feature/bugfix branch
* Commit and push until you are happy with your contribution
* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.