[](http://travis-ci.org/savonarola/pulse-meter)
# PulseMeter
PulseMeter is a gem for fast and convenient realtime aggregating of software internal stats through Redis.
## Features
PulseMeter is designed to provide the following features:
* Simple deployment. The only infrastructure resource you are required to have is Redis.
* Low resource consumption. Since different kinds of events are aggregated in Redis,
you are as light and fast as Redis is.
Event data is stored in constant space and expires over time.
* Focus on the client. To start gathering some metrics, you should only modify your client: create a sensor object
and send events to it. All aggregated data can be accessed immediately without any
sort of "server reconfiguration"
## Concept
The fundamental concept of PulseMeter is *sensor*. Sensor is some named piece of data in Redis which
can be updated through client side objects associated with this data. The semantics of the data can be
different: some counter, value, series of values, etc. There is no need to care about explicit creation this data:
one just creates a client object and writes data to it, e.g.
PulseMeter.redis = Redis.new
sensor = PulseMeter::Sensor::Counter.new :my_counter
sensor.event(5)
...
sensor.event(3)
After that the value associated with the counter is immediately available (through CLI, for example). Any other
client can access the associated counter by creating object with the same redis db and sensor name.
Sensors can be divided into two large groups.
### Static sensors
These are just single values which can be read by CLI, e.g. some counter or some value
representing current state of a resource (current free memory amount, current la etc.). Currently, the
following static sensors are available:
* Counter
* Hashed Counter
* Unique Counter
* Indicator
They have no web visualisation interface and they are assumed to be used by external visualisation tools.
### Timeline sensors
These sensors are series of values, one value for each consequent time interval. They
are available by CLI and have web visualisation interface. Examples of such sensors include: count of
requests to some resource per hour, the longest request to a database per minute, etc.
The following timeline sensors are available:
* Average value
* Counter
* Hashed counter
* Max value
* Min value
* Median value
* Percentile
* Unique counter
There are several caveats with timeline sensors:
* The value of a sensor for the last interval (which is not finished yet) is often not very useful.
When building a visualisation you may choose to display the last value or not.
* For some sensors (currently Median and Percentile) considerable amount of data should be stored for a
particular interval to obtain value for this interval. So it is a good idea to schedule
pulse reduce
command on a regular basis. This command reduces the stored data for passed intervals to single values,
so that they do not consume storage space.
## Client usage
Just create sensor objects and write data. Some examples below.
require 'pulse-meter'
PulseMeter.redis = Redis.new
# static sensor examples
counter = PulseMeter::Sensor::Counter.new :my_counter
counter.event(1)
counter.event(2)
puts counter.value
# prints
# 3
indicator = PulseMeter::Sensor::Indicator.new :my_value
indicator.event(3.14)
indicator.event(2.71)
puts indicator.value
# prints
# 2.71
hashed_counter = PulseMeter::Sensor::HashedCounter.new :my_h_counter
hashed_counter.event(:x => 1)
hashed_counter.event(:y => 5)
hashed_counter.event(:y => 1)
p hashed_counter.value
# prints
# {"x"=>1, "y"=>6}
# timeline sensor examples
requests_per_minute = PulseMeter::Sensor::Timelined::Counter.new(:my_t_counter,
:interval => 60, # count for each minute
:ttl => 24 * 60 * 60 # keep data one day
)
requests_per_minute.event(1)
requests_per_minute.event(1)
sleep(60)
requests_per_minute.event(1)
requests_per_minute.timeline(2 * 60).each do |v|
puts "#{v.start_time}: #{v.value}"
end
# prints somewhat like
# 2012-05-24 11:06:00 +0400: 2
# 2012-05-24 11:07:00 +0400: 1
max_per_minute = PulseMeter::Sensor::Timelined::Max.new(:my_t_max,
:interval => 60, # max for each minute
:ttl => 24 * 60 * 60 # keep data one day
)
max_per_minute.event(3)
max_per_minute.event(1)
max_per_minute.event(2)
sleep(60)
max_per_minute.event(5)
max_per_minute.event(7)
max_per_minute.event(6)
max_per_minute.timeline(2 * 60).each do |v|
puts "#{v.start_time}: #{v.value}"
end
# prints somewhat like
# 2012-05-24 11:07:00 +0400: 3.0
# 2012-05-24 11:08:00 +0400: 7.0
There is also an alternative and a bit more DRY way for sensor creation, management and usage using PulseMeter::Sensor::Configuration class. It is also convenient for creating a bunch of sensors from some configuration data.
require 'pulse-meter'
PulseMeter.redis = Redis.new
sensors = PulseMeter::Sensor::Configuration.new(
my_counter: {sensor_type: 'counter'},
my_value: {sensor_type: 'indicator'},
my_h_counter: {sensor_type: 'hashed_counter'},
my_t_counter: {
sensor_type: 'timelined/counter',
args: {
interval: 60, # count for each minute
ttl: 24 * 60 * 60 # keep data one day
}
},
my_t_max: {
sensor_type: 'timelined/max',
args: {
interval: 60, # count for each minute
ttl: 24 * 60 * 60 # keep data one day
}
}
)
sensors.my_counter(1)
sensors.my_counter(2)
puts sensors.sensor(:my_counter).value
sensors.my_value(3.14)
sensors.my_value(2.71)
puts sensors.sensor(:my_value).value
sensors.my_h_counter(:x => 1)
sensors.my_h_counter(:y => 5)
sensors.my_h_counter(:y => 1)
p sensors.sensor(:my_h_counter).value
sensors.my_t_counter(1)
sensors.my_t_counter(1)
sleep(60)
sensors.my_t_counter(1)
sensors.sensor(:my_t_counter).timeline(2 * 60).each do |v|
puts "#{v.start_time}: #{v.value}"
end
sensors.my_t_max(3)
sensors.my_t_max(1)
sensors.my_t_max(2)
sleep(60)
sensors.my_t_max(5)
sensors.my_t_max(7)
sensors.my_t_max(6)
sensors.sensor(:my_t_max).timeline(2 * 60).each do |v|
puts "#{v.start_time}: #{v.value}"
end
## Command line interface
Gem includes a tool pulse, which allows to send events to sensors, list them, etc.
You should pay attention to the command pulse reduce, which is generally should be
scheduled on a regular basis to keep data in Redis small.
To see available commands of this tool one can run the example above(see examples/readme\_client\_example.rb)
and run pulse help.
## Visualisation
PulseMeter comes with a simple DSL which allows to build a self-contained Rack application for
visualizing timeline sensor data.
The application is described by *Layout* which contains some general application options and a list of *Pages*.
Each page contain a list of *Widgets* (charts), and each widget is associated with several sensors, which produce
data series for the chart.
There is a minimal and a full example below.
### Minimal example
It can be found in examples/minimal folder. To run it, execute
bundle && cd examples/minimal && bundle exec foreman start (or just rake example:minimal)
at project root and visit
http://localhost:9292 at your browser.
client.rb just creates a timelined counter an sends data to it in an infinite loop.
require "pulse-meter"
PulseMeter.redis = Redis.new
sensor = PulseMeter::Sensor::Timelined::Counter.new(:simple_sample_counter,
:interval => 5,
:ttl => 60 * 60
)
while true
STDERR.puts "tick"
sensor.event(1)
sleep(Random.rand)
end
server.ru is a Rackup file creating a simple layout with one page and one widget on it, which displays
the sensor's data. The layout is converted to a rack application and launched.
require "pulse-meter/visualizer"
PulseMeter.redis = Redis.new
layout = PulseMeter::Visualizer.draw do |l|
l.title "Minimal App"
l.page "Main Page" do |p|
p.area "Live Counter",
sensor: :simple_sample_counter,
timespan: 5 * 60,
redraw_interval: 1
end
end
run layout.to_app
Procfile allows to launch both "client" script and the web server with foreman.
web: bundle exec rackup server.ru
sensor_data_generator: bundle exec ruby client.rb
### Full example with DSL explanation
It can be found in examples/full folder. To run it, execute
bundle && cd examples/full && bundle exec foreman start (or just rake example:full)
at project root and visit
http://localhost:9292 at your browser.
client.rb imitating users visiting some imaginary site.
require "pulse-meter"
PulseMeter.redis = Redis.new
sensors = PulseMeter::Sensor::Configuration.new(
requests_per_minute: {
sensor_type: 'timelined/counter',
args: {
annotation: 'Requests per minute',
interval: 60,
ttl: 60 * 60 * 24 # keep data one day
}
},
requests_per_hour: {
sensor_type: 'timelined/counter',
args: {
annotation: 'Requests per hour',
interval: 60 * 60,
ttl: 60 * 60 * 24 * 30 # keep data 30 days
}
},
# when ActiveSupport extentions are loaded, a better way is to write just
# :interval => 1.hour,
# :ttl => 30.days
errors_per_minute: {
sensor_type: 'timelined/counter',
args: {
annotation: 'Errors per minute',
interval: 60,
ttl: 60 * 60 * 24
}
},
errors_per_hour: {
sensor_type: 'timelined/counter',
args: {
annotation: 'Errors per hour',
interval: 60 * 60,
ttl: 60 * 60 * 24 * 30
}
},
longest_minute_request: {
sensor_type: 'timelined/max',
args: {
annotation: 'Longest minute requests',
interval: 60,
ttl: 60 * 60 * 24
}
},
shortest_minute_request: {
sensor_type: 'timelined/min',
args: {
annotation: 'Shortest minute requests',
interval: 60,
ttl: 60 * 60 * 24
}
},
perc90_minute_request: {
sensor_type: 'timelined/percentile',
args: {
annotation: 'Minute request 90-percent percentile',
interval: 60,
ttl: 60 * 60 * 24,
p: 0.9
}
}
)
agent_names = [:ie, :firefox, :chrome, :other]
agent_names.each do |agent|
sensors.add_sensor(agent,
sensor_type: 'timelined/counter',
args: {
annotation: "Requests from #{agent} browser",
interval: 60 * 60,
ttl: 60 * 60 * 24 * 30
}
)
end
while true
sensors.requests_per_minute(1)
sensors.requests_per_hour(1)
if Random.rand(10) < 1 # let "errors" sometimes occur
sensors.errors_per_minute(1)
sensors.errors_per_hour(1)
end
request_time = 0.1 + Random.rand
sensors.longest_minute_request(request_time)
sensors.shortest_minute_request(request_time)
sensors.perc90_minute_request(request_time)
agent_counter = sensors.sensor(agent_names.shuffle.first)
agent_counter.event(1)
sleep(Random.rand / 10)
end
A more complicated visualization
require "pulse-meter/visualizer"
PulseMeter.redis = Redis.new
layout = PulseMeter::Visualizer.draw do |l|
# Application title
l.title "Full Example"
# Use local time for x-axis of charts
l.use_utc false
# Color for values cut off
l.outlier_color '#FF0000'
# Transfer some global parameters to Highcharts
l.highchart_options({
tooltip: {
value_decimals: 2
}
})
# Add some pages
l.page "Request count" do |p|
# Add chart (of Highcharts `area' style, `spline', `pie' and `line' are also available)
p.area "Requests per minute" do |w|
# Plot :requests_per_minute values on this chart with black color
w.sensor :requests_per_minute, color: '#000000'
# Plot :errors_per_minute values on this chart with red color
w.sensor :errors_per_minute, color: '#FF0000'
# Plot values for the last hour
w.timespan 60 * 60
# Redraw chart every 10 seconds
w.redraw_interval 10
# Plot incomplete data
w.show_last_point true
# Meaning of the y-axis
w.values_label "Request count"
# Occupy half (5/10) of the page (horizontally)
w.width 5
# Transfer page-wide (and page-specific) options to Highcharts
p.highchart_options({
chart: {
height: 300
}
})
end
p.area "Requests per hour" do |w|
w.sensor :requests_per_hour, color: '#555555'
w.sensor :errors_per_hour, color: '#FF0000'
w.timespan 24 * 60 * 60
w.redraw_interval 10
w.show_last_point true
w.values_label "Request count"
w.width 5
end
end
l.page "Request times" do |p|
p.area "Requests time" do |w|
w.sensor :longest_minute_request
w.sensor :shortest_minute_request
w.sensor :perc90_minute_request
w.timespan 60 * 60
w.redraw_interval 10
w.show_last_point true
w.values_label "Time in seconds"
w.width 10
end
end
l.page "Browsers" do |p|
p.pie "Requests from browser" do |w|
[:ie, :firefox, :chrome, :other].each do |sensor|
w.sensor sensor
end
w.timespan 24 * 60 * 60
w.redraw_interval 10
w.show_last_point true
w.values_label "Request count"
w.width 10
end
p.highchart_options({
chart: {
height: 500
}
})
end
end
run layout.to_app
## Installation
Add this line to your application's Gemfile:
gem 'pulse-meter'
And then execute:
$ bundle
Or install it yourself as:
$ gem install pulse-meter
## Contributing
1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Added some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request