README.markdown in ustate-client-0.0.2 vs README.markdown in ustate-client-0.0.3
- old
+ new
@@ -9,15 +9,32 @@
host: A hostname, e.g. "api1", "foo.com",
service: e.g. "API port 8000 reqs/sec",
state: Any string less than 255 bytes, e.g. "ok", "warning", "critical",
time: The time that the service entered this state, in unix time,
description: Freeform text,
- metric_f: A floating-point number associated with this state, e.g. the number of reqs/sec
+ metric_f: A floating-point number associated with this state, e.g. the number of reqs/sec,
+ once: A boolean, described below.
}
-At http://showyou.com, we use UState to monitor the health and performance of hundreds of services across our infrastructure, including CPU, queries/second, latency bounds, disk usage, queues, and others.
+Normally, every state received by the server fires Index#on_state. When
+state.state changes, Index#on_state_change is called. You can, for example,
+register to send a single email whenever a state changes to :warning.
+:once states are transient. They fire Index#on_state and #on_state_once, but do
+*not* update the index. They can be used for events which are instantaneous;
+instead of sending {state: error} and {state: ok}, send {state: error,
+once:true}.
+
+For example, recoverable errors may not hang your application, but
+should be processed by the email notifier. Sending a :once state with
+the error description means you can receive an email for each error,
+instead of two for entering and exiting the error state.
+
+At http://showyou.com, we use UState to monitor the health and performance of
+hundreds of services across our infrastructure, including CPU, queries/second,
+latency bounds, disk usage, queues, and others.
+
UState also includes a simple dashboard Sinatra app.
Installing
==========
@@ -37,27 +54,94 @@
For the dashboard:
gem install sinatra thin erubis sass
-Getting started
-===============
+Demo
+====
To try it out, install all the gems above, and clone the repository. Start the server with
- bin/server [--host host] [--port port]
+ bin/server
UState listens on TCP socket host:port, and accepts connections from clients. Start a basic testing client with
- bin/test
+ bin/test
The tester spews randomly generated statistics at a server on the default local host and port. To see it in action, run the dashboard:
- cd lib/ustate/dash
- ../../../bin/dash
+ cd lib/ustate/dash
+ ../../../bin/dash
+Server
+======
+The server loads a file in the working directory named config.rb. Override with
+--config-file. Its contents are instance-evaled in the context of the current
+server. You can use this to extend ustate with additional behavior.
+
+Email
+-----
+
+config.rb:
+ # Email comes from this address (required):
+ emailer.from = 'ustate@your.net'
+
+ # Use this SMTP relay (default 127.0.0.1)
+ emailer.host = '123.4.56.7'
+
+ # Receive mail when a state transition matches any of ...
+ emailer.tell 'you@gmail.com', 'state = "error" or state = "critical"'
+ emailer.tell 'you@gmail.com', 'service =~ "mysql%"'
+
+Custom hooks
+------------
+
+config.rb:
+ # Log all states received to console.
+ index.on_state do |s|
+ p s
+ end
+
+ # Forward state transitions to another server.
+ require 'ustate/client'
+ client = UState::Client.new :host => '123.45.67.8'
+ index.on_state_change do |old, new|
+ client << new
+ end
+ index.on_state_once do |state|
+ client << state
+ end
+
+Client
+======
+
+You can use the git repo, or the gem.
+
+ gem install ustate-client
+
+Then:
+
+ require 'ustate'
+ require 'ustate/client'
+
+ # Create a client
+ c = UState::Client.new(
+ host: "my.host", # Default localhost
+ port: 1234 # Default 55956
+ )
+
+ # Insert a state
+ c << {
+ state: "ok",
+ service: "My service"
+ }
+
+ # Query for states
+ c.query.states # => [UState::State(state: 'ok', service: 'My service')]
+ c.query('state != "ok"').states # => []
+
The Dashboard
=============
The dashboard runs a file in the local directory: config.rb. That file can
override any configuration options on the Dash class (hence all Sinatra
@@ -88,22 +172,20 @@
You can also query states using a very basic expression language. The grammar is specified as a Parsable Expression Grammar in query_string.treetop. Examples include:
state = "ok"
(service =~ "disk%") or (state == "critical" and host =~ "%.trioptimum.com")
-Search queries will return a message with repeated States matching that expression. An empty expression matches all states.
+Search queries will return a message with repeated States matching that expression. An null expression will return no states.
Performance
===========
-It's Ruby. It ain't gonna be fast. However, on my 4-year-old core 2 duo, I see >600 inserts/sec or queries/sec. The client is fully threadsafe, and performs well concurrently. I will continue to tune UState for latency and throughput, and welcome patches.
+On a macbook pro 8,3, I see >1300 queries/sec or >1200 inserts/sec. The client is fully threadsafe, and performs well concurrently. I will continue to tune UState for latency and throughput, and welcome patches.
For large installations, I plan to implement a selective forwarder. Local ustate servers can accept high volumes of states from a small set of nodes, and forward updates at a larger granularity to supervisors, and so forth, in a tree. The query language should be able to support proxying requests to the most recent source of a state, so very large sets of services can be maintained at high granularity.
Goals
=====
-
-Immediately, I'll be porting our internal email alerter to UState. Users register for interest in certain types of states or transitions, and receive emails when those events occur.
In the medium term, I'll be connecting UState to Graphite (or perhaps another
graphing tool) for metrics archival and soft-realtime graphs. I have an
internal gnuplot system which is clunky and deserves retirement.