README.md in tobox-0.3.2 vs README.md in tobox-0.4.0

- old
+ new

@@ -18,10 +18,11 @@ - [Inbox](#inbox) - [Plugins](#plugins) - [Zeitwerk](#zeitwerk) - [Sentry](#sentry) - [Datadog](#datadog) + - [Stats](#stats) - [Supported Rubies](#supported-rubies) - [Rails support](#rails-support) - [Why?](#why) - [Development](#development) - [Contributing](#contributing) @@ -478,9 +479,72 @@ c.tracing.instrument :tobox end # tobox.rb plugin(:datadog) +``` + +<a id="markdown-datadog" name="stats"></a> +### Stats + +The `stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool). + +```ruby +plugin(:stats) +on_stats(5) do |stats_collector| # every 5 seconds + stats = stats_collector.collect + # + # stats => { + # pending_count: number of new events in the outbox table + # failing_count: number of events which have failed processing but haven't reached the threshold + # failed_count: number of events which have failed the max number of tries + # inbox_count: (if used) number of events marked as received in the inbox table + # } + # + # now you can send them to your statsd collector + # + StatsD.gauge('outbox_pending_backlog', stats[:pending_count]) +end +``` + +#### Bring your own leader election + +The stats collection runs on every `tobox` initiated. If you're launching it in multiple servers / containers / pods, this means you'll be collecting statistics about the same database on all of these instances. This may not be desirable, and you may want to do this collection in a single instance. This is not a problem that `tobox` can solve by itself, so you'll have to take care of that yourself. Still, here are some cheap recommendations. + +##### Postgres advisory locks + +If your database is PostgreSQL, you can leverage session-level [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) to ensure single-instance access to this functionality. `tobox` also exposes the database instance to the `on_stats` callback: + +```ruby +c.on_stats(5) do |stats_collector, db| + if db.get(Sequel.function(:pg_try_advisory_lock, 1)) + stats = stats_collector.collect + StatsD.gauge('outbox_pending_backlog', stats[:pending_count]) + end +end +``` + +If a server goes down, one of the remaining ones will acquire the lock and ensure stats processing. + +##### Redis distributed locks + +If you're already using [redis](https://redis.io/), you can use its distributed lock feature to achieve the goal: + +```ruby +# using redlock +c.on_stats(5) do |stats_collector, db| + begin + lock_info = lock_manager.lock("outbox", 5000) + + stats = stats_collector.collect + StatsD.gauge('outbox_pending_backlog', stats[:pending_count]) + + # extend to hold the lock for the next loop + lock_info = lock_manager.lock("outbox", 5000, extend: lock_info) + rescue Redlock::LockError + # some other server already has the lock, try later + end +end ``` <a id="markdown-supported-rubies" name="supported-rubies"></a> ## Supported Rubies