README.md in tobox-0.1.6 vs README.md in tobox-0.2.0
- old
+ new
@@ -1,30 +1,53 @@
# Tobox: Transactional outbox pattern implementation in ruby
[](http://rubygems.org/gems/tobox)
-[](https://gitlab.com/honeyryderchuck/tobox/pipelines?page=1&scope=all&ref=master)
-[](https://honeyryderchuck.gitlab.io/tobox/#_AllFiles)
+[](https://gitlab.com/os85/tobox/pipelines?page=1&scope=all&ref=master)
+[](https://os85.gitlab.io/tobox/#_AllFiles)
Simple, data-first events processing framework based on the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).
+<!-- TOC -->
+
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Configuration](#configuration)
+- [Event](#event)
+- [Features](#features)
+ - [Ordered event processing](#ordered-event-processing)
+- [Plugins](#plugins)
+ - [Zeitwerk](#zeitwerk)
+ - [Sentry](#sentry)
+ - [Datadog](#datadog)
+- [Supported Rubies](#supported-rubies)
+- [Rails support](#rails-support)
+- [Why?](#why)
+- [Development](#development)
+- [Contributing](#contributing)
+
+<!-- /TOC -->
+
+<a id="markdown-requirements" name="requirements"></a>
## Requirements
`tobox` requires integration with RDBMS which supports `SKIP LOCKED` functionality. As of today, that's:
* PostgreSQL 9.5+
* MySQL 8+
* Oracle
* Microsoft SQL Server
+<a id="markdown-installation" name="installation"></a>
## Installation
Add this line to your application's Gemfile:
```ruby
gem "tobox"
-# You'll also need to aadd the right database client gem for the target RDBMS
+# You'll also need to add the right database client gem for the target RDBMS
# ex, for postgresql:
#
# gem "pg"
# see more: http://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html
```
@@ -35,10 +58,12 @@
Or install it yourself as:
$ gem install tobox
+
+<a id="markdown-usage" name="usage"></a>
## Usage
1. create the `outbox` table in your application's database:
```ruby
@@ -78,15 +103,18 @@
BillingService.bill_user_account(user_data_hash)
end
on("user_updated") do |event|
# ...
end
+on("user_created", "user_updated") do |event|
+ # ...
+end
```
3. Start the `tobox` process
-```
+```bash
> bundle exec tobox -C path/to/tobox.rb -r path/to/file_requiring_application_code.rb
```
There is no API for event production yet (still TODO). It's recommended you write directly into the "outbox" table via database triggers (i.e. *insert into users table -> add user_created event"). Alternatively you can use `sequel` directly (`DB[:outbox].insert(...)`).
@@ -133,15 +161,16 @@
ON orders
FOR EACH ROW
EXECUTE PROCEDURE order_created_outbox_event();
```
+<a id="markdown-configuration" name="configuration"></a>
## Configuration
As mentioned above, configuration can be set in a particular file. The following options are configurable:
-### `environment``
+### `environment`
Sets the application environment (either "development" or "production"). Can be set directly, or via `APP_ENV` environment variable (defaults to "development").
### `database_uri`
@@ -226,10 +255,19 @@
```ruby
on_error_event { |event, exception| Sentry.capture_exception(exception) }
```
+### `on_error_worker { |error| }`
+
+callback executed when an exception was raised in the worker, before processing events.
+
+
+```ruby
+on_error_worker { |exception| Sentry.capture_exception(exception) }
+```
+
### `message_to_arguments { |event| }`
if exposing raw data to the `on` handlers is not what you'd want, you can always override the behaviour by providing an alternative "before/after fetcher" implementation.
```ruby
@@ -255,10 +293,15 @@
### log_level
Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
+### group_column
+
+Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
+
+<a id="markdown-event" name="event"></a>
## Event
The event is composed of the following properties:
* `:id`: unique event identifier
@@ -267,24 +310,61 @@
* `:after`: hash of the associated event data after event is emitted (can be `nil`)
* `:created_at`: timestamp of when the event is emitted
(*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
-## Rails support
+<a id="markdown-features" name="features"></a>
+## Features
-Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
+There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
-```bash
-> bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
+<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
+### Ordered event processing
+
+By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
+
+One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
+
+What you have to do is:
+
+1. add a "group id" column to the "outbox" table
+
+```ruby
+create_table(:outbox) do
+ primary_key :id
+ column :group_id, :integer
+ # The type is irrelevant, could also be :string, :uuid...
+ # ..
```
-In the `tobox` config, you can set the environment:
+2. set the "group_column" configuration
```ruby
-environment Rails.env
+# in your tobox.rb
+group_column :group_id
+index :group_id
```
+3. insert related outbox events with the same group id
+
+```ruby
+order = Order.new(
+ item_id: item.id,
+ price: 20_20,
+ currency: "EUR"
+)
+DB.transaction do
+ order.save
+ DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
+ DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
+end
+
+# "order_created" will be processed first
+# "billing_event_created" will only start processing once "order_created" finishes
+```
+
+<a id="markdown-plugins" name="plugins"></a>
## Plugins
`tobox` ships with a very simple plugin system. (TODO: add docs).
Plugins can be loaded in the config via `plugin`:
@@ -294,10 +374,11 @@
plugin(:plugin_name)
```
It ships with the following integrations.
+<a id="markdown-zeitwerk" name="zeitwerk"></a>
### Zeitwerk
(requires the `zeitwerk` gem.)
Plugin for the [zeitwerk](https://github.com/fxn/zeitwerk) auto-loader. It allows to set the autoload dirs, and seamlessly integrates code reloading in "development", and eagerloading in "production":
@@ -308,10 +389,11 @@
zeitwerk_loader do |loader|
loader.push_dir("path/to/handlers")
end
```
+<a id="markdown-sentry" name="sentry"></a>
### Sentry
(requires the `sentry-ruby` gem.)
Plugin for the [sentry](https://github.com/getsentry/sentry-ruby) ruby SDK for error tracking. It'll send all errors happening while processing events to Sentry.
@@ -319,10 +401,11 @@
```ruby
# tobox.rb
plugin(:sentry)
```
+<a id="markdown-datadog" name="datadog"></a>
### Datadog
(requires the `ddtrace` gem.)
Plugin for [datadog](https://github.com/DataDog/dd-trace-rb) ruby SDK. It'll generate traces for event handling.
@@ -335,14 +418,32 @@
# tobox.rb
plugin(:datadog)
```
+<a id="markdown-supported-rubies" name="supported-rubies"></a>
## Supported Rubies
All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
+
+<a id="markdown-rails-support" name="rails-support"></a>
+## Rails support
+
+Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
+
+```bash
+> bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
+```
+
+In the `tobox` config, you can set the environment:
+
+```ruby
+environment Rails.env
+```
+
+<a id="markdown-why" name="why"></a>
## Why?
### Simple and lightweight, framework (and programming language) agnostic
`tobox` event callbacks yield the data in ruby primitive types, rather than heavy ORM instances. This is by design, as callbacks may not rely on application code being loaded.
@@ -373,12 +474,14 @@
By using the database as the message broker, `tobox` can rely on good old transactions to ensure that data committed to the database has a corresponding event. This makes the delivery guarantee "exactly once".
(The actual processing may change this to "at least once", as issues may happen before the event is successfully deleted from the outbox. Still, "at least once" is acceptable and solvable using idempotency mechanisms).
+<a id="markdown-development" name="development"></a>
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+<a id="markdown-contributing" name="contributing"></a>
## Contributing
-Bug reports and pull requests are welcome on GitHub at https://gitlab.com/honeyryderchuck/tobox.
+Bug reports and pull requests are welcome on GitHub at https://gitlab.com/os85/tobox.