README.md in tobox-0.4.5 vs README.md in tobox-0.5.0

- old
+ new

@@ -11,14 +11,14 @@ - [Requirements](#requirements) - [Installation](#installation) - [Usage](#usage) - [Configuration](#configuration) - [Event](#event) -- [Features](#features) - - [Ordered event processing](#ordered-event-processing) - - [Inbox](#inbox) - [Plugins](#plugins) + - [Progress](#progress) + - [Event Grouping](#event-grouping) + - [Inbox](#inbox) - [Zeitwerk](#zeitwerk) - [Sentry](#sentry) - [Datadog](#datadog) - [Stats](#stats) - [Supported Rubies](#supported-rubies) @@ -202,15 +202,25 @@ ### `max_attempts` Maximum number of times a failed attempt to process an event will be retried (`10` by default). ```ruby -concurrency 4 +max_attempts 4 ``` **Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event. +### `exponential_retry_factor` + +Factor by which the number of seconds until an event can be retried will be exponentially calculated, i.e. 2 seconds on first attempt, then 4, then 8, then 16 (`2` by default). + +```ruby +exponential_retry_factor 2 +``` + +**Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event. + ### `concurrency` Number of workers processing events. ```ruby @@ -318,22 +328,11 @@ ### log_level Overrides the default log level ("info" when in "production" environment, "debug" otherwise). -### group_column -Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing). - -### inbox table - -Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox). - -### inbox column - -Defines the column in the outbox table which references the inbox table, when one is set. - <a id="markdown-event" name="event"></a> ## Event The event is composed of the following properties: @@ -343,40 +342,89 @@ * `:after`: hash of the associated event data after event is emitted (can be `nil`) * `:created_at`: timestamp of when the event is emitted (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.) -<a id="markdown-features" name="features"></a> -## Features -There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`. +<a id="markdown-plugins" name="plugins"></a> +## Plugins -<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a> -### Ordered event processing +`tobox` ships with a very simple plugin system. (TODO: add docs). +Plugins can be loaded in the config via `plugin`: + +```ruby +# tobox.rb +plugin(:plugin_name) +``` + +<a id="markdown-progress" name="progress"></a> +### Progress + +By default, the database transaction used to consume the event is kept open while the event is handled. While this ensures atomic event consumption, it may also cause overhead related to transaction management given enough load, particularly in cases where event handling time varies (i.e. throttled HTTP requests). + +The `:progress` plugin fixes this by releasing the databaase transaction after fetching the event. It does so by making the fetched event "invisible" for a certain period, during which the event must be successfully handled. + +Here's how to use it: + +```ruby +# in your tobox.rb +plugin :progress + +visibility_timeout 90 # default: 30 +``` + +3. insert related outbox events with the same group id + +```ruby +order = Order.new( + item_id: item.id, + price: 20_20, + currency: "EUR" +) +DB.transaction do + order.save + DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash) + DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash) +end + +# "order_created" will be processed first +# "billing_event_created" will only start processing once "order_created" finishes +``` + +#### Configuration + +##### `visibility_timeout` + +Timeout (in seconds) after which a previously marked-for-consumption event can be retried (default: `30`) + +<a id="markdown-event-grouping" name="event-grouping"></a> +### Event grouping + By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order. -One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration. +One solution is to have a single worker processing the "outbox" events. Another is to use the `:event_grouping` plugin. -What you have to do is: +All you have to do is: 1. add a "group id" column to the "outbox" table ```ruby create_table(:outbox) do primary_key :id - column :group_id, :integer - # The type is irrelevant, could also be :string, :uuid... + column :group_id, :integer # The type is irrelevant, could also be :string, :uuid... # .. + index :group_id ``` -2. set the "group_column" configuration +2. Enable the plugin ```ruby # in your tobox.rb -group_column :group_id -index :group_id +plugin :event_grouping + +group_column :group_id # by default already `:group_id` ``` 3. insert related outbox events with the same group id ```ruby @@ -392,46 +440,51 @@ end # "order_created" will be processed first # "billing_event_created" will only start processing once "order_created" finishes ``` + +#### Configuration + +##### `group_column` + +Defines the database column to be used for event grouping (`:group_id` by default). + <a id="inbox" name="inbox"></a> ### Inbox -`tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether). +Via the `:inbox` plugin, `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether). In order to do so, you'll have to: -1. add an "inbox" table in the database +1. add an "inbox" table in the database and the unique id reference in the outbox table: ```ruby create_table(:inbox) do column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP end -``` -2. add the unique id reference in the outbox table: - -```ruby create_table(:outbox) do primary_key :id column :type, :varchar, null: false column :inbox_id, :varchar, null: true # ... foreign_key :inbox_id, :inbox ``` -3. reference them in the configuration +2. Load the plugin and reference them in the configuration ```ruby # tobox.rb -inbox_table :inbox -inbox_column :inbox_id +plugin :inbox + +inbox_table :inbox # :inbox by default already +inbox_column :inbox_id # :inbox_id by default already ``` -4. insert related outbox events with an inbox id +3. insert related outbox events with an inbox id ```ruby order = Order.new( item_id: item.id, price: 20_20, @@ -444,24 +497,24 @@ end # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once. ``` -**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again. +#### Configuration -<a id="markdown-plugins" name="plugins"></a> -## Plugins -`tobox` ships with a very simple plugin system. (TODO: add docs). +##### inbox table -Plugins can be loaded in the config via `plugin`: +Defines the name of the table to be used for inbox (`:inbox` by default). -```ruby -# tobox.rb -plugin(:plugin_name) -``` +##### inbox column +Defines the column in the outbox table which references the inbox table (`:inbox_id` by default). + + +**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again. + It ships with the following integrations. <a id="markdown-zeitwerk" name="zeitwerk"></a> ### Zeitwerk @@ -574,10 +627,10 @@ ``` <a id="markdown-supported-rubies" name="supported-rubies"></a> ## Supported Rubies -All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby. +All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby. <a id="markdown-rails-support" name="rails-support"></a> ## Rails support