README.md in phobos-1.4.2 vs README.md in phobos-1.5.0
- old
+ new
@@ -27,10 +27,11 @@
1. [As library in another app](#usage-as-library)
1. [Configuration file](#usage-configuration-file)
1. [Instrumentation](#usage-instrumentation)
1. [Plugins](#plugins)
1. [Development](#development)
+1. [Test](#test)
## <a name="installation"></a> Installation
Add this line to your application's Gemfile:
@@ -118,11 +119,11 @@
```
### <a name="usage-consuming-messages-from-kafka"></a> Consuming messages from Kafka
Messages from Kafka are consumed using __handlers__. You can use Phobos __executors__ or include it in your own project [as a library](#usage-as-library), but __handlers__ will always be used. To create a handler class, simply include the module `Phobos::Handler`. This module allows Phobos to manage the life cycle of your handler.
-A handler must implement the method `#consume(payload, metadata)`.
+A handler is required to implement the method `#consume(payload, metadata)`.
Instances of your handler will be created for every message, so keep a constructor without arguments. If `consume` raises an exception, Phobos will retry the message indefinitely, applying the back off configuration presented in the configuration file. The `metadata` hash will contain a key called `retry_count` with the current number of retries for this message. To skip a message, simply return from `#consume`.
When the listener starts, the class method `.start` will be called with the `kafka_client` used by the listener. Use this hook as a chance to setup necessary code for your handler. The class method `.stop` will be called during listener shutdown.
@@ -160,19 +161,32 @@
# consume or skip message
end
end
```
+Finally, it is also possible to preprocess the message payload before consuming it using the `before_consume` hook which is invoked before `.around_consume` and `#consume`. The result of this operation will be assigned to payload, so it is important to return the modified payload. This can be very useful, for example if you want a single point of decoding Avro messages and want the payload as a hash instead of a binary.
+
+```ruby
+class MyHandler
+ include Phobos::Handler
+
+ def before_consume(payload)
+ # optionally preprocess payload
+ payload
+ end
+end
+```
+
Take a look at the examples folder for some ideas.
The hander life cycle can be illustrated as:
`.start` -> `#consume` -> `.stop`
or optionally,
- `.start` -> `.around_consume` [ `#consume` ] -> `.stop`
+ `.start` -> `#before_consume` -> `.around_consume` [ `#consume` ] -> `.stop`
### <a name="usage-producing-messages-to-kafka"></a> Producing messages to Kafka
`ruby-kafka` provides several options for publishing messages, Phobos offers them through the module `Phobos::Producer`. It is possible to turn any ruby class into a producer (including your handlers), just include the producer module, example:
@@ -301,11 +315,11 @@
__producer__ provides configurations for all producers created over the application, the options are the same for regular and async producers. All [options supported by `ruby-kafka`][ruby-kafka-producer] can be provided.
__consumer__ provides configurations for all consumer groups created over the application. All [options supported by `ruby-kafka`][ruby-kafka-consumer] can be provided.
-__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here
+__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here. Backoff can also be configured per listener.
__listeners__ is the list of listeners configured, each listener represents a consumers group
[ruby-kafka-client]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Ainitialize
[ruby-kafka-consumer]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Aconsumer
@@ -401,23 +415,40 @@
* Allows your system to quickly reprocess events in case of failures
## <a name="development"></a> Development
After checking out the repo:
-* make sure docker is installed and running
+* make sure `docker` is installed and running (for windows and mac this also includes `docker-compose`).
+* Linux: make sure `docker-compose` is installed and running.
* run `bin/setup` to install dependencies
-* run `sh utils/start-all.sh` to start the required kafka containers in the background
-* run `rspec` to run the tests
+* run `docker-compose up` to start the required kafka containers in a window
+* run `rspec` to run the tests in another window
You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
-The `utils` folder contain some shell scripts to help with the local Kafka cluster. It uses docker to start Kafka and zookeeper.
+## <a name="test"></a> Test
-```sh
-sh utils/start-all.sh
-sh utils/stop-all.sh
+Phobos exports a spec helper that can help you test your consumer. The Phobos lifecycle will conveniently be activated for you with minimal setup required.
+
+* `process_message(handler:, payload:, metadata:, encoding: nil)` - Invokes your handler with payload and metadata, using a dummy listener (encoding is optional).
+
+```ruby
+require 'spec_helper'
+
+describe MyConsumer do
+ let(:payload) { 'foo' }
+ let(:metadata) { 'foo' }
+
+ it 'consumes my message' do
+ expect(described_class).to receive(:around_consume).with(payload, metadata).once.and_call_original
+ expect_any_instance_of(described_class).to receive(:before_consume).with(payload).once.and_call_original
+ expect_any_instance_of(described_class).to receive(:consume).with(payload, metadata).once.and_call_original
+
+ process_message(handler: described_class, payload: payload, metadata: metadata)
+ end
+end
```
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/klarna/phobos.