README.md in phobos-1.4.2 vs README.md in phobos-1.5.0

- old
+ new

@@ -27,10 +27,11 @@ 1. [As library in another app](#usage-as-library) 1. [Configuration file](#usage-configuration-file) 1. [Instrumentation](#usage-instrumentation) 1. [Plugins](#plugins) 1. [Development](#development) +1. [Test](#test) ## <a name="installation"></a> Installation Add this line to your application's Gemfile: @@ -118,11 +119,11 @@ ``` ### <a name="usage-consuming-messages-from-kafka"></a> Consuming messages from Kafka Messages from Kafka are consumed using __handlers__. You can use Phobos __executors__ or include it in your own project [as a library](#usage-as-library), but __handlers__ will always be used. To create a handler class, simply include the module `Phobos::Handler`. This module allows Phobos to manage the life cycle of your handler. -A handler must implement the method `#consume(payload, metadata)`. +A handler is required to implement the method `#consume(payload, metadata)`. Instances of your handler will be created for every message, so keep a constructor without arguments. If `consume` raises an exception, Phobos will retry the message indefinitely, applying the back off configuration presented in the configuration file. The `metadata` hash will contain a key called `retry_count` with the current number of retries for this message. To skip a message, simply return from `#consume`. When the listener starts, the class method `.start` will be called with the `kafka_client` used by the listener. Use this hook as a chance to setup necessary code for your handler. The class method `.stop` will be called during listener shutdown. @@ -160,19 +161,32 @@ # consume or skip message end end ``` +Finally, it is also possible to preprocess the message payload before consuming it using the `before_consume` hook which is invoked before `.around_consume` and `#consume`. The result of this operation will be assigned to payload, so it is important to return the modified payload. This can be very useful, for example if you want a single point of decoding Avro messages and want the payload as a hash instead of a binary. + +```ruby +class MyHandler + include Phobos::Handler + + def before_consume(payload) + # optionally preprocess payload + payload + end +end +``` + Take a look at the examples folder for some ideas. The hander life cycle can be illustrated as: `.start` -> `#consume` -> `.stop` or optionally, - `.start` -> `.around_consume` [ `#consume` ] -> `.stop` + `.start` -> `#before_consume` -> `.around_consume` [ `#consume` ] -> `.stop` ### <a name="usage-producing-messages-to-kafka"></a> Producing messages to Kafka `ruby-kafka` provides several options for publishing messages, Phobos offers them through the module `Phobos::Producer`. It is possible to turn any ruby class into a producer (including your handlers), just include the producer module, example: @@ -301,11 +315,11 @@ __producer__ provides configurations for all producers created over the application, the options are the same for regular and async producers. All [options supported by `ruby-kafka`][ruby-kafka-producer] can be provided. __consumer__ provides configurations for all consumer groups created over the application. All [options supported by `ruby-kafka`][ruby-kafka-consumer] can be provided. -__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here +__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here. Backoff can also be configured per listener. __listeners__ is the list of listeners configured, each listener represents a consumers group [ruby-kafka-client]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Ainitialize [ruby-kafka-consumer]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Aconsumer @@ -401,23 +415,40 @@ * Allows your system to quickly reprocess events in case of failures ## <a name="development"></a> Development After checking out the repo: -* make sure docker is installed and running +* make sure `docker` is installed and running (for windows and mac this also includes `docker-compose`). +* Linux: make sure `docker-compose` is installed and running. * run `bin/setup` to install dependencies -* run `sh utils/start-all.sh` to start the required kafka containers in the background -* run `rspec` to run the tests +* run `docker-compose up` to start the required kafka containers in a window +* run `rspec` to run the tests in another window You can also run `bin/console` for an interactive prompt that will allow you to experiment. To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). -The `utils` folder contain some shell scripts to help with the local Kafka cluster. It uses docker to start Kafka and zookeeper. +## <a name="test"></a> Test -```sh -sh utils/start-all.sh -sh utils/stop-all.sh +Phobos exports a spec helper that can help you test your consumer. The Phobos lifecycle will conveniently be activated for you with minimal setup required. + +* `process_message(handler:, payload:, metadata:, encoding: nil)` - Invokes your handler with payload and metadata, using a dummy listener (encoding is optional). + +```ruby +require 'spec_helper' + +describe MyConsumer do + let(:payload) { 'foo' } + let(:metadata) { 'foo' } + + it 'consumes my message' do + expect(described_class).to receive(:around_consume).with(payload, metadata).once.and_call_original + expect_any_instance_of(described_class).to receive(:before_consume).with(payload).once.and_call_original + expect_any_instance_of(described_class).to receive(:consume).with(payload, metadata).once.and_call_original + + process_message(handler: described_class, payload: payload, metadata: metadata) + end +end ``` ## Contributing Bug reports and pull requests are welcome on GitHub at https://github.com/klarna/phobos.