# FlippRubyKafka

Flipp's best practices for using Kafka producers and consumers.
Built on top of Deimos, and also integrates
DataDog metrics and APM.  

See the Deimos gem for most configuration and usage.

## Installation

Add this line to your application's Gemfile:
```ruby
gem 'flipp-ruby-kafka', require: 'flipp_ruby_kafka'
```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install flipp-ruby-kafka

## Versioning

We use semver for this gem. Any change in previous behavior is denoted
with a bump in the minor version (0.4 -> 0.5). Patch versions are for
bugfixes or new functionality which does not affect existing code. You
should be locking your Gemfile to the minor version:

```ruby
gem 'flipp-ruby-kafka', '~> 0.4', require: 'flipp_ruby_kafka'
```

## Configuration

FlippRubyKafka comes with configuration out of the box for the Flipp
Microservice Platform. Your application configuration should look something
like this (in e.g. `config/initializers/flipp_ruby_kafka.rb`):

```ruby
  FlippRubyKafka.configure do
    producers.schema_namespace 'com.flipp.my_app'
    # For integration testing - see below
    kafkateria_url ENV['KAFKATERIA_URL']
  end
  FlippRubyKafka.configure_datadog
```

FlippRubyKafka also changes the default for a number of settings to make
your configuration smaller in the 80% case. The following are the setting
defaults:

Config name|Default
-----------|-------
logger|Clone of Rails logger
schema.backend|`:avro_validation` for development, `:avro_schema_registry` otherwise
schema.path|`"#{Rails.root}/app/schemas"`
schema.registry_url|`ENV['KAFKA_SCHEMA_REGISTRY'] \|\| 'http://localhost:8081'`
consumers.report_lag|true for staging/production, false for development/test
consumers.fatal_error|true if ActiveRecord connection is not active
kafka.seed_brokers|`ENV['KAFKA_SSL_BROKERS'] \|\| 'localhost:9092'`
kafka.ssl.enabled|true for staging/production, false for development/test
kafka.ssl.ca_cert|'/var/certs/kafka-broker-ca-public-cert'
kafka.ssl.client_cert|'/var/certs/kafka-client-public-key'
kafka.ssl.client_cert_key|'/var/certs/kafka-client-private-key-decrypted'
kafka.ssl.verify_hostname|false
kafkaeria_url|'http://localhost:3000'

## Running consumers

FlippRubyKafka aliases `rake deimos:start` to `rake phobos:start`
to support legacy applications.

## Testing

FlippRubyKafka has helper methods for sending API requests to
[Kafkateria](https://github.com/wishabi/kafkateria):

```ruby

kafkateria_delete_topic('my-topic')

kafkateria_produce_messages(
  topic: 'my-topic',
  num_messages: 5,
  traits: %i(indexed),
  values: { merchant_id: 1 }
)
```

Make sure to set `config.kafkateria_url` when configuring FlippRubyKafka
to enable this.

## Additional Tools

FlippRubyKafka provides generators to hook into the platform build's features. These generate JSON config files to be used by the build script:

* `rails g flipp_ruby_kafka:schema_validation` will generate a `schema_validation.json`
* `rails g flipp_ruby_kafka:topic_validation` will generate a `topic_validation.json`

You can run both scripts in your build step - as long as the files are persisted to your workspace, you can use them in your deploy or in your separate validation steps.