UPGRADING.md in deimos-ruby-2.0.0.pre.alpha1

- old
+ new
@@ -1,6 +1,243 @@
 # Upgrading Deimos
 
+## Upgrading to 2.x
+
+2.x is a major rewrite from 1.0. The underlying library has been changed from [Phobos](https://github.com/phobos/phobos) to [Karafka](https://karafka.io/). This change has given us an opportunity to fix some issues and deprecated code paths from version 1.0 as well as provide much more functionality by integrating more fully with the Karafka ecosystem.
+
+For a deeper dive into the internal changes, please see [...]().
+
+There are a number of breaking changes. We provide a `v2` generator to attempt to auto-fix many of these breaking changes automatically. To run the generator:
+
+    KARAFKA_BOOT_FILE=false rails g deimos:v2
+
+### Running Deimos
+
+Instead of running `rake deimos:start`, you now run your Kafka consumers the same way any Karafka consumers are run: `karafka server`.
+
+### Configuration
+
+In V1, Deimos configuration was all done in a single `Deimos.configure` block, including Kafka configs, consumers and producers:
+
+```ruby
+Deimos.configure do
+  producers.schema_namespace 'com.my-namespace'
+  kafka.seed_brokers ['my-broker:9092']
+
+  consumer do
+    class_name 'MyConsumer'
+    topic 'MyTopic'
+    session_timeout 30
+    schema 'MySchema'
+    key_config field: :id
+    namespace 'com.my-namespace'
+  end
+
+  producer do
+    class_name 'MyProducer'
+    topic 'MyTopic2'
+    schema 'MySchema2'
+    key_config none: true
+  end
+end  
+```
+
+In V2, the `Deimos.configure` block now only takes Deimos-specific settings, and is **not** used to configure producers and consumers. Kafka settings now go in the Karafka `kafka` setting method, and producers and consumers use Karafka [routing](https://karafka.io/docs/Routing/). There are Deimos-specific extensions to routing to apply to consumers and producers, either via a `defaults` block (applying to all consumers and producers) or in individual `topic` blocks:
+
+```ruby
+Deimos.configure do
+  producers.schema_namespace 'com.my-namespace'
+end
+
+class KarafkaApp < Karafka::App
+  setup do |config|
+    config.kafka = {
+      "bootstrap.servers": "my-broker:9092"
+    }
+  end
+  
+  routes.draw do
+    defaults do
+      namespace "com.my-namespace"
+    end
+
+    topic "MyTopic" do
+      # Karafka settings
+      consumer MyConsumer
+      kafka({"session.timeout.ms": 30_000})
+      # Deimos settings
+      schema "MySchema" # the res
+      key_config({field: id})
+    end
+    
+    topic "MyTopic2" do
+      # these are all Deimos settings since Karafka doesn't actually do per-topic producer configs
+      producer_class MyProducer
+      schema 'MySchema2'
+      key_config none: true
+    end
+  end
+end
+```
+
+This configuration must be in a file called `karafka.rb` at the root of your application. The V2 generator will generate this file for you. Without the generator, if you have this file and start up your app with the old `Deimos.configure` code, you will get notifications of the correct places to put these settings.
+
+
+### Removed deprecations
+
+The following were deprecated in version 1.x and are removed in 2.0.
+
+* The `kafka_producer` method for KafkaSource is no longer supported. Please use `kafka_producers`. (This is not addressed by the V2 generator.)
+
+```ruby
+# before:
+class MyRecord < ApplicationRecord
+  def kafka_producer
+    MyProducer
+  end
+end
+
+# after:
+class MyRecord < ApplicationRecord
+  def kafka_producers
+    [MyProducer]
+  end
+end
+```
+
+* The `record_attributes` method for ActiveRecordConsumer now must take two parameters, not one. (The V2 generator can fix this.)
+
+```ruby
+# before:
+class MyConsumer < Deimos::ActiveRecordConsumer
+  def record_attributes(payload)
+    # ...
+  end
+end
+
+# after:
+class MyConsumer < Deimos::ActiveRecordConsumer
+  def record_attributes(payload, key)
+    # ...
+  end
+end
+```
+
+* The `BatchConsumer` class has been removed. Please use the `Consumer` class.
+* You can no longer configure your application using a `phobos.yml` file. The V2 generator will not be able to work on apps using this approach.
+* Removed `test_consume_invalid_message` and `test_consume_batch_invalid_message` test helpers. These did not serve a useful purpose.
+* The following deprecated testing functions have been removed: `stub_producers_and_consumers!`, `stub_producer`, `stub_consumer`, `stub_batch_consumer`. These have not done anything in a long time.
+
+### Major breaking changes
+* Since Karafka only supports Ruby >= 3.0, that means Deimos also only supports those versions.
+* Deimos no longer supports a separate logger from Karafka. When you configure a Karafka logger, Deimos will use that logger for all its logging. (Deimos logs will be prefixed with a `[Deimos]` tag.)
+* The `:db` backend has been renamed to `:outbox`. All associated classes (like `DbProducer`) have likewise been renamed. The Rake task has also been renamed to `rake deimos:outbox`.
+* The `SchemaControllerMixin` has been removed as there was no serious usage for it.
+* `InlineConsumer` has been removed - Karafka Pro has an [Iterator API](https://karafka.io/docs/Pro-Iterator-API/) that does the same thing. There also has been no evidence that it was used (and was probably pretty buggy).
+* The `:test` backend has been removed and the `Deimos::TestHelpers` module is now largely powered by [karafka-testing](https://github.com/karafka/karafka-testing/). This means that you can no longer use `Deimos::Backends::Test.sent_messages` - you need to use `Deimos::TestHelpers.sent_messages`. (The V2 generator should fix this.)
+* Individual consumer and producer settings now live within Karafka route configuration. This means you can no longer call e.g. `consumer.schema` to retrieve this information, as settings are no longer stored directly on the consumer and producer objects (it is still available, but via different methods).
+* Consumers should no longer define a `consume` method, as the semantics have changed with Karafka. Instead, you can define a `consume_message` or `consume_batch` method. Both of these methods now take Karafka `Message` objects instead of hashes. The V2 generator can handle translating this for you, but if you create new consumers, you should take advantage of the Karafka functionality and use it first-class.
+* Phobos `delivery_method` is no longer relevant. Instead, specify an `each_message` setting for your consumer. If set to true, you should define a `consume_message` method. Otherwise, you should define a `consume_batch` method. (Note that this is the reverse from the previous default, which assumed `delivery_method: message`.) The V2 generator will create the correct setting for each consumer.
+
+```ruby
+# before:
+class MyConsumer < Deimos::Consumer
+  def consume(payload, metadata)
+    # payload and metadata are both hashes
+  end
+
+  # OR with delivery_method: inline_batch
+  def batch_consume(payloads, metadata)
+    # payloads is an array of hashes, metadata is a hash
+  end
+end
+
+# now:
+class MyConsumer < Deimos::Consumer
+  def consume_batch
+    payloads = messages.payloads # messages is an instance method and `payloads` will return the decoded hashes
+  end
+
+  # OR with batch(false)
+  def consume_message(message)
+    # message is a Karafka Message object
+    payload = message.payload
+    key = message.key # etc.
+  end
+end
+```
+
+### Metrics
+
+The following metrics have been **removed** in favor of Karafka's more robust [DataDog metrics](https://karafka.io/docs/Monitoring-and-Logging/#datadog-and-statsd-integration) and WaterDrop's [DataDog metrics](https://karafka.io/docs/WaterDrop-Monitoring-and-Logging/#datadog-and-statsd-integration):
+* `consumer_lag` (use `consumer.lags`)
+* `handler` (use `consumer.consumed.time_taken`)
+* `publish` (use `produced_sync` and `produced_async`)
+* `publish_error` (use `deliver.errors`)
+
+You will need to manually add the DataDog MetricsListener as shown in the above pages.
+
+The following metrics have been **renamed**:
+
+* `db_producer.insert` -> `outbox.insert`
+* `db_producer.process` -> `outbox.process`
+
+### Instrumentation
+
+Deimos's own instrumentation layer has been removed in favor of Karafka's. You can still subscribe to Deimos notifications - you simply do it via Karafka's monitor instead of Deimos's.
+
+```ruby
+# before:
+Deimos.subscribe('encode_messages') do |event|	
+    # ... 
+end
+
+# after:
+Karafka.monitor.subscribe('deimos.encode_messages') do |event|
+  # ...
+end
+```
+
+Note that Karafka's monitors do not support the legacy "splatted" subscribe:
+```ruby
+Deimos.subscribe("event") do |*args|
+  payload = ActiveSupport::Notifications::Event.new(*args).payload
+end
+```
+
+The following instrumentation events have been **removed** in favor of Karafka's [events](https://karafka.io/docs/Monitoring-and-Logging/#subscribing-to-the-instrumentation-events):
+
+* `produce_error` (use `error.occurred`)
+
+The following events have been **renamed**:
+* `encode_messages` -> `deimos.encode_message` (**note that only one message is instrumented at a time now**)
+* `db_producer.produce` -> `deimos.outbox.produce`
+* `batch_consumption.valid_records` -> `deimos.batch_consumption.valid_records`
+* `batch_consumption.invalid_records` -> `deimos.batch_consumption.invalid_records`
+
+### Additional breaking changes
+* `key_config` now defaults to `{none: true}` instead of erroring out if not set.
+* `fatal_error?` now receives a Karafka `messages` object instead of a payload hash or array of hashes.
+* `watched_attributes` has been moved from the corresponding ActiveRecord class to the ActiveRecordProducer class. The object being watched is passed into the method.
+* Removed `TestHelpers.full_integration_test!` and `kafka_test!` as Karafka does not currently support these use cases. If we need them back, we will need to put in changes to the testing library to support them.
+* `test_consume_message` and `test_consume_batch` used to not fully validate schemas when using the `:avro_validation` backend. Now these are fully validated, which may cause test errors when upgrading.
+
+### New functionality
+
+* When setting up a Datadog metrics client, you can pass `:karafka_namespace`, `:karafka_distribution_mode`, or `:rd_kafka_metrics` tags to specify the Karafka settings for Datadog metrics.
+- The `payload_log` setting now works for consumers as well as producers, as it is now a topic setting.
+- You can publish messages **without a Deimos Producer class**. Karafka producers take a hash with `:message`, `:topic`, `:key`, `:headers` and `:partition_key` keys. As long as the topic is configured in `karafka.rb`, you don't need a special class to send the message. You can simply call `Karafka.producer.produce()`.
+- The only features that are now available on the bare Producer (as opposed to ActiveRecordProducer) class are:
+    - Outbox backend
+    - Instance method to determine partition key (rather than passing it in)
+    - Using `Deimos.disable_producers`
+- If you need these features, you must continue using a `Deimos::Producer`.
+- You can now call `.produce(messages)` directly on a `Deimos::Producer` which allows for use of these features while still passing a Karafka message hash. This removes the need to add a `payload_key` key into your payload. This is now the recommended method to use in a Deimos Producer.
+
+### New deprecations
+* For testing, you no longer have to call `unit_test!` to get the right settings. It is handled automatically by Karafka. The only thing this method now does is set the schema backend to `:avro_validation`, and you can do that in a single line.
+* The `skip_expectation` and `call_original` arguments to `test_consume_message` and `test_consume_batch` have been deprecated and no longer need to be provided. The assumption is that `call_original` is always true.
+
 ## Upgrading from < 1.5.0 to >= 1.5.0
 
 If you are using Confluent's schema registry to Avro-encode your
 messages, you will need to manually include the `avro_turf` gem
 in your Gemfile now.