README.md in rocketjob-6.0.0.rc2 vs README.md in rocketjob-6.0.0.rc3

- old
+ new

@@ -15,24 +15,154 @@ ## Support * Questions? Join the chat room on Gitter for [rocketjob support](https://gitter.im/rocketjob/support) * [Report bugs](https://github.com/rocketjob/rocketjob/issues) -## Rocket Job v5 +## Rocket Job v6 - Support for Ruby v3 and Rails 6. -- Multiple output file support through extended `output_categories` capability. - - File output formats for each category. For example: CSV, PSV, JSON, etc. -- Support for AWS DocumentDB as the data store. +- Major enhancements in Batch job support: + - Direct built-in Tabular support for all input and output categories. + - Multiple output file support, each with its own settings for: + - Compression + - GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark). + - Encryption + - PGP, Symmetric Encryption. + - File format + - CSV, PSV, JSON, Fixed Format, xlsx. +- Significant error handling improvements, especially around throttle failures + that used to result in "hanging" jobs. +- Support AWS DocumentDB in addition to MongoDB as the data store. - Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid. -The following plugins have been deprecated and will be removed in Rocket Job v5.1 -- RocketJob::Batch::Tabular::Input -- RocketJob::Batch::Tabular::Output +### Upgrading to Rocket Job v6 +The following plugins have been deprecated and are no longer loaded by default. +- `RocketJob::Batch::Tabular::Input` +- `RocketJob::Batch::Tabular::Output` + +If your code relies on these plugins and you still want to upgrade to Rocket Job v6, +add the following require statement to any jobs that still use them: + +~~~ruby +require "rocket_job/batch/tabular" +~~~ + +It is important to migrate away from these plugins, since they will be removed in a future release. + +#### Upgrading Batch Jobs to Rocket Job v6 + +Rocket Job v6 replaces the array of symbol type for `input_categories` and `output_categories` +with an array of `RocketJob::Category::Input` and `RocketJob::Category::Output`. + +Jobs that added or modified the input or output categories need to be upgraded. For example: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + self.output_categories = [:main, :errors, :ignored] +end +~~~ + +Needs to be changed to: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + output_category name: :main + output_category name: :errors + output_category name: :ignored +end +~~~ + +##### slice_size, encrypt, compress + +These fields have been removed from the job itself: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + self.slice_sice = 1_000 + self.encrypt = true + self.compress = true +end +~~~ + +They are now specified on the `input_category` as follows: +- `slice_size` just moves under `input_category`. +- `encrypt` becomes an option to `serializer`. +- `compress` is now the default for all batch jobs so is not needed. + +If the serializer is set to `encrypt` then it is automatically compressed. + +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + input_category slice_sice: 1_000, serializer: :encrypt +end +~~~ + +##### collect_output, collect_nil_output + +The following fields have been moved from the job itself: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + self.collect_output = true + self.collect_nil_output = true +end +~~~ + +Into the corresponding `output_category`: +- `collect_output` no longer has any meaning. Output is collected anytime an `output_category` is defined. +- `collect_nil_output` is now the option `nils` on the `output_category. + It defaults to `false` so that by default any `nil` output from the `perform` method is not collected. +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + output_category nils: true +end +~~~ + +##### name + +For both `input_category` and `output_category`, when the `name` argument is not supplied +it defaults to `:main`. + +For Example: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + input_category name: :main, serializer: :encrypt + output_category name: :main +end +~~~ + +Is the same as: +~~~ruby +class MyJob < RocketJob::Job + include RocketJob::Batch + + input_category serializer: :encrypt + output_category +end +~~~ + +##### Existing and inflight jobs + +When migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the +deployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema. + +In flight jobs should not be affected, other than it is important to shutdown all running batch +servers _before_ running any new instances. + ## Rocket Job v4 -Rocket Job Pro is now open source and included in Rocket Job. +Rocket Job Pro is now fully open source and included in Rocket Job under the Apache License. The `RocketJob::Batch` plugin now adds batch processing capabilities to break up a single task into many concurrent workers processing slices of the entire job at the same time.