README.md in fluent-plugin-protobuf-http-0.3.0 vs README.md in fluent-plugin-protobuf-http-0.3.1
- old
+ new
@@ -1,26 +1,26 @@
# fluent-plugin-protobuf-http
[](https://github.com/iamazeem/fluent-plugin-protobuf-http/actions/workflows/ci.yml)
-[](https://github.com/iamAzeem/fluent-plugin-protobuf-http/blob/master/LICENSE)
+[](https://github.com/iamAzeem/fluent-plugin-protobuf-http/blob/master/LICENSE)
-[](https://rubygems.org/gems/fluent-plugin-protobuf-http)
+[](https://rubygems.org/gems/fluent-plugin-protobuf-http)
+[](https://www.buymeacoffee.com/iamazeem)
-
-
-
+## Overview
-[Fluentd](https://fluentd.org/) HTTP input plugin for Protocol Buffers.
+[Fluentd](https://fluentd.org/) HTTP input plugin for
+[Protocol Buffers](https://github.com/protocolbuffers/protobuf).
## Features
-* Automatic compilation of `.proto` files located in `proto_dir`
-* Incoming Format: Binary or JSON (`Content-Type`: `application/octet-stream` or
+- Automatic compilation of `.proto` files located in `proto_dir`
+- Incoming Format: Binary or JSON (`Content-Type`: `application/octet-stream` or
`application/json`)
-* Outgoing Format: Binary or JSON
-* Single and Batch message support
-* TLS Support with `<transport>` section and `https://` URL protocol prefix.
+- Outgoing Format: Binary or JSON
+- Single and Batch message support
+- TLS Support with `<transport>` section and `https://` URL protocol prefix.
For more details on TLS configuration, see this official
[example](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server#configuration-example).
## Installation
@@ -31,11 +31,11 @@
gem install fluent-plugin-protobuf-http
```
### Bundler
-Add following line to your Gemfile:
+Add the following line to your Gemfile:
```ruby
gem 'fluent-plugin-protobuf-http'
```
@@ -45,29 +45,29 @@
bundle
```
## Configuration
-* `bind` (string) (optional): The address to listen to.
- * Default: `0.0.0.0`
-* `port` (integer) (optional): The port to listen to.
- * Default: `8080`
-* `proto_dir` (string) (required): The directory path that contains the .proto files.
-* `in_mode` (enum) (optional): The mode of incoming (supported) events.
- * Modes: `binary`, `json`
- * Default: `binary`
-* `out_mode` (enum) (optional): The mode of outgoing (emitted) events.
- * Modes: `binary`, `json`
- * Default: `binary`
-* `tag` (string) (required): The tag for the event.
+- `bind` (string) (optional): The address to listen to.
+ - Default: `0.0.0.0`
+- `port` (integer) (optional): The port to listen to.
+ - Default: `8080`
+- `proto_dir` (string) (required): The directory path that contains the .proto files.
+- `in_mode` (enum) (optional): The mode of incoming (supported) events.
+ - Modes: `binary`, `json`
+ - Default: `binary`
+- `out_mode` (enum) (optional): The mode of outgoing (emitted) events.
+ - Modes: `binary`, `json`
+ - Default: `binary`
+- `tag` (string) (required): The tag for the event.
### `<transport>` section (optional) (single)
-* `protocol` (enum) (optional):
- * Protocols: `tcp`, `tls`
- * Default: `tcp`
- * For more details, see this official configuration
+- `protocol` (enum) (optional):
+ - Protocols: `tcp`, `tls`
+ - Default: `tcp`
+ - For more details, see this official configuration
[example](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server#configuration-example).
### Example
```text
@@ -148,12 +148,12 @@
string type = 1;
repeated Log batch = 2;
}
```
-IMPORTANT: The `Batch` message type is part of `log.proto`, it is not a separate
-file! You can choose any name for a batch message type.
+**IMPORTANT**: The `Batch` message type is part of `log.proto`, it is not a
+separate file! You can choose any name for a batch message type.
Here is the complete `log.proto` file:
```protobuf
syntax = "proto3";
@@ -261,13 +261,13 @@
<match debug.test>
@type stdout
</match>
```
-The incoming binary messages will be transformed to JSON for further consumption.
+The incoming binary messages will be converted to JSON for further consumption.
-### Single Message
+### Single Message Use-case
Test Parameters:
| input file | single message type |
|:----------:|:---------------------:|
@@ -295,14 +295,14 @@
2020-06-09 18:53:47 +0500 [info]: #0 [protobuf_http_input] [S] {binary} [127.0.0.1:41222, msgtype: service.logging.Log, size: 86 bytes]
2020-06-09 18:53:47.025638542 +0500 debug.test: {"message":"{\"context\":{\"timestamp\":\"2020-06-01T16:24:19Z\",\"hostOrIp\":\"192.168.xxx.xxx\",\"serviceName\":\"test\",\"user\":\"test\"},\"level\":\"INFO\",\"message\":\"This is a test log generated by [./log.rb].\"}"}
2020-06-09 18:53:47 +0500 [info]: #0 [protobuf_http_input] [S] {json} [127.0.0.1:41222, msgtype: service.logging.Log, size: 183 bytes]
```
-For sample Single message generation, see
-[this](https://github.com/iamAzeem/protobuf-log-sample).
+For generating sample Single messages, see
+https://github.com/iamAzeem/protobuf-log-sample.
-### Batch Message
+### Batch Message Use-case
Test Parameters:
| input file | batch message type | batch internal type | messages |
|:---------------:|:-----------------------:|:---------------------:|:--------:|
@@ -361,19 +361,48 @@
2020-06-09 19:07:09.738165527 +0500 debug.test: {"message":"{\"context\":{\"timestamp\":\"2020-06-08T17:27:05Z\",\"hostOrIp\":\"192.168.xxx.xxx\",\"serviceName\":\"test\",\"user\":\"test\"},\"level\":\"INFO\",\"message\":\"This is a test log generated by [./logbatch.rb].\"}"}
2020-06-09 19:07:09 +0500 [info]: #0 [protobuf_http_input] [B] {json} [127.0.0.1:41552, msgtype: service.logging.Batch] Batch received! [batch_type: service.logging.Log, batch_size: 5 messages]
```
For sample Batch message generation, see
-[this](https://gist.github.com/iamAzeem/a8a24092132e1741a76956192f2104cc).
+[this gist](https://gist.github.com/iamAzeem/a8a24092132e1741a76956192f2104cc).
+## CI Workflow
+
+The [CI workflow](ci..github/workflows/ci.yml) sets up the prerequisites. It
+builds and installs the plugin, and then runs the automated tests.
+
+To run tests locally, run:
+
+```shell
+bundle exec rake test
+```
+
+The [test](./test) directory contains the tests and the [input](./test/data)
+files.
+
+The code coverage is printed at the end using `simplecov`.
+
+## Known Issues
+
+- This plugin internally uses the HTTP server plugin
+ [helper](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server)
+ which has higher precedence for `async-http` over `webrick`. But,
+ [`webrick`](https://github.com/ruby/webrick) is required to run this. In an
+ environment where both are installed, `async-http` is automatically selected
+ causing runtime issues. To make this work, you need to uninstall `async-http`
+ i.e. `gem uninstall async-http`. See issue
+ [#10](https://github.com/iamazeem/fluent-plugin-protobuf-http/issues/10) for
+ more details where this was identified in Docker containers where both gems
+ were already installed.
+
## Contribute
-- Fork the project.
+- [Fork](https://github.com/iamazeem/fluent-plugin-protobuf-http/fork) the project.
- Check out the latest `main` branch.
-- Create a feature or bugfix branch from `main`.
+- Create a `feature` or `bugfix` branch from `main`.
- Commit and push your changes.
- Make sure to add and run tests locally: `bundle exec rake test`.
-- Run `rubocop` locally and fix all the lint warnings.
+- Run [Rubocop](https://github.com/rubocop/rubocop) and fix the lint errors.
- Submit the PR.
## License
[Apache 2.0](LICENSE)