README.md in fluent-plugin-querycombiner-0.0.0.pre vs README.md in fluent-plugin-querycombiner-0.0.1
- old
+ new
@@ -6,63 +6,219 @@
## Requirement
* a running Redis
+## Installation
-## Get started
+```
+$ fluent-gem install fluent-plugin-querycombiner
+```
+
+## Tutorial
+### Simple combination
+
+Suppose you have the sequence of event messages like:
+
```
-<match combiner.**>
+{
+ 'event_id': '01234567',
+ 'status': 'event-start',
+ 'started_at': '2001-02-03T04:05:06Z',
+}
+```
+
+and:
+
+```
+{
+ 'event_id': '01234567',
+ 'status': 'event-finish',
+ 'finished_at': '2001-02-03T04:15:11Z',
+}
+```
+
+Now you can combine these messages with this configuration:
+
+```
+<match event.**>
type query_combiner
tag combined.test
- flush_interval 0.5
-
+ # redis settings
host localhost
port 6379
db_index 0
- redis_retry 3
- query_identify session-id, task-id
- query_ttl 3 # sec
- buffer_size 10 # queries
+ query_identify event_id # field to combine together
+ query_ttl 3600 # messages time-to-live[sec]
+ buffer_size 1000 # max queries to store in redis
<catch>
- condition status == 'recog-init'
- replace time => time_init, status => status_init
+ condition status == 'event-start'
</catch>
- <prolong>
- condition status == 'recog-break'
- </prolong>
+ <dump>
+ condition status == 'event-finish'
+ </dump>
+</match>
+```
+
+Combined results will be:
+
+```
+{
+ "event_id": "01234567",
+ "status": "event-finish",
+ "started_at": "2001-02-03T04:05:06Z",
+ "finished_at": "2001-02-03T04:05:06Z"
+}
+```
+
+### Replace some field names
+
+If messages has the same fields, these are overwritten in the combination process. You can use `replace` sentence in `<catch>` and `<dump>` blocks to avoid overwrite such fields.
+
+For example, you have some event messages like:
+
+```
+{
+ 'event_id': '01234567',
+ 'status': 'event-start',
+ 'time': '2001-02-03T04:05:06Z',
+}
+```
+
+and:
+
+```
+{
+ 'event_id': '01234567',
+ 'status': 'event-finish',
+ 'time': '2001-02-03T04:15:11Z',
+}
+```
+
+You can keep `time` fields which defined both `event-start` and `event-finish` by using `replace` sentence.
+
+```
+<match event.**>
+ (...type, tag and redis configuration...)
+
+ query_identify event_id # field to combine together
+ query_ttl 3600 # messages time-to-live[sec]
+ buffer_size 1000 # max queries to store in redis
+
+ <catch>
+ condition status == 'event-start'
+ replace time => time_start
+
+ </catch>
+
<dump>
- condition status == 'recog-finish'
- replace time => time_finish, result => result_finish, status => status_finish
+ condition status == 'event-finish'
+ replace time => time_finish
</dump>
+</match>
+```
+
+Combined results will be:
+
+```
+{
+ "event_id": "01234567",
+ "status": "event-finish",
+ "time_start": "2001-02-03T04:05:06Z",
+ "time_finish": "2001-02-03T04:15:11Z"
+}
+```
+
+### \<release\> block
+
+In previous examples, messages with `"status": "event-start"` will be watched by plugin immediately.
+
+Suppose some error events occur and you don't want to watch or combine these messages.
+
+In this case `<release>` block will be useful.
+
+For example, your error messages are such like:
+
+```
+{
+ "event_id": "01234567",
+ "status": "event-error",
+ "time": "2001-02-03T04:05:06Z"
+}
+```
+
+Append this `<release>` block to the configuration and error events will not be watched or combined:
+
+```
<release>
- condition status == 'recog-error'
+ condition status == 'event-error'
</release>
+```
-</match>
+You cannot use `replace` sentence in the `<release>` block.
+
+
+### \<prolong\> block
+
+Suppose your `query_ttl` is **600** (10 minutes) and almost events are finished within **10 minutes**. But occasionally very-long events occur which finish about **1 hours**. These very-long events send `status: 'event-continue'` messages every 5 minutes for keep-alive.
+
+In this case you can use `<prolong>` block to reset expired time.
+
```
+ <prolong>
+ condition status == 'event-continue'
+ </prolong>
+```
+You cannot use `replace` sentence in the `<prolong>` block.
+
+Also you cannot combine messages which defined `<prolong>` blocks.
+
+
## Configuration
-#### host, port, db_index
+
+### tag
+The tag prefix for emitted event messages. By default it's `query_combiner`.
+
+### host, port, db_index
The basic information for connecting to Redis. By default it's **redis://127.0.0.1:6379/0**
-#### redis_retry
+### redis_retry
How many times should the plugin retry when performing a redis operation before raising a error.
By default it's 3.
-### session_timeout
-The inactive expire time in seconds. By default it's 1800 (30 minutes).
+### querl_ttl
+The inactive expire time in seconds. By default it's **1800** (30 minutes).
+### buffer_size
+The max queries to store in redis. By default it's **1000**.
-### tag
-The tag prefix for emitted event messages. By default it's `query_combiner`.
+### remove_interval
+The interval time to delete expired or overflowed queries which configured by `query_ttl` and `buffer_size`. By default it's 10 [sec].
+
+
+### redis_key_prefix
+
+The key prefix for data stored in Redis. By default it's `query_combiner:`.
+
+### query_identify
+
+Indicates how to extract the query identity from event record.
+It can be set as a single field name or multiple field names join by comma (`,`).
+
+
+## TODO
+
+- Multi-query combination
+- Support hyphen `-` and dollar `$` contained field names
+
## Copyright
Copyright:: Copyright (c) 2014- Takahiro Kamatani