README.rdoc in fluent-plugin-mongo-0.5.3 vs README.rdoc in fluent-plugin-mongo-0.6.0
- old
+ new
@@ -5,15 +5,15 @@
=== MongoOutput
Store fluent-event as MongoDB Document to MongoDB database.
MongoOutput set 'time' field to a document by default.
-You set -false- to 'include_time_key' parameter if you disable this behaivor.
+You set _false_ to 'include_time_key' parameter if you disable this behaivor.
-=== MongoOutputTagCollection
+==== tag mapped mode
-Tag mapping to MongoDB collection automatically.
+Tag mapped to MongoDB collection automatically.
=== MongoBackupOutput
Store fluent-event to local capped collection for backup.
@@ -33,45 +33,71 @@
# Following attibutes are optional
host fluenter
port 10000
# You can use 'capped' if you want to use capped collection
- capped true
+ capped
capped_size 100m
# Other buffer configurations here
</match>
-==== NOTE
+==== Tag mapped mode
-MongoDB's output plugins have the limitation of buffer size.
-Because MongoDB and Ruby-Driver checks the total object size at each insertion.
-If total object size gets over the size limitation,
-MongoDB returns error or Ruby-Driver raises an Exception.
+Use 'tag_mapped' parameter.
-So, MongoDB's output plugins reset :buffer_chunk_limit if configurated value is larger than above limitation.
-- Before v1.8, max of :buffer_chunk_limit is 2MB
-- After v1.8, max of :buffer_chunk_limit is 10MB
-
-=== MongoOutputTagCollection
-
-Use mongo_tag_collection type in match.
-
If tag name is "foo.bar", auto create collection "foo.bar" and insert data.
<match forward.*>
- type mongo_tag_collection
+ type mongo
database fluent
- # This configuration is used if tag not found. Default is 'untagged'
+ # If You use 'tag_mapped', then tag mapped mode enabled.
+ tag_mapped
+
+ # If tag is "forward.foo.bar", then prefix "forward." is removed.
+ # Collection name to insert is "foo.bar".
+ remove_tag_prefix forward.
+
+ # This configuration is used if tag not found. Default is 'untagged'.
collection misc
- # Tag "forward.foo.bar" remove prefix "forward.".
- # Collection Mapping name is "foo.bar".
- remove_prefix_collection forward.
+ # Other configurations here
</match>
+==== NOTE
+
+===== Broken data as a BSON
+
+Fluentd event sometimes has an invalid record as a BSON.
+In such case, mongo plugin marshals an invalid record using Marshal.dump
+and re-inserts its to same collection.
+
+Example:
+
+ {"key1": "invalid value", "key2": "valid value", "time": ISODate("2012-01-15T21:09:53Z") }
+
+to
+
+ {"__broken_data": "Marshal.dump result", "time": ISODate("2012-01-15T21:09:53Z") }
+
+mongo-ruby-driver cannot detect invalid attribute,
+so mongo plugin marshals all attributes excluding Fluentd keys(tag_key and time_key).
+
+If you ignores an invalid record, please set 'ignore_invalid_record true' in match.
+
+===== Buffer size limitation
+
+MongoDB's output plugins have the limitation of buffer size.
+Because MongoDB and Ruby-Driver checks the total object size at each insertion.
+If total object size gets over the size limitation,
+MongoDB returns error or Ruby-Driver raises an Exception.
+
+So, MongoDB's output plugins reset :buffer_chunk_limit if configurated value is larger than above limitation.
+- Before v1.8, max of :buffer_chunk_limit is 2MB
+- After v1.8, max of :buffer_chunk_limit is 10MB
+
=== MongoBackupOutput
Use mongo_backup type in match. mongo_backup alwalys use capped collection.
<match ...>
@@ -105,10 +131,10 @@
== TODO
=== More configuration
-- Create Index
+- Authentication
- Select insert or update
- etc
== Tool