README.rdoc in fluent-plugin-mongo-0.5.3 vs README.rdoc in fluent-plugin-mongo-0.6.0

- old
+ new

@@ -5,15 +5,15 @@ === MongoOutput Store fluent-event as MongoDB Document to MongoDB database. MongoOutput set 'time' field to a document by default. -You set -false- to 'include_time_key' parameter if you disable this behaivor. +You set _false_ to 'include_time_key' parameter if you disable this behaivor. -=== MongoOutputTagCollection +==== tag mapped mode -Tag mapping to MongoDB collection automatically. +Tag mapped to MongoDB collection automatically. === MongoBackupOutput Store fluent-event to local capped collection for backup. @@ -33,45 +33,71 @@ # Following attibutes are optional host fluenter port 10000 # You can use 'capped' if you want to use capped collection - capped true + capped capped_size 100m # Other buffer configurations here </match> -==== NOTE +==== Tag mapped mode -MongoDB's output plugins have the limitation of buffer size. -Because MongoDB and Ruby-Driver checks the total object size at each insertion. -If total object size gets over the size limitation, -MongoDB returns error or Ruby-Driver raises an Exception. +Use 'tag_mapped' parameter. -So, MongoDB's output plugins reset :buffer_chunk_limit if configurated value is larger than above limitation. -- Before v1.8, max of :buffer_chunk_limit is 2MB -- After v1.8, max of :buffer_chunk_limit is 10MB - -=== MongoOutputTagCollection - -Use mongo_tag_collection type in match. - If tag name is "foo.bar", auto create collection "foo.bar" and insert data. <match forward.*> - type mongo_tag_collection + type mongo database fluent - # This configuration is used if tag not found. Default is 'untagged' + # If You use 'tag_mapped', then tag mapped mode enabled. + tag_mapped + + # If tag is "forward.foo.bar", then prefix "forward." is removed. + # Collection name to insert is "foo.bar". + remove_tag_prefix forward. + + # This configuration is used if tag not found. Default is 'untagged'. collection misc - # Tag "forward.foo.bar" remove prefix "forward.". - # Collection Mapping name is "foo.bar". - remove_prefix_collection forward. + # Other configurations here </match> +==== NOTE + +===== Broken data as a BSON + +Fluentd event sometimes has an invalid record as a BSON. +In such case, mongo plugin marshals an invalid record using Marshal.dump +and re-inserts its to same collection. + +Example: + + {"key1": "invalid value", "key2": "valid value", "time": ISODate("2012-01-15T21:09:53Z") } + +to + + {"__broken_data": "Marshal.dump result", "time": ISODate("2012-01-15T21:09:53Z") } + +mongo-ruby-driver cannot detect invalid attribute, +so mongo plugin marshals all attributes excluding Fluentd keys(tag_key and time_key). + +If you ignores an invalid record, please set 'ignore_invalid_record true' in match. + +===== Buffer size limitation + +MongoDB's output plugins have the limitation of buffer size. +Because MongoDB and Ruby-Driver checks the total object size at each insertion. +If total object size gets over the size limitation, +MongoDB returns error or Ruby-Driver raises an Exception. + +So, MongoDB's output plugins reset :buffer_chunk_limit if configurated value is larger than above limitation. +- Before v1.8, max of :buffer_chunk_limit is 2MB +- After v1.8, max of :buffer_chunk_limit is 10MB + === MongoBackupOutput Use mongo_backup type in match. mongo_backup alwalys use capped collection. <match ...> @@ -105,10 +131,10 @@ == TODO === More configuration -- Create Index +- Authentication - Select insert or update - etc == Tool