= MongoDB plugin for Fluentd

fluent-plugin-mongo provides input and output plugins for {Fluentd}[http://fluentd.org/].

= Installation

== Gems

The gem is hosted at {Rubygems.org}[http://rubygems.org]. You can install the gem as follows:

    $ fluent-gem install fluent-plugin-mongo

= Plugins

== Output plugin

=== mongo

Store Fluentd event to MongoDB database.

==== Configuration

Use _mongo_ type in match.

    <match mongo.**>
      type mongo
      database fluent
      collection test

      # Following attibutes are optional
      host fluenter
      port 10000

      # Set 'capped' if you want to use capped collection
      capped
      capped_size 100m

      # Set 'user' and 'password' for authentication
      user handa
      password shinobu

      # Other buffer configurations here
    </match>

=== mongo(tag mapped mode)

Tag mapped to MongoDB collection automatically.

==== Configuration

Use _tag_mapped_ parameter in match of _mongo_ type.

If tag name is "foo.bar", auto create collection "foo.bar" and insert data.

    <match forward.*>
      type mongo
      database fluent

      # Set 'tag_mapped' if you want to use tag mapped mode.
      tag_mapped

      # If tag is "forward.foo.bar", then prefix "forward." is removed.
      # Collection name to insert is "foo.bar".
      remove_tag_prefix forward.

      # This configuration is used if tag not found. Default is 'untagged'.
      collection misc

      # Other configurations here
    </match>

=== mongo_replset

Replica Set version of mongo.

==== Configuration

Use _mongo_replset_ type in match.

    <match mongo.**>
      type mongo_replset
      database fluent
      collection logs

      # each node separated by ','
      nodes localhost:27017,localhost:27018,localhost:27019

      # num_retries is threshold at failover, default is 60.
      # If retry count reached this threshold, mongo plugin raises an exception.
      num_retries 30

      # following optional parameters passed to ReplSetConnection of mongo-ruby-driver.
      # See mongo-ruby-driver docs for more detail.
      #name replset_name
      #read secondary
      #refresh_mode sync
      #refresh_interval 60
    </match>

=== mongo_backup

Store Fluentd event to local capped collection for backup.

==== Configuration

Use _mongo_backup_ type in match. _mongo_backup_ alwalys use capped collection.

    <match ...>
      type mongo_backup
      capped_size 100m

      <store>
        type tcp
        host 192.168.0.13
        ...
      </store>
    </match>

== Input plugin

=== mongo_tail

Tail capped collection to input data.

==== Configuration

Use _mongo_tail_ type in source.

    <source>
      type mongo_tail
      database fluent
      collection capped_log

      tag app.mongo_log

      # Convert 'time'(BSON's time) to fluent time(Unix time).
      time_key time

      # You can store last ObjectId to tail over server's shutdown
      id_store_file /Users/repeatedly/devel/fluent-plugin-mongo/last_id
    </source>

= NOTE

== Broken data as a BSON

Fluentd event sometimes has an invalid record as a BSON.
In such case, Mongo plugin marshals an invalid record using Marshal.dump
and re-inserts its to same collection.

If passed following invalid record:

    {"key1": "invalid value", "key2": "valid value", "time": ISODate("2012-01-15T21:09:53Z") }

then Mongo plugin converts this record to following format:

    {"__broken_data": Marshal.dump result of {"key1": "invalid value", "key2": "valid value"}, "time": ISODate("2012-01-15T21:09:53Z") }

Mongo-Ruby-Driver cannot detect an invalid attribute,
so Mongo plugin marshals all attributes excluding Fluentd keys("tag_key" and "time_key").

If you want to ignore an invalid record, set _true_ to _ignore_invalid_record_ parameter in match.

    <match forward.*>
      ...

      # ignore invalid documents at write operation
      ignore_invalid_record true

      ...
    </match>

== Buffer size limitation

Mongo plugin has the limitation of buffer size.
Because MongoDB and mongo-ruby-driver checks the total object size at each insertion.
If total object size gets over the size limitation, then
MongoDB returns error or mongo-ruby-driver raises an exception.

So, Mongo plugin resets _buffer_chunk_limit_ if configurated value is larger than above limitation:
- Before v1.8, max of _buffer_chunk_limit_ is 2MB
- After v1.8, max of _buffer_chunk_limit_ is 10MB

= Tool

You can tail mongo capped collection.

    $ mongo-tail -f

= TODO

== More configuration

- Multi process
- etc

= Copyright

Copyright:: Copyright (c) 2011- Masahiro Nakagawa
License::   Apache License, Version 2.0