README.md in fluent-plugin-geoip-1.0.0 vs README.md in fluent-plugin-geoip-1.1.0

- old
+ new

@@ -1,10 +1,10 @@ # fluent-plugin-geoip [![Build Status](https://travis-ci.org/y-ken/fluent-plugin-geoip.png?branch=master)](https://travis-ci.org/y-ken/fluent-plugin-geoip) -Fluentd Output plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases. +Fluentd Filter/Output plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases. -fluent-plugin-geoip has bundled cost-free [GeoLite City database](http://dev.maxmind.com/geoip/legacy/geolite/) by default.<br /> +fluent-plugin-geoip has bundled cost-free [GeoLite2 Free Downloadable Databases](https://dev.maxmind.com/geoip/geoip2/geolite2/) and [GeoLite City database](http://dev.maxmind.com/geoip/legacy/geolite/) by default.<br /> Also you can use purchased [GeoIP City database](http://www.maxmind.com/en/city) ([lang:ja](http://www.maxmind.com/ja/city)) which costs starting from $50. The accuracy details for GeoLite City (free) and GeoIP City (purchased) has described at the page below. * http://www.maxmind.com/en/geolite_city_accuracy ([lang:ja](http://www.maxmind.com/ja/geolite_city_accuracy)) @@ -20,11 +20,11 @@ If you want to use this plugin with Fluentd v0.12.x or earlier use 0.8.x. ### Compatibility notice We've used Fluentd v1 API in this plugin since 1.0.0. -So we have dropped some features. +So we have dropped some features from GeoipOutput. See also [official document](http://docs.fluentd.org/v1.0/articles/plugin-update-from-v12) #### Fluent::Mixin::RewriteTagName @@ -105,123 +105,106 @@ $ sudo td-agent-gem install fluent-plugin-geoip ``` ## Usage -### For GeoipOutput +### For GeoipFilter +Note that filter version of geoip plugin does not have handling tag feature. + ```xml -<match access.apache> +<filter access.apache> @type geoip # Specify one or more geoip lookup field which has ip address (default: host) # in the case of accessing nested value, delimit keys by dot like 'host.ip'. geoip_lookup_key host # Specify optional geoip database (using bundled GeoLiteCity databse by default) - geoip_database "/path/to/your/GeoIPCity.dat" + # geoip_database "/path/to/your/GeoIPCity.dat" # Specify optional geoip2 database - # geoip2_database "/path/to/your/GeoLite2-City.mmdb" - # Specify backend library (geoip, geoip2_compat, geoip2_c) - backend_library geoip + # geoip2_database "/path/to/your/GeoLite2-City.mmdb" (using bundled GeoLite2-City.mmdb by default) + # Specify backend library (geoip2_c, geoip, geoip2_compat) + backend_library geoip2_c # Set adding field with placeholder (more than one settings are required.) <record> - latitude ${latitude["host"]} - longitude ${longitude["host"]} - country_code3 ${country_code3["host"]} - country ${country_code["host"]} - country_name ${country_name["host"]} - dma ${dma_code["host"]} - area ${area_code["host"]} - region ${region["host"]} - city ${city["host"]} + city ${city.names.en["host"]} + latitude ${location.latitude["host"]} + longitude ${location.longitude["host"]} + country ${country.iso_code["host"]} + country_name ${country.names.en["host"]} + postal_code ${postal.code["host"]} </record> - # Settings for tag - tag geoip.${tag[1]} - # To avoid get stacktrace error with `[null, null]` array for elasticsearch. skip_adding_null_record true - # Set log_level for fluentd-v0.10.43 or earlier (default: warn) - log_level info - - <buffer tag> - # Set buffering time (default: 0s) - flush_interval 1s - </buffer> -</match> + # Set @log_level (default: warn) + @log_level info +</filter> ``` #### Tips: how to geolocate multiple key ```xml -<match access.apache> +<filter access.apache> @type geoip geoip_lookup_key user1_host, user2_host <record> - user1_city ${city["user1_host"]} - user2_city ${city["user2_host"]} + user1_city ${city.names.en["user1_host"]} + user2_city ${city.names.en["user2_host"]} </record> - tag geoip.${tag[1]} -</match> +</filter> ``` #### Advanced config samples It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.<br /> -**Notice** v0 config will be deprecated in the future. - ``` -<match access.apache> +<filter access.apache> @type geoip geoip_lookup_key host <record> # lat lon as properties # ex. {"lat" => 37.4192008972168, "lon" => -122.05740356445312 } - location_properties '{ "lat" : ${latitude["host"]}, "lon" : ${longitude["host"]} }' + location_properties '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }' # lat lon as string # ex. "37.4192008972168,-122.05740356445312" - location_string ${latitude["host"]},${longitude["host"]} + location_string ${location.latitude["host"]},${location.longitude["host"]} # GeoJSON (lat lon as array) is useful for Kibana's bettermap. # ex. [-122.05740356445312, 37.4192008972168] - location_array '[${longitude["host"]},${latitude["host"]}]' + location_array '[${location.longitude["host"]},${location.latitude["host"]}]' </record> - tag geoip.${tag[1]} # To avoid get stacktrace error with `[null, null]` array for elasticsearch. skip_adding_null_record true -</match> +</filter> ``` On the case of using td-agent3 (v1-config), it have to quote `{ ... }` or `[ ... ]` block with quotation like below. ``` -<match access.apache> +<filter access.apache> @type geoip geoip_lookup_key host <record> - location_properties '{ "lat" : ${latitude["host"]}, "lon" : ${longitude["host"]} }' - location_string ${latitude["host"]},${longitude["host"]} - location_array '[${longitude["host"]},${latitude["host"]}]' + location_properties '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }' + location_string ${location.latitude["host"]},${location.longitude["host"]} + location_array '[${location.longitude["host"]},${location.latitude["host"]}]' </record> - remove_tag_prefix access. - tag geoip.${tag} skip_adding_null_record true -</match> +</filter> ``` -### For GeoipFilter +### For GeoipOutput -Note that filter version of geoip plugin does not have handling tag feature. - ```xml -<filter access.apache> +<match access.apache> @type geoip # Specify one or more geoip lookup field which has ip address (default: host) # in the case of accessing nested value, delimit keys by dot like 'host.ip'. geoip_lookup_key host @@ -233,58 +216,58 @@ # Specify backend library (geoip, geoip2_compat, geoip2_c) backend_library geoip # Set adding field with placeholder (more than one settings are required.) <record> - city ${city["host"]} - latitude ${latitude["host"]} - longitude ${longitude["host"]} - country_code3 ${country_code3["host"]} - country ${country_code["host"]} - country_name ${country_name["host"]} - dma ${dma_code["host"]} - area ${area_code["host"]} - region ${region["host"]} + latitude ${location.latitude["host"]} + longitude ${location.longitude["host"]} + country ${country.iso_code["host"]} + country_name ${country.names.en["host"]} + postal_code ${postal.code["host"]} + region ${subdivisions.0.iso_code["host"]} + region_name ${subdivisions.0.names.en["host"]} + city ${city.names.en["host"]} </record> + # Settings for tag + tag geoip.${tag[1]} + # To avoid get stacktrace error with `[null, null]` array for elasticsearch. skip_adding_null_record true - # Set log_level for fluentd-v0.10.43 or earlier (default: warn) - log_level info -</filter> + # Set @log_level (default: warn) + @log_level info + + <buffer tag> + # Set buffering time (default: 0s) + flush_interval 1s + </buffer> +</match> ``` ## Tutorial -### For GeoipOutput +### For GeoipFilter #### configuration ```xml <source> @type forward </source> -<match test.geoip> - @type copy - <store> - @type stdout - </store> - <store> - @type geoip - geoip_lookup_key host - <record> - lat ${latitude["host"]} - lon ${longitude["host"]} - country ${country_code["host"]} - </record> - tag debug.${tag[1]} - </store> -</match> +<filter test.geoip> + @type geoip + geoip_lookup_key host + <record> + city ${city.names.en["host"]} + lat ${location.latitude["host"]} + lon ${location.longitude["host"]} + </record> +</filter> -<match debug.**> +<match test.**> @type stdout </match> ``` #### result @@ -293,37 +276,48 @@ # forward record with Google's ip address. $ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip # check the result at stdout $ tail /var/log/td-agent/td-agent.log -2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"} -2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","lat":37.4192008972168,"lon":-122.05740356445312,"country":"US"} +2016-02-01 12:04:37 +0900 test.geoip: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312} ``` -For more details of geoip data format is described at the page below in section `GeoIP City Edition CSV Database Fields`.<br /> -http://dev.maxmind.com/geoip/legacy/csv/ +You can check geoip data format using [utils/dump.rb](https://github.com/okkez/fluent-plugin-geoip/utils/dump.rb). -### For GeoipFilter +``` +$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80 +$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80 +$ bundle exec ruby urils/dump.rb geoip 66.102.3.80 +``` +### For GeoipOutput + #### configuration ```xml <source> @type forward </source> -<filter test.geoip> - @type geoip - geoip_lookup_key host - <record> - city ${city["host"]} - lat ${latitude["host"]} - lon ${longitude["host"]} - </record> -</filter> +<match test.geoip> + @type copy + <store> + @type stdout + </store> + <store> + @type geoip + geoip_lookup_key host + <record> + lat ${location.latitude["host"]} + lon ${location.longitude["host"]} + country ${country.iso_code["host"]} + </record> + tag debug.${tag[1]} + </store> +</match> -<match test.**> +<match debug.**> @type stdout </match> ``` #### result @@ -332,42 +326,24 @@ # forward record with Google's ip address. $ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip # check the result at stdout $ tail /var/log/td-agent/td-agent.log -2016-02-01 12:04:37 +0900 test.geoip: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312} +2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"} +2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","lat":37.4192008972168,"lon":-122.05740356445312,"country":"US"} ``` -For more details of geoip data format is described at the page below in section `GeoIP City Edition CSV Database Fields`.<br /> -http://dev.maxmind.com/geoip/legacy/csv/ +You can check geoip data format using [utils/dump.rb](https://github.com/okkez/fluent-plugin-geoip/utils/dump.rb). +``` +$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80 +$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80 +$ bundle exec ruby urils/dump.rb geoip 66.102.3.80 +``` + ## Placeholders -### GeoIP legacy - -Provides these placeholders for adding field of geolocate results.<br /> -For more example of geolocating, you can try these websites like [Geo IP Address View](http://www.geoipview.com/) or [View my IP information](http://www.geoiptool.com/en/). - -| placeholder attributes | output example | type | note | -|--------------------------------|-------------------|--------------|------| -| ${city[lookup_field]} | "Ithaca" | varchar(255) | - | -| ${latitude[lookup_field]} | 42.4277992248535 | decimal | - | -| ${longitude[lookup_field]} | -76.4981994628906 | decimal | - | -| ${country_code3[lookup_field]} | "USA" | varchar(3) | - | -| ${country_code[lookup_field]} | "US" | varchar(2) | A two-character ISO 3166-1 country code | -| ${country_name[lookup_field]} | "United States" | varchar(50) | - | -| ${dma_code[lookup_field]} | 555 | unsigned int | **only for US** | -| ${area_code[lookup_field]} | 607 | char(3) | **only for US** | -| ${region[lookup_field]} | "NY" | char(2) | A two character ISO-3166-2 or FIPS 10-4 code | - -Further more specification available at http://dev.maxmind.com/geoip/legacy/csv/#GeoIP_City_Edition_CSV_Database_Fields - -Related configurations: - -* `backend_library`: `geoip` (default) -* `geoip_database`: path to your GeoLiteCity.dat - ### GeoIP2 You can get any fields in the [GeoLite2](http://dev.maxmind.com/geoip/geoip2/geolite2/) database and [GeoIP2 Downloadable Databases](http://dev.maxmind.com/geoip/geoip2/downloadable/). @@ -403,47 +379,143 @@ Related configurations: * `backend_library`: `geoip2_compat` or `geoip2_c` * `geoip2_database`: path to your GeoLite2-City.mmdb +### GeoIP legacy + +Provides these placeholders for adding field of geolocate results.<br /> +For more example of geolocating, you can try these websites like [Geo IP Address View](http://www.geoipview.com/) or [View my IP information](http://www.geoiptool.com/en/). + +| placeholder attributes | output example | type | note | +|--------------------------------|-------------------|--------------|------| +| ${city[lookup_field]} | "Ithaca" | varchar(255) | - | +| ${latitude[lookup_field]} | 42.4277992248535 | decimal | - | +| ${longitude[lookup_field]} | -76.4981994628906 | decimal | - | +| ${country_code3[lookup_field]} | "USA" | varchar(3) | - | +| ${country_code[lookup_field]} | "US" | varchar(2) | A two-character ISO 3166-1 country code | +| ${country_name[lookup_field]} | "United States" | varchar(50) | - | +| ${dma_code[lookup_field]} | 555 | unsigned int | **only for US** | +| ${area_code[lookup_field]} | 607 | char(3) | **only for US** | +| ${region[lookup_field]} | "NY" | char(2) | A two character ISO-3166-2 or FIPS 10-4 code | + +Further more specification available at http://dev.maxmind.com/geoip/legacy/csv/#GeoIP_City_Edition_CSV_Database_Fields + +Related configurations: + +* `backend_library`: `geoip` (default) +* `geoip_database`: path to your GeoLiteCity.dat + ## Parameters -### GeoipOutput +### GeoipFilter -* `include_tag_key` (default: false) -* `tag_key` +Note that filter version of `geoip` plugin does not have handling `tag` feature. -Add original tag name into filtered record using SetTagKeyMixin.<br /> -Further details are written at http://docs.fluentd.org/articles/in_exec +#### Plugin helpers -* `skip_adding_null_record` (default: false) +* [compat_parameters](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-compat_parameters) +* [inject](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-inject) +See also [Filter Plugin Overview](https://docs.fluentd.org/v1.0/articles/filter-plugin-overview) + +#### Supported sections + +* [Inject section configurations](https://docs.fluentd.org/v1.0/articles/inject-section) + +#### Parameters + +[Plugin Common Paramteters](https://docs.fluentd.org/v1.0/articles/plugin-common-parameters) + +**geoip_database** (string) (optional) + +* Default value: bundled database `GeoLiteCity.dat` + +Path to GeoIP database file. + +**geoip2_database** (string) (optional) + +* Default value: bundled database `GeoLite2-City.mmdb`. + +Path to GeoIP2 database file. + +**geoip_lookup_key** (string) (optional) + +* Default value: `host`. + +Specify one or more geoip lookup field which has IP address. + +**skip_adding_null_record** (bool) (optional) + +* Default value: `nil` + Skip adding geoip fields when this valaues to `true`. On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything. -* `tag` +**backend_library** (enum) (optional) -On using this option with tag placeholder like `tag geoip.${tag}` (test code is available at [test_out_geoip.rb](https://github.com/y-ken/fluent-plugin-geoip/blob/master/test/plugin/test_out_geoip.rb)). +* Available values: `geoip`, `geoip2_compat`, `geoip2_c` +* Default value: `geoip2_c`. -* `flush_interval` (default: 0 sec) +Set backend library. -Set buffering time to execute bulk lookup geoip. +### GeoipOutput -### GeoipFilter +#### Plugin helpers -Note that filter version of `geoip` plugin does not have handling `tag` feature. +* [event_emitter](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-event_emitter) +* [compat_parameters](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-compat_parameters) +* [inject](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-inject) -* `include_tag_key` (default: false) +See also [Output Plugin Overview](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) -Add original tag name into filtered record using SetTagKeyMixin.<br /> -Further details are written at http://docs.fluentd.org/articles/in_exec +#### Sections -* `skip_adding_null_record` (default: false) +* [Inject section configurations](https://docs.fluentd.org/v1.0/articles/inject-section) +* [Buffer section configurations](https://docs.fluentd.org/v1.0/articles/buffer-section) +#### Parameters + +[Plugin Common Paramteters](https://docs.fluentd.org/v1.0/articles/plugin-common-parameters) + +**geoip_database** (string) (optional) + +* Default value: bundled database `GeoLiteCity.dat` + +Path to GeoIP database file. + +**geoip2_database** (string) (optional) + +* Default value: bundled database `GeoLite2-City.mmdb`. + +Path to GeoIP2 database file. + +**geoip_lookup_key** (string) (optional) + +* Default value: `host`. + +Specify one or more geoip lookup field which has IP address. + +**skip_adding_null_record** (bool) (optional) + +* Default value: `nil` + Skip adding geoip fields when this valaues to `true`. On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything. +**backend_library** (enum) (optional) + +* Available values: `geoip`, `geoip2_compat`, `geoip2_c` +* Default value: `geoip2_c`. + +Set backend library. + +**tag** (string) (optional) + +On using this option with tag placeholder like `tag geoip.${tag}` (test code is available at [test_out_geoip.rb](https://github.com/y-ken/fluent-plugin-geoip/blob/master/test/plugin/test_out_geoip.rb)). + + ## Articles * [IPアドレスを元に位置情報をリアルタイムに付与する fluent-plugin-geoip v0.0.1をリリースしました #fluentd - Y-Ken Studio](http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-has-released)<br /> http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-has-released @@ -460,11 +532,9 @@ http://developer.smartnews.be/blog/2013/10/03/easy-data-analysis-using-fluentd-redshift-and-tableau/ ## TODO Pull requests are very welcome!! - -* support [GeoIP2](http://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/) ## Contributing 1. Fork it 2. Create your feature branch (`git checkout -b my-new-feature`)