README.md in fluent-plugin-bigquery-0.1.0 vs README.md in fluent-plugin-bigquery-0.2.0

- old
+ new

@@ -163,27 +163,28 @@ Note that the timestamp of logs and the date in the table id do not always match, because there is a time lag between collection and transmission of logs. ### Table schema -There are two methods to describe the schema of the target table. +There are three methods to describe the schema of the target table. 1. List fields in fluent.conf 2. Load a schema file in JSON. +3. Fetch a schema using BigQuery API The examples above use the first method. In this method, you can also specify nested fields by prefixing their belonging record fields. ```apache <match dummy> type bigquery - + ... time_format %s time_field time - + field_integer time,response.status,response.bytes field_string request.vhost,request.path,request.method,request.protocol,request.agent,request.referer,remote.host,remote.ip,remote.user field_float request.time field_boolean request.bot_access,request.loginsession </match> @@ -213,24 +214,42 @@ The second method is to specify a path to a BigQuery schema file instead of listing fields. In this case, your fluent.conf looks like: ```apache <match dummy> type bigquery - + ... - + time_format %s time_field time - + schema_path /path/to/httpd.schema field_integer time </match> ``` where /path/to/httpd.schema is a path to the JSON-encoded schema file which you used for creating the table on BigQuery. +The third method is to set `fetch_schema` to `true` to enable fetch a schema using BigQuery API. In this case, your fluent.conf looks like: + +```apache +<match dummy> + type bigquery + + ... + + time_format %s + time_field time + + fetch_schema true + field_integer time +</match> +``` + +If you specify multiple talbe in configuration file, plugin get all schema data from BigQuery and merge it. + NOTE: Since JSON does not define how to encode data of TIMESTAMP type, -you are still recommended to specify JSON types for TIMESTAMP fields as "time" field does in the example. +you are still recommended to specify JSON types for TIMESTAMP fields as "time" field does in the example, if you use second or third method. ## TODO * support Load API * with automatically configured flush/buffer options @@ -238,5 +257,10 @@ * support NULLABLE/REQUIRED/REPEATED field options in field list style of configuration * OAuth installed application credentials support * Google API discovery expiration * Error classes * check row size limits + +## Authors + +* @tagomoris: First author, original version +* KAIZEN platform Inc.: Maintener, Since 2014.08.19