README.md in fluent-plugin-bigquery-0.1.0 vs README.md in fluent-plugin-bigquery-0.2.0
- old
+ new
@@ -163,27 +163,28 @@
Note that the timestamp of logs and the date in the table id do not always match,
because there is a time lag between collection and transmission of logs.
### Table schema
-There are two methods to describe the schema of the target table.
+There are three methods to describe the schema of the target table.
1. List fields in fluent.conf
2. Load a schema file in JSON.
+3. Fetch a schema using BigQuery API
The examples above use the first method. In this method,
you can also specify nested fields by prefixing their belonging record fields.
```apache
<match dummy>
type bigquery
-
+
...
time_format %s
time_field time
-
+
field_integer time,response.status,response.bytes
field_string request.vhost,request.path,request.method,request.protocol,request.agent,request.referer,remote.host,remote.ip,remote.user
field_float request.time
field_boolean request.bot_access,request.loginsession
</match>
@@ -213,24 +214,42 @@
The second method is to specify a path to a BigQuery schema file instead of listing fields. In this case, your fluent.conf looks like:
```apache
<match dummy>
type bigquery
-
+
...
-
+
time_format %s
time_field time
-
+
schema_path /path/to/httpd.schema
field_integer time
</match>
```
where /path/to/httpd.schema is a path to the JSON-encoded schema file which you used for creating the table on BigQuery.
+The third method is to set `fetch_schema` to `true` to enable fetch a schema using BigQuery API. In this case, your fluent.conf looks like:
+
+```apache
+<match dummy>
+ type bigquery
+
+ ...
+
+ time_format %s
+ time_field time
+
+ fetch_schema true
+ field_integer time
+</match>
+```
+
+If you specify multiple talbe in configuration file, plugin get all schema data from BigQuery and merge it.
+
NOTE: Since JSON does not define how to encode data of TIMESTAMP type,
-you are still recommended to specify JSON types for TIMESTAMP fields as "time" field does in the example.
+you are still recommended to specify JSON types for TIMESTAMP fields as "time" field does in the example, if you use second or third method.
## TODO
* support Load API
* with automatically configured flush/buffer options
@@ -238,5 +257,10 @@
* support NULLABLE/REQUIRED/REPEATED field options in field list style of configuration
* OAuth installed application credentials support
* Google API discovery expiration
* Error classes
* check row size limits
+
+## Authors
+
+* @tagomoris: First author, original version
+* KAIZEN platform Inc.: Maintener, Since 2014.08.19