README.md in fluent-plugin-bigquery-1.0.0 vs README.md in fluent-plugin-bigquery-1.1.0
- old
+ new
@@ -29,48 +29,47 @@
## Configuration
### Options
-| name | type | required? | placeholder? | default | description |
-| :------------------------------------- | :------------ | :----------- | :---------- | :------------------------- | :----------------------- |
-| method | string | no | no | insert | `insert` (Streaming Insert) or `load` (load job) |
-| auth_method | enum | yes | no | private_key | `private_key` or `json_key` or `compute_engine` or `application_default` |
-| email | string | yes (private_key) | no | nil | GCP Service Account Email |
-| private_key_path | string | yes (private_key) | no | nil | GCP Private Key file path |
-| private_key_passphrase | string | yes (private_key) | no | nil | GCP Private Key Passphrase |
-| json_key | string | yes (json_key) | no | nil | GCP JSON Key file path or JSON Key string |
-| project | string | yes | yes | nil | |
-| dataset | string | yes | yes | nil | |
-| table | string | yes (either `tables`) | yes | nil | |
-| tables | array(string) | yes (either `table`) | yes | nil | can set multi table names splitted by `,` |
-| template_suffix | string | no | yes | nil | can use `%{time_slice}` placeholder replaced by `time_slice_format` |
-| auto_create_table | bool | no | no | false | If true, creates table automatically |
-| skip_invalid_rows | bool | no | no | false | Only `insert` method. |
-| max_bad_records | integer | no | no | 0 | Only `load` method. If the number of bad records exceeds this value, an invalid error is returned in the job result. |
-| ignore_unknown_values | bool | no | no | false | Accept rows that contain values that do not match the schema. The unknown values are ignored. |
-| schema | array | yes (either `fetch_schema` or `schema_path`) | no | nil | Schema Definition. It is formatted by JSON. |
-| schema_path | string | yes (either `fetch_schema`) | no | nil | Schema Definition file path. It is formatted by JSON. |
-| fetch_schema | bool | yes (either `schema_path`) | no | false | If true, fetch table schema definition from Bigquery table automatically. |
-| fetch_schema_table | string | no | yes | nil | If set, fetch table schema definition from this table, If fetch_schema is false, this param is ignored |
-| schema_cache_expire | integer | no | no | 600 | Value is second. If current time is after expiration interval, re-fetch table schema definition. |
-| field_string | string | no | no | nil | see examples. |
-| field_integer | string | no | no | nil | see examples. |
-| field_float | string | no | no | nil | see examples. |
-| field_boolean | string | no | no | nil | see examples. |
-| field_timestamp | string | no | no | nil | see examples. |
-| replace_record_key | bool | no | no | false | see examples. |
-| replace_record_key_regexp{1-10} | string | no | no | nil | see examples. |
-| convert_hash_to_json | bool | no | no | false | If true, converts Hash value of record to JSON String. |
-| insert_id_field | string | no | no | nil | Use key as `insert_id` of Streaming Insert API parameter. |
-| add_insert_timestamp | string | no | no | nil | Adds a timestamp column just before sending the rows to BigQuery, so that buffering time is not taken into account. Gives a field in BigQuery which represents the insert time of the row. |
-| allow_retry_insert_errors | bool | no | false | Retry to insert rows when an insertErrors occurs. There is a possibility that rows are inserted in duplicate. |
-| request_timeout_sec | integer | no | no | nil | Bigquery API response timeout |
-| request_open_timeout_sec | integer | no | no | 60 | Bigquery API connection, and request timeout. If you send big data to Bigquery, set large value. |
-| time_partitioning_type | enum | no (either day) | no | nil | Type of bigquery time partitioning feature(experimental feature on BigQuery). |
-| time_partitioning_expiration | time | no | no | nil | Expiration milliseconds for bigquery time partitioning. (experimental feature on BigQuery) |
+| name | type | required? | placeholder? | default | description |
+| :------------------------------------- | :------------ | :----------- | :---------- | :------------------------- | :----------------------- |
+| method | string | no | no | insert | `insert` (Streaming Insert) or `load` (load job) |
+| auth_method | enum | yes | no | private_key | `private_key` or `json_key` or `compute_engine` or `application_default` |
+| email | string | yes (private_key) | no | nil | GCP Service Account Email |
+| private_key_path | string | yes (private_key) | no | nil | GCP Private Key file path |
+| private_key_passphrase | string | yes (private_key) | no | nil | GCP Private Key Passphrase |
+| json_key | string | yes (json_key) | no | nil | GCP JSON Key file path or JSON Key string |
+| project | string | yes | yes | nil | |
+| dataset | string | yes | yes | nil | |
+| table | string | yes (either `tables`) | yes | nil | |
+| tables | array(string) | yes (either `table`) | yes | nil | can set multi table names splitted by `,` |
+| template_suffix | string | no | yes | nil | can use `%{time_slice}` placeholder replaced by `time_slice_format` |
+| auto_create_table | bool | no | no | false | If true, creates table automatically |
+| skip_invalid_rows | bool | no | no | false | Only `insert` method. |
+| max_bad_records | integer | no | no | 0 | Only `load` method. If the number of bad records exceeds this value, an invalid error is returned in the job result. |
+| ignore_unknown_values | bool | no | no | false | Accept rows that contain values that do not match the schema. The unknown values are ignored. |
+| schema | array | yes (either `fetch_schema` or `schema_path`) | no | nil | Schema Definition. It is formatted by JSON. |
+| schema_path | string | yes (either `fetch_schema`) | no | nil | Schema Definition file path. It is formatted by JSON. |
+| fetch_schema | bool | yes (either `schema_path`) | no | false | If true, fetch table schema definition from Bigquery table automatically. |
+| fetch_schema_table | string | no | yes | nil | If set, fetch table schema definition from this table, If fetch_schema is false, this param is ignored |
+| schema_cache_expire | integer | no | no | 600 | Value is second. If current time is after expiration interval, re-fetch table schema definition. |
+| insert_id_field | string | no | no | nil | Use key as `insert_id` of Streaming Insert API parameter. |
+| add_insert_timestamp | string | no | no | nil | Adds a timestamp column just before sending the rows to BigQuery, so that buffering time is not taken into account. Gives a field in BigQuery which represents the insert time of the row. |
+| allow_retry_insert_errors | bool | no | no | false | Retry to insert rows when an insertErrors occurs. There is a possibility that rows are inserted in duplicate. |
+| request_timeout_sec | integer | no | no | nil | Bigquery API response timeout |
+| request_open_timeout_sec | integer | no | no | 60 | Bigquery API connection, and request timeout. If you send big data to Bigquery, set large value. |
+| time_partitioning_type | enum | no (either day) | no | nil | Type of bigquery time partitioning feature(experimental feature on BigQuery). |
+| time_partitioning_expiration | time | no | no | nil | Expiration milliseconds for bigquery time partitioning. (experimental feature on BigQuery) |
+### Deprecated
+
+| name | type | required? | placeholder? | default | description |
+| :------------------------------------- | :------------ | :----------- | :---------- | :------------------------- | :----------------------- |
+| replace_record_key | bool | no | no | false | Use other filter plugin. |
+| replace_record_key_regexp{1-10} | string | no | no | nil | |
+
### Buffer section
| name | type | required? | default | description |
| :------------------------------------- | :------------ | :----------- | :------------------------- | :----------------------- |
| @type | string | no | memory (insert) or file (load) | |
@@ -111,9 +110,30 @@
| localtime | bool | no | true | |
| utc | bool | no | false | |
| timezone | string | no | nil | |
see. https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin_helper/inject.rb
+
+### Formatter section
+
+This section is for `load` mode only.
+If you use `insert` mode, used formatter is `json` only.
+
+Bigquery supports `csv`, `json` and `avro` format. Default is `json`
+I recommend to use `json` for now.
+
+For example.
+
+```
+source_format csv
+
+<format>
+ @type csv
+ fields col1, col2, col3
+</format>
+```
+
+see. https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin_helper/formatter.rb
## Examples
### Streaming inserts