README.md in fluent-plugin-bigquery-2.1.0 vs README.md in fluent-plugin-bigquery-2.2.0
- old
+ new
@@ -42,10 +42,11 @@
| auth_method | enum | yes | no | private_key | `private_key` or `json_key` or `compute_engine` or `application_default` |
| email | string | yes (private_key) | no | nil | GCP Service Account Email |
| private_key_path | string | yes (private_key) | no | nil | GCP Private Key file path |
| private_key_passphrase | string | yes (private_key) | no | nil | GCP Private Key Passphrase |
| json_key | string | yes (json_key) | no | nil | GCP JSON Key file path or JSON Key string |
+| location | string | no | no | nil | BigQuery Data Location. The geographic location of the job. Required except for US and EU. |
| project | string | yes | yes | nil | |
| dataset | string | yes | yes | nil | |
| table | string | yes (either `tables`) | yes | nil | |
| tables | array(string) | yes (either `table`) | yes | nil | can set multi table names splitted by `,` |
| auto_create_table | bool | no | no | false | If true, creates table automatically |
@@ -55,14 +56,14 @@
| fetch_schema | bool | yes (either `schema_path`) | no | false | If true, fetch table schema definition from Bigquery table automatically. |
| fetch_schema_table | string | no | yes | nil | If set, fetch table schema definition from this table, If fetch_schema is false, this param is ignored |
| schema_cache_expire | integer | no | no | 600 | Value is second. If current time is after expiration interval, re-fetch table schema definition. |
| request_timeout_sec | integer | no | no | nil | Bigquery API response timeout |
| request_open_timeout_sec | integer | no | no | 60 | Bigquery API connection, and request timeout. If you send big data to Bigquery, set large value. |
-| time_partitioning_type | enum | no (either day) | no | nil | Type of bigquery time partitioning feature(experimental feature on BigQuery). |
-| time_partitioning_field | string | no | no | nil | Field used to determine how to create a time-based partition(experimental feature on BigQuery). |
-| time_partitioning_expiration | time | no | no | nil | Expiration milliseconds for bigquery time partitioning. (experimental feature on BigQuery) |
-| time_partitioning_require_partition_filter | bool | no | no | false | If true, queries over this table require a partition filter that can be used for partition elimination to be specified. (experimental feature on BigQuery) |
+| time_partitioning_type | enum | no (either day) | no | nil | Type of bigquery time partitioning feature. |
+| time_partitioning_field | string | no | no | nil | Field used to determine how to create a time-based partition. |
+| time_partitioning_expiration | time | no | no | nil | Expiration milliseconds for bigquery time partitioning. |
+| clustering_fields | array(string) | no | no | nil | One or more fields on which data should be clustered. The order of the specified columns determines the sort order of the data. |
#### bigquery_insert
| name | type | required? | placeholder? | default | description |
| :------------------------------------- | :------------ | :----------- | :---------- | :------------------------- | :----------------------- |
@@ -431,11 +432,11 @@
Use placeholder.
```apache
<match dummy>
- @type bigquery_insert
+ @type bigquery_load
...
table accesslog$%Y%m%d
<buffer time>
@@ -444,10 +445,12 @@
...
</match>
```
But, Dynamic table creating doesn't support date partitioned table yet.
+And streaming insert is not allowed to insert with `$%Y%m%d` suffix.
+If you use date partitioned table with streaming insert, Please omit `$%Y%m%d` suffix from `table`.
### Dynamic table creating
When `auto_create_table` is set to `true`, try to create the table using BigQuery API when insertion failed with code=404 "Not Found: Table ...".
Next retry of insertion is expected to be success.
@@ -464,9 +467,11 @@
table accesslog_%Y_%m
...
</match>
```
+
+Also, you can create clustered table by using `clustering_fields`.
### Table schema
There are three methods to describe the schema of the target table.