README.md in embulk-output-bigquery-0.4.2 vs README.md in embulk-output-bigquery-0.4.3

- old
+ new

@@ -100,12 +100,12 @@ | encoding | string | optional | "UTF-8" | `UTF-8` or `ISO-8859-1` | | ignore_unknown_values | boolean | optional | false | | | allow_quoted_newlines | boolean | optional | false | Set true, if data contains newline characters. It may cause slow procsssing | | time_partitioning | hash | optional | `{"type":"DAY"}` if `table` parameter has a partition decorator, otherwise nil | See [Time Partitioning](#time-partitioning) | | time_partitioning.type | string | required | nil | The only type supported is DAY, which will generate one partition per day based on data loading time. | -| time_partitioning.expiration__ms | int | optional | nil | Number of milliseconds for which to keep the storage for a partition. partition | -| schema_update_options | array | optional | nil | List of `ALLOW_FIELD_ADDITION` or `ALLOW_FIELD_RELAXATION` or both. See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) | +| time_partitioning.expiration_ms | int | optional | nil | Number of milliseconds for which to keep the storage for a partition. partition | +| schema_update_options | array | optional | nil | (Experimental) List of `ALLOW_FIELD_ADDITION` or `ALLOW_FIELD_RELAXATION` or both. See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions). NOTE for the current status: `schema_update_options` does not work for `copy` job, that is, is not effective for most of modes such as `append`, `append_direct`, `replace`, `replace_backup` (except `delete_in_advance`) | ### Example ```yaml out: @@ -125,28 +125,29 @@ 5 modes are provided. ##### append -1. Load to temporary table. +1. Load to temporary table (Create and WRITE_APPEND in parallel) 2. Copy temporary table to destination table (or partition). (WRITE_APPEND) ##### append_direct -Insert data into existing table (or partition) directly. +1. Insert data into existing table (or partition) directly. (WRITE_APPEND in parallel) + This is not transactional, i.e., if fails, the target table could have some rows inserted. ##### replace -1. Load to temporary table. +1. Load to temporary table (Create and WRITE_APPEND in parallel) 2. Copy temporary table to destination table (or partition). (WRITE_TRUNCATE) ```is_skip_job_result_check``` must be false when replace mode ##### replace_backup -1. Load to temporary table. +1. Load to temporary table (Create and WRITE_APPEND in parallel) 2. Copy destination table (or partition) to backup table (or partition). (dataset_old, table_old) 3. Copy temporary table to destination table (or partition). (WRITE_TRUNCATE) ```is_skip_job_result_check``` must be false when replace_backup mode. @@ -314,11 +315,11 @@ ``` filters: - type: to_json column: {name: payload, type: string} - default_format: %Y-%m-%d %H:%M:%S.%6N + default_format: "%Y-%m-%d %H:%M:%S.%6N" out: type: bigquery payload_column_index: 0 # or, payload_column: payload ``` @@ -395,27 +396,15 @@ time_partitioning: type: DAY expiration_ms: 259200000 ``` -Use `schema_update_options` to allow the schema of the desitination table to be updated as a side effect of the load job as: +Use [Tables: patch](https://cloud.google.com/bigquery/docs/reference/v2/tables/patch) API to update the schema of the partitioned table, embulk-output-bigquery itself does not support it, though. +Note that only adding a new column, and relaxing non-necessary columns to be `NULLABLE` are supported now. Deleting columns, and renaming columns are not supported. -```yaml -out: - type: bigquery - table: table_name$20160929 - auto_create_table: true - time_partitioning: - type: DAY - expiration_ms: 259200000 - schema_update_options: - - ALLOW_FIELD_ADDITION - - ALLOW_FIELD_RELAXATION -``` - -It seems that only adding a new column, and relaxing non-necessary columns to be `NULLABLE` are supported now. -Deleting columns, and renaming columns are not supported. -See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) for details. +MEMO: [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) is available +to update the schema of the desitination table as a side effect of the load job, but it is not available for copy job. +Thus, it was not suitable for embulk-output-bigquery idempotence modes, `append`, `replace`, and `replace_backup`, sigh. ## Development ### Run example: