README.md in embulk-output-bigquery-0.4.2 vs README.md in embulk-output-bigquery-0.4.3
- old
+ new
@@ -100,12 +100,12 @@
| encoding | string | optional | "UTF-8" | `UTF-8` or `ISO-8859-1` |
| ignore_unknown_values | boolean | optional | false | |
| allow_quoted_newlines | boolean | optional | false | Set true, if data contains newline characters. It may cause slow procsssing |
| time_partitioning | hash | optional | `{"type":"DAY"}` if `table` parameter has a partition decorator, otherwise nil | See [Time Partitioning](#time-partitioning) |
| time_partitioning.type | string | required | nil | The only type supported is DAY, which will generate one partition per day based on data loading time. |
-| time_partitioning.expiration__ms | int | optional | nil | Number of milliseconds for which to keep the storage for a partition. partition |
-| schema_update_options | array | optional | nil | List of `ALLOW_FIELD_ADDITION` or `ALLOW_FIELD_RELAXATION` or both. See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) |
+| time_partitioning.expiration_ms | int | optional | nil | Number of milliseconds for which to keep the storage for a partition. partition |
+| schema_update_options | array | optional | nil | (Experimental) List of `ALLOW_FIELD_ADDITION` or `ALLOW_FIELD_RELAXATION` or both. See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions). NOTE for the current status: `schema_update_options` does not work for `copy` job, that is, is not effective for most of modes such as `append`, `append_direct`, `replace`, `replace_backup` (except `delete_in_advance`) |
### Example
```yaml
out:
@@ -125,28 +125,29 @@
5 modes are provided.
##### append
-1. Load to temporary table.
+1. Load to temporary table (Create and WRITE_APPEND in parallel)
2. Copy temporary table to destination table (or partition). (WRITE_APPEND)
##### append_direct
-Insert data into existing table (or partition) directly.
+1. Insert data into existing table (or partition) directly. (WRITE_APPEND in parallel)
+
This is not transactional, i.e., if fails, the target table could have some rows inserted.
##### replace
-1. Load to temporary table.
+1. Load to temporary table (Create and WRITE_APPEND in parallel)
2. Copy temporary table to destination table (or partition). (WRITE_TRUNCATE)
```is_skip_job_result_check``` must be false when replace mode
##### replace_backup
-1. Load to temporary table.
+1. Load to temporary table (Create and WRITE_APPEND in parallel)
2. Copy destination table (or partition) to backup table (or partition). (dataset_old, table_old)
3. Copy temporary table to destination table (or partition). (WRITE_TRUNCATE)
```is_skip_job_result_check``` must be false when replace_backup mode.
@@ -314,11 +315,11 @@
```
filters:
- type: to_json
column: {name: payload, type: string}
- default_format: %Y-%m-%d %H:%M:%S.%6N
+ default_format: "%Y-%m-%d %H:%M:%S.%6N"
out:
type: bigquery
payload_column_index: 0 # or, payload_column: payload
```
@@ -395,27 +396,15 @@
time_partitioning:
type: DAY
expiration_ms: 259200000
```
-Use `schema_update_options` to allow the schema of the desitination table to be updated as a side effect of the load job as:
+Use [Tables: patch](https://cloud.google.com/bigquery/docs/reference/v2/tables/patch) API to update the schema of the partitioned table, embulk-output-bigquery itself does not support it, though.
+Note that only adding a new column, and relaxing non-necessary columns to be `NULLABLE` are supported now. Deleting columns, and renaming columns are not supported.
-```yaml
-out:
- type: bigquery
- table: table_name$20160929
- auto_create_table: true
- time_partitioning:
- type: DAY
- expiration_ms: 259200000
- schema_update_options:
- - ALLOW_FIELD_ADDITION
- - ALLOW_FIELD_RELAXATION
-```
-
-It seems that only adding a new column, and relaxing non-necessary columns to be `NULLABLE` are supported now.
-Deleting columns, and renaming columns are not supported.
-See [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) for details.
+MEMO: [jobs#configuration.load.schemaUpdateOptions](https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.schemaUpdateOptions) is available
+to update the schema of the desitination table as a side effect of the load job, but it is not available for copy job.
+Thus, it was not suitable for embulk-output-bigquery idempotence modes, `append`, `replace`, and `replace_backup`, sigh.
## Development
### Run example: