README.md in embulk-input-gcs-0.2.5 vs README.md in embulk-input-gcs-0.2.6
- old
+ new
@@ -38,10 +38,11 @@
## Configuration
- **bucket** Google Cloud Storage bucket name (string, required)
- **path_prefix** prefix of target keys (string, either of "path_prefix" or "paths" is required)
- **paths** list of target keys (array of string, either of "path_prefix" or "paths" is required)
+* **path_match_pattern**: regexp to match file paths. If a file path doesn't match with this pattern, the file will be skipped (regexp string, optional)
- **incremental**: enables incremental loading(boolean, optional. default: true. If incremental loading is enabled, config diff for the next execution will include `last_path` parameter so that next execution skips files before the path. Otherwise, `last_path` will not be included.
- **auth_method** (string, optional, "private_key", "json_key" or "compute_engine". default value is "private_key")
- **service_account_email** Google Cloud Storage service_account_email (string, required when auth_method is private_key)
- **p12_keyfile** fullpath of p12 key (string, required when auth_method is private_key)
- **json_keyfile** fullpath of json_key (string, required when auth_method is json_key)
@@ -85,9 +86,24 @@
- {name: account, type: long}
- {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
- {name: purchase, type: timestamp, format: '%Y%m%d'}
- {name: comment, type: string}
out: {type: stdout}
+```
+
+To skip files using regexp:
+
+```yaml
+in:
+ type: gcs
+ bucket: my-gcs-bucket
+ path_prefix: logs/csv-
+ # ...
+ path_match_pattern: \.csv$ # a file will be skipped if its path doesn't match with this pattern
+ ## some examples of regexp:
+ #path_match_pattern: /archive/ # match files in .../archive/... directory
+ #path_match_pattern: /data1/|/data2/ # match files in .../data1/... or .../data2/... directory
+ #path_match_pattern: .csv$|.csv.gz$ # match files whose suffix is .csv or .csv.gz
```
## Authentication
There are three methods supported to fetch access token for the service account.