README.md in embulk-input-gcs-0.2.5 vs README.md in embulk-input-gcs-0.2.6

- old
+ new

@@ -38,10 +38,11 @@ ## Configuration - **bucket** Google Cloud Storage bucket name (string, required) - **path_prefix** prefix of target keys (string, either of "path_prefix" or "paths" is required) - **paths** list of target keys (array of string, either of "path_prefix" or "paths" is required) +* **path_match_pattern**: regexp to match file paths. If a file path doesn't match with this pattern, the file will be skipped (regexp string, optional) - **incremental**: enables incremental loading(boolean, optional. default: true. If incremental loading is enabled, config diff for the next execution will include `last_path` parameter so that next execution skips files before the path. Otherwise, `last_path` will not be included. - **auth_method** (string, optional, "private_key", "json_key" or "compute_engine". default value is "private_key") - **service_account_email** Google Cloud Storage service_account_email (string, required when auth_method is private_key) - **p12_keyfile** fullpath of p12 key (string, required when auth_method is private_key) - **json_keyfile** fullpath of json_key (string, required when auth_method is json_key) @@ -85,9 +86,24 @@ - {name: account, type: long} - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'} - {name: purchase, type: timestamp, format: '%Y%m%d'} - {name: comment, type: string} out: {type: stdout} +``` + +To skip files using regexp: + +```yaml +in: + type: gcs + bucket: my-gcs-bucket + path_prefix: logs/csv- + # ... + path_match_pattern: \.csv$ # a file will be skipped if its path doesn't match with this pattern + ## some examples of regexp: + #path_match_pattern: /archive/ # match files in .../archive/... directory + #path_match_pattern: /data1/|/data2/ # match files in .../data1/... or .../data2/... directory + #path_match_pattern: .csv$|.csv.gz$ # match files whose suffix is .csv or .csv.gz ``` ## Authentication There are three methods supported to fetch access token for the service account.