README.rdoc in fluent-plugin-s3-0.6.0.pre1 vs README.rdoc in fluent-plugin-s3-0.6.0

- old
+ new

@@ -19,11 +19,11 @@ <match pattern> type s3 aws_key_id YOUR_AWS_KEY_ID - aws_sec_key YOUR_AWS_SECRET/KEY + aws_sec_key YOUR_AWS_SECRET_KEY s3_bucket YOUR_S3_BUCKET_NAME s3_region ap-northeast-1 s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension} path logs/ buffer_path /var/log/fluent/s3 @@ -50,18 +50,20 @@ - %{path} - %{time_slice} - %{index} - %{file_extension} - %{uuid_flush} +- %{hex_random} to decide keys dynamically. %{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above. %{time_slice} is the time-slice in text that are formatted with *time_slice_format*. %{index} is the sequential number starts from 0, increments when multiple files are uploaded to S3 in the same time slice. %{file_extention} is always "gz" for now. -%{uuid_flush} a uuid that is replaced everytime the buffer will be flushed +%{uuid_flush} a uuid that is replaced for each buffer chunk to be flushed +%{hex_random} a random hex string that is replaced for each buffer chunk, not assured to be unique. This is used to follow a way of peformance tuning, `Add a Hex Hash Prefix to Key Name`, written in [Request Rate and Performance Considerations - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html). You can configure the length of string with a `hex_random_length` parameter (Default: 4). The default format is "%{path}%{time_slice}_%{index}.%{file_extension}". For instance, using the example configuration above, actual object keys on S3 will be something like: @@ -85,10 +87,12 @@ The {fluent-mixin-config-placeholders}[https://github.com/tagomoris/fluent-mixin-config-placeholders] mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the s3_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers. s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension} +[force_path_style] :force_path_style (Boolean) — default: false — When set to true, the bucket name is always left in the request URI and never moved to the host as a sub-domain. See Plugins::S3BucketDns for more details. + [store_as] archive format on S3. You can use serveral format: - gzip (default) - json - text @@ -170,9 +174,92 @@ - bucket_owner_read - bucket_owner_full_control To use cross-account access, you will need to create a bucket policy granting the specific access required. Refer to the {AWS documentation}[http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example3.html] for examples. + +[hex_random_length] The length of `%{hex_random}` placeholder. Default is 4 as written in [Request Rate and Performance Considerations - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html). + +[overwrite] Overwrite already existing path. Default is false, which raises an error if a s3 object of the same path already exists, or increment the `%{index}` placeholder until finding an absent path. + +=== assume_role_credentials + +Typically, you use AssumeRole for cross-account access or federation. + + <match *> + type s3 + + <assume_role_credentials> + role_arn ROLE_ARN + role_session_name ROLE_SESSION_NAME + </assume_role_credentials> + </match> + +See also: + +- {Using IAM Roles - AWS Identity and Access Management}[http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html] +- {Aws::STS::Client}[http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html] +- {Aws::AssumeRoleCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html] + +[role_arn (required)] The Amazon Resource Name (ARN) of the role to assume. + +[role_session_name (required)] An identifier for the assumed role session. + +[policy] An IAM policy in JSON format. + +[duration_seconds] The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value is set to 3600 seconds. + +[external_id] A unique identifier that is used by third parties when assuming roles in their customers' accounts. + +=== instance_profile_credentials + +Retrieve temporary security credentials via HTTP request. This is useful on EC2 instance. + + <match *> + type s3 + + <instance_profile_credentials> + ip_address IP_ADDRESS + port PORT + </instance_profile_credentials> + </match> + +See also: + +- {Aws::InstanceProfileCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html] +- {Temporary Security Credentials - AWS Identity and Access Management}[http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html] +- {Instance Metadata and User Data - Amazon Elastic Compute Cloud}[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html] + +[retries] Number of times to retry when retrieving credentials. Default is 5. + +[ip_address] Default is 169.254.169.254. + +[port] Default is 80. + +[http_open_timeout] Default is 5. + +[http_read_timeout] Default is 5. + +=== shared_credentials + +This loads AWS access credentials from local ini file. This is useful for local developing. + + <match *> + type s3 + + <shared_credentials> + path PATH + profile_name PROFILE_NAME + </shared_credentials> + </match> + +See also: + +- {Aws::SharedCredentials}[http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html] + +[path] Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials". + +[profile_name] Defaults to 'default' or `ENV['AWS_PROFILE']`. == IAM Policy The following is an example for a minimal IAM policy needed to write to an s3 bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).