README.rdoc in fluent-plugin-s3-0.2.6 vs README.rdoc in fluent-plugin-s3-0.3.0
- old
+ new
@@ -20,10 +20,11 @@
aws_key_id YOUR_AWS_KEY_ID
aws_sec_key YOUR_AWS_SECRET/KEY
s3_bucket YOUR_S3_BUCKET_NAME
s3_endpoint s3-ap-northeast-1.amazonaws.com
+ s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
path logs/
buffer_path /var/log/fluent/s3
time_slice_format %Y%m%d-%H
time_slice_wait 10m
@@ -36,20 +37,60 @@
[s3_bucket (required)] S3 bucket name.
[s3_endpoint] s3 endpoint name. Example, Tokyo region is "s3-ap-northeast-1.amazonaws.com".
+[s3_object_key_format] The format of S3 object keys. You can use several built-in variables:
+
+- %{path}
+- %{time_slice}
+- %{index}
+- %{file_extension}
+
+to decide keys dynamically.
+
+%{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above.
+%{time_slice} is the time-slice in text that are formatted with *time_slice_format*.
+%{index} is the sequential number starts from 0, increments when multiple files are uploaded to S3 in the same time slice.
+%{file_extention} is always "gz" for now.
+
+The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
+
+For instance, using the example configuration above, actual object keys on S3 will be something like:
+
+ "logs/20130111-22_0.gz"
+ "logs/20130111-23_0.gz"
+ "logs/20130111-23_1.gz"
+ "logs/20130112-00_0.gz"
+
+With the configuration:
+
+ s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
+ path log
+ time_slice_format %Y%m%d-%H
+
+You get:
+
+ "log/events/ts=20130111-22/events_0.gz"
+ "log/events/ts=20130111-23/events_0.gz"
+ "log/events/ts=20130111-23/events_1.gz"
+ "log/events/ts=20130112-00/events_0.gz"
+
+The {fluent-mixin-config-placeholders}[https://github.com/tagomoris/fluent-mixin-config-placeholders] mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the s3_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
+
+ s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
+
+[auto_create_bucket] Create S3 bucket if it does not exists. Default is true.
+
[path] path prefix of the files on S3. Default is "" (no prefix).
[buffer_path (required)] path prefix of the files to buffer logs.
[time_slice_format] Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
[time_slice_wait] The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
[utc] Use UTC instead of local time.
-
-The actual path on S3 will be: "{path}{time_slice_format}_{sequential_number}.gz"
== Copyright
Copyright:: Copyright (c) 2011 Sadayuki Furuhashi