:plugin: s3 :type: input :default_codec: plain /////////////////////////////////////////// START - GENERATED VARIABLES, DO NOT EDIT! /////////////////////////////////////////// :version: %VERSION% :release_date: %RELEASE_DATE% :changelog_url: %CHANGELOG_URL% :include_path: ../../../../logstash/docs/include /////////////////////////////////////////// END - GENERATED VARIABLES, DO NOT EDIT! /////////////////////////////////////////// [id="plugins-{type}s-{plugin}"] === S3 input plugin include::{include_path}/plugin_header.asciidoc[] ==== Description Stream events from files from a S3 bucket. Each line from each file generates an event. Files ending in `.gz` are handled as gzip'ed files. Files that are archived to AWS Glacier will be skipped. [id="plugins-{type}s-{plugin}-options"] ==== S3 Input Configuration Options This plugin supports the following configuration options plus the <> described later. [cols="<,<,<",options="header",] |======================================================================= |Setting |Input type|Required | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|Yes | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No |======================================================================= Also see <> for a list of options supported by all input plugins.   [id="plugins-{type}s-{plugin}-access_key_id"] ===== `access_key_id` * Value type is <> * There is no default value for this setting. This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order: 1. Static configuration, using `access_key_id` and `secret_access_key` params in logstash plugin config 2. External credentials file specified by `aws_credentials_file` 3. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` 4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY` 5. IAM Instance Profile (available when running inside EC2) [id="plugins-{type}s-{plugin}-aws_credentials_file"] ===== `aws_credentials_file` * Value type is <> * There is no default value for this setting. Path to YAML file containing a hash of AWS credentials. This file will only be loaded if `access_key_id` and `secret_access_key` aren't set. The contents of the file should look like this: [source,ruby] ---------------------------------- :access_key_id: "12345" :secret_access_key: "54321" ---------------------------------- [id="plugins-{type}s-{plugin}-backup_add_prefix"] ===== `backup_add_prefix` * Value type is <> * Default value is `nil` Append a prefix to the key (full path including file name in s3) after processing. If backing up to another (or the same) bucket, this effectively lets you choose a new 'folder' to place the files in [id="plugins-{type}s-{plugin}-backup_to_bucket"] ===== `backup_to_bucket` * Value type is <> * Default value is `nil` Name of a S3 bucket to backup processed files to. [id="plugins-{type}s-{plugin}-backup_to_dir"] ===== `backup_to_dir` * Value type is <> * Default value is `nil` Path of a local directory to backup processed files to. [id="plugins-{type}s-{plugin}-bucket"] ===== `bucket` * This is a required setting. * Value type is <> * There is no default value for this setting. The name of the S3 bucket. [id="plugins-{type}s-{plugin}-delete"] ===== `delete` * Value type is <> * Default value is `false` Whether to delete processed files from the original bucket. [id="plugins-{type}s-{plugin}-endpoint"] ===== `endpoint` * Value type is <> * There is no default value for this setting. The endpoint to connect to. By default it is constructed using the value of `region`. This is useful when connecting to S3 compatible services, but beware that these aren't guaranteed to work correctly with the AWS SDK. [id="plugins-{type}s-{plugin}-exclude_pattern"] ===== `exclude_pattern` * Value type is <> * Default value is `nil` Ruby style regexp of keys to exclude from the bucket [id="plugins-{type}s-{plugin}-additional_settings"] ===== `additional_settings` * Value type is <> * Default value is `{}` Key-value pairs of settings and corresponding values used to parametrize the connection to s3. See full list in https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html[the AWS SDK documentation]. Example: [source,ruby] input { s3 { "access_key_id" => "1234" "secret_access_key" => "secret" "bucket" => "logstash-test" "additional_settings" => { "force_path_style" => true "follow_redirects" => false } } } [id="plugins-{type}s-{plugin}-include_object_properties"] ===== `include_object_properties` * Value type is <> * Default value is `false` Whether or not to include the S3 object's properties (last_modified, content_type, metadata) into each Event at `[@metadata][s3]`. Regardless of this setting, `[@metdata][s3][key]` will always be present. [id="plugins-{type}s-{plugin}-interval"] ===== `interval` * Value type is <> * Default value is `60` Interval to wait between to check the file list again after a run is finished. Value is in seconds. [id="plugins-{type}s-{plugin}-prefix"] ===== `prefix` * Value type is <> * Default value is `nil` If specified, the prefix of filenames in the bucket must match (not a regexp) [id="plugins-{type}s-{plugin}-proxy_uri"] ===== `proxy_uri` * Value type is <> * There is no default value for this setting. URI to proxy server if required [id="plugins-{type}s-{plugin}-region"] ===== `region` * Value type is <> * Default value is `"us-east-1"` The AWS Region [id="plugins-{type}s-{plugin}-role_arn"] ===== `role_arn` * Value type is <> * There is no default value for this setting. The AWS IAM Role to assume, if any. This is used to generate temporary credentials, typically for cross-account access. See the https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html[AssumeRole API documentation] for more information. [id="plugins-{type}s-{plugin}-role_session_name"] ===== `role_session_name` * Value type is <> * Default value is `"logstash"` Session name to use when assuming an IAM role. [id="plugins-{type}s-{plugin}-secret_access_key"] ===== `secret_access_key` * Value type is <> * There is no default value for this setting. The AWS Secret Access Key [id="plugins-{type}s-{plugin}-session_token"] ===== `session_token` * Value type is <> * There is no default value for this setting. The AWS Session token for temporary credential [id="plugins-{type}s-{plugin}-sincedb_path"] ===== `sincedb_path` * Value type is <> * Default value is `nil` Where to write the since database (keeps track of the date the last handled file was added to S3). The default will write sincedb files to in the directory '{path.data}/plugins/inputs/s3/' If specified, this setting must be a filename path and not just a directory. [id="plugins-{type}s-{plugin}-temporary_directory"] ===== `temporary_directory` * Value type is <> * Default value is `"/tmp/logstash"` Set the directory where logstash will store the tmp files before processing them. [id="plugins-{type}s-{plugin}-watch_for_new_files"] ===== `watch_for_new_files` * Value type is <> * Default value is `true` Whether or not to watch for new files. Disabling this option causes the input to close itself after processing the files from a single listing. [id="plugins-{type}s-{plugin}-common-options"] include::{include_path}/{type}.asciidoc[] :default_codec!: