# Logstash Output Charrington

Logstash Output Plugin to handle batching messages and dynamically create and alter tables.

Forked From [logstash-output-jdbc](https://github.com/theangryangel/logstash-output-jdbc)
## Configure
Example Logstash Config
```bash
input {
  kafka {
    auto_offset_reset => earliest
    bootstrap_servers => "${LS_KAFKA_BOOTSTRAP_SERVERS:kafka:9092}"
    codec => json
    group_id => "winston_ls_pg_sinks_cg"
    topics => [
      "orwell.analytics.v1.json"
    ]
  }
}

filter {
  if "_jsonparsefailure" in [tags] {
    drop { }
  }

  ruby {
    code => "
      hash = event.to_hash
      hash.each do |k,v|
        if v == nil
          event.remove(k)
        end
      end
    "
  }
}

output {
  charrington {
    driver_jar_path => "${DRIVER_PATH:/usr/share/logstash/vendor/jar/jdbc/postgresql-42.2.5.jar}"
    connection_string => "jdbc:postgresql://${WINSTON_DATABASE_HOST}:5432/${WINSTON_DATABASE}?user=${WINSTON_DATABASE_USERNAME}&password=${WINSTON_DATABASE_PASSWORD}"
    schema => "dea"
  }

  if "${LS_SHOW_DEBUG_OUTPUT}" == "true" {
    stdout { codec => rubydebug }
  }
}
```

## Build & Publish

**TLDR**
```bash
bundle exec rake build                          # build gem
gem push logstash-output-charrington-x.x.x.gem  # publish
```

Everything in the [vendor](./vendor) directory (including `test-jars` if it exists) will be packaged with the generated gem.
Avoid including unnecessary dependencies, by removing the vendor directory and regenerating it with the rake task. 
See the first command below: 
 
```bash
rm -r vendor && bundle exec rake install_jars       # clean vendor directory and re-install runtime jar dependencies
gem build logstash-output-charrington.gemspec       # build
gem push logstash-output-charrington-x.x.x.gem      # publish
gem owner --add <email> logstash-output-charrington # add another authorized publisher
```

## Testing
**NOTE:** Downloading test_jars requires maven and Java. You can set them up with asdf: [asdf-java](https://github.com/skotchpine/asdf-java), [asdf-maven](https://github.com/skotchpine/asdf-maven)
* There is currently an [open bug](https://github.com/skotchpine/asdf-maven/pull/17) with how `asdf-maven` sets `$JAVA_HOME` on a mac. Edit your `~/.asdf/plugins/maven/bin/exec-env` file to fix the bug 

```bash
bundle exec rake install_test_jars      # install test runtime jar dependencies
bundle exec rspec
```

#### Dangling Embedded Postgres
**TLDR**
```bash
lsof -PiTCP -sTCP:LISTEN | grep postgres | awk '{print $2}' | xargs kill
```
Sometimes the embedded postgres JAR can leave dangling open processes. This is obvious when you have `export TEST_DEBUG=true`
and the test output contains a stacktrace that starts with `[2019-06-21T15:18:06,734][ERROR][ru.yandex.qatools.embed.postgresql.PostgresProcess] Failed to read PID file (File '/var/folders/...`.
To resolve this issue, check for dangling processes using `lsof -PiTCP -sTCP:LISTEN | grep postgres`, then you can kill the process ids

#### Tracks Table
**Create Statment for Driver is Postgres and Transform is Redshift**
 * See `@@redshift_tracks_columns` in [insert.rb](./lib/logstash/outputs/charrington/insert.rb) for column list
```sql
create table <schema>.tracks
(
	id varchar(512) not null constraint tracks_pkey primary key,
	anonymous_id varchar(512),
	app_name varchar(512),
	context_campaign_content varchar(512)
	context_campaign_medium varchar(512),
	context_campaign_name varchar(512),
	context_campaign_source varchar(512),
	context_ip varchar(512),
	context_library_name varchar(512),
	context_library_version varchar(512),
	context_page_path varchar(512),
	context_page_referrer varchar(512),
	context_page_search varchar(512),
	context_page_title varchar(512),
	context_page_url varchar(512),
	context_user_agent varchar(512),
	event varchar(512),
	event_text varchar(512),
	original_timestamp timestamp,
	received_at timestamp,
	segment_dedupe_id varchar(512),
	sent_at timestamp,
	timestamp timestamp,
	user_id varchar(512),
	user_uid varchar(512),
	uuid bigint,
	uuid_ts timestamp default ('now'::text)::timestamp without time zone,
)

```

#### Resources
* [logstash-devutils](https://rubygems.org/gems/logstash-devutils) gem with logstash helper methods and tools
    * [spec_helper.rb source](https://github.com/elastic/logstash-devutils/blob/master/lib/logstash/devutils/rspec/spec_helper.rb)  
    * [logstash_helpers.rb source](https://github.com/elastic/logstash-devutils/blob/master/lib/logstash/devutils/rspec/logstash_helpers.rb)
* [logstash-output-file spec test](https://github.com/logstash-plugins/logstash-output-file/blob/master/spec/outputs/file_spec.rb)