# remote_syslog Ruby daemon & sender

Lightweight Ruby daemon to tail one or more log files and transmit UDP syslog 
messages to a remote syslog host (centralized log aggregation).

remote_syslog generates UDP packets itself instead of depending on a system 
syslog daemon, so its configuration doesn't affect system-wide 
logging - syslog is just the transport.

Uses:

* collecting logs from servers & daemons which don't natively support syslog
* when reconfiguring the system logger is less convenient than a 
purpose-built daemon (e.g., automated app deployments)
* aggregating files not generated by daemons (e.g., package manager logs)

The library can also be used to generate one-off log messages from Ruby code.

Tested with the hosted log management service [Papertrail] and should work for 
transmitting to any syslog server.


## Installation

Install the gem, which includes a binary called "remote_syslog":

    $ [sudo] gem install remote_syslog

Optionally, create a log_files.yml with the log file paths to read and the 
host/port to log to (see examples/log_files.yml.example). These can also be 
specified as arguments to the remote_syslog daemon. More below.


## Usage

    $ remote_syslog -h
    Usage:   remote_syslog [options] <path to add'l log 1> .. <path to add'l log n>

    Example: remote_syslog -c configs/logs.yml -p 12345 /var/log/mysqld.log

    Options:
        -c, --configfile PATH            Path to config (/etc/log_files.yml)
        -d, --dest-host HOSTNAME         Destination syslog hostname or IP (logs.papertrailapp.com)
        -p, --dest-port PORT             Destination syslog port (514)
        -D, --no-detach                  Don't daemonize and detach from the terminal
        -f, --facility FACILITY          Facility (user)
            --hostname HOST              Local hostname to send from
        -P, --pid-dir DIRECTORY          Directory to write .pid file in (/var/run/)
            --pid-file FILENAME          PID filename (<program name>.pid)
            --parse-syslog               Parse file as syslog-formatted file
        -s, --severity SEVERITY          Severity (notice)
            --strip-color                Strip color codes
            --tls                        Connect via TCP with TLS
        -h, --help                       Show this message
    

## Example

Daemonize, collecting from files mentioned in `./config/logs.yml` as well as
`/var/log/mysqld.log`:

    $ remote_syslog -c configs/logs.yml -p 12345 /var/log/mysqld.log

Stay attached to the terminal, look for and use `/etc/log_files.yml` if it
exists, write PID to `/tmp/remote_syslog.pid`, and send with facility local0:

    $ remote_syslog -d a.server.com -f local0 -P /tmp /var/log/mysqld.log

remote_syslog will daemonize by default. A sample init file is in the gem as 
remote_syslog.init.d. You may be able to:

    $ cp examples/remote_syslog.init.d /etc/init.d/remote_syslog

## Sending messages securely ##

If the receiving system supports sending syslog over TCP with TLS, you can
pass the `--tls` option when running `remote_syslog`:

    $ remote_syslog --tls -p 1234 /var/log/mysqld.log


## Configuration

By default, the gem looks for a configuration in /etc/log_files.yml.

The gem comes with a sample config.  Optionally:

    $ cp examples/log_files.yml.example /etc/log_files.yml
    
log_files.yml has filenames to log from (as an array) and hostname and port 
to log to (as a hash). Wildcards are supported using * and standard shell 
globbing. Filenames given on the command line are additive to those in 
the config file.

Only 1 destination server is supported; the command-line argument wins. 

    files: [/var/log/httpd/access_log, /var/log/httpd/error_log, /var/log/mysqld.log, /var/run/mysqld/mysqld-slow.log]
    destination:
      host: logs.papertrailapp.com
      port: 12345


### Optional: Parse fields from messages written by syslogd

This is not needed for most configurations.

In cases where logs from multiple programs are in the same file (for example, 
``/var/log/messages``), the log line may include text that is not part of the 
log message, like a timestamp, hostname, or program name. remote_syslog can 
parse the program, hostname, and/or message text so that the message has 
accurate metadata.

To do that, add an optional top-level configuration option `parse_fields` 
with the name of a predefined regex (by remote_syslog) or a regex string. To 
use the predefined regex for standard syslog messages, provide:

    parse_fields: syslog

The `syslog` regex is `(\w+ \d+ \S+) (\S+) ([^:]+): (.*)`. It parses this:

    Jul 18 08:25:08 hostname programname[1234]: The log message

Or provide `parse_fields: rfc3339` to parse high-precision RFC 3339
timestamps like:
    2011-07-16T08:25:08.651413-07:00 hostname programname[1234]: The log message
    
Or provide your own regex that includes these 4 backreferences, in order: 
timestamp, system name, program name, message. Match and return empty 
strings for any empty positions where the log value should be ignored.
For example, in the log:

    something-meaningless The log message

You could ignore the first word, returning 3 empty values then the log 
message with:

    parse_fields: "something-meaningless ()()()(.*)"

Per-file parsing is not supported. Run multiple instances.


### Optional: Run multiple instances

Run multiple instances to support more than one message-specific file format 
(concurrently) or to specify distinct syslog hostnames. To do so, provide
an alternative PID filename as a command-line option to additional 
instance(s), such as:

    --pid-file remote_syslog_2.pid
    

## Reporting bugs

1. See whether the issue has already been reported: <https://github.com/papertrail/remote_syslog/issues/>
2. If you don't find one, create an issue with a repro case.


## Contributing

Once you've made your great commits:

1. [Fork][fk] remote_syslog
2. Create a topic branch - `git checkout -b my_branch`
3. Commit the changes without changing the Rakefile or other files unrelated to your enhancement.
4. Push to your branch - `git push origin my_branch`
5. Create a Pull Request or an [Issue][is] with a link to your branch
6. That's it!

[fk]: http://help.github.com/forking/
[is]: https://github.com/papertrail/remote_syslog/issues/
[Papertrail]: http://papertrailapp.com/