# MuchKeys ─────────▄──────────────▄ ────────▌▒█───────────▄▀▒▌ ────────▌▒▒▀▄───────▄▀▒▒▒▐ ───────▐▄▀▒▒▀▀▀▀▄▄▄▀▒▒▒▒▒▐ ─────▄▄▀▒▒▒▒▒▒▒▒▒▒▒█▒▒▄█▒▐ ───▄▀▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▀██▀▒▌ ──▐▒▒▒▄▄▄▒▒▒▒▒▒▒▒▒▒▒▒▒▀▄▒▒▌ ──▌▒▒▐▄█▀▒▒▒▒▄▀█▄▒▒▒▒▒▒▒█▒▐ ─▐▒▒▒▒▒▒▒▒▒▒▒▌██▀▒▒▒▒▒▒▒▒▀▄▌ ─▌▒▀▄██▄▒▒▒▒▒▒▒▒▒▒▒░░░░▒▒▒▒▌ ─▌▀▐▄█▄█▌▄▒▀▒▒▒▒▒▒░░░░░░▒▒▒▐ ▐▒▀▐▀▐▀▒▒▄▄▒▄▒▒▒▒▒░░░░░░▒▒▒▒▌ ▐▒▒▒▀▀▄▄▒▒▒▄▒▒▒▒▒▒░░░░░░▒▒▒▐ ─▌▒▒▒▒▒▒▀▀▀▒▒▒▒▒▒▒▒░░░░▒▒▒▒▌ ─▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▐ ──▀▄▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▄▒▒▒▒▌ ────▀▄▒▒▒▒▒▒▒▒▒▒▄▄▄▀▒▒▒▒▄▀ ───▐▀▒▀▄▄▄▄▄▄▀▀▀▒▒▒▒▒▄▄▀ ──▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▀▀ MuchKeys lets you store your application keys in consul and then leverages many conventions to create a pleasant API. Keys in this context can mean application settings, knobs, dials, passwords, api keys or anything. It's primary use is in production where it will check to see if a ENV variable exists first and then search consul for a key based on an opinionated hierarchy convention. You will want to read below about the convention and assumptions first to see if this works for you. MuchKeys also has a way of using encrypted secrets stored in consul. MuchKeys has a command line interface to help you encrypt or read secrets stored in consul. ## Installation Add this line to your application's Gemfile: ```ruby gem 'muchkeys' ``` And then execute: ``` $ bundle ``` Or install it yourself as: ``` $ gem install muchkeys ``` ## Usage Use a snippet like this below in your YAML config files and anywhere else to use a central configuration store while retaining control of your local development environment. ``` <%= MUCHKEYS['widget_api_key'] %> ``` Now the `widget_api_key` is coming from a central location. Devs can override `widget_api_key` with an ENV setting. `export WIDGET_API_KEY="development_key76"` MuchKeys defers to ENV when it sees one set. MuchKeys will look in consul (a key/value store) for keys (settings/secrets/knobs to turn). It searches in an order of convention. It is also assuming you want to use git2consul to enable your developers to add their own keys. Git2consul syncs a git repo to consul. So let's walk through what happens when a developer is adding a new feature to an app. The developer is integrating an API from reddit into an app called `feed_reader`. So now they have a reddit API key and this is something new to the app. * The developer adds twitter_api_key to `.env` from the dotenv gem to control their own development environment. Staging and production have no idea this has happened. * The developer adds/commits a file to `/feed_reader/twitter_api_key` * git2consul is sync'd to consul and their new key is there. (This might happen through cron or some other way) * They use their key in a configuration file or in the app: `<%= MUCHKEYS['redis_api_key'] %>` * When they deploy to staging/production the key is there and the `feed_reader` app doesn't explode. * Ops wasn't involved. Developers are empowered. MuchKeys does this magic through a search order. It searches consul for the key in certain order: (notice that the key isn't prefixed when you look up `twitter_api_key` with `MUCHKEYS['twitter_api_key']`) It will search a consul hierarchy looking for `twitter_api_key` in this search order: ``` git/feed_reader/secrets git/feed_reader/config git/shared/secrets git/shared/config ``` The above `feed_reader` is detected from `Rails.application`. If you don't have a Rails app, then you can set the application name in the configure block: ``` MuchKeys.configure do |config| config.application_name = "rack_app" end ``` The order and paths that it searches is configurable: ``` MuchKeys.configure do |config| config.search_paths = %W( app_name/keys app_name/secrets shared/keys shared/secrets ) end ``` This would look for keys and settings in consul under these paths instead: ``` app_name/keys app_name/secrets shared/keys shared/secrets ``` Anything that has /secrets/ in it is going to be assumed to be a secret. There is an assumption that you have your keys and secrets organized like this (one deep nesting). This is configurable. ``` # if you don't put your secrets in something like shared/secrets/ # but shared/passwords/ MuchKeys.configure do |config| config.secrets_path_hint = "passwords/" end ``` ### Using in a Rails App Add to your Gemfile. ``` gem 'muchkeys' ``` Run `bundle install`. If you don't need to change your hostname (production you do not need to) or any other settings, then you are done. If you need to override defaults, create an initializer. ``` # config/initializers/muchkeys.rb MuchKeys.configure do |config| # your settings if desired end ``` Then, use in YAML files, in the app or in other initializers. ``` # example of centralizing a database password in database.yml # blog/config/database.yml production: <<: *default username: MUCHKEYS['database_user'] # set in consul at git/blog/config/database_user password: MUCHKEYS['database_password'] # set in consul at git/blog/secrets/database_password ``` In the above example, `database_password` needs to be encrypted with an SSL certificate and then set in consul (paste the PEM text). The private and public key are needed to decrypt and these keys should be stored in `/.keys/blog.pem`. With these conventions set once, you won't have to do anything else. ### Encrypted Secrets Generate an SSL cert or use an existing one. You will need the private and public key to encrypt and the public key to decrypt. ```bash openssl req -new -newkey rsa:4096 -nodes -x509 -keyout /tmp/self_signed_test.pem -out /tmp/self_signed_test.pem ``` ```ruby MuchKeys.configure do |config| config.public_key = "/tmp/self_signed_test.pem" config.private_key = "/tmp/self_signed_test.pem" end puts MuchKeys::Secret.encrypt_string("bacon is just ok").to_s # => -----BEGIN PKCS7----- # => MIICuwYJKoZIhvcNAQcDoIICrDCCAqgCAQAxggJuMIICagIBADBSMEUxCzAJ ... # => ... # Copy and paste this into consul under a key: ie: secrets/fake # Now you can fetch the secret with the same public key. MuchKeys::Secret.get_consul_secret('secrets/fake') # => bacon is just ok ``` Inside your app: ```ruby # inside a rails initializer or other setup file you have MuchKeys.configure do |config| config.consul_url = "http://myrealhost" # Default is "http://localhost:8500" config.public_key = "path to public .pem" # this is only required if you are encrypting secrets config.private_key = "path to private .pem" # this is only required if you are encrypting secrets end # then inside your YAML or app: MuchKeys.fetch_key('number_of_threads') # goes to git/config/myapp/number_of_threads in consul ``` ## Automagic Certificates MuchKeys can find certs automatically in `~/.keys`. `MuchKeys.fetch_key("git/waffles/secrets/a_password")` will try to decrypt `a_password` with a cert called `~/.keys/waffles.pem`. The private key can be in the same file but should be protected with file permissions `0600`. ## CLI Example of decrypted an encrypted key: ``` $ muchkeys -d --public_key=~/.keys/staging.pem --private_key=~/.keys/staging.pem --consul_url=http://consul.domain:8500 --consul_key git/pants/secrets/some_password ``` ### Other Usage You can fetch individual keys if you want. ```bash ruby -e 'require "muchkeys"; puts MuchKeys.fetch_key("mail_server")' # => smtp.example.com (from consul) ``` ```bash mail_server=muffin.ninja.local ruby -e 'require "muchkeys"; puts MuchKeys.fetch_key("mail_server")' # => muffin.ninja.local (from ENV) ``` ### Keys to Store You will probably just want to store things that change between environments in consul. If `database_pool_size` never changes between development and production then don't store it in consul. If `sidekiq_number_of_workers` is 1 in development, 2 in staging and 16 in production, then this is a good thing to store in consul because you'll have a central place that all app servers can read from. You could store things in rails' `environments/` path but then you'll need to do a deploy to change a setting and some things might be shared between apps like URLs, api keys, secrets and scaling settings. ### Limits and Caveats Searching multiple paths in consul is because consul is hierarchical (or can be) and ENV is not. Making an easy to use API that looks and behaves like `ENV` was one of the goals to make it more familiar to developers. So worst case, by default, MuchKeys will search consul 4 times to find a key. Encrypted secrets can be slightly more clunky. Developers may not have signing keys, in that case, an admin or someone from ops maybe has to manage the secrets. ### Compared to Other Tools It's sort of the opposite of `envconsul` or similar to `dotenv` designed for use in production. * Dotenv doesn't centralize keys. * envconsul has a auto restart the launched process on change feature, this project does not. ### Future It'd be nice if this project wasn't so tied to consul. I don't think it's impossible to decouple it. Etcd basically behaves the same afaik so we could add an etcd adapter. ## License The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).