# awsborn This Gem lets you define and launch a server cluster on Amazon EC2. The `launch` operation is idempotent, i.e., it only launches instances that are not running. ## Installation gem install awsborn ## Synopsis #! /usr/bin/env ruby require 'rubygems' require 'awsborn' # If not set, will be picked up from ENV['AMAZON_ACCESS_KEY_ID'] Awsborn.access_key_id = 'AAAAAAAAAAAA' # Can be picked up from ENV['AMAZON_SECRET_ACCESS_KEY'] or Keychain (OS X) Awsborn.secret_access_key = 'KKKKKKKKKKKK' # Logs to `awsborn.log` in current dir with level INFO by default. # To suppress logging, do `Awsborn.logger = Logger.new('/dev/null')` Awsborn.logger.level = Logger::DEBUG class LogServer < Awsborn::Server instance_type :m1_small image_id 'ami-2fc2e95b', :sudo_user => 'ubuntu' security_group 'Basic web' keys 'keys/*.pub' bootstrap_script 'chef-bootstrap.sh' monitor true end log_servers = LogServer.cluster do domain 'releware.net' server :log_a, :zone => :eu_west_1a, :disk => {:sdf => "vol-a857b8c1"}, :ip => 'log-a' server :log_b, :zone => :eu_west_1b, :disk => {:sdf => "vol-aa57b8c3"}, :ip => 'log-b' end log_servers.launch log_servers.each do |server| system "rake cook server=#{server.host_name}" end ## How it works It's really simple: 1. Define a server type (`LogServer` in the example above). 2. Define a cluster of server instances. 3. Launch the cluster. See *Launching a cluster* below for details. ### Defining a server type Server types can be named anything but must inherit from `Awsborn::Server`. Servers take five different directives: * `instance_type` -- a symbol. One of `:m1_small`, `:m1_large`, `:m1_xlarge`, `:m2_2xlarge`, `:m2_4xlarge`, `:c1_medium`, `:c1_xlarge`, `cc1.4xlarge`, and `t1.micro`. * `image_id` -- a valid EC2 AMI or a hash `{:i386 => 'ami-3232', :x64 => 'ami-6464'}` Specify `:sudo_user` as an option if the AMI does not allow you to log in as root. * `security_group` -- a security group that you own. * `keys` -- one or more globs to public ssh key files. When the servers are running, `root` will be able to log in using any one of these keys. * `bootstrap_script` -- path to a script which will be run on each instance as soon as it is started. Use it to bootstrap `chef` and let `chef` take it from there. A sample bootstrap script is included towards the end of this document. A server cannot be started without `instance_type`, `image_id` and `security_group`. The don't have to be defined for the server type though, but can equally well be added as options to the specific servers. ### Defining a cluster The `cluster` method accepts two commands, `domain` and `server`. `domain` is optional and is just a way to avoid repetition in the `server` commands. `server` takes a name (which can be used as a key in the cluster, e.g. `log_servers[:log_a]`) and a hash: Mandatory keys: * `:zone` -- the availability zone for the server. One of `:us_east_1a`, `:us_east_1b`, `:us_east_1c`, `:us_east_1d`, `:us_west_1a`, `:us_west_1b`, `:eu_west_1a`, `:eu_west_1b`, `:ap-southeast-1a`, `:ap-southeast-1b`. * `:disk` -- a hash of `device => volume-id`. Awsborn uses the disks to tell if a server is running or not (see *Launching a cluster*). `volume-id` can also be an array `[volume-id, :format]`, in which case `the_server.format_disk_on_device?(device)` will return `true`. See `contrib/cookbooks/ec2-ebs` for example usage. Optional keys: * `:ip` -- a domain name which translates to an elastic ip. If the domain name does not contain a full stop (dot) and `domain` has been specified above, the domain is added. Keys that override server type settings: * `:instance_type` * `:sudo_user` * `:security_group` * `:keys` * `:bootstrap_script` * `:image_id` ### Launching a cluster The `launch` method on the cluster checks to see if each server is running by checking if the server's disks are attached to an EC2 instance, i.e., the server is defined by its content, not by its AMI or ip address. Servers that are not running are started by calling the `start` method on the server, which does the following: def start (key_pair) launch_instance(key_pair) update_known_hosts install_ssh_keys(key_pair) if keys if elastic_ip associate_address update_known_hosts end bootstrap if bootstrap_script attach_volumes end The `key_pair` is a temporary key pair that is used only for launching this cluster. In case of a failed launch, the private key is available as `/tmp/temp_key_*`. ## A bootstrapping script This is what we use with the AMI above: #!/bin/bash echo '------------------' echo 'Bootstrapping Chef' echo aptitude -y update aptitude -y install gcc g++ curl build-essential \ libxml-ruby libxml2-dev \ ruby irb ri rdoc ruby1.8-dev libzlib-ruby libyaml-ruby libreadline-ruby \ libruby libruby-extras libopenssl-ruby \ libdbm-ruby libdbi-ruby libdbd-sqlite3-ruby \ sqlite3 libsqlite3-dev libsqlite3-ruby curl -L 'http://rubyforge.org/frs/download.php/69365/rubygems-1.3.6.tgz' | tar zxf - cd rubygems* && ruby setup.rb --no-ri --no-rdoc ln -sfv /usr/bin/gem1.8 /usr/bin/gem gem install chef ohai --no-ri --no-rdoc --source http://gems.opscode.com --source http://gems.rubyforge.org echo echo 'Bootstrapping Chef - done' echo '-------------------------' ## Running with Chef Solo Awsborn integrates with [Chef Solo](http://github.com/opscode/chef). An example from reality: Awsborn.access_key_id = 'AKIAJHL53MPX7LPCKIFQ' class LogServer < Awsborn::Server instance_type :m1_small image_id 'ami-2fc2e95b', :sudo_user => 'ubuntu' security_group 'Basic web' keys '../keys/*' bootstrap_script 'chef-bootstrap.sh' monitor true cluster do domain 'releware.net' server :log_a, :zone => :eu_west_1a, :disk => {:sdf => "vol-a857b8c1"}, :ip => 'log-a' server :log_b, :zone => :eu_west_1b, :disk => {:sdf => "vol-aa57b8c3"}, :ip => 'log-b' end cluster :test do domain 'releware.net' server :test, :zone => :eu_west_1b, :disk => {:sdf => "vol-abababab"}, :ip => 'log-test' end def chef_dna { :user => "lumberjack", :host => host_name, :users => [ { :username => "lumberjack", :authorized_keys => key_data, :gid => 1001, :uid => 1001, :sudo => true, } ], :packages => %w[ vim heirloom-mailx ], :gems => [ "rake" ], :ebs_volumes => [ {:device => "sdf", :path => "/apps"} ], :recipes => [ "packages", "users", "sudo", "openssh", "ec2-ebs", "git", "gems", "lumberjack" ] } end end Just add a `Rakefile` in the same directory: require 'awsborn' include Awsborn::Chef::Rake require './servers' Put your cookbooks directory in the same directory as the `Rakefile` or in its parent directory. Run `rake` to start all servers and run Chef on each of them. Other rake tasks include: * `rake chef` - Run chef on all servers, or the ones specified with `host=name1,name2`. * `rake chef:debug` - Ditto, but with chef's log level set to `debug`. * `rake start` - Start all servers (or host=name1,name2) but don't run `chef`. * `rake start cluster=test` - Start the servers in the named cluster. Default is the first unnamed cluster. ## Bugs and surprising features * There is a race condition in launch: Immediately after the launch request (`launch_instance`) we call `instance_running?`, but occasionally, AWS will not recognise the instance id yet (`InvalidInstanceID.NotFound: The instance ID 'i-e9005283' does not exist`) * Running with `rake` suppresses stack traces which makes it hard to debug rare errors. The rake tasks or entry points should catch exceptions and print the stack trace. * `t1_micro` instances will select an i386 image when `image_id` is specified as a hash. To run them with x64 images, specify the image name on the server directive. ### Untested features * Launching without ssh keys. (I suppose certain AMIs would support that.) ## To do / Could do * Tests. * Dynamic discovery of instance types and availability zones. * Hack RightAws to verify certificates. * Elastic Load Balancing. * Launch servers in parallel. ## Note on Patches/Pull Requests * Fork the project. * Make your feature addition or bug fix. * Add tests for it. This is important so I don't break it in a future version unintentionally. * At the moment, the gem does not have much tests. I intend to fix that, and I won't accept patches without tests. * Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull) * Send me a pull request. Bonus points for topic branches. ## Changes ### New in 0.7.0 * Allow several security groups per server. * Specify `individual_security_group true` to add a unique-ish security group named _`ServerClass servername`_. This security group is created on the fly if it doesn't exist. * Server clusters can have names. ### New in 0.6.0 * Uses `right_aws` version 2. ### New in 0.5.4, 0.5.5 * Run `rake list` to see a list of running servers. ### New in 0.5.2, 0.5.3 * Run `rake upgrade_chef host=foo` to upgrade chef on server `foo`. ### New in 0.5.1 * When awsborn uploads cookbooks and configuration, it looks for a `cookbooks` directory in the parent directory if none is found in the `Rakefile`'s directory. ### New in 0.5.0 * Support for cc1.4xlarge and t1.micro instance types. ## History This gem is inspired by [Awsymandias](http://github.com/bguthrie/awsymandias) which was sort of what I needed but was too complicated to fix, partly because the documentation was out of date. Big thanks for the inspiration and random code snippets. ## Copyright Copyright (c) 2010 ICE House & David Vrensk. See LICENSE for details.