== A hands-on tutorial for learning AutomateIt AutomateIt is an open source tool for automating the setup and maintenance of servers, applications and their dependencies. This hands-on guide will teach you to use AutomateIt and explain where to find more detailed instructions. It's recommended that you see the Screenshots[http://automateit.org/screenshots] of AutomateIt in action to get a quick idea of what AutomateIt is all about. AutomateIt is feature-complete, exceeds the capabilities of similar products, and ensures its quality with a self-test suite. However, this is a young product and users are expected to be technically proficient, willing to accept rough spots, to work through problems and upgrade frequently. Please sign up for RSS change notifications[http://automateit.org/changes] so you know when upgrades are available. === Ruby AutomateIt is written using the Ruby programming language. If you haven't used Ruby, you'll find it easy to learn and a pleasure to use. If you already know the basics of Perl, Python, PHP or Java, you'll be able to pick up Ruby almost instantly. Although AutomateIt provides much of the structure and commands needed, you still need to know the basic Ruby syntax to get by. Some Ruby resources: * Ruby's official documentation page: http://www.ruby-lang.org/en/documentation/ * The online "Ruby user's guide" is a gentle introduction to the Ruby language: http://www.ruby-doc.org/docs/UsersGuide/rg/ * The online "Ruby syntax" is a condensed, one-page reference of the language's syntax: http://www.ruby-doc.org/docs/ruby-doc-bundle/Manual/man-1.4/syntax.html * The online "Programming Ruby" site provides a fairly detailed walk-through of the language: http://www.ruby-doc.org/docs/ProgrammingRuby/ * The "Practical Ruby for System Administration" book is a gentle introduction for sysadmins, which may be much easier for them to understand than a book aimed at software engineers, and it covers many system administration automation topics that'll be of use for Automateit recipes: http://www.apress.com/book/view/1590598210 * The "Programming Ruby" book provides a much more complete language reference than any online reference: http://www.pragmaticprogrammer.com/titles/ruby/index.html === Typographical conventions AutomateIt's technical documentation uses the following typographical conventions: * +automateit+ -- A file, command or variable. * :verbosity -- A symbol, usually used in an options hash. * AutomateIt::ShellManager -- A class, with a link to its documentation. You can also find a link to this class's documentation in the "Classes" pane on the left. * AutomateIt::ShellManager#sh -- A method, with a link to its documentation. You can also find a link to this method's documentation in the "Methods" pane on the left. * Ruby code: puts "Ruby code" * Unix shell command, although the prompt may be left off when obvious: you@host:myproject> echo "Unix command run from the 'myproject' directory" * AutomateIt interactive shell command, although the prompt may be left off when obvious: ai> puts "I'm in the Interpreter" === Glossary AutomateIt uses a number of unique terms: * *Recipe* -- A file that contains AutomateIt commands. * *Project* -- A special directory that contains related recipes and helper files. * *Interpreter* -- The part of AutomateIt that runs commands. * *Plugin* -- A part of AutomateIt that describes related features. * *Driver* -- An implementation of a plugin. There are often multiple drivers per plugin. === Interactive shell AutomateIt comes with an interactive shell, which is useful for exploring commands and developing recipes. Here's what an interactive shell session looks like (text following the # symbol explains the line and is not actually part of the session): you@host:tmp> automateit # Start the AutomateIt interactive shell => AutomateIt Shell v0.5.0 # Welcome message from AutomateIt ai> puts "Hello world!" # AutomateIt's prompt and a user command Hello world! # Output from the "puts" command => nil # Return value of the "puts" command ai> self.class # Prompt and another user command => AutomateIt::Interpreter # Return value of "self.class" ai> # Press to exit the shell you@host:tmp> # We're back to the Unix shell === Recipe files The Interpreter can run recipe files that contain AutomateIt commands. For example, create a /tmp/hello.rb file that contains: puts "Hello, I'm an #{self.class}" Then run the recipe: you@host:tmp> automateit /tmp/hello.rb Hello, I'm an AutomateIt::Interpreter === String evaluation The Interpreter can also evaluate commands as strings: you@host:tmp> automateit -e 'puts self.class' AutomateIt::Interpreter === Exploring the Interpreter's unique methods The Interpreter provides some unique methods: ai> unique_methods => ["account_manager", "address_manager", "cd", "chmod", ... The names are Interpreter methods. Names ending with +_manager+ are methods for accessing plugins, while the rest are normal methods. The AutomateIt::Interpreter documentation provides a list of these methods and links to their individual documentation. For example, one of the commands listed is +pwd+, which can be used like this: ai> pwd => "/tmp" AutomateIt provides many commands that work just like the Unix shell commands you already know, so you'll be productive quickly. === Conditional execution AutomateIt executes only the commands needed to achieve a desired state, which makes recipes repeatable, reusable and maintainable. Eliminating the distinction between setup and maintenance means the same command can perform both tasks without concern for the host's condition. A properly-written recipe can be run and re-run immediately afterwards, and it will do nothing the second time because all changes have already been applied. An example can make this clearer -- again the text following the # symbol explains the commands and is not part of the session: you@host:tmp> automateit # Start the AutomateIt interactive shell => AutomateIt Shell v0.5.0 # Welcome message ai> mkdir "asdfasdf" # Create a directory called "asdfasdf" ** mkdir asdfasdf # Message showing that directory was created => ["asdfasdf"] # Return value with array of directories created ai> mkdir "asdfasdf" # Try to create the same directory again => false # Nothing happened, the directory already exists ai> rmdir "asdfasdf" # Remove the directory ** rmdir asdfasdf # Message showing that directory was removed => ["asdfasdf"] # Return value with array of directories removed ai> rmdir "asdfasdf" # Try to remove the directory again => false # Nothing happened, there's no directory to remove Notice how the second time +mkdir+ was run above, it returned +false+ and didn't create a directory? That's the conditional execution in action, realizing the action has already been performed. Similarly, the +rmdir+ only ran the first time, but returned +false+ and took no action when the directory was already gone. You can use the return values of these commands to write sophisticated logic of your own based on what actions took place. === Plugins AutomateIt uses an extensible plugin architecture to group together related commands: * AutomateIt::AccountManager -- Manipulates users and groups. * AutomateIt::AddressManager -- Manipulates host's network addresses. * AutomateIt::DownloadManager -- Downloads files. * AutomateIt::EditManager -- Edits files and strings. * AutomateIt::FieldManager -- Queries configuration variables. * AutomateIt::PackageManager -- Manipulates software packages. * AutomateIt::PlatformManager -- Queries platform, such as its OS version. * AutomateIt::ServiceManager -- Manipulates services, such as Unix daemons. * AutomateIt::ShellManager -- Manipulates files and executes Unix commands. * AutomateIt::TagManager -- Groups hosts by role and queries membership. * AutomateIt::TemplateManager -- Renders templates to files. Plugins can be accessed from the Interpreter like this: ai> shell_manager.pwd => "/tmp" The most common plugin methods have aliased shortcuts. For example, +pwd+ is the alias for shell_manager.pwd. These are documented in AutomateIt::Interpreter's "Aliased methods." === Drivers Each plugin has one or more drivers that implement its functionality. For example: * AutomateIt::ShellManager -- A plugin for running shell commands. * AutomateIt::ShellManager::Portable -- A portable but limited-functionality driver that implements the ShellManager's methods. * AutomateIt::ShellManager::Unix -- A full-featured driver implementing ShellManager's methods that only runs on Unix-like systems. === Plugins provide APIs, drivers provide implementations Plugins describe consistent interfaces for related features, and drivers implement this API for different tools. For example, AutomateIt provides a common API for all packaging tools: AutomateIt::PackageManager. It provides drivers for packaging tools like APT, YUM, Gem, Egg and others. To install a package called +foo+ with the +apt+ driver: ai> package_manager.install "foo", :with => :apt AutomateIt will check if +foo+ is installed, install it if needed, or do nothing if the package is present. This API is the same for all packaging tools, making it easy to get work done by using high-level AutomateIt commands instead of cryptic tool-specific commands. Although AutomateIt uses the low-level tools, it uses them with best-practices approaches and hides the senseless complexity from the user. What's the big deal? Consider how one would install packages from the Unix shell. Most packaging tools are pathologically dysfunctional and make it bafflingly difficult to programmatically install or uninstall packages, or tell if a package is installed. Many don't use exit values and require complex output parsing. Others require user-input even when it's obvious that none is needed. Almost all make it necessary to write conditional code because they'll either fail with errors if told to install an existing package, destroy an existing setup, or install duplicate packages. Writing Unix shell code to handle all these quirks is frustrating and risky. Here's a sample Unix shell command for installing a package using one of the simplest, most reasonable tools available -- although note that unlike AutomateIt, this shell command can't handle multiple packages, is slower, can't be previewed, and has no consistent error handling: if dpkg-query -W --showformat '${Status}\n' foo 2>&1 | \ egrep -q '(^| )installed'; then apt-get install -y -q some_package_name < /dev/null done Now compare that hideous command to the simple, clear and consistent AutomateIt command before it. AutomateIt's plugins and drivers are easy to install and write. As more are written, more people will hopefully be freed from needlessly convoluted low-level commands, and able to get simple things done simply. AutomateIt's consistent API, multiple drivers, sane defaults and conditional-checking make it easier to write clear and maintainable recipes than using the low-level directly commands from the Unix shell. === Driver auto-detection AutomateIt can automatically detect the most suitable driver for each plugin command. For example, the AutomateIt::PackageManager plugin has drivers called AutomateIt::PackageManager::APT and AutomateIt::PackageManager::YUM. On a Debian system, which uses the apt-get packaging tool, AutomateIt will default to using the AutomateIt::PackageManager::APT driver: ai> package_manager.installed? "apache2" => true === Using a specific driver Sometimes it's necessary to specify the driver to use. The recommended way to do this is to pass a :with => :driver_name option to the plugin command. For example, tell the package manager to use the Gem driver: ai> package_manager.installed? "automateit", :with => :gem => true You can also completely bypass the plugin and its auto-detection to directly interact with the driver: ai> package_manager[:gem].installed? "automateit" => true === Projects A project is a special directory that contains related recipes and helper files. Although it's possible to run recipe files without a project, a project provides many useful features, described in the AutomateIt::Project documentation. A project is created by specifying the directory to create: you@host:tmp> cd /tmp you@host:tmp> automateit --create myproject ** mkdir -p myproject => Creating AutomateIt project at: myproject ** cd myproject ** mkdir config ** cd config => Rendering 'tags.yml' because of it doesn't exist => Rendering 'fields.yml' because of it doesn't exist => Rendering 'automateit_env.rb' because of it doesn't exist ** cd - ** mkdir dist ** cd dist => Rendering 'README.txt' because of it doesn't exist ** cd - ** mkdir lib ** cd lib => Rendering 'README.txt' because of it doesn't exist ** cd - ** mkdir recipes ** cd recipes => Rendering 'README.txt' because of it doesn't exist ** cd - => DONE! This creates a /tmp/myproject directory with the newly-created project. It contains directories, each with a README.txt file explaining the directory's purpose, and individual files like tags.yml that contain comments with basic usage instructions. The project creator is an AutomateIt recipe, so it's smart enough to only execute the commands needed. If the project creator is re-run against an existing project, it won't make any changes because none are needed: you@host:tmp> automateit --create myproject => Found AutomateIt project at: myproject => DONE! === Project recipes To associate a recipe with a project, put it into the project's +recipes+ directory. For example, create a recipes/hello_project.rb file with these contents: puts "Hello, this is: " + project And run it: you@host:myproject> automateit recipes/hello_project.rb Hello, this is: /tmp/myproject The Interpreter automatically loads the project. The +project+ method contains the project path and is only available when executing a recipe associated with a project. === Fields A project's config/fields.yml file is meant to store configuration constants, like custom paths for applications. Fields abstract configuration variables for recipes, improving maintainability by separating data and logic. Fields can also be easily queried from Unix using the +aifield+ command. More information about fields and +aifield+ can be found in AutomateIt::FieldManager. For example, consider a fields file with the following contents: myuser: dhh myapp: path: /var/www/rails These fields can be used from the interactive shell like this: you@host:myproject> pwd /tmp/myproject you@host:myproject> automateit -p . # (1) Load project from current directory => AutomateIt Shell v0.5.0 ai> lookup :myuser # (2) Lookup string value by symbol key => "dhh" ai> lookup "myuser" # (3) Lookup string value by string key => "dhh" ai> lookup "myapp" # (4) Lookup hash value by string key => {"path"=>"/var/www/rails"} ai> lookup "myapp#path" # (5) Lookup string value by compound key => "/var/www/rails" To load a project's fields, the AutomateIt interactive shell needs to know which project to load. On the line annotated (1), automateit -p . tells it to load the project from the "." directory, the current directory. The argument can be an absolute or relative path. The command accepts other arguments and environmental options, run automateit --help for details. Fields can be queried by string (2) or symbol (3) keys. The +lookup+ command can return any associated data, usually a string, but it can be a complex type, such as a hash (4). The compound-key (5) syntax is a convenient syntax for looking up nested keys in a hash. For example, lookup "myapp#path" is a shortcut for lookup("myapp")["path"] -- these query the +myapp+ hash and return the value of its +path+ key. The Interpreter loads a project and its fields automatically for recipes, so there is no need to specify the -p option. For example, a project recipe file recipes/hello_fields.rb contains: puts lookup("myapp#path") This recipe will load its project and fields automatically: you@host:myproject> automateit recipes/hello_fields.rb /var/www/rails === Tags A project's config/tags.yml file describes tags assigned to hosts, grouping together hosts by their roles and attributes. For example, this tags file describes a "desktops" tag with three hosts named "satori", "sunyata" and "michiru", and another tag named "notebooks" with other hosts: desktops: - satori - sunyata - michiru notebooks: - rheya - avijja The tags can be accessed like this when run on a host named "satori": you@satori:myproject> automateit -p . ai> tags => ["satori", "desktops", "localhost", ...] # Tags for this host ai> tagged?("desktops") # Is this host tagged with "desktops"? => true ai> tagged?("notebooks") => false ai> tagged?(:satori) # Strings and symbols are treated the same. => true ai> tagged?("satori") => true ai> tagged?("satori || desktops") # Query by simple boolean expression => true ai> tagged?("(satori || desktops) && !notebooks") # Query by complex boolean expression => true Role-based behavior can be demonstrated by creating a file called recipes/hello_tags.rb: if tagged?("localhost") puts "I'm a localhost!" end if tagged?("desktops") puts "Special commands to execute on desktops" elsif tagged?("notebooks") puts "Special commands to execute on notebooks" end Here are the results when executed on a host called "satori": you@satori:myproject> automateit recipes/hello_tags.rb I'm a localhost! Special commands to execute on desktops Notice how the +notebooks+ section wasn't executed? Tags make it possible to create sophisticated recipes that run commands on only the appropriate systems. More documentation on tags can be found in AutomateIt::TagManager. === Previewing commands AutomateIt lets you preview commands the recipe will run without actually letting them actually modify the system. The Interpreter has a boolean that determines if it'll make changes to your system. This one variable can be called from two different ways: * writing? -- Will it write changes? * noop? -- Will it not write changes and just preview them? The +noop+ means "no-operation" Here's an example with the interactive shell, with comments for annotations: ai> noop true # Enter noop mode ai> noop? # Currently in noop mode? => true # Yes, in noop mode ai> writing? # Writing to disk? The opposite of noop => false # No, not writing ai> mkdir "foo" # Try to create a directory ** mkdir foo # Message showing directory will be creating => ["foo"] # Return value with directories to create ai> File.directory? "foo" # Was the directory actually made? => false # No, noop mode prevented the change Recipes can take advantage of the noop mode as well, consider a mkdir_example.rb recipe: mkdir "foo" To run this recipe with noop mode, pass the -n option to the Interpreter shell: you@host:tmp> automateit -n mkdir_example.rb ** mkdir foo you@host:tmp> ls foo ls: foo: No such file or directory Notice how the directory wasn't actually created? This is great because you can see exactly what commands AutomateIt will run without actually having it apply the changes. *WARNING*: Previewing code can be dangerous. Read previews.txt[link:files/docs/previews_txt.html] for instructions on how to write code that can be safely previewed. === Conclusion I hope you enjoy working with AutomateIt and look forward to hearing about your experiences with it. Drivers, patches, documentation and ideas are welcome. Thank you for taking the time to read this! - Igal Koshevoy (igal@pragmaticraft.com)