= Opskeleton 
Ronen Narkis, <narkisr@gmail.com>
v0.9.2
:toc: left
:!numbered:
:idseparator: -
:idprefix:
:source-highlighter: pygments
:pygments-style: friendly
:sectlinks:

== Introduction

Opskelaton is an opinionated bootstrap tool for local Sandbox projects, it aims to solve the following common issues:

* Developing Puppet/Chef modules/cookbooks on master machines which results with 'It works on my master/server' approach.
* Large monolithic Puppet/Chef code bases, code isn't modular or reusable.
* Implicit/Missing dependencies including: Ruby version, OS, gems, modules/cookbooks.
* Manual steps in setting up and maintaining such projects.
* Non standard layout, projects missing README and LICENSE files, no clear separation between developed and dependant code.
* Lacking development guidelines (for example extracting general modules and exporting them).
* No continues build, linting and testing, provisioning code is second class citizen.
 
Opskeleton comes to solve these issues by introducing a decentralized development work flow with pre-defined layout, packaging and dependency management.

Currently Opskeleton supports Puppet and Chef, this guide is segmented to common functionality the respective separate sections for Puppet and Chef.

== Usage

=== Installation

Perquisites (on Ubuntu):

* Vagrant 1.7.x
* RVM
* Ruby >= 2.1.2

```bash 
 $ rvm use system
 $ sudo gem install opskeleton
``` 

=== Boxes

Opskeleton recommends the use of https://github.com/box-cutter[box-cutter] in order to create Vagrant boxes in a consistent manner (as no free hosting solution currently exist):
```bash
# make sure to have latest packer
$ packer version
Packer v0.6.0
$ git clone git@github.com:box-cutter/ubuntu-vm.git
$ cd ubuntu-vm
# Edit Makefile.local
$ cat Makefile.local
# Makefile.local
CM := puppet
CM_VERSION := 3.6.1
$ make virtualbox/ubuntu1404
```
A useful convention for Box names:

```bash
ubuntu-14.04_puppet-3.6.1 ([os]-[version]_[provisioner]-[version])
```

== Puppet

=== Introduction

Opskeleton supports the creation of Puppet based projects supporting dependency management (using librarian-puppet), linting and testing.


=== Structure

==== Layout

Opskeleton generates the complete folder structure for projects:

image:https://raw.githubusercontent.com/opskeleton/opskeleton/master/img/puppet-layout.png[width=30%,hight=50%]

==== Lifecycle

Opskelaton defines a module life cycle:

1. Internal non reusable modules (usually specific to a client site) go under static-modules
2. If we create a general reusable module which is ready for prime time we pull out to a new git repository.
3. The extracted module is added (using link:https://github.com/rodjek/librarian-puppet[librarian-puppet]) back as a third party module under modules folder.

Life cycle scheme:

image:https://raw.githubusercontent.com/opskeleton/opskeleton/master/img/puppet-cycle.png[width='40%',hight='50%']

Creating new (static) modules is easy as:

```bash
$ opsk module foo
```

Each generated module will contain puppet-rspec with matching Rakefile. 

==== Pushing changes

Making changes to third party modules is quite easy once librarian-puppet installed them locally (you can push only git based modules):

```ruby
forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs/stdlib'
mod 'puppetlabs/apt'

mod 'strings/artifactory', 
   :git => 'git://github.com/pulling-strings/puppet-artifactory.git'

mod 'rip/module-data',
  :git => 'git://github.com/ripienaar/puppet-module-data.git'
```

Its best practice to use git protocol (read only) which makes pushing changes from multiple modules a bit tedious, Opskeleton fixes that:


We can list the changes:

```bash 
$ opsk uncommited
Listing changes for modules/artifactory:

changed files:

- metadata.json

added files:

untracked files:
```

We can commit them (providing a commit message per module or --message for all):
```bash
# We hacked a number of submodules, now we commit
$ opsk commit
Listing changes for modules/artifactory:

changed files:

- metadata.json

added files:

untracked files:

Commit the changes under modules/artifactory? (y/n) y
Commit message:
This is a nice change
```

For more commit options:

```bash
$ opsk help commit

Usage:
  opsk commit [message]

Options:
  [--message=MESSAGE]  # optional commit message
  [--all], [--no-all]  # commit all

commit each changed puppet module under modules folder
```

Once commits are made we can push the changes:

```bash
$ opsk push
Push modules/artifactory? (y/n) y
pushing modules/artifactory ..
```

Opsk will add a remote writable repository for each submodule substituting the readonly protocol with a write enabled one, the default protocol is ssh (customizable by using --protocol).

For more options: 

```bash 
$ opsk help push

Usage:
  opsk push

Options:
  [--protocol=PROTOCOL]  # remote ssh protocol (https or ssh)
                         # Default: ssh
  [--dry], [--no-dry]    # dry mode
  [--all], [--no-all]    # push all without asking

push each changed puppet module under modules folder
```
=== Testing

Opskelaton supports two levels of testing:

* Static module testing that includes rspec and linting.
* Integration testing using http://serverspec.org/[serverspec]and Vagrant.

```bash
# linting all static modules
$ rake lint
# rspecing 
$ rake modspec
# running serverspec
$ rake spec
```

=== Packaging 
Opskelaton fully supports deployment and portable execution of sandboxes on non Vagrant environments:

```bash
$ opsk generate_puppet foo ubuntu-13.10
$ cd foo-sandbox
# The package version file
$ cat opsk.yaml
--- 
  version: '0.0.1'
  name: foo

# post bundle and gem install ..
$ opsk package
      create  pkg/foo-sandbox-0.0.1
      create  pkg/foo-sandbox-0.0.1/scripts
      create  pkg/foo-sandbox-0.0.1/scripts/lookup.rb
       chmod  pkg/foo-sandbox-0.0.1/scripts/lookup.rb
      create  pkg/foo-sandbox-0.0.1/scripts/run.sh
       chmod  pkg/foo-sandbox-0.0.1/scripts/run.sh
      create  pkg/foo-sandbox-0.0.1/manifests/site.pp
       exist  pkg
$ ls pkg
foo-sandbox-0.0.1  foo-sandbox-0.0.1.tar.gz
```
The packaging process creates a portable tar file that can be run on any machine with puppet installed via the bundled run.sh:

```bash 
$ tar -xvzf foo-sandbox-0.0.1.tar.gz
$ cd foo-sandbox-0.0.1 
$ sudo ./run.sh
```

An external node classifier based runner is also available under scripts/run.sh, this runner expects to get a <hostname>.yaml input file with the required node classes.

=== Updating
Keeping you box up to date with latest opsk version is easy, just re-generate it again and resolve conflicts by answering y/n:
```bash
# Moving to latest opsk
$ gem update opskeleton
# foo box already exists
$ opsk generate foo <vagrant-box>
 exist  foo-sandbox
    conflict  foo-sandbox/Vagrantfile
Overwrite /home/ronen/code/foo-sandbox/Vagrantfile? (enter "h" for help) [Ynaqdh]
```

=== Vagrant
Opskeleton generates a Vagrant file with couple of enhancements:
 
* VAGRANT_BRIDGE (default eth0) for setting up public bridge on the go.
* PUPPET_ENV (default dev) for setting puppet environment.
* Puppet options preset to match modules and hiera folders.

=== Docker
The only assumption that Opskelaton makes is that the target host will have Pupppet installed, this enables us to create docker images from our sandboxes quite easily:

```bash
# creates dockerfiles/<host> and fig.yml
$ opsk dockerize
$ opsk package
# grabs the opsk tar file
$ sudo fig build
```

=== Benchmarking
Tracking the speed of our provisioning code is important for keeping a consistent level of service with the produced sandboxes, enabling benchmarking:

```bash 
$ opsk generate_puppet redis ubuntu-14.04 --bench-enable
# install imagemagic before bundle install
$ sudo apt-get install imagemagick libmagickwand-dev
$ rake serverspec:redis
# with each run more result lines will be recorded
$ cat benchmark.json
{"total":656,"host":"redis","revision":"5d03a41ade9fc3dd5296d4119ccb0b0ad8290b9e","time":"2014-12-17 02:57:45 +0200"}
# add it to git for tracking
$ git add benchmark.json
```

Now after a number of runs we could plot and view the results of a single host or of all the hosts side by side:

```bash 
$ rake plot:hosts plot:per_hosts
# resulting png files 
$ google-chrome plots
```

== Chef

=== Introduction
Opskelaton fully supports Chef based projects it offers similar features to the Puppet based sandboxes with additional support for roles, environments and cookbooks.

Creating out first sandbox

```bash
$ opsk generate_chef redis ubuntu-14.04
$ cd redis-sandbox
```

=== Structure

==== layout
Opskelaton creates the complete folder structure fine tuned to match best practices:

Folder layout:

image:https://raw.githubusercontent.com/opskeleton/opskeleton/master/img/chef-layout.png[width='30%',hight='50%']

==== lifecycle

Opskelaton defines a simple cookbook life cycle:

 1. Internal non reusable cookbooks (usually specific to a client site) go under static-cookbooks
 2. If we create a general reusable cookbook which is ready for prime time we pull out to a new git repository.
 3. The extracted cookbook is added back as a third party (using link:https://github.com/applicationsonline/librarian-chef[librarian-chef]) cookbook under cookbooks folder).

Life cycle scheme:

image:https://raw.githubusercontent.com/opskeleton/opskeleton/master/img/chef-cycle.png[width='40%',hight='50%']

Creating new cookbooks is easy as:

```bash
$ opsk cookbook foo
```

=== Testing

Opskelaton supports testing/linting:

* Static cookbook testing that includes rspec and food-critic. (TBD)
* Integration testing using [serverspec](http://serverspec.org/) and Vagrant.

```bash
# running serverspec
$ rake spec
```

=== Packaging 
Opskelaton fully supports deployment and portable execution of sandboxes on non Vagrant environments:

```bash
$ opsk generate_chef foo ubuntu-14.04.
$ cd foo-sandbox
# The package version file
$ cat opsk.yaml

--- 
  version: '0.0.1'
  name: redis
  includes: 
    - Cheffile
    - cookbooks
    - static-cookbooks
    - dna.json
    - environments
    - Gemfile
    - Gemfile.lock
    - opsk.yaml
    - roles
    - LICENSE-2.0.txt
    - run.sh
    - boot.sh
    - solo.rb

# post bundle and gem install ..
$ opsk package
	create  pkg/foo-sandbox-0.0.1
	create  pkg/foo-sandbox-0.0.1/scripts
	create  pkg/foo-sandbox-0.0.1/scripts/lookup.rb
	 chmod  pkg/foo-sandbox-0.0.1/scripts/lookup.rb
	create  pkg/foo-sandbox-0.0.1/scripts/run.sh
	 chmod  pkg/foo-sandbox-0.0.1/scripts/run.sh
	 exist  pkg
$ ls pkg
foo-sandbox-0.0.1  foo-sandbox-0.0.1.tar.gz
```
The packaging process creates a portable tar file that can be run on any machine with chef-solo installed via the bundled run.sh:

```bash 
$ tar -xvzf foo-sandbox-0.0.1.tar.gz
$ cd foo-sandbox-0.0.1 
# expects to get the chef environment
$ sudo ./run.sh dev
```

=== Updating

Keeping you box up to date with latest opsk version is easy, just re-generate it again and resolve conflicts by answering y/n:
```bash
# Moving to latest opsk
$ gem update opskeleton
# foo box already exists
$ opsk generate_chef foo <vagrant-box>
 exist  foo-sandbox
    conflict  foo-sandbox/Vagrantfile
Overwrite /home/ronen/code/foo-sandbox/Vagrantfile? (enter "h" for help) [Ynaqdh]
```

=== Vagrant
Opskeleton generates a Vagrant file with couple of enhancements:
 
* CHEF_ENV (default dev) for setting chef environment.
* Default role (sandbox name) created under roles/{type}.rb
* static-cookbooks/cookbooks roles/environments folders are set.

== Deployment

The packaged tar files can be consumed using any tool and protocol (http, s3 etc),  opsk has built in support for deploying public sandboxes into:

* Bintray (make sure to  https://github.com/narkisr/bintray-deploy#usage[configure] the bintray API key):

```bash 
$ opsk package
$ opsk deploy_bintray <bintray-repo>
   deployed foo-sandbox-0.0.1.tar.gz to http://dl.bintray.com/narkisr/<bintray-repo>/foo-sandbox-0.0.1.tar.gz
```

* S3 (Make sure to configure s3 section under ~/.configuration.rb):

```bash 
$ opsk package
$ opsk deploy_s3 <bucket> <path>
   deployed foo-sandbox-0.0.1.tar.gz to opsk-boxes/foo/foo-sandbox-0.0.1.tar.gz
```

```ruby
Configuration.for('s3') {
  access_key ''
  secret_key ''
  region ''
}
```

* Scp (Make sure to configure scp section under ~/.configuration.rb):

```bash 
$ opsk package
$ opsk deploy_scp bar
   deployed foo-sandbox-0.0.1.tar.gz to foo@opsk-boxes:/var/boxes
```

```ruby
Configuration.for('scp') {
  bar {
   host 'opsk-boxes'
   user 'foo'
   dest '/var/boxes'
   # optional
   port 2222
 }
}
```