# -*- mode: org; mode: auto-fill; -*-
#+STARTUP:	showeverything

* Org Converge

[[https://secure.travis-ci.org/wallyqs/org-converge.png?branch=master]]

** Description

This is an experiment of using Org mode to
create documentable reproducible runs, borrowing some ideas
of what is possible to do with tools like =chef-solo=,
=rake=, =foreman=, etc...

** Installing

: gem install org-converge

** Motivation

[[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]] has proven to be flexible enough to write
[[http://www.jstatsoft.org/v46/i03][reproducible research papers]]. 
Then, I believe that configuring and setting up
a server for example, is something that could also be done using
the same approach, given that /converging/ the configuration 
is a kind of run that one ought to be able to reproduce.

** Usage

To run the blocks in parallel:

#+begin_src sh
org-converge path/to/setup-file.org --runmode=parallel
#+end_src

or sequentially...

#+begin_src sh
org-converge path/to/setup-file.org --runmode=sequential
#+end_src

By including ~:after~ or ~:before~ arguments to a block,
it is possible to make the blocks depend on one another.

#+begin_src sh
org-converge path/to/setup-file.org --runmode=chained
#+end_src

In case the blocks have results blocks, it is possible to run
the blocks and matched against the results with the ~org-spec~ helper command:

#+begin_src sh
org-spec path/to/file-spec.org
#+end_src

*** Other commands available

: org-run        # alias for org-converge
: org-tangle     # just tangles the contents without running the blocks

** How it works

Org Converge uses a liberally extended version of Org Babel
features in order to give support for converging the configuration
of a server, and the [[https://github.com/wallyqs/org-ruby][Ruby implementation of the Org mode parser]] 
which makes integrating with useful Ruby tools like foreman, rake, rspec and ohai more straightforward.

For example, using Org Babel we can easily spread config
files on a server by writing the following on a ~server.org~ file.

#+begin_src sh
,#+begin_src yaml :tangle /etc/component.yml
multitenant: false
status_port: 10004
,#+end_src
#+end_src

And then configure it by running it as follows, (considering we have
the correct permissions for tangling at =/etc/component.yml=): 

#+begin_src sh
sudo org-converge server.org
#+end_src

Next, let's say that we no only one want to set the configured templates,
but that we also want to install some packages. In that case, we
should be able to do the following:
(Note: Currently, only named blocks would be run)

#+begin_src sh
,** Configuring the component
　
,#+begin_src yaml :tangle /etc/component.yml
multitenant: false
status_port: 10004
,#+end_src  

,** Installing the dependencies

Need the following so that ~bundle install~ can compile 
the native extensions correctly.
 
# Giving the block a name would make it run
　
,#+name: build_essentials
,#+begin_src sh
apt-get install build-essentials -y
,#+end_src
　
Then the following should work:
　
,#+name: bundle_install　
,#+begin_src sh
cd project_path
bundle install
,#+end_src
#+end_src

Since we are using Org mode syntax, it is possible to reuse this setup file by including it.

#+begin_src sh
,#+TITLE: Another setup

Include the code blocks from the server into this:
,#+include: "server.org"

,#+name: install_org_mode
,#+begin_src sh
apt-get install org-mode -y
,#+end_src
#+end_src

#+end_src

** Examples

Currently there is support for the following kind of runs:

- runs where the blocks need to run sequentially (~--runmode=sequential~) ::
  
  Each code block is part of a step to be ran
  
- runs where the blocks need to be run in parallel (~--runmode=parallel~) ::
  
  One example of this is having what is supposed to be a distributed system running locally for development (where ~foreman~ would be used).
  
- runs where the blocks need to be run in sequence according to defined dependencies (~--runmode=chained~) ::
  
  Set of runs that are usually covered by using something like rake, make, etc...

- runs where the blocks are run and matched against the expected results for testing (~--runmode=spec~) ::
  
  Each block is run and there is an assertion to check whether the contents in ~#+RESULTS~ block match
  
Besides being able to specify which kind of run to use through an option, it is also possible 
to define this within the Org mode file itself as an in buffer setting:

#+begin_src org
  ,#+TITLE:   Defining the runmode as an in buffer setting 
  ,#+runmode: sequential
#+end_src

*** Parallel runs

The following is an example of running 3 processes
in parallel by defining them as code blocks from 
an Org mode file:

#+begin_src sh
  ,#+TITLE: Running Org babel processes in parallel
  　
  ,* Print with different languages
   　　
  ,#+name: hello_from_bash
  ,#+begin_src sh :shebang #!/bin/bash
  while true; do echo "hello world from bash"; sleep 1; done
  ,#+end_src
　  　 
  ,#+name: hello_from_ruby
  ,#+begin_src ruby :shebang #!/usr/local/bin/ruby
  $stdout.sync = true
  loop { puts "hello world from ruby" ; sleep 1 }
  ,#+end_src
  　 　
  ,#+name: hello_from_python
  ,#+begin_src python :shebang #!/usr/bin/python
  import time
  import sys
  for i in range(0,100):
    print "hello world from python"
    sys.stdout.flush()
    time.sleep(1)
  ,#+end_src   
#+end_src

We store this in a file named =hello.org= and then run it as follows:

#+begin_src sh
org-run hello.org
#+end_src

This would produce an output similar to the following:

#+begin_src sh
[2014-05-04T19:23:40 +0900] Tangling 0 files...
[2014-05-04T19:23:40 +0900] Tangling succeeded!
[2014-05-04T19:23:40 +0900] Tangling 3 scripts within directory: /Users/wallyqs/repos/org-converge/run...
[2014-05-04T19:23:40 +0900] Running code blocks now! (3 runnable blocks found in total)
[2014-05-04T19:23:40 +0900] hello_from_bash       (4664) -- started with pid 4664
[2014-05-04T19:23:40 +0900] hello_from_ruby       (4665) -- started with pid 4665
[2014-05-04T19:23:40 +0900] hello_from_python     (4666) -- started with pid 4666
[2014-05-04T19:23:40 +0900] hello_from_bash       (4664) -- hello world from bash
[2014-05-04T19:23:41 +0900] hello_from_ruby       (4665) -- hello world from ruby
[2014-05-04T19:23:41 +0900] hello_from_python     (4666) -- hello world from python
[2014-05-04T19:23:42 +0900] hello_from_ruby       (4665) -- hello world from ruby
#+end_src

Also possible to specify the name of the block to be ran:

#+begin_src sh
org-run hello.org --name from_ruby
#+end_src

*** Spec mode

In case the Org mode file has a results block which represents the expected result, 
there is an ~org-spec~ command which can be useful to check whether there is change.
For example, given the following file stored in ~test.org~:

#+begin_src sh
  ,#+TITLE:   Expected results example
  　
  ,#+name: hello
  ,#+begin_src ruby :results output
  10.times do 
    puts "hola"
  end
  ,#+end_src
  　
  ,#+RESULTS: hello
  ,#+begin_example
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  ,#+end_example
#+end_src

We can be able to verify whether this is still correct by running ~org-spec test.org~

#+begin_src sh
Checking results from 'hello' code block:	OK
#+end_src

As an example, let's say that the behavior of the original code block changed, and now says hello 5 times instead. 
In that case the output would be as follows:

#+begin_src diff
Checking results from 'hello' code block:	DIFF
@@ -1,11 +1,6 @@
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
+hello
+hello
+hello
+hello
+hello
#+end_src

** Contributing

The project is in very early development at this moment, but if you
feel that it is interesting enough, please create a ticket to start
the discussion.