# -*- mode: org; mode: auto-fill; -*-
#+STARTUP:	showeverything

* Org Converge

  [[https://secure.travis-ci.org/wallyqs/org-converge.png?branch=master]]

** Description

   A framework to create documented reproducible runs using [[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]],
   borrowing several ideas of what is possible to do with tools
   like =chef-solo=, =rake=, =foreman= and =capistrano=.

** Install

   : gem install org-converge

** Motivation

   [[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]] has proven to be flexible enough to write [[http://www.jstatsoft.org/v46/i03][reproducible research papers]],
   then I believe that configuring and setting up a server for
   example, is something that could also be done using
   the same approach, given that /converging/ the configuration
   is a kind of run that one ought to be able to reproduce.

   Taking the original Emacs implementation as reference,
   Org Converge uses the [[https://github.com/wallyqs/org-ruby][Ruby implementation of the Org mode parser]]
   to implement and enhance the functionality from Org Babel 
   by adding helpers to define the properties of a run while taking advantage
   of what is already there in the Ruby ecosystem.

** Usage examples

   Org Converge supports the following kind of runs below:

*** Parallel runs

    Each one of the code blocks is run on an independent process.
    This is akin to having a =Procfile= based application, where
    one of the runner command would be to start a web application
    and the other one to start the a worker processes.

    In the following example, we are defining 2 code blocks, one 
    that would be run using Ruby and one more that would be run in Python.
    Notice that the Python block has =:procs= set to 2, meaning that 
    it would spawn 2 processes for this.

#+begin_src sh
  ,#+TITLE: Sample parallel run
  　
  ,#+name: infinite-worker-in-ruby
  ,#+begin_src ruby
  $stdout.sync = true
  loop { puts "working!"; sleep 1; }
  ,#+end_src
  　
  ,#+name: infinite-worker-in-python
  ,#+begin_src python :procs 2
  import sys
  import time
  
  while True:
    print "working too"
    sys.stdout.flush()
    time.sleep(1)
  ,#+end_src
#+end_src

    The above example can be run with the following:
    
    : org-run procfile-example.org

    Sample output of the run:

#+begin_src sh
[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- started with pid 19648
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- started with pid 19649
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- started with pid 19650
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- working too
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- working too
[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- working!
[2014-06-07T18:05:49 +0900] infinite-worker-in-python:1 -- working too
[2014-06-07T18:05:49 +0900] infinite-worker-in-python:2 -- working too
#+end_src

*** Sequential runs

    In case the code blocks form part of a runlist that should be 
    ran sequentially, it is possible to do this by specifying the
    ~runmode=sequential~ option.

#+begin_src sh
  ,#+TITLE:   Sample sequential run
  ,#+runmode: sequential
  　
  ,#+name: first
  ,#+begin_src sh
  echo "first"
  ,#+end_src
  　
  ,#+name: second
  ,#+begin_src sh
  echo "second"
  ,#+end_src
  　
  ,#+name: third
  ,#+begin_src sh
  echo "third"
  ,#+end_src
  　
  ,#+name: fourth
  ,#+begin_src sh
  echo "fourth"
  ,#+end_src
#+end_src

  In order to specify that this is to be run sequentially, 
  we set the runmode option in the command line:

  : org-run runlist-example.org --runmode=sequential

  Another way of specifying this is via the Org mode file itself:

  #+begin_src org
  ,#+TITLE:   Defining the runmode as an in buffer setting 
  ,#+runmode: sequential
  #+end_src

  Sample output:

#+begin_src sh
  [2014-06-07T18:10:33 +0900] first     -- started with pid 19845
  [2014-06-07T18:10:33 +0900] first     -- first
  [2014-06-07T18:10:33 +0900] first     -- exited with code 0
  [2014-06-07T18:10:33 +0900] second    -- started with pid 19846
  [2014-06-07T18:10:33 +0900] second    -- second
  [2014-06-07T18:10:33 +0900] second    -- exited with code 0
  [2014-06-07T18:10:33 +0900] third     -- started with pid 19847
  [2014-06-07T18:10:33 +0900] third     -- third
  [2014-06-07T18:10:33 +0900] third     -- exited with code 0
  [2014-06-07T18:10:33 +0900] fourth    -- started with pid 19848
  [2014-06-07T18:10:33 +0900] fourth    -- fourth
  [2014-06-07T18:10:33 +0900] fourth    -- exited with code 0
#+end_src

*** Configuration management runs

    For example, using Org Babel tangling functionality we can spread
    config files on a server by writing the following on a ~server.org~ file...

#+begin_src sh
　
Configuration for a component that shoul be run in multitenant mode:
　
,#+begin_src yaml :tangle /etc/component.yml
multitenant: false
status_port: 10004
,#+end_src
#+end_src

    Then run:

   : sudo org-tangle server.org
   
   *Note:* The above tangle command is a reimplementation of the tangling functionality from Org mode using Ruby.
   If you are interested in tangling from the command line without losing functionality and have available a local Emacs Org mode install,
   the following project can be useful: <https://github.com/ngirard/org-noweb>

*** Idempotent runs

    A run can have idempotency checks (similar to how the execute resource from [[http://docs.opscode.com/resource_execute.html][Chef]] works).

    An example of this, would be when installing packages.  In this example,
    we want to install the =build-essential= package once, and skip it in following runs:

#+begin_src sh
  ,** Installing the dependencies
  　
  Need the following so that ~bundle install~ can compile 
  the native extensions correctly.
  　　
  ,#+name: build-essential-installed
  ,#+begin_src sh
  dpkg -l | grep build-essential
  ,#+end_src
  　 　
  ,#+name: build_essentials
  ,#+begin_src sh :unless build-essential-installed
  apt-get install build-essential -y
  ,#+end_src
　　
  ,#+name: bundle_install
  ,#+begin_src sh
  cd project_path
  bundle install
  ,#+end_src
#+end_src

  Furthermore,since we are using Org mode syntax, it is possible 
  to reuse this setup file by including it into another Org file:

#+begin_src sh
  ,#+TITLE: Another setup
  　
  Include the code blocks from the server into this:
 　　
  ,#+include: "server.org"
  　
  ,#+name: install_org_mode
  ,#+begin_src sh
  apt-get install org-mode -y
  ,#+end_src
#+end_src
#+end_src

Since this a run that involves converging into a state,
it would be run sequentially with idempotency checks applied:

: sudo org-converge setup.org

*** Dependencies based runs

    In this type of runs we use the =:after= and =:before=
    header arguments to specify the prerequisites for a code block to run,
    similar to some of the functioality provided by tools like =rake= 
    (Behind the scenes, these arguments create =Rake= tasks)

    In order for this kind of run to work, it has to be specified
    what is the task that we are converging to by using 
    the =#+final_task:= in buffer setting:

#+begin_src sh
  ,#+TITLE:           Linked tasks example
  ,#+runmode:         tasks
  ,#+final_task:      final
  　
  ,#+name: second
  ,#+begin_src sh :after first
  for i in `seq 5 10`; do 
    echo $i >> out.log
  done
  ,#+end_src
  　
  ,#+name: first
  ,#+begin_src ruby
  5.times { |n| File.open("out.log", "a") {|f| f.puts n } }
  ,#+end_src
  　
  ,#+name: final
  ,#+begin_src python :after second :results output
  print "Wrapping up with Python in the end"
  f = open('out.log', 'a')
  f.write('11')
  f.close()
  ,#+end_src
  　
  ,#+name: prologue
  ,#+begin_src sh :before first :results output
  echo "init" > out.log
  ,#+end_src
#+end_src

    : org-run chained-example.org --runmode=chained

    Instead of using =--runmode= options, it is also possible to just declare in buffer
    that the Org file should be run chained mode.

    #+begin_src org
    ,#+TITLE:   Defining the runmode as an in buffer setting 
    ,#+runmode: chained
    #+end_src
    
    Sample output:

#+begin_src sh
[2014-06-07T18:14:25 +0900] Running final task: final
[2014-06-07T18:14:25 +0900] prologue  -- started with pid 20035
[2014-06-07T18:14:25 +0900] prologue  -- exited with code 0
[2014-06-07T18:14:25 +0900] first     -- started with pid 20036
[2014-06-07T18:14:26 +0900] first     -- exited with code 0
[2014-06-07T18:14:26 +0900] second    -- started with pid 20038
[2014-06-07T18:14:26 +0900] second    -- exited with code 0
[2014-06-07T18:14:26 +0900] final     -- started with pid 20040
[2014-06-07T18:14:26 +0900] final     -- Wrapping up with Python in the end
[2014-06-07T18:14:26 +0900] final     -- exited with code 0
#+end_src

*** Remote runs

    For any of the cases above, it is also possible to specify
    whether the code blocks should be run remotely on another node.
    This is done by using =:dir= in the code block header argument.

#+begin_src sh
  ,#+sshidentityfile:	vagrant/keys/vagrant
  ,#+name: remote-bash-code-block
  　
  ,#+begin_src sh :results output :dir /vagrant@127.0.0.1#2222:/tmp
  random_number=$RANDOM
  for i in `seq 1 10`; do echo "[$random_number] Running script is $0 being run from `pwd`"; done
  ,#+end_src
#+end_src

  Note that in order for the above to work, it is also needed to set identity to be used by ssh.

*** Asserted runs

    In case the Org mode file has a results block which represents the expected result, 
    there is an ~org-spec~ command which can be useful to check whether there was a change
    that no longer makes the results from the Org file valid. Example:

#+begin_src sh
  ,#+TITLE:   Expected results example
  　　
  ,#+name: hello
  ,#+begin_src ruby :results output
  10.times do 
    puts "hola"
  end
  ,#+end_src
  　　
  ,#+RESULTS: hello
  ,#+begin_example
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  hola
  ,#+end_example
#+end_src

  We can be able to verify whether this is still correct by running:

  : org-spec test.org

#+begin_src sh
Checking results from 'hello' code block:	OK
#+end_src

  As an example, let's say that the behavior of the original code block changed,
  and now says hello 5 times instead. 
  In that case the output would be as follows:

#+begin_src diff
Checking results from 'hello' code block:	DIFF
@@ -1,11 +1,6 @@
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
+hello
+hello
+hello
+hello
+hello
#+end_src

** Contributing

The project is still in very early development and a proof of concept at this moment.
But if you feel that it is interesting enough, please create a ticket to start
the discussion.