README.rdoc in rubu-0.0.1 vs README.rdoc in rubu-0.0.2

@@ -4,59 +4,287 @@
 programs. {Rubu} is in practice a replacement for Make and Rake type
 of tools. {Rubu} is targeted to provide means for creating flexible
 build environments.
 
 Make and Rake are rule and recipe based, and they are very effective
-when the build process can be captured to these rules and are fairly
+when the build process can be captured to these rules and is fairly
 fixed. {Rubu} includes declarations which correspond to rules, but the
 declarations are lower level and provide more control over the
 behavior. {Rubu} also provides direct control over when the rules are
 executed.
 
 Make and Rake are more compact for the simplest build environments,
 but there is a break-even point where {Rubu} becomes more
 convenient. This is likely to happen when the user needs many
-exceptions to basic rules, also when other tasks than just build task
+exceptions to basic rules, also when other tasks than just build tasks
 are needed.
 
 {Rubu} library can be used from any Ruby program, since it is just a
 library. It can also be part of a larger program as less significant
 part of the overall functionality. From maintenance point of view this
 can be a win in many cases.
 
+The easiest way to get started with {Rubu} is to study examples (See:
+"Example" chapter below). However for longer term the main concepts
+are important as well (See: "Concepts" chapter below).
 
-== Concepts
 
-TBD
-
-
 == Example
 
 Example of {Rubu} build program is placed in:
 
     example/bin/rubu_example
 
 It builds a "hello world" type program which is split into two C
 source files. The main file is hand written and the other file is
 generated with a script. File generation is part of the build program.
 
-Various executions of the build is captured in:
+Different execution variations of the build program is captured in:
 
     example/runme
 
 Please see example executions within the script and run the script in
 the "example" directory:
 
     cd example
     runme
 
-Also take a look into the build program:
+Also take a look into the build program itself:
 
     cat example/bin/rubu_example
 
 There are comments which highlight purpose of the build program
 content.
 
+
+== Concepts
+
+{Rubu} Build Program, RBP, is a Ruby program that uses the {Rubu}
+library. Typically command line options are accepted to give options
+for the build process.
+
+Build programs are about collecting source files and transforming them
+into targets. This process is explicit in {Rubu}. Each file, whether
+it's source or target, is collected within the {Rubu} build
+program. The files are converted into *Mark* objects. Basically *Mark*
+corresponds to a file, but it includes methods that make the build
+process convenient.
+
+*Step* is build action that transforms source file(s) to target
+file(s). For example, a C source file compilation is a *Step* where
+one source file is converted to a corresponding object file. *Step* is
+given source *Mark* and target *Mark* as arguments. Hence this type of
+*Step* a is one-to-one mapping.
+
+Linking multiple objects to an executable is a many-to-one
+mapping. This is performed by a *Step* where there are multiple
+sources and one target, i.e. objects and the executable respectively.
+
+*Step* class (or object) has "step" method which includes the *Move*
+that is needed to create the target. *Move* is either a Shell or a
+Ruby command. *Move* is executed only if necessary. Typically the
+*Step* is sensitive to source being newer that the target, just as
+with Make and Rake. However, it is possible to have a *Step* that is
+sensitive to file content, for example. Timestamp might change, but if
+the content is the same as previously, no *Move* is made.
+
+*Step* is typically a simple command. In some cases *Step* can include
+multiple *Moves*. These can be performed in sequence or in parallel,
+with a multicore CPU.
+
+*Trail* is a collection of *Steps*. *Trail* has a name, and it can be
+references by another *Trail*. *Trail* can be sequential or
+parallel. For example, C source to object compilation can be done in
+parallel, but linking of objects to executable must occur after all
+objects have been created.
+
+{Rubu} Build Program includes all of the above, and the last step is
+to select which *Trail* is executed. There should be at least
+"default" *Trail*, and typically there is also a "clean" *Trail*.
+
+{Rubu} supports the configuration spaces: *Order*, *Var*, and
+*Info*. *Order* is meant to be used for controlling {Rubu} itself. For
+example if user wants to see all executed command, then "verbose"
+option should be selected.
+
+*Var* is used to control how *Steps* behave and also details of *Move*
+internals. *Info* is meant for runtime generated information. The
+division is semantic and user might choose to combine everything to
+*Var* space, for example.
+
+To summarize and highlight the relationships between concepts, see the
+diagram below:
+
+    RBP => Trial+ => Step+ [s/p] => Move+ [s/p] => Source*, Target*
+        => Order+
+        => Var*
+        => Info*
+
+    * = Zero or more
+    + = One or more
+    s = Serial
+    p = Parallel
+
+RBP includes *Trials*, *Orders*, and possibly *Vars* and
+*Infos*. *Trial* includes one or more *Steps*, which can be executed
+in sequence or in parallel.
+
+*Step* includes one or more *Moves*, and they can also be sequential
+or parallel. *Move* has typically at least one *Source* and at least
+one *Target*. However, there are exceptions as well.
+
+
+The suggested ordering within {Rubu} Build Program is:
+
+[configuration] Process command line arguments and assign options that
+                are used to control the build process.
+
+[file collection] Collect source and target files either using Ruby
+                  with glob pattern, direct references, or through
+                  manifest files. Convert file names to *Mark*
+                  objects.
+
+[step definition] Define *Steps*. Inherit one of the *Step* types
+                  (classes) and define at least the "step" method.
+
+[trail definition] Define *Trails*. Create all possible flows that
+                   user wants to execute. Use hierarchy for your
+                   benefit in order to make RBP maintenance as light
+                   as possible.
+
+[trail selection] Find the selected *Trail* or *Trails* and execute.
+
+
+== Configuration
+
+*Order* space default are set by the {Rubu} library and user can
+override the default based on command line parameters if needed.
+
+Set default values for *Var* space. After default setting, override
+the defaults with possible control from command line arguments.
+
+Finally setup *Trails* by calling:
+
+    Trail.setup
+
+
+== File collection
+
+If the build environment has clear and clean directory structure, it
+is typically easiest to collect files programmatically. For example,
+to get all C source files:
+
+    cee_files = Mark.glob( "#{Var[:source_dir]}/*.c" )
+
+This creates a list of *Mark* objects which are source files. The
+corresponding object files can be created with "peer" method:
+
+    obj_files = cee_files.peer( Var[ :build_dir ], '.o' )
+
+This method invocation takes an Array of *Mark* objects and calls
+"peer" method for them. "peer" method changes the directory and
+extension, and returns a new *Mark* object. In some cases also the
+"basename" of the file is changed. This is achieved by giving the
+"peer" method a third argument specifying the new "basename".
+
+
+== Step definition
+
+*Step* is defined by inheriting of the {Rubu} *Step* classes and
+defining the custom "step" method. If a specific setup for the *Step*
+is required, then an additional "setup" method can be defined as well.
+
+    class CleanUp < StepAlways
+        def step
+            shrun "rm -f #{Var[ :build_dir ]}/*.o"
+        end
+    end
+
+
+"setup" method is typically used to change the command according to
+user configuration. "step" method is the action for *Step*. {Rubu}
+executes it, if necessary. For example, if *Step* is a *StepAged*
+class, then the "step" method is only called if the source *Mark* is
+newer than the target *Mark*.
+
+"step" method may include only one command or a set of commands. If
+one command is needed, it can be simply executed:
+
+    def step
+        shrun "rm -f #{Var[ :build_dir ]}/*.o"
+    end
+
+If there are multiple commands, then the commands can be grouped for
+sequential ("walk") or parallel execution ("fork"). 
+
+    def step
+        fork do
+            shuse "rm -f #{Info[ :gen_files ].rpath}"
+            shuse "rm -f #{Var[ :build_dir ]}/*.o"
+        end
+    end
+
+Note that the commands are defined with "shuse". The logic is that we
+don't execute the commands right away as in "shrun". Instead we just
+register the commands to the group. When the group definition is
+closed ("end" keyword), it is complete, and we execute it either
+sequentially or in parallel.
+
+
+== Trail definition
+
+*Trail* is a collection of *Steps* or other *Trails*, or a mix of
+both. For example, in order to compile a set of C source files to
+object files, we might define:
+
+    Fork.form 'compile-cee' do
+        GccCompileFile.usezip( cee_files, obj_files )
+    end
+
+This creates a parallel ("Fork") *Trail* called
+"compile-cee". Collection of *Steps* is created with "usezip"
+method. "usezip" performs two operations: it creates a zip type mix of
+"cee_files" and "obj_files". The "zip" method of Ruby Array is
+used. "zip" creates an Array of *Mark* pairs. Each pair is passed to a
+"GccCompileFile" *Step* object. Finally we have a collection of
+*Steps* which are executed in parallel when "compile-cee" is invoked.
+
+Here the defined *Trail* from above is referenced in order to create a
+*Trail* for the complete compile flow:
+
+    Walk.form 'default' do
+        pick 'compile-cee'
+        GccLinkExe.use( obj_files, exe_file )
+    end
+
+This *Trail* definition defines that we want to first execute
+"compile-cee" and then a *Step* which will create an executable from
+object file collection. "GccLinkExe" *Step* is called with "use"
+method, which will register the *Step* to the enclosing *Trail*,
+"default".
+
+
+== Trail selection
+
+The final stage in RBP execution is to use command line control for
+selecting the executed *Trails* and then execute them. The above
+definition (from previous section) is executed with:
+
+    Trail.run( 'default' )
+
+
+== Final notes
+
+{Rubu} treats single *Mark* objects and Array of *Mark* objects the
+"same" way. In practice this means that the most common methods for
+*Mark* objects are implemented for Array as well. Hence, you can call
+a *Mark* method for an Array, and they will actually be applied for
+each individual *Mark* in the collection.
+
+*Step* has sources and targets. Even if a step has only one source and
+one target, the actual container is an Array. If you want to refer to
+the source, i.e. the only existing source, you can use the "source"
+method. Same applies to targets (i.e. use the "target" method).
 
 
 == Testing
 
 For this version, example is the testcase.