README.md in elasticity-2.5.6 vs README.md in elasticity-2.6
- old
+ new
@@ -1,15 +1,13 @@
[](http://badge.fury.io/rb/elasticity)
-**(February 3, 2013)** Taking requests! I have a few ideas for what might be cool features though I'd rather work on what the community wants. Go ahead and file an issue!
+**(August 17, 2013)** Taking requests! I have a few ideas for what might be cool features though I'd rather work on what the community wants. Go ahead and file an issue!
Elasticity provides programmatic access to Amazon's Elastic Map Reduce service. The aim is to conveniently abstract away the complex EMR REST API and make working with job flows more productive and more enjoyable.
-**Travis has been flaky, failing builds before they start. "Trust me", it's green :)**
+[](http://travis-ci.org/rslifka/elasticity) 1.9.2, 1.9.3, 2.0.0
-[](http://travis-ci.org/rslifka/elasticity) REE, 1.8.7, 1.9.2, 1.9.3
-
Elasticity provides two ways to access EMR:
* **Indirectly through a JobFlow-based API**. This README discusses the Elasticity API.
* **Directly through access to the EMR REST API**. The less-discussed hidden darkside... I use this to enable the Elasticity API. RubyDoc can be found at the RubyGems [auto-generated documentation site](http://rubydoc.info/gems/elasticity/frames). Be forewarned: Making the calls directly requires that you understand how to structure EMR requests at the Amazon API level and from experience I can tell you there are more fun things you could be doing :) Scroll to the end for more information on the Amazon API.
@@ -20,11 +18,11 @@
```
or in your Gemfile
```
-gem 'elasticity', '~> 2.5'
+gem 'elasticity', '~> 2.6'
```
This will ensure that you protect yourself from API changes, which will only be made in major revisions.
# Roughly, What Am I Getting Myself Into?
@@ -61,14 +59,15 @@
1. Create a job flow.
1. Specify options.
1. (optional) Configure instance groups.
1. (optional) Add bootstrap actions.
- 1. Add steps.
+ 1. (optional) Add steps.
1. (optional) Upload assets.
1. Run the job flow.
1. (optional) Add additional steps.
+ 1. (optional) Wait for the job flow to complete.
1. (optional) Shutdown the job flow.
## 1 - Create a Job Flow
Only your AWS credentials are needed.
@@ -105,10 +104,11 @@
jobflow.action_on_failure = 'TERMINATE_JOB_FLOW'
jobflow.keep_job_flow_alive_when_no_steps = false
jobflow.ami_version = 'latest'
jobflow.hadoop_version = '1.0.3'
jobflow.log_uri = nil
+jobflow.enable_debugging = false # Requires a log_uri to enable
jobflow.ec2_key_name = nil
jobflow.ec2_subnet_id = nil
jobflow.visible_to_all_users = false
jobflow.placement = 'us-east-1a'
@@ -197,13 +197,13 @@
```ruby
action = Elasticity::HadoopFileBootstrapAction.new('s3n://my-bucket/job-config.xml')
jobflow.add_bootstrap_action(action)
```
-## 5 - Add Steps
+## 5 - Add Steps (optional)
-Each type of step has ```#name``` and ```#action_on_failure``` fields that can be overridden. Apart from that, steps are configured differently - exhaustively described below.
+Each type of step has ```#name``` and ```#action_on_failure``` fields that can be specified. Apart from that, steps are configured differently - exhaustively described below.
### Adding a Pig Step
```ruby
# Path to the Pig script
@@ -315,10 +315,28 @@
## 8 - Add Additional Steps (optional)
Steps can be added to a running jobflow just by calling ```#add_step``` on the job flow exactly how you add them prior to submitting the job.
-## 9 - Shut Down the Job Flow (optional)
+## 9 - Wait For the Job Flow to Complete (optional)
+
+Elasticity has the ability to block until the status of a job flow is not STARTING or RUNNING. There are two flavours. Without a status callback:
+
+```ruby
+# Blocks until status changes
+jobflow.wait_for_completion
+```
+
+And with a status callback, providing the elapsed time and an instance of ```Elasticity::JobFlowStatus``` so you can inspect the progress of the job.
+
+```ruby
+# Blocks until status changes, calling back every 60 seconds
+jobflow.wait_for_completion do |elapsed_time, job_flow_status|
+ puts "Waiting for #{seconds}, jobflow status: #{status.state}"
+end
+```
+
+## 10 - Shut Down the Job Flow (optional)
By default, job flows are set to terminate when there are no more running steps. You can tell the job flow to stay alive when it has nothing left to do:
```ruby
jobflow.keep_job_flow_alive_when_no_steps = true