# Ufo - Build Docker Containers and Ship Them to AWS ECS ## Quick Introduction Ufo is a simple tool that makes building and shipping Docker containers to [AWS ECS](https://aws.amazon.com/ecs/) super easy. A summary of steps `ufo ship` takes: 1. builds a docker image.  2. generates and registers the ECS template definition.  3. deploys the ECS template definition to the specified service. Ufo deploys a task definition that is created via a template generator which is fully controllable. ## Installation $ gem install ufo You will need a working version of docker installed as ufo calls the docker command. ## Usage When using ufo if the ECS service does not yet exist, it will automatically be created for you. If you are relying on this tool to create the cluster, you still need to associate ECS Container Instances to the cluster yourself. First initialize ufo files within your project. Let's say you have an `hi` app. ``` $ git clone https://github.com/tongueroo/hi $ cd hi $ ufo init --app hi --cluster stag --image tongueroo/hi Setting up ufo project... created: ./bin/deploy exists: ./Dockerfile created: ./ufo/settings.yml created: ./ufo/task_definitions.rb created: ./ufo/templates/main.json.erb created: ./.env Starter ufo files created. $ ``` Take a look at the `ufo/settings.yml` file to see that it holds some default configuration settings so you don't have to type out these options every single time. ```yaml image: tongueroo/hi service_cluster: default: stag # default cluster hi-web: stag hi-clock: stag hi-worker: stag ``` The `image` value is the name that ufo will use for the Docker image name. The `service_cluster` mapping provides a way to set default service to cluster mappings so that you do not have to specify the `--cluster` repeatedly. Example: ``` ufo ship hi-web --cluster hi-cluster ufo ship hi-web # same as above because it is configured in ufo/settings.yml ufo ship hi-web --cluster special-cluster # overrides any setting default fallback. ``` ## Task Definition ERB Template and DSL Generator Ufo task definitions are is written in a template generator DSL to provide full control of the task definition that gets uploaded for each service. We'll go over a simple example. Here is the ERB template for `ufo/templates/main.json.erb`: ```json { "family": "<%= @family %>", "containerDefinitions": [ { "name": "<%= @name %>", "image": "<%= @image %>", "cpu": <%= @cpu %>, <% if @memory %> "memory": <%= @memory %>, <% end %> <% if @memory_reservation %> "memoryReservation": <%= @memory_reservation %>, <% end %> <% if @container_port %> "portMappings": [ { "containerPort": "<%= @container_port %>", "protocol": "tcp" } ], <% end %> "command": <%= @command.to_json %>, <% if @environment %> "environment": <%= @environment.to_json %>, <% end %> <% if @awslogs_group %> "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "<%= @awslogs_group %>", "awslogs-region": "<%= @awslogs_region || 'us-east-1' %>", "awslogs-stream-prefix": "<%= @awslogs_stream_prefix %>" } }, <% end %> "essential": true } ] } ``` The instance variable values are specified in `ufo/task_definitions.rb`. Here's the ```ruby task_definition "hi-web" do source "main" # will use ufo/templates/main.json.erb variables( family: task_definition_name, # image: tongueroo/hi:ufo-[timestamp]=[sha] image: helper.full_image_name, environment: env_file('.env.prod') name: "web", container_port: helper.dockerfile_port, command: ["bin/web"] ) end ``` As you can see above, the task\_definitions.rb file has some special variables and helper methods available. These helper methods provide useful contextual information about the project. For example, one of the variable provides the exposed port in the Dockerfile of the project. Here is a list of the important ones: * **helper.full\_image\_name** — The full docker image name that ufo builds. The “base” portion of the docker image name is defined in ufo/settings.yml. For example, the base portion is "tongueroo/hi" and the full image name is tongueroo/hi:ufo-[timestamp]-[sha]. So the base does not include the Docker tag and the full image name does include the tag. * **helper.dockerfile\_port** — Exposed port extracted from the Dockerfile of the project.  * **env_file** — This method takes an .env file which contains a simple key value list of environment variables and converts the list to the proper task definition json format. The 2 classes which provide these special helper methods are in [ufo/dsl.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl.rb) and [ufo/dsl/helper.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl/helper.rb). Refer to these classes for the full list of the special variables and methods. ### Customizing Templates If you want to change the template then you can follow the example in the generated ufo files. For example, if you want to create a template for the worker service. 1. Create a new template under ufo/templates/worker.json.erb. 2. Change the source in the `task_definition` using "worker" as the source. 3. Add variables. ### ufo ship Ufo uses the aforementioned files to build task definitions and then ship to them to AWS ECS. To execute the ship process run: ```bash ufo ship hi-web --cluster stag ``` Note, if you have configured `ufo/settings.yml` to map hi-web to the stag cluster using the service_cluster option the command becomes simply: ```bash ufo ship hi-web ``` When you run `ufo ship hi-web`: 1. It builds the docker image. 2. Generates a task definition and registers it. 3. Updates the ECS service to use it. If the ECS service hi-web does not yet exist, ufo will create the service for you. If the service has a container name web, you'll get prompted to create an ELB and specify a target group arn. The ELB and target group must already exist. You can bypass the prompt and specify the target group arn as part of the command. The elb target group can only be associated when the service gets created for the first time. If the service already exists then the `--target-group` parameter just gets ignored and the ECS task simply gets updated. Example: ```bash ufo ship hi-web --target-group=arn:aws:elasticloadbalancing:us-east-1:12345689:targetgroup/hi-web/jdisljflsdkjl ``` ### Shipping Multiple Services with bin/deploy A common pattern is to have 3 processes: web, worker, and clock. This is very common in rails applcations. The web process handles web traffic, the worker process handles background job processing that would be too slow and potentially block web requests, and a clock process is typically used to schedule recurring jobs. These processes use the same codebase, or same docker image, but have slightly different run time settings. For example, the docker run command for a web process could be [puma](http://puma.io/) and the command for a worker process could be [sidekiq](http://sidekiq.org/). Environment variables are sometimes different also. The important key is that the same docker image is used for all 3 services but the task definition for each service is different. This is easily accomplished with the `bin/deploy` wrapper script that the `ufo init` command initially generates. The starter script example shows you how you can use ufo to generate one docker image and use the same image to deploy to all 3 services. Here is an example `bin/deploy` script: ```bash #!/bin/bash -xe ufo ship hi-worker --cluster stag --no-wait ufo ship hi-clock --cluster stag --no-wait --no-docker ufo ship hi-web --cluster stag --no-docker ``` The first `ufo ship hi-worker` command build and ships docker image to ECS, but the following two `ufo ship` commands use the `--no-docker` flag to skip the `docker build` step. `ufo ship` will use the last built docker image as the image to be shipped. For those curious, this is stored in `ufo/docker_image_name_ufo.txt`. ### Service and Task Names Convention Ufo assumes a convention that service\_name and the task\_name are the same. If you would like to override this convention then you can specify the task name. ``` ufo ship hi-web --task my-task ``` This means that in the task_definition.rb you will also defined it with `my-task`. For example: ```ruby task_definition "my-task" do source "web" # this corresponds to the file in "ufo/templates/web.json.erb" variables( family: "my-task", .... ) end ``` ### Running Tasks in Pieces The `ufo ship` command goes through a few stages: building the docker image, registering the task defiintions and updating the ECS service. The CLI exposes each of the steps as separate commands. Here is now you would be able to run each of the steps in pieces. Build the docker image first. ```bash ufo docker build ufo docker build --push # will also push to the docker registry ``` Build the task definitions. ```bash ufo tasks build ufo tasks register # will register all genreated task definitinos in the ufo/output folder ``` Skips all the build docker phases of a deploy sequence and only update the service with the task definitions. ```bash ufo ship hi-web --no-docker ``` Note if you use the `--no-docker` option you should ensure that you have already push a docker image to your docker register. Or else the task will not be able to spin up because the docker image does not exist. I recommend that you normally use `ufo ship` most of the time. ## Automated Docker Images Clean Up Ufo can be configured to automatically clean old images from the ECR registry after the deploy completes. I normally set `~/.ufo/settings.yml` like so: ```yaml ecr_keep: 3 ``` Automated Docker images clean up only works if you are using ECR registry. ## Scale There is a convenience wrapper that simple executes `aws ecs update-service --service [SERVICE] --desired-count [COUNT]` ``` ufo scale hi-web 1 ``` ## Destroy To scale down the service and destroy it: ``` ufo destroy hi-web ``` ### More Help ``` ufo help ``` ## Contributing Bug reports and pull requests are welcome on GitHub at [https://github.com/tongueroo/ufo/issues](https://github.com/tongueroo/ufo/issues).