# Creating a Security Test
This document walks through the steps for creating a security test and integrating it into a Norad development box. It leverages the norad command line tool, which can be installed by doing:
```
$ gem install norad_cli
```
Other dependencies include a working [Norad Development Box](https://norad-dev-box-url) and [Docker](https://www.docker.com/community-edition).
# Security Test Development Steps
Assuming the dependencies described above are installed. Creating and testing a security test is straightforward. First, decided if you will be starting an entirely new repository to house security tests. If a new repository will be created, decided on a descriptive name, for instance asig-security, and use the norad repo create command:
```
$ norad repo create asig-security
Initializing a new norad security test repository
create asig-security/base
create asig-security/spec
License the repo under Apache 2? y
Who is the copyright holder (Default: Cisco Systems, Inc.)?
create asig-security/LICENSE
create asig-security/.gitlab.ci.yml
create asig-security/.gitignore
create asig-security/CONTRIBUTING.md
create asig-security/README.md
```
The command asks if the tests should be licensed under Apache 2 and for the copyright holder. When in doubt, answer no to the licensing question. The 'repo create' command creates a new git repository conforming to the Norad standards for layout and ci.
If you will not be creating a new repostiory simply clone an existing Norad security test repo.
```
$ git clone git@norad-gitlab.cisco.com:norad/asig-hackathon-security-tests.git
```
Once a repository is selected, either by creating a new one or cloning an existing one, change into the root of that repository before starting test development.
```
$ cd asig-hackathon-security-tests
```
Now, for this example, we will be creating a security test using the ping utility. The norad CLI tool includes a subcommand, sectest, to aid developers with creating, testing, validating, and integrating a security tool into Norad. Help for the sectest subcommand is available:
```
$ norad help sectest
Commands:
norad sectest build # Build all sectest images and specs for the entire repository
norad sectest build:all SECTESTNAME # Build sectest images for SECTESTNAME and all testing images for SECTESTNAME
norad sectest build:image SECTESTNAME # Build the docker image for the security test SECTESTNAME
norad sectest build:specs SECTESTNAME # Build the spec images (test images) for the security test SECTESTNAME
norad sectest execute SECTESTNAME # Execute SECTESTNAME against an arbitrary target
norad sectest help [COMMAND] # Describe subcommands or one specific subcommand
norad sectest scaffold TESTNAME # Create a new security test with standard files + testing
norad sectest seed # Create the containers.rb seed to import into the api
norad sectest spec # Run all rspec tests for the entire repo (all sectests)
norad sectest spec:image SECTESTNAME # Run the rspec tests for SECTESTNAME
norad sectest validate # Validate all manifest.yml and readme.md
norad sectest validate:image SECTESTNAME # Validate SECTESTNAME manifest.yml and readme.md
```
To start off, we use the scaffold command to create all of the required files to create a security test. The scaffold subcommand includes several options:
```
$ norad sectest scaffold --help
Usage:
norad scaffold TESTNAME
Options:
-t, [--test-type=TEST_TYPE] # The security test type, Options: [authenticated|web_application|brute_force|ssl_crypto|ssh_crypto|whole_host]
# Default: whole_host
-r, [--registry=REGISTRY] # The Docker registry to store docker images
# Default: norad-registry.cisco.com:5000
-v, [--version=VERSION] # The version of the security test
# Default: latest
-b, [--base-image=BASE_IMAGE] # Base Docker image to use (i.e. FROM field in the Dockerfile)
# Default: norad-registry.cisco.com:5000/norad:0.0.1
-c, [--configurable], [--no-configurable] # Is the security test configurable (e.g. Qualys username and password)
Create a new security test with standard files + testing
```
First, the TESTNAME will be whatever name you would like to assign to the security test. For this example, we will use pinger. The TESTNAME should not collide with any folder names under the sectests folder. Next, you must decide on the security test's test-type, registry, version, base-image, and if it will be configurable options. The test-type defines ......., available test tests are:
* authenticated:
* web_application:
* brute_force:
* ssl_crypto:
* ssh_crypto:
* whole_host:
For the pinger security test, we are simply checking the host, so the test-type be the default value, whole_host. The registry option permits the user to change where the security test will be stored, our test can use the default value (and most tests will). The version and base-image options allow you to set the initial version of the security test and a Docker image to start from. The default options are safe to use, unless, for the base image, you know of a different Docker image to use for a base. The configurable option sets whether the sectest will have additional command line options w/ defaults set in its manifest.yml. This new utility will not have any additional options, so there is no need to use the -c flag.
Therefore, for the new pinger sectest, the scaffold command is:
```
$ norad sectest scaffold pinger
create sectests/pinger/Dockerfile
create sectests/pinger/README.md
create sectests/pinger/manifest.yml
create sectests/pinger/pinger-wrapper.rb
create spec/pinger/pinger_spec.rb
create spec/pinger/targets/Dockerfile.secure
create spec/pinger/targets/Dockerfile.vulnerable
```
The scaffolding creates a new directory in sectests, pinger, housing the files necessary to create a new test. These files are:
* pinger/Dockerfile: alter this file to install new software packages
* pinger/README.md: describes the new security test and any options. This file is used to generate user facing documentation
* pinger/manifest.yml: YAML file to describe the security test. This file is used for importing information into a Norad instance
* pinger/pinger-wrapper.rb: A ruby script where execution begins when the security test is executed.
Scaffolding also creates tests and test targets for the new security test. The test targets, a secure and a vulnerable target, are defined by the Dockerfiles in spec/pinger/targets/. The [rspec](http://rspec.info/) tests to run for the new security test (to check if it is working properly) are defined in the file spec/pinger/pinger_spec.rb.
After a new security test is scaffolded, a developer alters the corresponding Dockerfile, (e.g. sectests/pinger/Dockerfile) to install any required dependencies. For instance, maybe you need to install a new tool (e.g. nmap) to develop the security test. In this case, the developer would add a new RUN command into the Dockerfile (sectests/pinger/Dockerfile):
```
RUN apt-get -y install nmap
```
Please refer to the [Dockerfile reference](https://docs.docker.com/engine/reference/builder/#parser-directives) for all the available commands.
Once you have added all of the necessary commands to the new Dockerfile (sectests/pinger/Dockerfile), then you can try building the new sectest container with the norad sectest build command from the repo's root directory. For example, to build our test security tool:
```
$ norad sectest build:image pinger
Building image pinger...
{"stream":"Step 1/5 : FROM norad-registry.cisco.com:5000/norad:0.0.1\n"}
{"stream":" ---\u003e e01961118951\n"}
{"stream":"Step 2/5 : COPY pinger-wrapper.rb /pinger-wrapper.rb\n"}
{"stream":" ---\u003e fde55e5e2b8a\n"}
{"stream":"Removing intermediate container 6ab5c3c7f4c1\n"}
{"stream":"Step 3/5 : RUN chmod 755 /pinger-wrapper.rb\n"}
{"stream":" ---\u003e Running in 2431f94971e2\n"}
{"stream":" ---\u003e 3294126d1b23\n"}
{"stream":"Removing intermediate container 2431f94971e2\n"}
{"stream":"Step 4/5 : WORKDIR /\n"}
{"stream":" ---\u003e 92ef3e0a4eb9\n"}
{"stream":"Removing intermediate container c83183b4a0f7\n"}
{"stream":"Step 5/5 : ENTRYPOINT /pinger-wrapper.rb\n"}
{"stream":" ---\u003e Running in 8b83ca556636\n"}
{"stream":" ---\u003e 53b3a4bfbbf9\n"}
{"stream":"Removing intermediate container 8b83ca556636\n"}
{"stream":"Successfully built 53b3a4bfbbf9\n"}
```
The resulting image is stored on your machine and can be viewed by using docker image list command:
```
$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
norad-registry.cisco.com:5000/pinger latest 53b3a4bfbbf9 About a minute ago 496 MB
```
After that, update the manifest.yml (e.g. sectests/pinger/manifest.yml) and README.md (e.g. sectests/pinger/README.md) files. Updates to these files can be validated using the sectest validate commands:
```
$ norad sectest validate:image pinger
Looking for README.md in: sectests/pinger...
OK
Testing for variant READMEs...
No variants for this tool
OK
.
Finished in 0.00142 seconds (files took 0.14096 seconds to load)
1 example, 0 failures
Looking for README.md in: sectests/pinger...
OK
Testing for variant READMEs...
No variants for this tool
OK
.Looking for valid manifest in: ...
OK
Testing for valid name...
OK
Testing for existence of version...
OK
Testing for existence of registry...
OK
Testing for valid prog args...
OK
Testing for test types...
OK
Testing for configurability...
OK
Testing for variants...
No variants for this repo
OK
.
Finished in 0.00587 seconds (files took 0.1604 seconds to load)
3 examples, 0 failures
```
Next, the security tools wrapper script should be updated to perform the actual security checks (e.g. sectests/pinger/pinger-wrapper.rb). The wrapper script is responsible for spawning any tools (e.g. ping, nmap, etc.), parsing the results, and posting them back to Norad. For the pinger example, we can use an already prepared wrapper script:
```ruby
#!/usr/bin/env ruby
require 'norad_beacon'
def run(args)
timeout = 3600 # set timeout for runner to 1 hour
# Allocate a runner
runner = NoradBeacon::Runner.new('ping', [ '-c', '1', args].flatten, timeout)
# Execute the runner
runner.execute(true)
# Ensure the tool created results
runner.parse_results do |fh|
id = '1'
title = 'Ping Test'
description = 'This test will ping a host to determine if it is alive.'
cvss = 'no_impact'
raw_output = fh.read() + "Exit Code: #{runner.exit_code}"
status = raw_output =~ /100% packet loss/ ? 'fail' : 'pass'
# Add the result to the runner's result set
# Note: Multiple results can be added, they will show up individually
runner.result_set.add(NoradBeacon::Result.new(id, status, raw_output, title, description, cvss))
end
rescue Exception => e
puts "An exception occurred: #{e.inspect}"
puts e.backtrace
status = 'error'
raw_output = 'Internal error occurred'
title = 'Failed to run the tests'
description = 'Internal error occurred'
runner.result_set.add(NoradBeacon::Result.new('0', status, raw_output, title, description))
ensure
# Save the results to Norad
NoradBeacon::NoradAPI.post_results(runner.result_set)
end
run(ARGV)
```
For this walkthrough, copy the above code to sectests/pinger/pinger-wrapper.rb. The pinger docker image will need to rebuilt to incorporate the changes to the wrapper script. Simply rerun the norad sectest build:image pinger command.
After the build succeeds, it is time to test the new security tests. There are two methods for testing. The first method is to run the security test container against a target, and the second is to use Norad's rspec testing system. To run against a target, use the command:
FIXME: norad sectest execute SECTESTNAME ARGUMENTS
The second method is more involved but necessary for the image to be merged into Norad's approved images. Norad's CI system runs validation tests against security tests to ensure the test performs as expected. The CI system automatically spins up a vulnerable image (defined in spec/SECTESTNAME/targets/Dockerfile.vulnerable) and a secure image (defined in spec/SECTESTNAME/targets/Dockerfile.secure) where SECTESTNAME is the name of the security test (e.g. pinger). The secure and vulnerable images are defined by the developer. For your testing, please define a vulnerable and a secure container by altering their respective Dockerfiles. When you are ready to build these two images, norad sectest build:specs command can be used:
```
$ norad sectest build:specs pinger
.....
Building image spec/pinger/targets/Dockerfile.secure...
{"stream":"Step 1/5 : FROM ubuntu:14.04\n"}
{"stream":" ---\u003e 7c09e61e9035\n"}
{"stream":"Step 2/5 : RUN apt-get -y update\n"}
{"stream":" ---\u003e Using cache\n"}
{"stream":" ---\u003e b2bc3253c8c3\n"}
{"stream":"Step 3/5 : RUN apt-get -y install openssh-server\n"}
{"stream":" ---\u003e Using cache\n"}
{"stream":" ---\u003e b23b83d7dc9c\n"}
{"stream":"Step 4/5 : RUN mkdir /var/run/sshd \u0026\u0026 chmod 0755 /var/run/sshd\n"}
{"stream":" ---\u003e Using cache\n"}
{"stream":" ---\u003e 3f7fe39c33ec\n"}
{"stream":"Step 5/5 : CMD /usr/sbin/sshd -D\n"}
{"stream":" ---\u003e Using cache\n"}
{"stream":" ---\u003e 28da9b415f86\n"}
{"stream":"Successfully built 28da9b415f86\n"}
....
```
The tests to run against the secure and vulnerable images are defined by the file spec/SECTESTNAME/SECTESTNAME_spec.rb (e.g. spec/pinger/pinger_spec.rb). These tests are written in (rspec)[http://rspec.info/], and the scaffolding command created some example tests for reference. For this example, the default rspec tests will work. For now, run the nroad cli testing command to view the results:
```
$ norad sectest spec:image pinger
```
At the end of the run, you will see:
```
Finished in 33.22 seconds (files took 0.14089 seconds to load)
6 examples, 1 failure
Failed examples:
rspec ./spec/pinger/pinger_spec.rb:24 # Pinger for vulnerable machine should report a failure
```
Digging into the reason why the test "Pinger for vulnerable machine should report a failure" failed, the console output shows:
```
expected: "fail"
got: "pass"
```
Since this test is merely running a ping to check connectivity, the vulnerable host is up during the test, so the parsing logic in sectests/pinger/pinger-wrapper.rb, specifically:
```ruby
raw_output = fh.read() + "Exit Code: #{runner.exit_code}"
status = raw_output =~ /100% packet loss/ ? 'fail' : 'pass'
```
is marking the status incorrectly for the vulnerable machine. If you would like to see verbose logging from tests (includes stdout/stderr from the containers as wrapper scripts), run use the spec:image verbose flag:
```
$ norad sectest spec:image -v pinger
```
Once all of the tests have passed, it is time to generate a seed file to import into a development machine. A seed file can be generated by running:
```
$ norad sectest seed
```
The seed command generates a file, `containers.rb`, which can be used to import the tests into the Norad Web Application (e.g. [Norad Dev-Box](https://gitlab.com/norad/dev-box) or
production machine). Follow the instructions below to import the file:
* Upload `containers.rb` to your dev-box or production machine.
* From the directory containing the `api` code run the following:
$ rails runner path/to/your/containers.rb
# Helpful Docker Commands
* `docker images` - Determine the images available
* `docker ps` - Show containers currently running
* `docker kill 8b83ca556636` - Remove a currently running container
# References
* [Docker Command Line Reference](https://docs.docker.com/engine/reference/commandline/docker/)
* [Dockerfile Reference](https://docs.docker.com/engine/reference/builder/)
* [RSpec](http://rspec.info/)