# What tests can be run?

[[_TOC_]]

## The two types of QA tests

First of all, the first thing to choose is whether you want to run orchestrated
tests (various Docker containers are spun up and tests are run against them,
also from a specific Docker container) or instance-level tests (tests are run
from your host machine against a live instance: a local GDK installation or a staging/production instance).

Ultimately, orchestrated tests run instance-level tests, the difference being
that these tests are run from a specific Docker container instead of from your
host machine.

## Orchestrated tests

Orchestrated tests are run with the `gitlab-qa` binary (from the
`gitlab-qa` gem), or in the `gitlab-qa` project, with the `exe/gitlab-qa` binary
(useful if you're working on the `gitlab-qa` project itself and want to test
your changes).

These tests spin up Docker containers specifically to run tests against them.
Orchestrated tests are usually used for features that involve external services
or complex setup (e.g. LDAP, Geo etc.), or for generic Omnibus checks (ensure
our Omnibus package works, can be updated / upgraded to EE etc.).

For more details on the internals, please read the
[How it works](./how_it_works.md) documentation.

## Supported GitLab environment variables

All environment variables used by GitLab QA should be defined in [`lib/gitlab/qa/runtime/env.rb`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/lib/gitlab/qa/runtime/env.rb).

|        Variable       | Default   | Description           | Required |
|-----------------------|-----------|-----------------------|----------|
| `GITLAB_USERNAME`       | `root`  | Username to use when signing into GitLab. | Yes|
| `GITLAB_PASSWORD`       | `5iveL!fe` | Password to use when signing into GitLab. | Yes|
| `GITLAB_FORKER_USERNAME`|-  | Username to use for forking a project. | Yes|
| `GITLAB_FORKER_PASSWORD`|-  | Password to use for forking a project. | Yes|
| `GITLAB_QA_USERNAME_1`  |-  | Username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_1`  |-  | Password for `GITLAB_QA_USERNAME_1` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_QA_USERNAME_2`  |-  | Another username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_2`  |-  | Password for `GITLAB_QA_USERNAME_2` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_QA_USERNAME_3`  |-  | Another username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_3`  |-  | Password for `GITLAB_QA_USERNAME_3` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_QA_USERNAME_4`  |-  | Another username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_4`  |-  | Password for `GITLAB_QA_USERNAME_4` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_QA_USERNAME_5`  |-  | Another username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_5`  |-  | Password for `GITLAB_QA_USERNAME_5` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_QA_USERNAME_6`  |-  | Another username available in environments where signup is disabled. | No|
| `GITLAB_QA_PASSWORD_6`  |-  | Password for `GITLAB_QA_USERNAME_6` available in environments where signup is disabled (e.g. staging.gitlab.com). | No|
| `GITLAB_LDAP_USERNAME`  |-  | LDAP username to use when signing into GitLab. | No|
| `GITLAB_LDAP_PASSWORD`  |-  | LDAP password to use when signing into GitLab. | No|
| `GITLAB_ADMIN_USERNAME` |-  | Admin username to use when adding a license. | No|
| `GITLAB_ADMIN_PASSWORD` |-  | Admin password to use when adding a license. | No|
| `GITLAB_SANDBOX_NAME`   | `gitlab-qa-sandbox` | The sandbox group name the test suite is going to use. | No|
| `GITLAB_QA_ACCESS_TOKEN`|-  | A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. An existing token that is valid on [`Test::Instance::Staging`](#testinstancestaging) scenario to retrieve the staging can be found in the shared 1Password vault. |No|
| `GITLAB_QA_ADMIN_ACCESS_TOKEN` |-  | A valid personal access token with the `api` scope from a user with admin access. Used for API access as an admin during tests. | No|
| `GITLAB_QA_CONTAINER_REGISTRY_ACCESS_TOKEN` | - | A valid personal access token with the `read_registry` scope. Used to [access the container registry on `registry.gitlab.com` when tests run in a CI job that _is not_ triggered via another pipeline](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/364addb83e7b136ff0f9d8719ca9553d290aa9ab/lib/gitlab/qa/release.rb#L152). For example, if you manually run a [new Staging pipeline](https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/new), this token will be used. | No |
| `EE_LICENSE` |-  | Enterprise Edition license. | No|
| `QA_ARTIFACTS_DIR` |`/tmp/gitlab-qa`| Path to a directory where artifacts (logs and screenshots) for failing tests will be saved. | No|
| `DOCKER_HOST` |`http://localhost`| Docker host to run tests against. | No|
| `WEBDRIVER_HEADLESS` |-  | When running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run. | No|
| `QA_ADDITIONAL_REPOSITORY_STORAGE` |-  | The name of additional, non-default storage to be used with tests tagged `repository_storage`, run via the `Test::Instance::RepositoryStorage` scenario.  Note: Admin access is required to change repository storage. | No|
| `QA_PRAEFECT_REPOSITORY_STORAGE` |-  | The name of repository storage using Praefect. Note: Admin access is required to change repository storage. | No|
| `QA_COOKIES` |-  | Optionally set to "cookie1=value;cookie2=value" in order to add a cookie to every request. This can be used to set the canary cookie by setting it to "gitlab_canary=true". | No|
| `QA_DEBUG` |-  | Set to `true` to verbosely log page object actions. Note: if enabled be aware that sensitive data might be logged. If an input element has a QA selector with `password` in the name, data entered into the input element will be masked. If the element doesn't have `password` in its name it won't be masked. | No|
| `QA_LOG_PATH` |-  | Path to output debug logging to. If not set logging will be output to STDOUT. | No|
| `QA_CAN_TEST_GIT_PROTOCOL_V2` | `true` | Set to `false` to skip tests that require Git protocol v2 if your environment doesn't support it. | No|
| `QA_CAN_TEST_ADMIN_FEATURES` | `true` | Set to `false` to skip tests that require admin access. | No|
| `QA_CAN_TEST_PRAEFECT` | `true` | Set to `false` to skip tests that require Praefect to be running. | No|
| `QA_DISABLE_RSPEC_RETRY` |-  | Set to `true` to turn off retrying tests on failure. | No|
| `QA_SIMULATE_SLOW_CONNECTION` |-  | Set to `true` to configure Chrome's network settings to simulate a slow connection. | No|
| `QA_SLOW_CONNECTION_LATENCY_MS` | `2000` | The additional latency (in ms) of the simulated slow connection. | No|
| `QA_SLOW_CONNECTION_THROUGHPUT_KBPS` | `32` | The maximum throughput (in kbps) of the simulated slow connection. | No|
| `QA_SKIP_PULL` | `false` | Set to `true` to skip pulling docker images (e.g., to use one you built locally). | No|
| `QA_GENERATE_ALLURE_REPORT` | `false` | When running on CI, set to `true` to generate allure reports | No|
| `QA_EXPORT_TEST_METRICS` | `true` | When running on CI, set to `true` to export test metrics to influxdb | No|
| `QA_INFLUXDB_URL` |- | Influxdb url for test metrics reporting | No|
| `QA_INFLUXDB_TOKEN` |- | Influxdb token for test metrics reporting | No|
| `QA_RUN_TYPE` |- | QA run type like `staging-full`, `canary`, `production` etc. Used in test metrics reporting | No|
| `GITHUB_USERNAME` |-  | Username for authenticating with GitHub. | No|
| `GITHUB_PASSWORD` |-  | Password for authenticating with GitHub. | No|
| `GITLAB_QA_LOOP_RUNNER_MINUTES` | `1` | Minutes to run and repeat a spec while using the '--loop' option; default value is 1 minute. | No|
| `CI_SERVER_PERSONAL_ACCESS_TOKEN` |-  | Personal access token of the server that is running the CI pipeline. This is used for pulling CI_RUNNER information in certain tests. | No|
| `GEO_MAX_FILE_REPLICATION_TIME` | `120` | Maximum time that a test will wait for a replicated file to appear on a Geo secondary node. | No|
| `GEO_MAX_DB_REPLICATION_TIME` | `120` | Maximum time that a test will wait for database data to appear on a Geo secondary node. | No|
| `JIRA_ADMIN_USERNAME` |-  | Username for authenticating with Jira server as admin. | No|
| `JIRA_ADMIN_PASSWORD` |-  | Password for authenticating with Jira server as admin. | No|
| `CACHE_NAMESPACE_NAME` | `true` | Cache namespace name for groups. | No|
| `DEPLOY_VERSION` |- | The version of GitLab being tested against. | No|
| `GITLAB_QA_USER_AGENT` |- | The browser user-agent to use instead of the default Chrome user-agent. | No|
| `GEO_FAILOVER` | `false` | Set to `true` when a Geo secondary site has been promoted to a Geo primary site. | No|
| `GITLAB_INITIAL_ROOT_PASSWORD` | `5iveL!fe` | Initial root password for Omnibus installations | No|

## [Supported Remote Grid environment variables](./running_against_remote_grid.md)

## Running tests with a feature flag enabled

It is possible to enable or disable a feature flag before running tests.
To test a Gitlab image with a feature flag enabled, run this command:
```
$ gitlab-qa Test::Instance::Image gitlab/gitlab-ee:12.4.0-ee.0 --enable-feature feature_flag_name
```

To run a test with feature flag disabled, run this command:
```
$ gitlab-qa Test::Instance::Image gitlab/gitlab-ee:12.4.0-ee.0 --disable-feature feature_flag_name
```

You an also test a GitLab image multiple times with different feature flag settings:
```
$ gitlab-qa Test::Instance::Image gitlab/gitlab-ee:12.4.0-ee.0 --disable-feature feature_flag_name --enable-feature feature_flag_name
```
This will first disable `feature_flag_name` flag and run the tests and then enable it and run the tests again.

You can pass any number of feature flag settings. The tests will run once for each setting.

See the [QA framework documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/README.md#running-tests-with-a-feature-flag-enabled-or-disabled)
for information on running the tests with different feature flag settings from the QA framework.

## Running tests with multiple feature flags set

The options above allow you to enable or disable a single feature flag at a time. However, if you want to set more than
one feature flag at the same time you'll need to use `--set-feature-flags` instead.

The desired state must be set individually for each feature flag in a comma-separated list. For example to disable a feature flag
named `feature-one` and enable another named `feature-two`, use the following parameters:

```shell
--set-feature-flags feature-one=disable,feature-two=enable
```

Those parameters will instruct GitLab QA to set both feature flags before running the suite of tests.

## Specifying the GitLab version

In each of the examples below, it is possible to test a specific version of GitLab
by providing the full image name, or an abbreviation followed by the image tag.

For example, to test GitLab version `12.4.0-ee`, the image tag is [`12.4.0-ee.0`](https://hub.docker.com/layers/gitlab/gitlab-ee/12.4.0-ee.0/images/sha256-65df19d8abbb0debdccb64bfe96871563806098cd84da0b818ae8cfdd928b9aa)
and so you could run the tests with the command:

```
$ gitlab-qa Test::Instance::Image gitlab/gitlab-ee:12.4.0-ee.0
```

Or with the command:

```
$ gitlab-qa Test::Instance::Image EE:12.4.0-ee.0
```

If you only provide the abbreviation, it will run the tests against the latest nightly image.

For example, the following command would use the image named `gitlab/gitlab-ee:nightly`

```
$ gitlab-qa Test::Instance::Image EE
```

To run EE tests, the `EE_LICENSE` environment variable needs to be set:

`$ export EE_LICENSE=$(cat /path/to/GitLab.gitlab_license)`

## Specifying the GitLab QA image to use

By default, `gitlab-qa` infers the QA image to use based on the GitLab image.
For instance, if you run the following:

```
$ gitlab-qa Test::Instance::Image gitlab/gitlab-ee:12.4.0-ee.0
```

Then, `gitlab-qa` would infer `gitlab/gitlab-ee-qa:12.4.0-ee.0` as the QA image
based on the GitLab image (note the `-qa` suffix in the image name).

In some cases, you'll want to use a specific QA image instead of letting
`gitlab-qa` infer the QA image name from the GitLab image. Such cases can be
when you're doing local debugging/testing and you want to control the QA image
name, or in the CI where the QA image might be built by a project (e.g.
`gitlab-org/gitlab`, and the GitLab image might be built by another project
(e.g. `gitlab-org/omnibus-gitlab-mirror`).

To specify the QA image to use, pass the `--qa-image QA_IMAGE` option,
as follows:

```
$ gitlab-qa Test::Instance::Image --qa-image registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:branch-name EE
```

Additionally, setting the `$QA_IMAGE` environment variable achieve the same result,
without needing to pass the `--qa-image` option:

```
$ export QA_IMAGE="registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:branch-name"
$ gitlab-qa Test::Instance::Image EE
```

## Running a specific test (or set of tests)

In most of the scenarios listed below, if you don't want to run all the tests
it's possible to specify one or more tests. The framework uses RSpec, so tests can be
specified as you would when using RSpec.

For example, the following would run `create_merge_request_spec.rb`:

```shell
$ gitlab-qa Test::Instance::Image EE -- qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb
```

While the following would run all Create UI tests:

```shell
$ gitlab-qa Test::Instance::Image EE -- qa/specs/features/browser_ui/3_create
```

And the following would run all Create API tests as well as UI tests:

```shell
$ gitlab-qa Test::Instance::Image EE -- qa/specs/features/browser_ui/3_create qa/specs/features/api/3_create
```

### Running tests for transient bugs

A suite of tests have been written to test for [transient bugs](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#transient-bugs).
Those tests are tagged `:transient` and therefore can be run via:

```shell
$ gitlab-qa Test::Instance::Image EE -- --tag transient
```

If you would like to run the transient bug tests against a specific GitLab instance such as your GDK, you can use the following command:

```shell
$ gitlab-qa Test::Instance::Any EE https://your.instance.gitlab -- --tag transient
```

## Examples

### `Test::Instance::Image CE|EE|<full image address>`

This tests that a GitLab Docker container works as expected by running
instance-level tests against it.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the `Test::Instance`
scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
GitLab project).

Example:

```
$ gitlab-qa Test::Instance::Image CE
```

### `Test::Omnibus::Image CE|EE|<full image address>`

This tests that a GitLab Docker container can start without any error.

This spins up a GitLab Docker container based on the given edition or image:

- `gitlab/gitlab-ce:nightly` for `CE`
- `gitlab/gitlab-ee:nightly` for `EE`
- the given custom image for `<full image address>`

Example:

```
$ gitlab-qa Test::Omnibus::Image CE
```

### `Test::Omnibus::Update CE|EE|<full image address>`

This tests that:

- the GitLab Docker `latest` container works as expected by running
  instance-level tests against it (see `Test::Instance::Image` above)
- it can be updated to a new (`nightly` or `<full image address>`) container
- the new GitLab container still works as expected by running
  `Test::Instance::Image` against it

Example:

```
# Update from gitlab/gitlab-ce:latest to gitlab/gitlab-ce:nightly
$ gitlab-qa Test::Omnibus::Update CE

# Update from gitlab/gitlab-ee:latest to gitlab/gitlab-ee:nightly
$ gitlab-qa Test::Omnibus::Update EE

# Update from gitlab/gitlab-ce:latest to gitlab/gitlab-ce:my-custom-tag
$ gitlab-qa Test::Omnibus::Update gitlab/gitlab-ce:my-custom-tag
```

### `Test::Omnibus::Upgrade CE|<full image address>`

This tests that:

- the GitLab Docker container works as expected by running instance-level tests
  against it (see `Test::Instance::Image` above)
- it can be upgraded to a corresponding EE container
- the new GitLab container still works as expected by running
  `Test::Instance::Image` against it

Example:

```
# Ugrade from gitlab/gitlab-ce:nightly to gitlab/gitlab-ee:nightly
$ gitlab-qa Test::Omnibus::Upgrade CE

# Ugrade from gitlab/gitlab-ce:my-custom-tag to gitlab/gitlab-ee:my-custom-tag
$ gitlab-qa Test::Omnibus::Upgrade gitlab/gitlab-ce:my-custom-tag
```

### `Test::Integration::Geo EE|<full image address>`

This tests that two GitLab Geo instances work as expected.

The scenario spins up a primary and secondary GitLab Geo nodes, and verifies
that the replications (repository, attachments, project rename etc.) work as
expected.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`QA::EE::Scenario::Test::Geo` scenario (located under
[`gitlab-org/gitlab-ee@qa/qa/ee/scenario/test/geo.rb`][test-geo] in the GitLab
EE project).

**Required environment variables:**

- `EE_LICENSE`: A valid EE license.

Example:

```
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
$ gitlab-qa Test::Integration::Geo EE
```

[test-geo]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/ee/scenario/test/geo.rb

### `Test::Integration::GitalyCluster CE|EE|<full image address>`

This tests [Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/praefect.html),
a clustered configuration of the Gitaly repository storage service.

The scenario configures and starts several docker containers to
represent the [recommended minimum configuration](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#requirements-for-configuring-a-gitaly-cluster)
of a Gitaly Cluster.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance::All` scenario with the `:gitaly_cluster` tag.

Example:

```
$ gitlab-qa Test::Integration::GitalyCluster EE
```

### `Test::Integration::LDAPNoTLS CE|EE|<full image address>`

This tests that a GitLab instance works as expected with an external
LDAP server with TLS not enabled.

The scenario spins up an OpenLDAP server, seeds users, and verifies
that LDAP-related features work as expected.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::LDAPNoTLS` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_no_tls.rb`][test-integration-ldap-no-tls]
in the GitLab project).

In EE, both the GitLab standard and LDAP credentials are needed:

1. The first is used to login as an Admin to enter in the EE license.
2. The second is used to conduct LDAP-related tasks

**Required environment variables:**

- [For EE only] `EE_LICENSE`: A valid EE license.

Example:

```
$ gitlab-qa Test::Integration::LDAPNoTLS CE

# For EE
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Integration::LDAPNoTLS EE
```

[test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_no_tls.rb

### `Test::Integration::LDAPTLS CE|EE|<full image address>`

This tests that a TLS enabled GitLab instance works as expected with an external TLS enabled LDAP server.
The self signed TLS certificate used for the Gitlab instance and the private key is located at: [`gitlab-org/gitlab-qa@tls_certificates/gitlab`][test-integration-ldap-tls-certs]

The certificate was generated with openssl using this command:
```
openssl req -x509 -newkey rsa:4096 -keyout gitlab.test.key -out gitlab.test.crt -days 3650 -nodes -subj "/C=US/ST=CA/L=San Francisco/O=GitLab/OU=Org/CN=gitlab.test"
```

The scenario spins up a TLS enabled OpenLDAP server, seeds users, and verifies
that LDAP-related features work as expected.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::LDAPTLS` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_tls.rb`][test-integration-ldap-tls]
in the GitLab project).

In EE, both the GitLab standard and LDAP credentials are needed:

1. The first is used to login as an Admin to enter in the EE license.
2. The second is used to conduct LDAP-related tasks

**Required environment variables:**

- [For EE only] `EE_LICENSE`: A valid EE license.

Example:

```
$ gitlab-qa Test::Integration::LDAPTLS CE

# For EE
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Integration::LDAPTLS EE
```

[test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_tls.rb
[test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab/blob/master/tls_certificates/gitlab

### `Test::Integration::LDAPNoServer EE|<full image address>`

This configures a GitLab instance for use with LDAP but does not
spin up an LDAP server in a docker container.

The LDAP server is created at runtime by the spec so that
the test can provide the fixture data for the LDAP server
as needed.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::LDAPNoServer` scenario (located under
[`gitlab-org/gitlab@qa/qa/scenario/test/integration/ldap_no_server.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_no_server.rb)
in the GitLab project).

In GiLab project, both the GitLab standard and LDAP credentials are needed:

1. The first is used to login as an Admin to enter in the GitLab license.
2. The second is used to conduct LDAP-related tasks

**Required environment variables:**

- `EE_LICENSE`: A valid Enterprise license.

Example:

```
$ export EE_LICENSE=$(cat /path/to/GitLab.gitlab_license)

$ gitlab-qa Test::Integration::LDAPNoServer EE
```

### `Test::Integration::GroupSAML EE|<full image address>`

This tests that Group SAML login works as expected with an external SAML identity provider (idp).

This scenario spins up a SAML idp provider and verifies that a user is able to login to a group
in GitLab that has SAML SSO enabled.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::GroupSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/ee/scenario/test/integration/group_saml.rb`][test-integration-group-saml] in the GitLab EE project).

[test-integration-group-saml]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/ee/scenario/test/integration/group_saml.rb

**Required environment variables:**

- `EE_LICENSE`: A valid EE license.

Example:

```
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Integration::GroupSAML EE
```

### `Test::Integration::InstanceSAML CE|EE|<full image address>`

This tests that a GitLab instance works as expected with an external
SAML identity provider (idp).

This scenario spins up a SAML idp provider and verifies that a user is able to login to GitLab instance
using SAML.

To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab project).

[test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/instance_saml.rb

**Required environment variables:**

- [For EE only] `EE_LICENSE`: A valid EE license.

Example:

```
$ gitlab-qa Test::Integration::InstanceSAML CE

# For EE
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Integration::InstanceSAML EE
```

### `Test::Integration::Mattermost CE|EE|<full image address>`

This tests that a GitLab instance works as expected when enabling the embedded
Mattermost server (see `Test::Instance::Image` above).

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::Mattermost` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
in the GitLab project).

**Required environment variables:**

- [For EE only] `EE_LICENSE`: A valid EE license.

Example:

```
$ gitlab-qa Test::Integration::Mattermost CE

# For EE
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Integration::Mattermost EE
```

[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/mattermost.rb

### `Test::Integration::Packages CE|EE|<full image address>`

**Note: This Scenario no longer exists.  See https://gitlab.com/gitlab-org/gitlab-qa/-/merge_requests/662**

To run Packages tests, you may [configure Omnibus](configuring_omnibus.md) to use the [Packages](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/lib/gitlab/qa/runtime/omnibus_configurations/packages.rb) configurator.

Example:

```
$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)

$ gitlab-qa Test::Instance::Image EE --omnibus-config packages
```

This tests the GitLab Package Registry feature by setting
`gitlab_rails['packages_enabled'] = true` in the Omnibus configuration
before starting the GitLab container.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance::All` scenario with the `--tag packages` RSpec parameter,
which runs only the tests with `:packages` metadata.

**Required environment variables:**

- `EE_LICENSE`: A valid EE license.

### `Test::Integration::Praefect CE|EE|<full image address>`

This tests [Praefect](https://docs.gitlab.com/ee/administration/gitaly/praefect.html),
which is a reverse-proxy for Gitaly. It sets the Omnibus configuration
to use Praefect as the default storage backed by a single Gitaly node
before starting the GitLab container.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance::All` scenario.

Example:

```
$ gitlab-qa Test::Integration::Praefect EE
```

### `Test::Integration::SMTP CE|EE|<full image address>`

This tests SMTP notification email delivery from Gitlab by using
[MailHog](https://github.com/mailhog/MailHog) as MTA.
It starts up a Docker container for MailHog and sets the Omnibus configuration
to use it for SMTP delivery. The MailHog container will expose the configured
port for SMTP delivery, and also another port for the HTTP MailHog API used for
querying the delivered messages.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::SMTP` scenario.

Example:

```
$ gitlab-qa Test::Integration::SMTP CE
```

### `Test::Integration::Jira CE|EE|<full image address>`

This tests that a GitLab instance works as expected with an external
Jira server.
It starts up a Docker container for Jira Server and another container
for GitLab.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Integration::Jira` scenario.

**Required environment variables:**

- [For EE only] `EE_LICENSE`: A valid EE license.
- `JIRA_ADMIN_USERNAME`: Username for authenticating with Jira server as admin.
- `JIRA_ADMIN_PASSWORD`: Password for authenticating with Jira server as admin.

Example:

```
$ export JIRA_ADMIN_USERNAME=<jira_admin_username>
$ export JIRA_ADMIN_PASSWORD=<jira_admin_password>

# For EE
$ export EE_LICENSE=$(cat /path/to/GitLab.gitlab_license)

$ gitlab-qa Test::Integration::Jira EE
```

### `Test::Integration::Actioncable CE|EE|<full image address>`

This tests the real-time assignees feature by setting
`actioncable['enable'] = true` in the Omnibus configuration
before starting the GitLab container.

To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance::All` scenario with the `--tag actioncable` RSpec parameter,
which runs only the tests with `:actioncable` metadata.

Example:

```
$ gitlab-qa Test::Integration::Actioncable CE
```

### `Test::Instance::Any CE|EE|<full image address>:nightly|latest|any_tag http://your.instance.gitlab`

This tests that a live GitLab instance works as expected by running tests
against it.

To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
in the GitLab project).

Example:

```
$ export GITLAB_USERNAME=your_username
$ export GITLAB_PASSWORD=your_password

# Runs the QA suite for an instance running GitLab CE 10.8.1
$ gitlab-qa Test::Instance::Any CE:10.8.1-ce https://your.instance.gitlab

# Runs the QA suite for an instance running GitLab EE 10.7.3
$ gitlab-qa Test::Instance::Any EE:10.7.3-ee https://your.instance.gitlab

# You can even pass a gitlab-{ce,ee}-qa image directly
$ gitlab-qa Test::Instance::Any registry.gitlab.com:5000/gitlab/gitlab-ce-qa:v11.1.0-rc12 https://your.instance.gitlab
```

### `Test::Instance::Staging`

This scenario tests that the [`staging.gitlab.com`](https://staging.gitlab.com)
works as expected by running tests against it.

To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
in the GitLab project).

**Required environment variables:**

- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
  This is used to retrieve the version that staging is currently running.
  This can be found in the shared 1Password vault.

**Optional environment variables:**

- `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
  `gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
  This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
  If the variable isn't present, the QA image from Docker Hub will be used.
  This can be found in the shared 1Password vault.

Example:

```
$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
$ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
$ export GITLAB_USERNAME="gitlab-qa"
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"

$ gitlab-qa Test::Instance::Staging
```

### `Test::Instance::StagingRef`

This scenario tests that the [`Staging Ref`](https://staging-ref.gitlab.com)
works as expected by running tests against it.

To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
in the GitLab project).

**Required environment variables:**

- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
  This is used to retrieve the version that staging is currently running.
  Staging Ref QA users credentials can be found in the shared GitLab QA 1Password vault.

**Optional environment variables:**

- `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
  `gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
  This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
  If the variable isn't present, the QA image from Docker Hub will be used.
  This can be found in the shared GitLab QA 1Password vault.

Example:

```
$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
$ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
$ export GITLAB_USERNAME="gitlab-qa"
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"

$ gitlab-qa Test::Instance::StagingRef
```

### `Test::Instance::StagingGeo`

This scenario tests that the Geo staging deployment (with [`staging.gitlab.com`](https://staging.gitlab.com) as the primary site and [`geo.staging.gitlab.com`](https://geo.staging.gitlab.com) as the secondary site) works as expected by running tests tagged `:geo` against it. This is done by spinning up a GitLab QA (`gitlab/gitlab-qa`) container and running the `QA::EE::Scenario::Test::Geo` scenario. Note that the Geo setup steps in the `QA::EE::Scenario::Test::Geo` scenario are skipped when testing a live Geo deployment.

**Required user properties:**

- The user must provide OAuth authorization on the secondary site before running Geo tests. This can be done via the authorization modal that appears after logging into the secondary node for the first time.

- Some Geo tests require the user to have Admin access level (for example, the Geo Nodes API tests)

**Required environment variables:**

- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
  This is used to retrieve the version that staging is currently running.
  This can be found in the shared 1Password vault.

**Optional environment variables:**

- `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
  `gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
  This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
  If the variable isn't present, the QA image from Docker Hub will be used.
  This can be found in the shared 1Password vault.

Example:

```
$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
$ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
$ export GITLAB_USERNAME="gitlab-qa"
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"

$ gitlab-qa Test::Instance::StagingGeo
```

### `Test::Instance::Production`

This scenario functions the same as `Test::Instance::Staging`
but will run tests against [`gitlab.com`](https://gitlab.com).

In release 11.6 it is possible to test against the canary stage of production
by setting `QA_COOKIES=gitlab_canary=true`. This adds a cookie
to all web requests which will result in them being routed
to the canary fleet.

### `Test::Instance::Preprod`

This scenario functions the same as `Test::Instance::Staging`
but will run tests against [`pre.gitlab.com`](https://pre.gitlab.com).

Note that [`pre.gitlab.com`](https://pre.gitlab.com) is used as an Interim
Performance Testbed and [will be replaced with the actual testbed in the future](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/60).

### `Test::Instance::Release`

This scenario functions the same as `Test::Instance::Staging`
but will run tests against [`release.gitlab.net`](https://release.gitlab.net).

### `Test::Instance::Smoke`

This scenario will run a limited amount of tests selected from the test suite tagged by `:smoke`.
Smoke tests are quick tests that ensure that some basic functionality of GitLab works.

To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
container is spun up and tests are run from it by running the
`Test::Instance::Smoke` scenario (located under
[`gitlab-org/gitlab/qa/qa/scenario/test/smoke.rb`][smoke-instance] in the
in the GitLab project).

Example:

```
$ gitlab-qa Test::Instance::Smoke ee:<tag> https://staging.gitlab.com
```

### `Test::Instance::RepositoryStorage`

This scenario will run a limited number of tests that are tagged with `:repository_storage`.

These tests verify features related to multiple repository storages.

**Required environment variables:**

- `QA_ADDITIONAL_REPOSITORY_STORAGE`: The name of the non-default repository storage.

Example:

```
$ export QA_ADDITIONAL_REPOSITORY_STORAGE=secondary

$ gitlab-qa Test::Instance::RepositoryStorage
```

### `Test::Instance::Airgapped`

This scenario will run tests from the test suite against an airgapped instance.
The airgapped instance is set up by using `iptables` in the GitLab container to block network traffic other than testable ports, and by using runners
in a shared internal network.

Example:

```
# For EE
$ export EE_LICENSE=$(cat /path/to/gitlab_license)

$ gitlab-qa Test::Instance::Airgapped EE -- --tag smoke
```

### `Test::Instance::Image CE|EE|<full image address> --omnibus-config object_storage`

This orchestrated scenario runs tests from the test suite against a GitLab instance which has object storage configured instead of using local storage. The omnibus configuration `object_storage` ([configurator](configuring_omnibus.md)), uses Minio and `object_storage_aws` uses an AWS S3 account with credentials configured in the pipeline as CI Variables (available on 1Password). Requires:

* `AWS_S3_REGION` - Region where the bucket is created. To optimize latency, minimize costs, or address regulatory requirements, choose any AWS Region that is geographically close to you.
* `AWS_S3_KEY_ID` - Key ID for the user accessing the bucket.
* `AWS_S3_ACCESS_KEY` - Access Key for the user accessing the bucket.
* `AWS_S3_BUCKET_NAME` - The name of the bucket where objects are stored. After a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted.

### `Test::Instance::Geo EE|<full image address>:nightly|latest|any_tag http://geo-primary.gitlab http://geo-secondary.gitlab`

This scenario will run tests tagged `:geo` against a live Geo deployment, by spinning up a GitLab QA (`gitlab/gitlab-qa`)
container and running the `QA::EE::Scenario::Test::Geo` scenario. Note that the Geo setup steps in the `QA::EE::Scenario::Test::Geo` scenario are skipped when testing a live Geo deployment. The URLs for the primary site and secondary site of the live Geo deployment must be provided as command line arguments.

**Required user properties:**

- The user must provide OAuth authorization on the secondary site before running Geo tests. This can be done via the authorization modal that appears after signing into the secondary node for the first time.

- Some Geo tests require the user to have Admin access level (for example, the Geo Nodes API tests)

Example:

```
$ export GITLAB_USERNAME="gitlab-qa"
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"

$ gitlab-qa Test::Instance::Geo EE https://primary.gitlab.com https://secondary.gitlab.com
```
----

[Back to README.md](../README.md)

[test-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/all.rb
[smoke-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/smoke.rb