README.md in knapsack_pro-1.4.0 vs README.md in knapsack_pro-1.5.0
- old
+ new
@@ -96,11 +96,13 @@
- [How to debug branch names?](#how-to-debug-branch-names)
- [Supported CI providers](#supported-ci-providers)
- [Info for CircleCI users](#info-for-circleci-users)
- [CircleCI and knapsack_pro Queue Mode](#circleci-and-knapsack_pro-queue-mode)
- [Info for Travis users](#info-for-travis-users)
- - [Info for semaphoreapp.com users](#info-for-semaphoreappcom-users)
+ - [Info for semaphoreci.com users](#info-for-semaphorecicom-users)
+ - [Semaphore 2.0](#semaphore-20)
+ - [Semaphore 1.0](#semaphore-10)
- [Info for buildkite.com users](#info-for-buildkitecom-users)
- [Info for GitLab CI users](#info-for-gitlab-ci-users)
- [GitLab CI `>= 11.5`](#gitlab-ci--115)
- [GitLab CI `< 11.5` (old GitLab CI)](#gitlab-ci--115-old-gitlab-ci)
- [Info for codeship.com users](#info-for-codeshipcom-users)
@@ -119,10 +121,11 @@
- [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation)
- [Why knapsack_pro freezes / hangs my CI (for instance Travis)?](#why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis)
- [Queue Mode problems](#queue-mode-problems)
- [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail)
- [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail)
+ - [Why when I use Queue Mode for RSpec then my rake tasks are run twice?](#why-when-i-use-queue-mode-for-rspec-then-my-rake-tasks-are-run-twice)
- [Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?](#why-when-i-use-queue-mode-for-rspec-then-i-see-error-superclass-mismatch-for-class)
- [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard)
- [Why all test files have 0.1s time execution for my CI build in user dashboard?](#why-all-test-files-have-01s-time-execution-for-my-ci-build-in-user-dashboard)
- [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result)
- [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests)
@@ -580,13 +583,15 @@
#### Environment variables for debugging gem
This is only for maintainer of knapsack_pro gem. Not for the end users.
-`KNAPSACK_PRO_ENDPOINT` - Default value is `https://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com).
+* `KNAPSACK_PRO_ENDPOINT` - Default value is `https://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com).
-`KNAPSACK_PRO_MODE` - Default value is `production`. When mode is `development` then endpoint is `http://api.knapsackpro.test:3000`. When mode is `test` then endpoint is `http://api-staging.knapsackpro.com`.
+* `KNAPSACK_PRO_MODE` - Default value is `production` and then endpoint is `https://api.knapsackpro.com`.
+ * When mode is `development` then endpoint is `http://api.knapsackpro.test:3000`.
+ * When mode is `test` then endpoint is `https://api-staging.knapsackpro.com`.
### Passing arguments to rake task
#### Passing arguments to rspec
@@ -863,12 +868,109 @@
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token
More info about global and matrix ENV configuration in [travis docs](https://docs.travis-ci.com/user/customizing-the-build/#build-matrix).
-#### Info for semaphoreapp.com users
+#### Info for semaphoreci.com users
+##### Semaphore 2.0
+
+knapack_pro gem supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in `.semaphore/semaphore.yml` config file.
+
+* You need to set `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC`. If you don't want to commit secrets in yml file then you can [follow this guide](https://docs.semaphoreci.com/article/66-environment-variables-and-secrets).
+* You need to create as many jobs with unique names (Node 0 - Knapsack Pro, Node 1 - Knapsack Pro etc) as many parallel jobs you want to run. If your test suite is long you should use more parallel jobs.
+* If you have 2 parallel jobs you need to set `KNAPSACK_PRO_CI_NODE_TOTAL=2` for each job.
+* You need to set job index starting from 0 like `KNAPSACK_PRO_CI_NODE_INDEX=0` for Node 0.
+
+Below you can find full Semaphore CI 2.0 config for Rails project.
+
+```yaml
+# .semaphore/semaphore.yml
+# Use the latest stable version of Semaphore 2.0 YML syntax:
+version: v1.0
+
+# Name your pipeline. In case you connect multiple pipelines with promotions,
+# the name will help you differentiate between, for example, a CI build phase
+# and delivery phases.
+name: Demo Rails 5 app
+
+# An agent defines the environment in which your code runs.
+# It is a combination of one of available machine types and operating
+# system images.
+# See https://docs.semaphoreci.com/article/20-machine-types
+# and https://docs.semaphoreci.com/article/32-ubuntu-1804-image
+agent:
+ machine:
+ type: e1-standard-2
+ os_image: ubuntu1804
+
+# Blocks are the heart of a pipeline and are executed sequentially.
+# Each block has a task that defines one or more jobs. Jobs define the
+# commands to execute.
+# See https://docs.semaphoreci.com/article/62-concepts
+blocks:
+ - name: Setup
+ task:
+ env_vars:
+ - name: RAILS_ENV
+ value: test
+ jobs:
+ - name: bundle
+ commands:
+ # Checkout code from Git repository. This step is mandatory if the
+ # job is to work with your code.
+ # Optionally you may use --use-cache flag to avoid roundtrip to
+ # remote repository.
+ # See https://docs.semaphoreci.com/article/54-toolbox-reference#libcheckout
+ - checkout
+ # Restore dependencies from cache.
+ # Read about caching: https://docs.semaphoreci.com/article/54-toolbox-reference#cache
+ - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
+ # Set Ruby version:
+ - sem-version ruby 2.6.1
+ - bundle install --jobs=4 --retry=3 --path vendor/bundle
+ # Store the latest version of dependencies in cache,
+ # to be used in next blocks and future workflows:
+ - cache store gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock) vendor/bundle
+
+ - name: RSpec tests
+ task:
+ env_vars:
+ - name: RAILS_ENV
+ value: test
+ - name: PGHOST
+ value: 127.0.0.1
+ - name: PGUSER
+ value: postgres
+ - name: KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC
+ value: your_api_token_here
+ # This block runs two jobs in parallel and they both share common
+ # setup steps. We can group them in a prologue.
+ # See https://docs.semaphoreci.com/article/50-pipeline-yaml#prologue
+ prologue:
+ commands:
+ - checkout
+ - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
+ # Start Postgres database service.
+ # See https://docs.semaphoreci.com/article/54-toolbox-reference#sem-service
+ - sem-service start postgres
+ - sem-version ruby 2.6.1
+ - bundle install --jobs=4 --retry=3 --path vendor/bundle
+ - bundle exec rake db:setup
+
+ jobs:
+ - name: Node 0 - Knapsack Pro
+ commands:
+ - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec
+
+ - name: Node 1 - Knapsack Pro
+ commands:
+ - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec
+```
+
+##### Semaphore 1.0
+
Knapsack Pro supports semaphoreapp ENVs `SEMAPHORE_THREAD_COUNT` and `SEMAPHORE_CURRENT_THREAD`. The only thing you need to do is set up knapsack_pro rspec/cucumber/minitest/test_unit command for as many threads as you need. Here is an example:
```bash
# Thread 1
## Step for RSpec
@@ -1344,10 +1446,29 @@
association :task
end
end
```
+##### Why when I use Queue Mode for RSpec then my rake tasks are run twice?
+
+Why rake tasks are being ran twice in Queue Mode? If you have tests for your rake task then you want to ensure you clear the rake task before loading it inside of test file.
+In Queue Mode the rake task could be already loaded and loading it again in test file may result in running the task twice.
+
+```ruby
+before do
+ # Clear rake task from memory if it was already loaded.
+ # This ensures rake task is loaded only once in knapsack_pro Queue Mode.
+ Rake::Task[task_name].clear if Rake::Task.task_defined?(task_name)
+
+ # loaad rake task only once here
+ Rake.application.rake_require("tasks/dummy")
+ Rake::Task.define_task(:environment)
+end
+```
+
+Here is the full [example how to test rake task along with dummy rake task](https://github.com/KnapsackPro/rails-app-with-knapsack_pro/commit/9f068e900deb3554bd72633e8d61c1cc7f740306) from our example rails project.
+
##### Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?
You may see error like:
```
@@ -2020,12 +2141,14 @@
#### How to retry failed tests (flaky tests)?
Flaky (nondeterministic) tests, are tests that exhibit both a passing and a failing result with the same code.
-You can use [only failures option in RSpec](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests.
+I recommend to use [rspec-retry](https://github.com/NoRedInk/rspec-retry) gem that can retry failing test. It can be useful for randomly failing features specs. For instance you can configure it to retry 3 times features test before marking it as failing.
+Alternative way is to use built into [RSpec only failures option](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests.
+
Please add to your RSpec configuration:
```ruby
RSpec.configure do |c|
c.example_status_persistence_file_path = "tmp/rspec_examples.txt"
@@ -2144,10 +2267,10 @@
The test file pattern and exclude pattern support any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob).
#### How to use CodeClimate with knapsack_pro?
-You can check CodeClimate docs about [parallel and multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-and-multiple-test-suites).
+You can check CodeClimate docs about [parallel tests](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-tests) and [multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-multiple-test-suites).
#### How to run knapsack_pro only on a few parallel CI nodes instead of all?
You may want to run knapack_pro only on a few CI nodes when you would like to run a different job on other CI nodes.