README.md in knapsack_pro-1.2.0 vs README.md in knapsack_pro-1.2.1

- old
+ new

@@ -100,11 +100,11 @@ - [Info for CircleCI users](#info-for-circleci-users) - [CircleCI and knapsack_pro Queue Mode](#circleci-and-knapsack_pro-queue-mode) - [Info for Travis users](#info-for-travis-users) - [Info for semaphoreapp.com users](#info-for-semaphoreappcom-users) - [Info for buildkite.com users](#info-for-buildkitecom-users) - - [Info for Gitlab CI users](#info-for-gitlab-ci-users) + - [Info for GitLab CI users](#info-for-gitlab-ci-users) - [GitLab CI `>= 11.5`](#gitlab-ci--115) - [GitLab CI `< 11.5` (old GitLab CI)](#gitlab-ci--115-old-gitlab-ci) - [Info for codeship.com users](#info-for-codeshipcom-users) - [Info for Heroku CI users](#info-for-heroku-ci-users) - [Info for Solano CI users](#info-for-solano-ci-users) @@ -116,21 +116,24 @@ - [Common problems](#common-problems) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) - [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed) - [Why I see HEAD as branch name in user dashboard for Build metrics for my API token?](#why-i-see-head-as-branch-name-in-user-dashboard-for-build-metrics-for-my-api-token) + - [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation) - [Queue Mode problems](#queue-mode-problems) - [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail) - [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail) + - [Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?](#why-when-i-use-queue-mode-for-rspec-then-i-see-error-superclass-mismatch-for-class) - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) - [Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue.](#parallel-tests-cucumber-and-rspec-with-cucumber-failures-exit-ci-node-early-leaving-fewer-ci-nodes-to-finish-rspec-queue) - [Why when I reran the same build (same commit hash, etc) on Codeship then no tests would get executed in Queue Mode?](#why-when-i-reran-the-same-build-same-commit-hash-etc-on-codeship-then-no-tests-would-get-executed-in-queue-mode) + - [Why knapsack_pro hangs / freezes / is stale i.e. for Codeship in Queue Mode?](#why-knapsack_pro-hangs--freezes--is-stale-ie-for-codeship-in-queue-mode) - [How to find seed in RSpec output when I use Queue Mode for RSpec?](#how-to-find-seed-in-rspec-output-when-i-use-queue-mode-for-rspec) - [General questions](#general-questions) - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - [for knapack_pro regular mode](#for-knapack_pro-regular-mode) - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) @@ -883,39 +886,41 @@ * [Buildkite Rails Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) * [Buildkite Rails Docker Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro) If you want to use Buildkite retry single agent feature to retry just failed tests on particular agent (CI node) then you should set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). -#### Info for Gitlab CI users +#### Info for GitLab CI users -Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to [Secret Variables](https://gitlab.com/help/ci/variables/README.md#secret-variables) in `Gitlab CI Settings -> CI/CD Pipelines -> Secret Variables`. +Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to [Secret Variables](https://gitlab.com/help/ci/variables/README.md#secret-variables) in `GitLab CI Settings -> CI/CD Pipelines -> Secret Variables`. ##### GitLab CI `>= 11.5` ```yaml test: parallel: 2 - # RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split) - script: bundle exec rake knapsack_pro:queue:rspec + # Knapsack Pro Regular Mode (deterministic test suite split) + script: bundle exec rake knapsack_pro:rspec # Other commands you could use: # Knapsack Pro Regular Mode (deterministic test suite split) - # bundle exec rake knapsack_pro:rspec # bundle exec rake knapsack_pro:cucumber # bundle exec rake knapsack_pro:minitest + # bundle exec rake knapsack_pro:test_unit + # bundle exec rake knapsack_pro:spinach # Knapsack Pro Queue Mode (dynamic test suite split) + # bundle exec rake knapsack_pro:queue:rspec # bundle exec rake knapsack_pro:queue:minitest ``` Here you can find info [how to configure the GitLab parallel CI nodes](https://docs.gitlab.com/ee/ci/yaml/#parallel). ##### GitLab CI `< 11.5` (old GitLab CI) -Gitlab CI does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same `test` stage. Below is relevant part of `.gitlab-ci.yml` configuration for 2 parallel jobs. +GitLab CI does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same `test` stage. Below is relevant part of `.gitlab-ci.yml` configuration for 2 parallel jobs. ``` # .gitlab-ci.yml stages: - test @@ -1218,10 +1223,24 @@ knapack_pro uses git command `git -C /home/user/project_dir rev-parse --abbrev-ref HEAD` to detect branch name. See [source of knapack_pro](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/repository_adapters/git_adapter.rb). In most of cases it's good way to detect branch name. But if your CI provider during CI build checkouts to specific git commit then git cannot provide the name of the branch. In such scenario you would see `HEAD` as your branch name. It is good enough situation and knapack_pro will work correctly. The benefit of knowing exactly the branch name allows KnapsackPro API to better track history of test files timing changes across branches in order to better do split of test suite. The difference should be rather very small so it's not a problem that you have `HEAD` as branch name. If you would like to see exact branch name instead of `HEAD` in your `build metrics` history in [user dashboard](https://knapsackpro.com/dashboard) then you can explicitly provide the branch name with `KNAPSACK_PRO_BRANCH` for each CI build. +#### Why Capybara feature tests randomly fail when using CI parallelisation? + +It can happen that when you use CI parallelisation then your CI machine is overloaded and some of Capybara feature specs may randomly fail when tested website loaded slowly. + +You can try to increase default Capybara max wait time from 2 seconds to something bigger like 5 seconds to ensure the Capybara will wait longer till the website is loaded before marking test as failed. + +```ruby +# spec/rails_helper.rb +Capybara.default_max_wait_time = 5 # in seconds +``` + +For instance, this tip might be helpful for Heroku CI users who use Heroku dynos with lower performance. + + #### Queue Mode problems ##### Why when I use Queue Mode for RSpec then my tests fail? knapsack_pro Queue Mode uses `RSpec::Core::Runner` feature that allows [running specs multiple times with different runner options in the same process](https://relishapp.com/rspec/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process). @@ -1263,10 +1282,64 @@ association :task end end ``` +##### Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`? + +You may see error like: + +``` +TypeError: + superclass mismatch for class BatchClass +``` + +when you have 2 test files like this one: + +``` +# spec/a_spec.rb +class BaseBatchClass +end + +module Mock + module FakeModels + class BatchClass < BaseBatchClass + def args + end + end + end +end + +describe 'A test of something' do + it do + end +end +``` + +``` +# spec/b_spec.rb +class DifferentBaseBatchClass +end + +module Mock + module FakeModels + # Note the base class is different here! + class BatchClass < DifferentBaseBatchClass + def args + end + end + end +end + +describe 'B test of something' do + it do + end +end +``` + +Instead of mocking like shown above you could use [RSpec stub_const](https://relishapp.com/rspec/rspec-mocks/docs/mutating-constants) to solve error `superclass mismatch for class BatchClass`. + ##### Why I don't see collected time execution data for my build in user dashboard? If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure: * you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb` @@ -1339,9 +1412,17 @@ This way knapsack_pro won’t use build ID provided by Codeship and each build will be treated as a unique. This should be good enough solution for most users. There is one edge case with that solution. Please note that the knapsack_pro gem doesn't have a CI build ID in order to generate a queue for each particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit. To avoid this you should specify a unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. + +##### Why knapsack_pro hangs / freezes / is stale i.e. for Codeship in Queue Mode? + +Some users with larger projects notice that in Queue Mode knapsack_pro ruby process hangs, for instance for CodeShip CI users. + +It happens due too big RSpec log output in Queue Mode. To produce less logs on output you can change log level to `KNAPSACK_PRO_LOG_LEVEL=warn`. + +Learn more about [log levels](#how-can-i-change-log-level). ##### How to find seed in RSpec output when I use Queue Mode for RSpec? In output for RSpec in knapack_pro Queue Mode you may see message: