README.md in knapsack_pro-0.28.0 vs README.md in knapsack_pro-0.28.1

- old
+ new

@@ -97,10 +97,12 @@ - [FAQ](#faq) - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - [for knapack_pro regular mode](#for-knapack_pro-regular-mode) - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) + - [for knapack_pro regular mode](#for-knapack_pro-regular-mode-1) + - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode-1) - [How can I change log level?](#how-can-i-change-log-level) - [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level) - [A. Create multiple small test files](#a-create-multiple-small-test-files) - [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file) - [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project) @@ -109,10 +111,15 @@ - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) + - [How to use junit formatter?](#how-to-use-junit-formatter) + - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) + - [How many API keys I need?](#how-many-api-keys-i-need) + - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) + - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) <!-- END doctoc generated TOC please keep comment here to allow auto update --> @@ -339,10 +346,17 @@ bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" Note if you will run queue mode command for the first time it might be slower. The second build should have better optimal test suite split. +If you will encounter problem with stack level too deep then you may want to ensure you load your dependencies only once in `spec/rails_helper.rb` or `spec/spec_helper.rb`. The Queue Mode may load multiple times the `rails_helper.rb` hence the problem. For instance the problem occurs for capybara-screenshot gem. Here is the example how you should load the gem. + + unless ENV['KNAPSACK_PRO_RSPEC_DEPENDENCIES_LOADED'] + ENV['KNAPSACK_PRO_RSPEC_DEPENDENCIES_LOADED'] = 'true' + require 'capybara-screenshot/rspec' + end + ### Additional info about queue mode * You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode. There might be some cached test suite splits for git commits you run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). @@ -699,18 +713,20 @@ #### for knapsack_pro queue mode There are a few ways to reproduce tests executed on CI node in your development environment. -* At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run. +* At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run. I recommend this approach. * For each intermediate request to Knapsack Pro API queue you will also find example of command to run a subset of tests fetched from API. This might be helpful when you use `--order random` for rspec and you would like to reproduce the tests with the same seed. * You can also retry tests and record the time execution data for them again for the particular CI node. Note you must be checkout on the same branch and git commit as your CI node was. To retry the particular CI node do this on your machine: + RACK_ENV=test \ + RAILS_ENV=test \ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ KNAPSACK_PRO_CI_NODE_TOTAL=2 \ KNAPSACK_PRO_CI_NODE_INDEX=0 \ @@ -726,22 +742,30 @@ (...) Randomized with seed 123 You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node: + RACK_ENV=test \ + RAILS_ENV=test \ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ KNAPSACK_PRO_CI_NODE_TOTAL=2 \ KNAPSACK_PRO_CI_NODE_INDEX=0 \ KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" ### What happens when Knapsack Pro API is not available/not reachable temporarily? -knapsack_pro gem has fallback behaviour and it will split test files across CI nodes based on popular test directory names. +#### for knapack_pro regular mode +knapack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds til it switch to fallback behaviour and it will split test files across CI nodes based on popular test directory names. + +#### for knapsack_pro queue mode + +knapack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds til it fails. + ### How can I change log level? You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment variable. KNAPSACK_PRO_LOG_LEVEL=warn bundle exec rake knapsack_pro:rspec @@ -861,9 +885,65 @@ You can see the list of all pending test files at the end of knapack_pro queue mode command. ### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again? No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once. + +### How to use junit formatter? + +You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter). +Here you can find example how to generate `rspec.xml` file with junit format and at the same time show normal documentation format output for RSpec. + + # Regular Mode + bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" + + # Queue Mode + # The xml report will contain all tests executed across intermediate test subset runs based on queue + bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" + +### Why I see API error commit_hash parameter is required? + + ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]} + +When Knapsack Pro API returns error like above the problem is because you use CI provider not supported by knapack_pro which means +knapack_pro gem cannot determine the git commit hash and branch name. To fix this problem you can do: + +* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) +* if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) + +### How many API keys I need? + +Basically you need as many API keys as you have steps in your build. + +Here is example: + +* Step 1. API_KEY_A for `bundle exec rake knapsack_pro:cucumber` +* Step 2. API_KEY_B for `bundle exec rake knapsack_pro:rspec` +* Step 3. API_KEY_C for `KNAPSACK_PRO_TEST_FILE_PATTERN="specs/features/*_spec.rb" bundle exec rake knapsack_pro:rspec` +* Step 4. API_KEY_D for `bundle exec rake knapsack_pro:rspec[--tag tagA]` +* Step 5. API_KEY_E for `bundle exec rake knapsack_pro:rspec[--tag ~tagA]` +* Step 6. API_KEY_F for `bundle exec rake knapsack_pro:queue:rspec` + +Note: + +* If you specified `KNAPSACK_PRO_TEST_FILE_PATTERN` then you run subset of your whole test suite hence you need separate API key because we want to track only tests for this subset. +* If you pass `--tag tagA` or `--tag ~tagA` then you run subset of your whole test suite hence you need separate API key. +* If you use regular or queue mode then you need separate API key for each mode. + +### What is optimal order of test commands? + +I recommend to run first the test commands in the regular mode and later the commands in the queue mode. + + * Step 1. `bundle exec rake knapsack_pro:cucumber` (regular mode) + * Step 2. `bundle exec rake knapsack_pro:queue:rspec` (queue mode) + +Thanks to that when for some reason the tests executed for cucumber in regular mode will not be well balanced across CI nodes (for instance when one of CI node has bad performance) then the rspec tests executed later in the queue mode will autobalance your build. + +### Why my tests are executed twice in queue mode? Why CI node runs whole test suite again? + +This may happen when one of your CI node started work when all other CI nodes already executed whole test suite. +This slow CI node will initialize a new queue hence the tests executed twice. To solve this problem you should set `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`. +Please [read this](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). ## Gem tests ### Spec