README.md in knapsack_pro-1.11.0 vs README.md in knapsack_pro-1.12.0

- old
+ new

@@ -156,29 +156,31 @@ - [How to use junit formatter?](#how-to-use-junit-formatter) - [How to use junit formatter with knapsack_pro regular mode?](#how-to-use-junit-formatter-with-knapsack_pro-regular-mode) - [How to use junit formatter with knapsack_pro queue mode?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode) - [How to use junit formatter with knapsack_pro queue mode when CI nodes use common local drive?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive) - [Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode?](#why-tmprspec_final_resultsxml-is-corrupted-when-i-use-junit-formatter-with-knapsack_pro-queue-mode) + - [How to use junit formatter with knapsack_pro queue mode in Cucumber?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode-in-cucumber) - [How to use JSON formatter for RSpec?](#how-to-use-json-formatter-for-rspec) - [How to use RSpec JSON formatter with knapsack_pro Queue Mode?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode) - [How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive) - [How many API keys I need?](#how-many-api-keys-i-need) - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) + - [What hooks are supported in Queue Mode?](#what-hooks-are-supported-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [parallel_tests with knapsack_pro on parallel CI nodes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes) - [parallel_tests with knapsack_pro on single CI machine](#parallel_tests-with-knapsack_pro-on-single-ci-machine) - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests) - [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories) - [Why I don't see all test files being recorded in user dashboard](#why-i-dont-see-all-test-files-being-recorded-in-user-dashboard) - [Why when I use 2 different CI providers then not all test files are executed?](#why-when-i-use-2-different-ci-providers-then-not-all-test-files-are-executed) - [How to run only RSpec feature tests or non feature tests?](#how-to-run-only-rspec-feature-tests-or-non-feature-tests) - [How to exclude tests from running them?](#how-to-exclude-tests-from-running-them) - [How to run a specific list of test files or only some tests from test file?](#how-to-run-a-specific-list-of-test-files-or-only-some-tests-from-test-file) - - [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro) - [How to run knapsack_pro only on a few parallel CI nodes instead of all?](#how-to-run-knapsack_pro-only-on-a-few-parallel-ci-nodes-instead-of-all) + - [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro) - [How to use simplecov in Queue Mode?](#how-to-use-simplecov-in-queue-mode) - [Do I need to use separate API token for Queue Mode and Regular Mode?](#do-i-need-to-use-separate-api-token-for-queue-mode-and-regular-mode) - [How to stop running tests on the first failed test (fail fast tests in RSpec)?](#how-to-stop-running-tests-on-the-first-failed-test-fail-fast-tests-in-rspec) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) @@ -435,30 +437,36 @@ See how it works and what problems can be solved with Queue Mode https://youtu.be/hUEB1XDKEFY ### How to use queue mode? -Please use a separate API token for queue mode from one used already for regular mode. +Please don't use the same API token to run tests in Regular Mode and Queue Mode at the same time for your daily work. -Use this command to run queue mode: +Only when you setup your project for the first time use the same API token and please record whole test suite with Regular Mode then change knapsack pro command to Queue Mode and keep using the same API token. +Thanks to that your first CI build run in Queue Mode will use timing data recorded with Regular Mode to run tests in Queue Mode faster for the very first run. +Use this command to run Queue Mode: + ```bash # RSpec >= 3.x bundle exec rake knapsack_pro:queue:rspec # Minitest bundle exec rake knapsack_pro:queue:minitest + +# Cucumber +bundle exec rake knapsack_pro:queue:cucumber ``` -If the above command fails then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: +If the above command fails for RSpec then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: ```bash bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" ``` -Note: when you run queue mode command for the first time it might be slower. -The second build should have a more optimal test suite split. +Note: when you run Queue Mode command for the first time without recording tests first in Regular Mode then CI build might be slower (especially for Cucumber). +The second CI build should have optimal test suite split with faster tests distribution across CI nodes in Queue Mode. __Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.__ If you use the capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). @@ -542,10 +550,11 @@ At this moment the queue mode works for: * RSpec * Minitest +* Cucumber ## Extra configuration for CI server ### Info about ENV variables @@ -666,10 +675,11 @@ ```bash knapsack_pro rspec "--tag custom_tag_name --profile" knapsack_pro queue:rspec "--tag custom_tag_name --profile" knapsack_pro cucumber "--name feature" +knapsack_pro queue:cucumber "--name feature" knapsack_pro minitest "--verbose --pride" knapsack_pro queue:minitest "--verbose" knapsack_pro test_unit "--verbose" knapsack_pro spinach "--arg_name value" ``` @@ -799,10 +809,17 @@ name: Minitest via knapsack_pro Queue Mode command: | # export word is important here! export RAILS_ENV=test bundle exec rake "knapsack_pro:queue:minitest[--verbose]" + +- run: + name: Cucumber via knapsack_pro Queue Mode + command: | + # export word is important here! + export RAILS_ENV=test + bundle exec rake knapsack_pro:queue:cucumber ``` Please remember to add additional containers for your project in CircleCI settings. ##### CircleCI and knapsack_pro Queue Mode @@ -1062,10 +1079,11 @@ # bundle exec rake knapsack_pro:spinach # Knapsack Pro Queue Mode (dynamic test suite split) # bundle exec rake knapsack_pro:queue:rspec # bundle exec rake knapsack_pro:queue:minitest + # bundle exec rake knapsack_pro:queue:cucumber ``` Here you can find info [how to configure the GitLab parallel CI nodes](https://docs.gitlab.com/ee/ci/yaml/#parallel). ##### GitLab CI `< 11.5` (old GitLab CI) @@ -1085,20 +1103,23 @@ stage: test script: - export KNAPSACK_PRO_CI_NODE_INDEX=0 # Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split) - bundle exec rake knapsack_pro:cucumber + # or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split) + - bundle exec rake knapsack_pro:queue:cucumber # RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split) # It will autobalance build because it is executed after Cucumber tests. - bundle exec rake knapsack_pro:queue:rspec # second CI node running in parallel test_ci_node_1: stage: test script: - export KNAPSACK_PRO_CI_NODE_INDEX=1 - bundle exec rake knapsack_pro:cucumber + - bundle exec rake knapsack_pro:queue:cucumber - bundle exec rake knapsack_pro:queue:rspec ``` #### Info for codeship.com users @@ -1110,10 +1131,13 @@ # first CI node running in parallel # Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split) KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber +# or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split) +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:cucumber + # RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split) # It will autobalance build because it is executed after Cucumber tests. KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec ``` @@ -1123,10 +1147,13 @@ # second CI node running in parallel # Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split) KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber +# or use Cucumber tests in Knapsack Pro Queue Mode (dynamic test suite split) +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:cucumber + # RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split) # It will autobalance build because it is executed after Cucumber tests. KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec ``` @@ -2014,10 +2041,19 @@ ###### Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode? The `tmp/rspec_final_results.xml` might be corrupted due syntax error in your test suite. First check if your test suite is green. Another reason might be that you did not configure the junit formatter as shown in the example for Queue Mode. Please check above 2 questions & answers explaing that. +###### How to use junit formatter with knapsack_pro queue mode in Cucumber? + +Please provide in `--out` argument directory path where xml files for each test file will be created. It must be a directory in order to work in Queue Mode because in Queue Mode the Cucumber test runner is executed multiple times. +Each time for set of tests fetched from Queue so it means multiple xml files will be created in junit format. + +```bash +bundle exec rake "knapsack_pro:queue:cucumber[--format junit --out tmp/test-reports/cucumber/queue_mode/]" +``` + #### How to use JSON formatter for RSpec? ##### How to use RSpec JSON formatter with knapsack_pro Queue Mode? You need to specify `format` and `out` argument (it's important to provide both). @@ -2158,10 +2194,65 @@ # we are not able to determine which after(:suite) block will be called as the last one # due to the fact the Knapsack Pro Queue Mode allocates tests in dynamic way. end ``` +#### What hooks are supported in Queue Mode? + +* RSpec in knapsack_pro Queue Mode supports hooks: + +```ruby +# spec_helper.rb or rails_helper.rb +KnapsackPro::Hooks::Queue.before_queue do |queue_id| + print 'Before Queue Hook - run before test suite' +end + +# this will be run after set of tests fetched from Queue has been executed +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + print 'After Subset Queue Hook - run after subset of test suite' +end + +KnapsackPro::Hooks::Queue.after_queue do |queue_id| + print 'After Queue Hook - run after test suite' +end +``` + +* Minitest in knapsack_pro Queue Mode supports hooks: + +```ruby +# test/test_helper.rb +KnapsackPro::Hooks::Queue.before_queue do |queue_id| + print 'Before Queue Hook - run before test suite' +end + +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + print 'After Subset Queue Hook - run after subset of test suite' +end + +KnapsackPro::Hooks::Queue.after_queue do |queue_id| + print 'After Queue Hook - run after test suite' +end +``` + +* Cucumber in knapsack_pro Queue Mode supports hooks: + +```ruby +# features/support/knapsack_pro.rb +KnapsackPro::Hooks::Queue.before_queue do |queue_id| + print 'Before Queue Hook - run before test suite' +end + +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + print 'After Subset Queue Hook - run after subset of test suite' +end + +# this hook is not supported and won't run +KnapsackPro::Hooks::Queue.after_queue do |queue_id| + print 'After Queue Hook - run after test suite' +end +``` + #### How to run knapsack_pro with parallel_tests gem? ##### parallel_tests with knapsack_pro on parallel CI nodes You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands per CI node. @@ -2434,16 +2525,10 @@ Note when you set `KNAPSACK_PRO_TEST_FILE_LIST` then below environment variables are ignored: * `KNAPSACK_PRO_TEST_FILE_PATTERN` * `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN` -#### How to use CodeClimate with knapsack_pro? - -You can check CodeClimate docs about [parallel tests](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-tests) and [multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-multiple-test-suites). - -You can also read our article [how to merge CodeClimate reports for parallel jobs (CI nodes)](https://docs.knapsackpro.com/2019/how-to-merge-codeclimate-reports-for-parallel-jobs-ci-nodes). - #### How to run knapsack_pro only on a few parallel CI nodes instead of all? You may want to run knapsack_pro only on a few CI nodes when you would like to run a different job on other CI nodes. For instance, you have 3 parallel CI nodes. You would like to run knapsack_pro only on two CI nodes. The last CI node you want to use for the different job like running linters etc. @@ -2459,9 +2544,15 @@ We decrease the number of CI node total by 1 that knapsack_pro can see. This way you can run tests with knapsack_pro only on two CI nodes. On the 3rd CI node, you can run other things like linters etc. If you would like to check what is the CI node total ENV variable name exposed by your CI provider you can check that in your CI provider environment variables docs or preview the [ENV variables that knapsack_pro can read](https://github.com/KnapsackPro/knapsack_pro-ruby/tree/master/lib/knapsack_pro/config/ci) for supported CI providers. + +#### How to use CodeClimate with knapsack_pro? + +You can check articles about CodeClimate configuration with knapsack_pro gem: +* [CodeClimate and CircleCI 2.0 parallel builds config for RSpec with SimpleCov and JUnit formatter](https://docs.knapsackpro.com/2019/codeclimate-and-circleci-2-0-parallel-builds-config-for-rspec-with-simplecov-and-junit-formatter) +* [How to merge CodeClimate reports for parallel jobs (CI nodes) on Semaphore CI 2.0](https://docs.knapsackpro.com/2019/how-to-merge-codeclimate-reports-for-parallel-jobs-ci-nodes). #### How to use simplecov in Queue Mode? If you would like to make [simplecov](https://github.com/colszowka/simplecov) gem work with knapsack_pro Queue Mode to correctly track code coverage for parallel CI nodes please do: