README.md in knapsack_pro-0.41.0 vs README.md in knapsack_pro-0.42.0

- old
+ new

@@ -101,10 +101,11 @@ - [Info for Jenkins users](#info-for-jenkins-users) - [FAQ](#faq) - [Common problems](#common-problems) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [Queue Mode problems](#queue-mode-problems) + - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) @@ -126,10 +127,11 @@ - [How many API keys I need?](#how-many-api-keys-i-need) - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) + - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Gem tests](#gem-tests) @@ -360,12 +362,16 @@ bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" Note if you will run queue mode command for the first time it might be slower. The second build should have better optimal test suite split. +Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes. + If you use capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). +If you use rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). + ### Additional info about queue mode * You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode. There might be some cached test suite splits for git commits you run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). @@ -854,10 +860,18 @@ * if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) * if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) #### Queue Mode problems +##### Why I don't see collected time execution data for my build in user dashboard? + +If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure: + +* you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb` +* you explicitly set `RAILS_ENV=test` on your CI nodes (for instance you use Docker then please set `RAILS_ENV`) +* knapsack_pro Queue Mode saves temporary files with collected time execution data in `your_rails_project/tmp/knapsack_pro/queue/`. Please ensure you don't clean `tmp` directory in your tests so knapsack_pro can publish time execution data to Knapsack Pro API server. + ##### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result? The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests). When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. @@ -1265,9 +1279,33 @@ ``` KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=2 PARALLEL_TESTS_CONCURRENCY=2 KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=3 PARALLEL_TESTS_CONCURRENCY=2 (tests output here) ``` + +#### How to retry failed tests (flaky tests)? + +Flaky (nondeterministic) tests, are tests that exhibit both a passing and a failing result with the same code. + +You can use [only failures option in RSpec](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests. + +Please add to your RSpec configuration: + +``` +RSpec.configure do |c| + c.example_status_persistence_file_path = "tmp/rspec_examples.txt" +end +``` + +Then you can execute rspec with only failed tests after main knapsack_pro command finish. + +``` +# Run knapack_pro in Queue Mode and it will save failed tests in tmp/rspec_examples.txt +bundle exec rake knapsack_pro:queue:rspec + +# run only failed tests from tmp/rspec_examples.txt +bundle exec rspec --only-failures +``` ### Questions around data usage and security #### What data is sent to your servers?