README.md in knapsack_pro-0.53.0 vs README.md in knapsack_pro-0.54.0

- old
+ new

@@ -109,18 +109,21 @@ - [Info for Jenkins users](#info-for-jenkins-users) - [FAQ](#faq) - [Common problems](#common-problems) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) + - [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed) - [Queue Mode problems](#queue-mode-problems) - [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail) + - [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail) - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) + - [Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue.](#parallel-tests-cucumber-and-rspec-with-cucumber-failures-exit-ci-node-early-leaving-fewer-ci-nodes-to-finish-rspec-queue) - [General questions](#general-questions) - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - [for knapack_pro regular mode](#for-knapack_pro-regular-mode) - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) @@ -139,10 +142,15 @@ - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests) - [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories) + - [Why I don't see all test files being recorded in user dashboard](#why-i-dont-see-all-test-files-being-recorded-in-user-dashboard) + - [Why when I use 2 different CI providers then not all test files are executed?](#why-when-i-use-2-different-ci-providers-then-not-all-test-files-are-executed) + - [How to run only RSpec feature tests or non feature tests?](#how-to-run-only-rspec-feature-tests-or-non-feature-tests) + - [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro) + - [How to run knapsack_pro only on a few parallel CI nodes instead of all?](#how-to-run-knapsack_pro-only-on-a-few-parallel-ci-nodes-instead-of-all) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Gem tests](#gem-tests) @@ -389,20 +397,24 @@ Please use a separate API token for queue mode from one used already for regular mode. Use this command to run queue mode: + # RSpec bundle exec rake knapsack_pro:queue:rspec + # Minitest + bundle exec rake knapsack_pro:queue:minitest + If the above command fails then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" Note: when you run queue mode command for the first time it might be slower. The second build should have a more optimal test suite split. -Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes. +__Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.__ If you use the capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). If you use the rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). @@ -450,18 +462,26 @@ It means when you run test suite or just retry single CI node again for the same commit hash and a total number of nodes and for the same branch then you will get exactly the same test suite split as it was when you run the build for the first time. Thanks to that when tests on one of your node failed you can retry the node with exactly the same subset of tests that were run on the node in the first place. - Note when fixed queue split is enabled then you can run tests in a dynamic way only once for particular commit hash and a total number of nodes and for the same branch. + * Note when fixed queue split is enabled then you can run tests in a dynamic way only once for particular commit hash and a total number of nodes and for the same branch. - When Knapsack Pro API server has already information about previous queue split then the information will be used. You will see at the beginning of the knapsack command the log with info that queue name is nil because it was not generated this time. You will get the list of all test files that were executed on the particular CI node in the past. + * When Knapsack Pro API server has already information about previous queue split then the information will be used. You will see at the beginning of the knapsack command the log with info that queue name is nil because it was not generated this time. You will get the list of all test files that were executed on the particular CI node in the past. - [knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]} + ``` + [knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]} + ``` - To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ. + * Knapsack Pro is fault-tolerant and can withstand possible CI instance preemptions (shut down) when you use highly affordable CI nodes like [Google Cloud Preemptible VMs](https://cloud.google.com/preemptible-vms/) or [Amazon EC2 Spot Instances](https://aws.amazon.com/ec2/spot/). When you retry failed CI node or when your CI provider will do auto retry then the knapsack_pro will run tests previosly served to CI node that failed. After that it will try to consume the test files from the Queue if there are remaining test files that were not yet executed. You will see in the logs info that you retry the tests if the `queue_name` has prefix `retry-dead-ci-node`: + ``` + [knapsack_pro] {"queue_name"=>"retry-dead-ci-node:queue-id", "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]} + ``` + + * To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ. + #### KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests) * `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=true` (default) By default, the knapack_pro will monkey patch [RSpec Formatters](https://www.relishapp.com/rspec/rspec-core/v/2-6/docs/command-line/format-option) in order to @@ -579,10 +599,11 @@ $ knapsack_pro rspec "--tag custom_tag_name --profile" $ knapsack_pro queue:rspec "--tag custom_tag_name --profile" $ knapsack_pro cucumber "--name feature" $ knapsack_pro minitest "--verbose --pride" + $ knapsack_pro queue:minitest "--verbose" $ knapsack_pro test_unit "--verbose" $ knapsack_pro spinach "--arg_name value" This is optional way of using knapsack_pro when you don't want to add it to `Gemfile`. @@ -696,10 +717,17 @@ name: RSpec via knapsack_pro Queue Mode command: | # export word is important here! export RAILS_ENV=test bundle exec rake "knapsack_pro:queue:rspec[--format documentation]" + +- run: + name: Minitest via knapsack_pro Queue Mode + command: | + # export word is important here! + export RAILS_ENV=test + bundle exec rake "knapsack_pro:queue:minitest[--verbose]" ``` Please remember to add additional containers for your project in CircleCI settings. ##### CircleCI and knapsack_pro Queue Mode @@ -899,10 +927,12 @@ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec ``` Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `Environment` page of your project settings in Codeship. +If you want to use Codeship retry single CI node feature to retry just failed tests on particular CI node then you should set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + #### Info for Heroku CI users You can parallelize your tests on Heroku CI by configuring `app.json`. You can set how many parallel dynos with tests you want to run with `quantity` value. @@ -1043,10 +1073,14 @@ LoadError: cannot load such file -- spec_helper then problem might be related to the fact you specified complex `KNAPSACK_PRO_TEST_FILE_PATTERN` and knapsack_pro gem cannot detect correct main test directory with spec_helper. You should set `KNAPSACK_PRO_TEST_DIR=spec`. Please [read also example](#how-can-i-run-tests-from-multiple-directories). +#### Why my CI build fails when I use Test::Unit even when all tests passed? + +Please ensure you are actually using only Test::Unit runner. You may use some hybrid of Test::Unit and Minitest. Ensure you are not loading Minitest. + #### Queue Mode problems ##### Why when I use Queue Mode for RSpec then my tests fail? knapsack_pro Queue Mode uses `RSpec::Core::Runner` feature that allows [running specs multiple times with different runner options in the same process](https://relishapp.com/rspec/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process). @@ -1055,10 +1089,43 @@ If you have custom things that are not common in how typical RSpec spec looks like then the RSpec feature won't be able to handle it between test suite subset runs. In that case you need to resolve failed tests in a way that allows RSpec to run the tests. Feel free to [ask me for help](https://knapsackpro.com/contact). You can learn more about [recent RSpec team changes](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/42) that was backported into knapsack_pro. +##### Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail? + +You can use [knapsack_pro binary](#knapsack-pro-binary) instead of rake task version to solve problem: + +``` +# knapsack_pro binary for Queue Mode +$ bundle exec knapsack_pro queue:rspec +``` + +Other solution is to check if your factories for FactoryBot/FactoryGirl use the same methods as Rake DSL and remove problematic part of the code. + +The use of implicit association `task` can cause a problem. + +```ruby +# won't work in knapsack_pro Queue Mode +FactoryBot.define do + factory :assignment do + task + end +end +``` + +Workaround is to replace `task` with explicit association: + +```ruby +# this will work in knapack_pro Queue Mode +FactoryBot.define do + factory :assignment do + association :task + end +end +``` + ##### Why I don't see collected time execution data for my build in user dashboard? If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure: * you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb` @@ -1104,10 +1171,27 @@ end ``` Here is [fix PR](https://github.com/mattheworiordan/capybara-screenshot/pull/205) to official capybara-screenshot repository and the explanation of the problem. +##### Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue. + +If you run tests in 2 steps like: + +* Step 1. `bundle exec rake knapsack_pro:cucumber` (regular mode) +* Step 2. `bundle exec rake knapsack_pro:queue:rspec` (queue mode) + +and your CI provider is configured to fail fast when one of the steps fails then in the case when the first step with Cucumber fails on one of CI nodes then the second step with RSpec in Queue Mode won't start on the CI node that failed fast. + +It means the other CI nodes that will run the second step for RSpec in Queue Mode will consume the whole RSpec Queue so your whole CI build will take more than typical CI build when all Cucumber tests are green. + +You should configure your CI provider to not fail fast the Cucumber step. + +CI providers tips: + +* If you use CircleCI 2.0 you can use `when=always` flag. Read more [here](https://discuss.circleci.com/t/parallel-tests-cuc-rspec-w-failures-exit-early-leaving-less-workers-to-finish/18081). + ### General questions #### How to run tests for particular CI node in your development environment ##### for knapack_pro regular mode @@ -1534,9 +1618,85 @@ require_relative 'spec_helper' # bad - won't work require 'spec_helper' ``` + +#### Why I don't see all test files being recorded in user dashboard + +If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro does not track test files with empty content or when the test file contains only pending tests. + +The test files with pending tests are executed so you will see it in RSpec output but just not recorded in Knapsack Pro API because there is nothing to record time for. + +#### Why when I use 2 different CI providers then not all test files are executed? + +Please ensure you use 2 different API token per test suite. If you use 2 CI providers for instance CircleCI and TravisCI at the same time and you run the RSpec test suite then you need to have separate API token for RSpec executed on CircleCI and a separate API token for RSpec test suite executed on the TravisCI. + +#### How to run only RSpec feature tests or non feature tests? + +**Option 1: RSpec tags** + +You can run just feature tests this way. You need to generate a separate API token for it. + +``` +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS bundle exec rake "knapsack_pro:queue:rspec[--tag type:feature]" +``` + +If you would like to run only non feature tests then use negation `~type:feature`: + +``` +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_NON_FEATURE_TESTS bundle exec rake "knapsack_pro:queue:rspec[--tag ~type:feature]" +``` + +Note above examples are for knapack_pro Queue Mode and when you will run tests you may notice that all test files are run by RSpec but only tests specified by tag like `tag type:feature` will be executed. Basically RSpec will just load all files but run just specified tags. + +**Option 2: specify directory pattern** + +Another approach is to explicitly specify which files should be executed. + +Run all specs from multiple directories except `spec/features` directory which is not listed below. +If you would like to run additional directory please add it after comma in `KNAPSACK_PRO_TEST_FILE_PATTERN`. +Ensure the list of directories match your spec directory structure. + +``` +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_NON_FEATURE_TESTS \ +KNAPSACK_PRO_TEST_DIR=spec \ +KNAPSACK_PRO_TEST_FILE_PATTERN="{spec/*_spec.rb,spec/controllers/**/*_spec.rb,spec/mailers/**/*_spec.rb,spec/models/**/*_spec.rb,spec/presenters/**/*_spec.rb,spec/requests/**/*_spec.rb,spec/routing/**/*_spec.rb,spec/services/**/*_spec.rb,spec/workers/**/*_spec.rb,spec/jobs/**/*_spec.rb}" \ +bundle exec rake knapsack_pro:queue:rspec +``` + +When you would like to run tests only from `spec/features` directory then run: + +``` +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS \ +KNAPSACK_PRO_TEST_DIR=spec \ +KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \ +bundle exec rake knapsack_pro:queue:rspec +``` + +#### How to use CodeClimate with knapsack_pro? + +You can check CodeClimate docs about [parallel and multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-and-multiple-test-suites). + +#### How to run knapsack_pro only on a few parallel CI nodes instead of all? + +You may want to run knapack_pro only on a few CI nodes when you would like to run a different job on other CI nodes. + +For instance, you have 3 parallel CI nodes. You would like to run knapack_pro only on two CI nodes. The last CI node you want to use for the different job like running linters etc. + +In such case, you can override the number of total CI nodes available by your CI provider. For instance, Heroku CI provider exposes in ENV variables `CI_NODE_TOTAL=3`. + +You can then run knapsack_pro command this way on the first and the second CI node: + +``` +KNAPSACK_PRO_CI_NODE_TOTAL=$((CI_NODE_TOTAL-1)) bundle exec rake knapsack_pro:rspec +``` + +We decrease the number of CI node total by 1 that knapack_pro can see. This way you can run tests with knapsack_pro only on two CI nodes. +On the 3rd CI node, you can run other things like linters etc. + +If you would like to check what is the CI node total ENV variable name exposed by your CI provider you can check that in your CI provider environment variables docs +or preview the [ENV variables that knapack_pro can read](https://github.com/KnapsackPro/knapsack_pro-ruby/tree/master/lib/knapsack_pro/config/ci) for supported CI providers. ### Questions around data usage and security #### What data is sent to your servers?