README.md in knapsack_pro-1.10.1 vs README.md in knapsack_pro-1.11.0

- old
+ new

@@ -118,10 +118,11 @@ - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) - [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed) - [Why I see HEAD as branch name in user dashboard for Build metrics for my API token?](#why-i-see-head-as-branch-name-in-user-dashboard-for-build-metrics-for-my-api-token) - [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation) - [Why knapsack_pro freezes / hangs my CI (for instance Travis)?](#why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis) + - [Why tests hitting external API fail?](#why-tests-hitting-external-api-fail) - [Queue Mode problems](#queue-mode-problems) - [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail) - [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail) - [Why when I use Queue Mode for RSpec then my rake tasks are run twice?](#why-when-i-use-queue-mode-for-rspec-then-my-rake-tasks-are-run-twice) - [Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?](#why-when-i-use-queue-mode-for-rspec-then-i-see-error-superclass-mismatch-for-class) @@ -171,14 +172,16 @@ - [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories) - [Why I don't see all test files being recorded in user dashboard](#why-i-dont-see-all-test-files-being-recorded-in-user-dashboard) - [Why when I use 2 different CI providers then not all test files are executed?](#why-when-i-use-2-different-ci-providers-then-not-all-test-files-are-executed) - [How to run only RSpec feature tests or non feature tests?](#how-to-run-only-rspec-feature-tests-or-non-feature-tests) - [How to exclude tests from running them?](#how-to-exclude-tests-from-running-them) + - [How to run a specific list of test files or only some tests from test file?](#how-to-run-a-specific-list-of-test-files-or-only-some-tests-from-test-file) - [How to use CodeClimate with knapsack_pro?](#how-to-use-codeclimate-with-knapsack_pro) - [How to run knapsack_pro only on a few parallel CI nodes instead of all?](#how-to-run-knapsack_pro-only-on-a-few-parallel-ci-nodes-instead-of-all) - [How to use simplecov in Queue Mode?](#how-to-use-simplecov-in-queue-mode) - [Do I need to use separate API token for Queue Mode and Regular Mode?](#do-i-need-to-use-separate-api-token-for-queue-mode-and-regular-mode) + - [How to stop running tests on the first failed test (fail fast tests in RSpec)?](#how-to-stop-running-tests-on-the-first-failed-test-fail-fast-tests-in-rspec) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Gem tests](#gem-tests) @@ -1407,10 +1410,30 @@ Timecop.return end end ``` +#### Why tests hitting external API fail? + +If you use knapsack_pro and you have tests that do real HTTP requests to external API you need to ensure your tests can be run across parallel CI nodes. + +Let's say you have tests that do requests to Stripe API or any other API. Before running each test you want to make sure Stripe Sandbox is clean up so you have removed all fake subscriptions and customers from Stripe Sandbox. + +```ruby +# RSpec hook +before(:each) do + Stripe::Subscription.all.each { |sub| sub.delete } + Stripe::Customer.all.each { |customer| customer.delete } +end +``` + +But this will cause a problem when 2 different test files will run on 2 different CI nodes at the same time and this hook will be called. You will remove subscriptions and customers while another parallel test was running. Simply speaking you have tests that are written in a way that you can't run them in parallel. + +To fix that you can think of: +* using [VCR](https://github.com/vcr/vcr) gem to record HTTP requests and then instead of doing real HTTP requests just reply recorded requests. +* maybe you could write your tests in a way when you generate some fake customers or subscriptions with fake id and each test has different customer id so there will be no conflict when 2 tests are run at the same time. + #### Queue Mode problems ##### Why when I use Queue Mode for RSpec then my tests fail? knapsack_pro Queue Mode uses `RSpec::Core::Runner` feature that allows [running specs multiple times with different runner options in the same process](https://relishapp.com/rspec/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process). @@ -1419,10 +1442,22 @@ If you have custom things that are not common in how typical RSpec spec looks like then the RSpec feature won't be able to handle it between test suite subset runs. In that case you need to resolve failed tests in a way that allows RSpec to run the tests. Feel free to [ask me for help](https://knapsackpro.com/contact). You can learn more about [recent RSpec team changes](https://github.com/KnapsackPro/knapsack_pro-ruby/pull/42) that was backported into knapsack_pro. +To solve failing tests in Queue Mode you can check: + +* you use full namespacing. If you see error like `NameError: uninitialized constant MyModule::ModelName` then in some cases a top-level constant would be matched if the code hadn't been loaded for the scoped constant. Try to use full namespacing `::SomeModule::MyModule::ModelName` etc. +* you can try to use binary version of knapsack_pro instead of running it via rake task. This helps if your rake tasks mess up with tests and make knapsack_pro Queue Mode fail. [See example](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail): + + ```bash + # Knapsack Pro Queue Mode run via binary + bundle exec knapsack_pro queue:rspec "--profile 10 --format progress" + ``` + +* You can check below questions for common reasons of failing tests in Queue Mode + ##### Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail? You can use [knapsack_pro binary](#knapsack-pro-binary) instead of rake task version to solve problem: ```bash @@ -2310,14 +2345,18 @@ require 'spec_helper' ``` #### Why I don't see all test files being recorded in user dashboard -If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro does not track test files with empty content or when the test file contains only pending tests. +If you open `Build metrics` for particular API token at [user dashboard](https://knapsackpro.com/dashboard) and you don't see all time execution data recorded for all test files then you should know that knapsack_pro version older than [`1.0.2`](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#102) does not track test files with empty content or when the test file contains only pending tests. The test files with pending tests are executed so you will see it in RSpec output but just not recorded in Knapsack Pro API because there is nothing to record time for. +We recommend to update to the latest version of knapsack_pro. + +Please check also this question [why you may don't see time execution data](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) in your dashboard. + #### Why when I use 2 different CI providers then not all test files are executed? Please ensure you use 2 different API token per test suite. If you use 2 CI providers for instance CircleCI and TravisCI at the same time and you run the RSpec test suite then you need to have separate API token for RSpec executed on CircleCI and a separate API token for RSpec test suite executed on the TravisCI. #### How to run only RSpec feature tests or non feature tests? @@ -2381,10 +2420,24 @@ bundle exec rake knapsack_pro:queue:rspec ``` The test file pattern and exclude pattern support any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob). +#### How to run a specific list of test files or only some tests from test file? + +:information_source: If you don't want to use the pattern [`KNAPSACK_PRO_TEST_FILE_PATTERN`](#how-can-i-run-tests-from-multiple-directories) to define a list of tests to run then read below. + +If you want to run a specific list of test files that are explicitly defined by you or auto-generated by some kind of script you created then please use: + +`KNAPSACK_PRO_TEST_FILE_LIST=spec/features/dashboard_spec.rb,spec/models/user.rb:10,spec/models/user.rb:29` + +Note `KNAPSACK_PRO_TEST_FILE_LIST` must be a list of test files comma separated. You can provide line number for tests inside of spec file in case of RSpec (this way you can run only one test or a group of tests from RSpec spec file). You can provide the same file a few times with different test line number. + +Note when you set `KNAPSACK_PRO_TEST_FILE_LIST` then below environment variables are ignored: +* `KNAPSACK_PRO_TEST_FILE_PATTERN` +* `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN` + #### How to use CodeClimate with knapsack_pro? You can check CodeClimate docs about [parallel tests](https://docs.codeclimate.com/docs/configuring-test-coverage#section-parallel-tests) and [multiple test suites](https://docs.codeclimate.com/docs/configuring-test-coverage#section-multiple-test-suites). You can also read our article [how to merge CodeClimate reports for parallel jobs (CI nodes)](https://docs.knapsackpro.com/2019/how-to-merge-codeclimate-reports-for-parallel-jobs-ci-nodes). @@ -2430,9 +2483,35 @@ #### Do I need to use separate API token for Queue Mode and Regular Mode? I recommend to record timing of a new test suite with `API token A` and knapsack_pro Regular Mode. After you recorded test suite timing then you should use the `API token A` to run your tests in knapsack_pro Queue Mode. This way Queue Mode will leverage test suite timing recorded in a fast way with Regular Mode so the first run in Queue Mode won't be slow due to recording test files timing for the first time. When you want to go back from Queue Mode to Regular Mode then the fact of using the same API token could cause edge cases that some builds might not be well balanced in Regular Mode. That is why I recommend using separate API token for Regular Mode and Queue Mode. If you plan to use only Queue Mode then no worry. + +#### How to stop running tests on the first failed test (fail fast tests in RSpec)? + +If you want to stop running tests as soon as one of it fails then you can pass [--fail-fast](https://relishapp.com/rspec/rspec-core/docs/command-line/fail-fast-option) RSpec option to knapsack_pro: + +``` +# Regular Mode +bundle exec rake "knapsack_pro:rspec[--fail-fast]" + +# Queue Mode +bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]" +``` + +You may add a parameter to tell RSpec to stop running the test suite after N failed tests, for example: `--fail-fast=3`. + +``` +Note there is no = sign on purpose here: + +# Regular Mode +bundle exec rake "knapsack_pro:rspec[--fail-fast 3]" + +# Queue Mode +bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]" +``` + +There is a downside to it. If you stop running tests then tests that were never run will have no recorded timing of execution and because of that, the future CI build might have tests split across CI nodes in no optimal way. ### Questions around data usage and security #### What data is sent to your servers?