README.md in knapsack_pro-0.27.0 vs README.md in knapsack_pro-0.28.0

- old
+ new

@@ -107,10 +107,11 @@ - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) + - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) @@ -345,22 +346,30 @@ * You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode. There might be some cached test suite splits for git commits you run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). * If you are not using one of [supported CI providers](#supported-ci-providers) then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit. - To avoid this you can specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. + To avoid this you should specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. * Note that in the Queue Mode by default you cannot retry the failed CI node with exactly the same subset of tests that were run on the CI node in the first place. It's possible in regular mode ([read more](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)). If you want to have similar behavior in Queue Mode you need to explicitly [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). By default the Queue Mode works this way: - * If you retry the failed build and your all CI nodes will start a new build then there will be a new dynamic test suite split across CI nodes. + * If you retry the failed build and your all CI nodes will start a new build then there will be a new dynamic test suite split across CI nodes. The reason is that the most of the CI providers schedule a new CI build with different ID when you retry CI build. They retry all CI nodes again. In that case you don't have to worry with below edge cases because the CI build ID will be different so a new queue will be initialized on Knapsack Pro API side and all retried CI node will connect to that queue. - * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. The retried CI node will be connected to the queue consumed by still running CI nodes. + Edge cases: - * In the case when there are no running CI nodes then your retried CI node will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run. + * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The retried CI node will be connected to the queue consumed by still running CI nodes. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes already finished work. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The fact is all CI nodes finished work so the queue was consumed. + * If you retry CI node in first hour since the CI build started for the first time then the retried CI node won't execute tests because the queue was consumed. There is important reason why it works like that. For instance some CI providers like Buildkite allows to start CI node later than the others so sometimes the particular CI node may start work while all other CI nodes finished work. In that case we don't want to run tests on the CI node because queue was already consumed. We don't know whether the CI node is part of the build or it is retried CI node hence the 1 hour lock on initializing a new queue. + * If you retry CI node after 1 hour since the CI build started for the first time then the retried CI node will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + + * When you use unsupported CI provider by knapack_pro gem or you forget to set unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` per CI build then: + * when you retry single CI node then it will initialize a new queue and it will run whole test suite from the queue because there will be no other CI nodes running connected to the queue. The order of tests on retried CI node will be different than on the first run. + * when you retry all CI nodes then a new queue will be initialized and all CI nodes will connect to it. + ### Extra configuration for Queue Mode #### KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node) * `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=false` (default) @@ -380,10 +389,12 @@ When Knapsack Pro API server has already information about previous queue split then the information will be used. You will see at the beginning of the knapsack command the log with info that queue name is nil because it was not generated this time. You will get the list of all test files that were executed on the particular CI node in the past. [knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]} + To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ. + ### Supported test runners in queue mode At this moment the queue mode works for: * RSpec @@ -686,39 +697,47 @@ If you were running your tests with `--order random` on your CI then you can additionaly pass seed param with proper value in above command (`--seed 123`). #### for knapsack_pro queue mode -To retry the particular CI node do this on your machine: +There are a few ways to reproduce tests executed on CI node in your development environment. - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ - KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ - KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ - KNAPSACK_PRO_CI_NODE_TOTAL=2 \ - KNAPSACK_PRO_CI_NODE_INDEX=0 \ - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ - bundle exec rake "knapsack_pro:queue:rspec" +* At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run. -If you were running your tests with `--order random` on your CI like this: +* For each intermediate request to Knapsack Pro API queue you will also find example of command to run a subset of tests fetched from API. This might be helpful when you use `--order random` for rspec and you would like to reproduce the tests with the same seed. - bundle exec rake "knapsack_pro:queue:rspec[--order random]" +* You can also retry tests and record the time execution data for them again for the particular CI node. Note you must be checkout on the same branch and git commit as your CI node was. -Then you can find the seed number visible in rspec output: + To retry the particular CI node do this on your machine: - (...) - Randomized with seed 123 + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ + KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ + KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ + KNAPSACK_PRO_CI_NODE_TOTAL=2 \ + KNAPSACK_PRO_CI_NODE_INDEX=0 \ + KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ + bundle exec rake "knapsack_pro:queue:rspec" -You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node: + If you were running your tests with `--order random` on your CI like this: - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ - KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ - KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ - KNAPSACK_PRO_CI_NODE_TOTAL=2 \ - KNAPSACK_PRO_CI_NODE_INDEX=0 \ - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ - bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" + bundle exec rake "knapsack_pro:queue:rspec[--order random]" + Then you can find the seed number visible in rspec output: + + (...) + Randomized with seed 123 + + You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node: + + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ + KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ + KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ + KNAPSACK_PRO_CI_NODE_TOTAL=2 \ + KNAPSACK_PRO_CI_NODE_INDEX=0 \ + KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ + bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" + ### What happens when Knapsack Pro API is not available/not reachable temporarily? knapsack_pro gem has fallback behaviour and it will split test files across CI nodes based on popular test directory names. ### How can I change log level? @@ -831,9 +850,16 @@ RSpec collects information about failed tests and presents it at the end of RSpec result. When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. This means RSpec will remember failed tests so far and it will present them at the end of each executed test subset. You can see the list of all failed test files at the end of knapack_pro queue mode command. + +### Why when I use Queue Mode for RSpec then I see multiple times the same pending tests? + +RSpec collects information about pending tests and presents it at the end of RSpec result. +When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. +This means RSpec will remember pending tests so far and it will present them at the end of each executed test subset. +You can see the list of all pending test files at the end of knapack_pro queue mode command. ### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again? No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once.