README.md in knapsack_pro-1.15.0 vs README.md in knapsack_pro-1.16.0

- old
+ new

@@ -40,12 +40,16 @@ The knapsack_pro has also [queue mode](#queue-mode) to get an optimal test suite split. ## FAQ -You can see list of questions for common problems and tips in below [Table of Contents](#table-of-contents). Scroll 1 page down and you will see the FAQ in the table of contents. +__NEW:__ Up to date [FAQ for knapsack_pro gem can be found here](https://knapsackpro.com/faq/knapsack_pro_client/knapsack_pro_ruby). +__OLD:__ This README also contains FAQ questions but we keep adding new info only to our new FAQ page mentioned above. + +We keep this old FAQ in README to not break old links spread across the web. You can see old FAQ list of questions for common problems and tips in below [Table of Contents](#table-of-contents). Scroll 1 page down and you will see the FAQ in the table of contents. + # Requirements `>= Ruby 2.0.0` # Table of Contents @@ -173,10 +177,11 @@ - [percy-capybara gem version < 4 (old)](#percy-capybara-gem-version--4-old) - [percy-capybara gem version >= 4](#percy-capybara-gem-version--4) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [What hooks are supported in Queue Mode?](#what-hooks-are-supported-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) + - [Should I use parallel_tests gem (what are pitfalls)?](#should-i-use-parallel_tests-gem-what-are-pitfalls) - [parallel_tests with knapsack_pro on parallel CI nodes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes) - [parallel_tests with knapsack_pro on single CI machine](#parallel_tests-with-knapsack_pro-on-single-ci-machine) - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests) - [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories) - [Why I don't see all test files being recorded in user dashboard](#why-i-dont-see-all-test-files-being-recorded-in-user-dashboard) @@ -867,17 +872,35 @@ end # Here is additional configuration to ensure the xml report will be visible by CircleCI KnapsackPro::Hooks::Queue.after_queue do |queue_id| # Metadata collection - # https://circleci.com/docs/1.0/test-metadata/#metadata-collection-in-custom-test-steps + # https://circleci.com/docs/2.0/collect-test-data/#metadata-collection-in-custom-test-steps if File.exist?(FINAL_RSPEC_XML_REPORT) && ENV['CIRCLE_TEST_REPORTS'] FileUtils.cp(FINAL_RSPEC_XML_REPORT, "#{ENV['CIRCLE_TEST_REPORTS']}/rspec.xml") end end ``` +Ensure you have in CircleCI config yml + +```yaml +- run: + name: RSpec via knapsack_pro Queue Mode + command: | + export CIRCLE_TEST_REPORTS=/tmp/test-results + mkdir $CIRCLE_TEST_REPORTS + bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" + +# collect reports +- store_test_results: + path: /tmp/test-results +- store_artifacts: + path: /tmp/test-results + destination: test-results +``` + #### Info for Travis users You can parallelize your builds across virtual machines with [travis matrix feature](http://docs.travis-ci.com/user/speeding-up-the-build/#parallelizing-your-builds-across-virtual-machines). Edit `.travis.yml` ```yaml @@ -1210,11 +1233,11 @@ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec ``` Remember to add API tokens like `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` and `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `Environment` page of your project settings in Codeship. -If you want to use Codeship retry single CI node feature to retry just failed tests on particular CI node then you should set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). +CodeShip uses the same build number if you restart a build. Because of that you need to set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node) in order to be able to restart CI build in Queue Mode. #### Info for Heroku CI users You can parallelize your tests on Heroku CI by configuring `app.json`. @@ -2646,10 +2669,27 @@ end ``` #### How to run knapsack_pro with parallel_tests gem? +##### Should I use parallel_tests gem (what are pitfalls)? + +If you plan to use parallel_tests please be careful how many parallel processes with running tests you will start on a single CI node. +Often it happens that running 2 or more processes with tests using parallel_tests gem on the CI node that has low performance leads to slower execution of test suite. You can accidentally make your whole test suite running slower by using parallel_tests if you have not enough powerful CI server (slow CPU, not enough RAM, slow disk). + +If you use parallel_tests and knapsack_pro you can see recorded tests timing in Knapsack Pro [user dashboard](https://knapsackpro.com/dashboard). See the "Build metrics" link next to your API token and check the last recorded CI build time. You will be able to see there how long each test files took to execute. If you notice that after adding parallel_tests gem your test files started to take more time than before it means you overloaded your CI server. + +You should: + +* reduce the number of parallel processes in parallel_tests gem +* or buy a more powerful CI node to allow running more parallel processes (vertical scaling) +* or don't use parallel_tests gem at all (recommended) + +In case of tests execution time increase (slower tests) I recommend using more parallel nodes offered by your CI provider to scale your tests horizontally. Basically, adding parallel CI nodes instead of vertically adding more CPU/RAM to CI node is a better option. Parallel_tests gem has mixed output results from the parallel processes so it's easier to just browse tests output from parallel CI nodes when you scale horizontally by using knapsack_pro without parallel_tests. + +If you want to use parallel_tests you can use it with Knapsack Pro Queue Mode to auto-balance tests split across parallel processes started by parallel_tests gem. See below tips on how to do it on [many parallel CI nodes where each node starts many parallel_tests processes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes) or [on a single powerful CI server](#parallel_tests-with-knapsack_pro-on-single-ci-machine). + ##### parallel_tests with knapsack_pro on parallel CI nodes You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands per CI node. Let's consider this example. We have 2 CI node. On each CI node we want to run 2 concurrent knapsack_pro commands by parallel_tests gem (`PARALLEL_TESTS_CONCURRENCY=2`). @@ -2802,10 +2842,10 @@ bundle exec rspec --only-failures ``` #### How can I run tests from multiple directories? -The test file pattern config option supports any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob) and can be configured to pull test files from multiple directories. An example of this when using RSpec would be `""{spec,engines/*/spec}/**/*_spec.rb" "`. For complex cases like this, the test directory can't be extracted and must be specified manually using the `KNAPSACK_PRO_TEST_DIR` environment variable: +The test file pattern config option supports any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob) and can be configured to pull test files from multiple directories. An example of this when using RSpec would be `"{spec,engines/*/spec}/**/*_spec.rb"`. For complex cases like this, the test directory can't be extracted and must be specified manually using the `KNAPSACK_PRO_TEST_DIR` environment variable: ``` # This is example where you have in engines directory multiple projects # and each project directory has a spec folder and you would like to run tests for it. # You want to use the spec_helper from the main spec directory.