README.md in knapsack_pro-1.3.0 vs README.md in knapsack_pro-1.4.0

- old
+ new

@@ -18,16 +18,10 @@ * [Spinach](https://github.com/codegram/spinach) * [Turnip](https://github.com/jnicklas/turnip) __Would you like to try knapsack_pro gem?__ You can [get an API token here](http://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=get_api_token). -## Is knapsack_pro gem free? - -* If your __project is open source__ then you can use Knapsack Pro for free. Please let me know via email (arturtrzop@gmail.com) and I will mark your account on KnapsackPro.com as open source. - -* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to a paid plan is a good way to validate that and a way to get support from happy users. Maybe you will be the next one who will join and support the project. Thanks! - # How does knapsack_pro work? ## Basics Basically it will track your branches, commits and for how many CI nodes you are running tests. @@ -44,10 +38,14 @@ * knapsack_pro will run the set of test files which it got from API * once tests are finished, knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs The knapsack_pro has also [queue mode](#queue-mode) to get an optimal test suite split. +## FAQ + +You can see list of questions for common problems and tips in below [Table of Contents](#table-of-contents). Scroll 1 page down and you will see the FAQ in the table of contents. + # Requirements `>= Ruby 2.0.0` # Table of Contents @@ -117,15 +115,17 @@ - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) - [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed) - [Why I see HEAD as branch name in user dashboard for Build metrics for my API token?](#why-i-see-head-as-branch-name-in-user-dashboard-for-build-metrics-for-my-api-token) - [Why Capybara feature tests randomly fail when using CI parallelisation?](#why-capybara-feature-tests-randomly-fail-when-using-ci-parallelisation) + - [Why knapsack_pro freezes / hangs my CI (for instance Travis)?](#why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis) - [Queue Mode problems](#queue-mode-problems) - [Why when I use Queue Mode for RSpec then my tests fail?](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail) - [Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail?](#why-when-i-use-queue-mode-for-rspec-then-factorybotfactorygirl-tests-fail) - [Why when I use Queue Mode for RSpec then I see error `superclass mismatch for class`?](#why-when-i-use-queue-mode-for-rspec-then-i-see-error-superclass-mismatch-for-class) - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) + - [Why all test files have 0.1s time execution for my CI build in user dashboard?](#why-all-test-files-have-01s-time-execution-for-my-ci-build-in-user-dashboard) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) @@ -148,10 +148,13 @@ - [How to use junit formatter?](#how-to-use-junit-formatter) - [How to use junit formatter with knapsack_pro regular mode?](#how-to-use-junit-formatter-with-knapsack_pro-regular-mode) - [How to use junit formatter with knapsack_pro queue mode?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode) - [How to use junit formatter with knapsack_pro queue mode when CI nodes use common local drive?](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive) - [Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode?](#why-tmprspec_final_resultsxml-is-corrupted-when-i-use-junit-formatter-with-knapsack_pro-queue-mode) + - [How to use JSON formatter for RSpec?](#how-to-use-json-formatter-for-rspec) + - [How to use RSpec JSON formatter with knapsack_pro Queue Mode?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode) + - [How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive) - [How many API keys I need?](#how-many-api-keys-i-need) - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) @@ -191,13 +194,13 @@ end ``` And then execute: +```bash +bundle install ``` -$ bundle install -``` If you are not using Rails then add this line at the bottom of `Rakefile`: ```ruby # Add this only if you are not using Rails. @@ -236,11 +239,10 @@ Ensure you have `require false` in your Gemfile for webmock gem (see below) when VCR is hooked into it. That ensures that the webmock configuration in `spec_helper.rb` (above) is loaded properly. ```ruby # Gemfile - group :test do gem 'vcr' gem 'webmock', require: false gem 'fakeweb', require: false # example when you use fakeweb end @@ -348,41 +350,45 @@ #### Set knapsack_pro command to execute tests On your CI server run this command for the first CI node. Update `KNAPSACK_PRO_CI_NODE_INDEX` for the next one. - # Step for RSpec - $ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec - # Step for Cucumber - $ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber - # Step for Minitest - $ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest +# Step for Minitest +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest - # Step for test-unit - $ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit +# Step for test-unit +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit - # Step for Spinach - $ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +# Step for Spinach +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +``` You can add `KNAPSACK_PRO_TEST_FILE_PATTERN` if your tests are not in default directory. For instance: - # Step for RSpec - $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_specs/**{,/*/**}/*_spec.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec +KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_specs/**{,/*/**}/*_spec.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec - # Step for Cucumber - $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber +KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber - # Step for Minitest - $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest +# Step for Minitest +KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest - # Step for test-unit - $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit +# Step for test-unit +KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit - # Step for Spinach - $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +# Step for Spinach +KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +``` __Tip:__ If you use one of the supported CI providers then instead of the above steps you should [take a look at this](#supported-ci-providers). __Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported). @@ -421,19 +427,23 @@ Please use a separate API token for queue mode from one used already for regular mode. Use this command to run queue mode: - # RSpec >= 3.x - bundle exec rake knapsack_pro:queue:rspec +```bash +# RSpec >= 3.x +bundle exec rake knapsack_pro:queue:rspec - # Minitest - bundle exec rake knapsack_pro:queue:minitest +# Minitest +bundle exec rake knapsack_pro:queue:minitest +``` If the above command fails then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: - bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" +```bash +bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" +``` Note: when you run queue mode command for the first time it might be slower. The second build should have a more optimal test suite split. __Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes.__ @@ -519,10 +529,11 @@ ### Supported test runners in queue mode At this moment the queue mode works for: * RSpec +* Minitest ## Extra configuration for CI server ### Info about ENV variables @@ -567,69 +578,89 @@ * you start your tests not at the same time across your CI nodes. For instance, one of the CI node finished faster than the other CI node started. This would change the seed for the second CI node that started later. #### Environment variables for debugging gem -`KNAPSACK_PRO_ENDPOINT` - Default value is `http://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com). +This is only for maintainer of knapsack_pro gem. Not for the end users. -`KNAPSACK_PRO_MODE` - Default value is `production`. When mode is `development` then endpoint is `http://api.knapsackpro.dev:3000`. When mode is `test` then endpoint is `http://api-staging.knapsackpro.com`. +`KNAPSACK_PRO_ENDPOINT` - Default value is `https://api.knapsackpro.com` which is endpoint for [Knapsack Pro API](http://docs.knapsackpro.com). +`KNAPSACK_PRO_MODE` - Default value is `production`. When mode is `development` then endpoint is `http://api.knapsackpro.test:3000`. When mode is `test` then endpoint is `http://api-staging.knapsackpro.com`. + ### Passing arguments to rake task #### Passing arguments to rspec Knapsack Pro allows you to pass arguments through to rspec. For example if you want to run only specs that have the tag `focus`. If you do this with rspec directly it would look like: - $ bundle exec rake rspec --tag focus +```bash +bundle exec rake rspec --tag focus +``` To do this with Knapsack Pro you simply add your rspec arguments as parameters to the knapsack_pro rake task. - $ bundle exec rake "knapsack_pro:rspec[--tag focus]" +```bash +bundle exec rake "knapsack_pro:rspec[--tag focus]" +``` #### Passing arguments to cucumber Add arguments to knapsack_pro cucumber task like this: - $ bundle exec rake "knapsack_pro:cucumber[--name feature]" +```bash +bundle exec rake "knapsack_pro:cucumber[--name feature]" +``` #### Passing arguments to minitest Add arguments to knapsack_pro minitest task like this: - $ bundle exec rake "knapsack_pro:minitest[--arg_name value]" +```bash +bundle exec rake "knapsack_pro:minitest[--arg_name value]" +``` For instance to run verbose tests: - $ bundle exec rake "knapsack_pro:minitest[--verbose]" +```bash +bundle exec rake "knapsack_pro:minitest[--verbose]" +``` #### Passing arguments to test-unit Add arguments to knapsack_pro test-unit task like this: - $ bundle exec rake "knapsack_pro:test_unit[--arg_name value]" +```bash +bundle exec rake "knapsack_pro:test_unit[--arg_name value]" +``` For instance to run verbose tests: - $ bundle exec rake "knapsack_pro:test_unit[--verbose]" +```bash +bundle exec rake "knapsack_pro:test_unit[--verbose]" +``` #### Passing arguments to spinach Add arguments to knapsack_pro spinach task like this: - $ bundle exec rake "knapsack_pro:spinach[--arg_name value]" +```bash +bundle exec rake "knapsack_pro:spinach[--arg_name value]" +``` ### Knapsack Pro binary You can install knapsack_pro globally and use binary. For instance: - $ knapsack_pro rspec "--tag custom_tag_name --profile" - $ knapsack_pro queue:rspec "--tag custom_tag_name --profile" - $ knapsack_pro cucumber "--name feature" - $ knapsack_pro minitest "--verbose --pride" - $ knapsack_pro queue:minitest "--verbose" - $ knapsack_pro test_unit "--verbose" - $ knapsack_pro spinach "--arg_name value" +```bash +knapsack_pro rspec "--tag custom_tag_name --profile" +knapsack_pro queue:rspec "--tag custom_tag_name --profile" +knapsack_pro cucumber "--name feature" +knapsack_pro minitest "--verbose --pride" +knapsack_pro queue:minitest "--verbose" +knapsack_pro test_unit "--verbose" +knapsack_pro spinach "--arg_name value" +``` This is optional way of using knapsack_pro when you don't want to add it to `Gemfile`. ### Test file names encryption @@ -644,11 +675,13 @@ Before you enable test file encryption please ensure you are using fresh API key. You should not use the same API key for encrypted and non encrypted test suite. You can generate API key for your test suite in [your dashboard](https://knapsackpro.com). Next step is to generate salt which will be used to encrypt test files or branch names. - $ bundle exec rake knapsack_pro:salt +```bash +bundle exec rake knapsack_pro:salt +``` Add to your CI server generated environment variable `KNAPSACK_PRO_SALT`. #### How to enable test file names encryption? @@ -656,11 +689,13 @@ #### How to debug test file names? If you need to check what is the encryption hash for particular test file you can check that with the rake task: - $ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_test_file_names[rspec] +```bash +KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_test_file_names[rspec] +``` You can pass the name of test runner like `rspec`, `minitest`, `test_unit`, `cucumber`, `spinach` as argument to rake task. #### How to enable branch names encryption? @@ -676,15 +711,17 @@ #### How to debug branch names? If you need to check what is the encryption hash for particular branch then use the rake task: - # show all local branches and respective hashes - $ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names +```bash +# show all local branches and respective hashes +$ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names - # show hash for branch provided as argument to rake task - $ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names[not-encrypted-branch-name] +# show hash for branch provided as argument to rake task +$ KNAPSACK_PRO_SALT=xxx bundle exec rake knapsack_pro:encrypted_branch_names[not-encrypted-branch-name] +``` ### Supported CI providers #### Info for CircleCI users @@ -726,11 +763,11 @@ parallel: true # Caution: there are 8 spaces indentation! ``` Here is another example for CircleCI 2.0 platform. -```YAML +```yaml # CircleCI 2.0 # some tests that are not balanced and executed only on first CI node - run: case $CIRCLE_NODE_INDEX in 0) npm test ;; esac @@ -830,56 +867,60 @@ #### Info for semaphoreapp.com users Knapsack Pro supports semaphoreapp ENVs `SEMAPHORE_THREAD_COUNT` and `SEMAPHORE_CURRENT_THREAD`. The only thing you need to do is set up knapsack_pro rspec/cucumber/minitest/test_unit command for as many threads as you need. Here is an example: - # Thread 1 - ## Step for RSpec - bundle exec rake knapsack_pro:rspec - ## Step for Cucumber - bundle exec rake knapsack_pro:cucumber - ## Step for Minitest - bundle exec rake knapsack_pro:minitest - ## Step for test-unit - bundle exec rake knapsack_pro:test_unit - ## Step for Spinach - bundle exec rake knapsack_pro:spinach +```bash +# Thread 1 +## Step for RSpec +bundle exec rake knapsack_pro:rspec +## Step for Cucumber +bundle exec rake knapsack_pro:cucumber +## Step for Minitest +bundle exec rake knapsack_pro:minitest +## Step for test-unit +bundle exec rake knapsack_pro:test_unit +## Step for Spinach +bundle exec rake knapsack_pro:spinach - # Thread 2 - ## Step for RSpec - bundle exec rake knapsack_pro:rspec - ## Step for Cucumber - bundle exec rake knapsack_pro:cucumber - ## Step for Minitest - bundle exec rake knapsack_pro:minitest - ## Step for test-unit - bundle exec rake knapsack_pro:test_unit - ## Step for Spinach - bundle exec rake knapsack_pro:spinach +# Thread 2 +## Step for RSpec +bundle exec rake knapsack_pro:rspec +## Step for Cucumber +bundle exec rake knapsack_pro:cucumber +## Step for Minitest +bundle exec rake knapsack_pro:minitest +## Step for test-unit +bundle exec rake knapsack_pro:test_unit +## Step for Spinach +bundle exec rake knapsack_pro:spinach +``` Tests will be split across threads. Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. #### Info for buildkite.com users Knapsack Pro supports buildkite ENVs `BUILDKITE_PARALLEL_JOB_COUNT` and `BUILDKITE_PARALLEL_JOB`. The only thing you need to do is to configure the parallelism parameter in your build step and run the appropiate command in your build - # Step for RSpec - bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec +bundle exec rake knapsack_pro:rspec - # Step for Cucumber - bundle exec rake knapsack_pro:cucumber +# Step for Cucumber +bundle exec rake knapsack_pro:cucumber - # Step for Minitest - bundle exec rake knapsack_pro:minitest +# Step for Minitest +bundle exec rake knapsack_pro:minitest - # Step for test-unit - bundle exec rake knapsack_pro:test_unit +# Step for test-unit +bundle exec rake knapsack_pro:test_unit - # Step for Spinach - bundle exec rake knapsack_pro:spinach +# Step for Spinach +bundle exec rake knapsack_pro:spinach +``` Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. Here you can find article [how to set up a new pipeline for your project in Buildkite and configure Knapsack Pro](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) and 2 example repositories for Ruby/Rails projects: @@ -918,11 +959,11 @@ ##### GitLab CI `< 11.5` (old GitLab CI) GitLab CI does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same `test` stage. Below is relevant part of `.gitlab-ci.yml` configuration for 2 parallel jobs. -``` +```yaml # .gitlab-ci.yml stages: - test variables: @@ -952,11 +993,11 @@ Codeship does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each [parallel test pipeline](https://documentation.codeship.com/basic/builds-and-configuration/parallel-tests/#using-parallel-test-pipelines). Below is an example for 2 parallel test pipelines. Configure test pipelines (1/2 used) -``` +```bash # first CI node running in parallel # Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split) KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber @@ -965,11 +1006,11 @@ KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec ``` Configure test pipelines (2/2 used) -``` +```bash # second CI node running in parallel # Cucumber tests in Knapsack Pro Regular Mode (deterministic test suite split) KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber @@ -992,11 +1033,11 @@ You need to specify also the environment variable with API token for Knapsack Pro. For any sensitive environment variables (like Knapsack Pro API token) that you do not want in your `app.json` manifest, you can add them to your pipeline’s Heroku CI settings. Note the [Heroku CI Parallel Test Runs](https://devcenter.heroku.com/articles/heroku-ci-parallel-test-runs) are in Beta and you may need to ask Heroku support to enabled it for your project. -``` +```json # app.json { "environments": { "test": { "formation": { @@ -1022,107 +1063,115 @@ #### Info for Solano CI users [Solano CI](https://www.solanolabs.com) does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same CI build. - # Step for RSpec for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec - # Step for RSpec for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec +# Step for RSpec for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec - # Step for Cucumber for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber - # Step for Cucumber for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber - # Step for Minitest for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest - # Step for Minitest for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest +# Step for Minitest for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest +# Step for Minitest for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest - # Step for test-unit for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit - # Step for test-unit for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit +# Step for test-unit for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit +# Step for test-unit for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit - # Step for Spinach for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach - # Step for Spinach for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach +# Step for Spinach for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +# Step for Spinach for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach +``` Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. #### Info for AppVeyor users [AppVeyor](https://www.appveyor.com) does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same CI build. - # Step for RSpec for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec - # Step for RSpec for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:rspec +# Step for RSpec for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:rspec - # Step for Cucumber for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber - # Step for Cucumber for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:cucumber +# Step for Cucumber for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:cucumber - # Step for Minitest for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest - # Step for Minitest for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest +# Step for Minitest for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest +# Step for Minitest for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:minitest - # Step for test-unit for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit - # Step for test-unit for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit +# Step for test-unit for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:test_unit +# Step for test-unit for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:test_unit - # Step for Spinach for first CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach - # Step for Spinach for second CI node - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach +# Step for Spinach for first CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach +# Step for Spinach for second CI node +KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:spinach +``` Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. #### Info for snap-ci.com users Knapsack Pro supports snap-ci.com ENVs `SNAP_WORKER_TOTAL` and `SNAP_WORKER_INDEX`. The only thing you need to do is to configure number of workers for your project in configuration settings in order to enable parallelism. Next thing is to set below commands to be executed in your stage: - # Step for RSpec - bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec +bundle exec rake knapsack_pro:rspec - # Step for Cucumber - bundle exec rake knapsack_pro:cucumber +# Step for Cucumber +bundle exec rake knapsack_pro:cucumber - # Step for Minitest - bundle exec rake knapsack_pro:minitest +# Step for Minitest +bundle exec rake knapsack_pro:minitest - # Step for test-unit - bundle exec rake knapsack_pro:test_unit +# Step for test-unit +bundle exec rake knapsack_pro:test_unit - # Step for Spinach - bundle exec rake knapsack_pro:spinach +# Step for Spinach +bundle exec rake knapsack_pro:spinach +``` Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. #### Info for cirrus-ci.org users Knapsack Pro supports cirrus-ci.org ENVs `CI_NODE_TOTAL` and `CI_NODE_INDEX`. The only thing you need to do is to configure number of parallel CI nodes for your project. Next thing is to set one of below commands to be executed on each parallel CI node: - # Step for RSpec - bundle exec rake knapsack_pro:rspec +```bash +# Step for RSpec +bundle exec rake knapsack_pro:rspec - # Step for Cucumber - bundle exec rake knapsack_pro:cucumber +# Step for Cucumber +bundle exec rake knapsack_pro:cucumber - # Step for Minitest - bundle exec rake knapsack_pro:minitest +# Step for Minitest +bundle exec rake knapsack_pro:minitest - # Step for test-unit - bundle exec rake knapsack_pro:test_unit +# Step for test-unit +bundle exec rake knapsack_pro:test_unit - # Step for Spinach - bundle exec rake knapsack_pro:spinach +# Step for Spinach +bundle exec rake knapsack_pro:spinach +``` Please remember to set up API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. Here is example for [`.cirrus.yml` configuration file](https://cirrus-ci.org/examples/#ruby). @@ -1131,11 +1180,11 @@ In order to run parallel jobs with Jenkins you should use Jenkins Pipeline. You can learn basics about it in the article [Parallelism and Distributed Builds with Jenkins](https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins). Here is example `Jenkinsfile` working with Jenkins Pipeline. -``` +```groovy timeout(time: 60, unit: 'MINUTES') { node() { stage('Checkout') { checkout([/* checkout code from git */]) @@ -1236,11 +1285,24 @@ Capybara.default_max_wait_time = 5 # in seconds ``` For instance, this tip might be helpful for Heroku CI users who use Heroku dynos with lower performance. +#### Why knapsack_pro freezes / hangs my CI (for instance Travis)? +[Freeze error can occur for example on Travis CI](https://docs.travis-ci.com/user/common-build-problems/#ruby-tests-frozen-and-cancelled-after-10-minute-log-silence). +The `timecop` gem can result in sporadic freezing due to issues with ordering calls of `Timecop.return`, `Timecop.freeze`, and `Timecop.travel`. For instance, if using RSpec, ensure to have a `Timecop.return` configured to run after all examples: + +```ruby +# in, e.g. spec/spec_helper.rb +RSpec.configure do |c| + c.after(:all) do + Timecop.return + end +end +``` + #### Queue Mode problems ##### Why when I use Queue Mode for RSpec then my tests fail? knapsack_pro Queue Mode uses `RSpec::Core::Runner` feature that allows [running specs multiple times with different runner options in the same process](https://relishapp.com/rspec/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process). @@ -1253,11 +1315,11 @@ ##### Why when I use Queue Mode for RSpec then FactoryBot/FactoryGirl tests fail? You can use [knapsack_pro binary](#knapsack-pro-binary) instead of rake task version to solve problem: -``` +```bash # knapsack_pro binary for Queue Mode $ bundle exec knapsack_pro queue:rspec ``` Other solution is to check if your factories for FactoryBot/FactoryGirl use the same methods as Rake DSL and remove problematic part of the code. @@ -1293,11 +1355,11 @@ superclass mismatch for class BatchClass ``` when you have 2 test files like this one: -``` +```ruby # spec/a_spec.rb class BaseBatchClass end module Mock @@ -1313,11 +1375,11 @@ it do end end ``` -``` +```ruby # spec/b_spec.rb class DifferentBaseBatchClass end module Mock @@ -1344,10 +1406,18 @@ * you have `Knapsack::Adapters::RspecAdapter.bind` in your `rails_helper.rb` or `spec_helper.rb` * you explicitly set `RAILS_ENV=test` on your CI nodes (for instance you use Docker then please set `RAILS_ENV`) * knapsack_pro Queue Mode saves temporary files with collected time execution data in `your_rails_project/tmp/knapsack_pro/queue/`. Please ensure you don't clean `tmp` directory in your tests so knapsack_pro can publish time execution data to Knapsack Pro API server. +##### Why all test files have 0.1s time execution for my CI build in user dashboard? + +If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open CI build for your last git commit you should see there info about collected time execution data from all CI nodes. If you see all test files have 0.1s time execution then please ensure: + +* you don't clean up `tmp` directory in your tests (for instance in RSpec hooks like `before` or `after`) so knapsack_pro can publish measured time execution data to Knapsack Pro API server. knapsack_pro Queue Mode saves temporary files with collected time execution data in `your_rails_project/tmp/knapsack_pro/queue/`. + +The 0.1s is a default time execution used when test file is an empty file or its content are all pending tests. + ##### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result? The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests). When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. @@ -1376,11 +1446,11 @@ ##### How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec? Please use fixed version of capybara-screenshot. -``` +```ruby # Gemfile group :test do gem 'capybara-screenshot', github: 'mattheworiordan/capybara-screenshot', branch: 'master' end ``` @@ -1431,11 +1501,11 @@ > > INFO -- : [knapsack_pro] bundle exec rspec --default-path spec "spec/a_spec.rb" "spec/b_spec.rb" The seed number is used by RSpec only when you tell it, you need to provide argument `--order random`: -``` +```bash bundle exec rake "knapsack_pro:queue:rspec[--order random]" ``` then in RSpec output you will see something like: @@ -1443,11 +1513,11 @@ Randomized with seed 11055 ``` You can use the seed number to run tests in development: -``` +```bash bundle exec rspec --seed 11055 --default-path spec "spec/a_spec.rb" "spec/b_spec.rb" ``` If you don't use RSpec argument `--order random` then you don't need to provide `--seed` number when you want to reproduce tests in development. @@ -1458,30 +1528,36 @@ ##### for knapack_pro regular mode In your development environment you can debug tests that were run on the particular CI node. For instance to run subset of tests for the first CI node with specified seed you can do. - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ - KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ - KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ - KNAPSACK_PRO_CI_NODE_TOTAL=2 \ - KNAPSACK_PRO_CI_NODE_INDEX=0 \ - bundle exec rake "knapsack_pro:rspec[--seed 123]" +```bash +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ +KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ +KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ +KNAPSACK_PRO_CI_NODE_TOTAL=2 \ +KNAPSACK_PRO_CI_NODE_INDEX=0 \ +bundle exec rake "knapsack_pro:rspec[--seed 123]" +``` Above example is for RSpec. You can use respectively rake task name and token environment variable when you want to run tests for minitest, test_unit, cucumber or spinach. It should work when all CI nodes finished work and sent time execution data to Knapsack Pro API. You can visit [user dashboard](https://knapsackpro.com/dashboard) to preview particular CI build and ensure time execution data were collected from all CI nodes. If at least one CI node has not sent time execution data to the Knapsack Pro API then you should check below solution. Check test runner output on particular CI node you would like to retry in development. You should see at the beginning of rspec command an output that can be copied and executed in development. - /Users/ubuntu/.rvm/gems/ruby-2.4.0/gems/rspec-core-3.4.4/exe/rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec +``` +/Users/ubuntu/.rvm/gems/ruby-2.4.0/gems/rspec-core-3.4.4/exe/rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec +``` Command similar to above can be executed in your development this way: - bundle exec rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec +```bash +bundle exec rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec +``` If you were running your tests with `--order random` on your CI then you can additionaly pass seed param with proper value in above command (`--seed 123`). ##### for knapsack_pro queue mode @@ -1493,40 +1569,46 @@ * You can also retry tests and record the time execution data for them again for the particular CI node. Note you must be checkout on the same branch and git commit as your CI node was. To retry the particular CI node do this on your machine: - RACK_ENV=test \ - RAILS_ENV=test \ - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ - KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ - KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ - KNAPSACK_PRO_CI_NODE_TOTAL=2 \ - KNAPSACK_PRO_CI_NODE_INDEX=0 \ - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ - bundle exec rake "knapsack_pro:queue:rspec" + ```bash + RACK_ENV=test \ + RAILS_ENV=test \ + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ + KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ + KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ + KNAPSACK_PRO_CI_NODE_TOTAL=2 \ + KNAPSACK_PRO_CI_NODE_INDEX=0 \ + KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ + bundle exec rake "knapsack_pro:queue:rspec" + ``` If you were running your tests with `--order random` on your CI like this: - bundle exec rake "knapsack_pro:queue:rspec[--order random]" + ```bash + bundle exec rake "knapsack_pro:queue:rspec[--order random]" + ``` Then you can find the seed number visible in rspec output: (...) Randomized with seed 123 You can pass the seed in your local environment to reproduce the tests in the same order as they were executed on CI node: - RACK_ENV=test \ - RAILS_ENV=test \ - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ - KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ - KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ - KNAPSACK_PRO_CI_NODE_TOTAL=2 \ - KNAPSACK_PRO_CI_NODE_INDEX=0 \ - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ - bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" + ```bash + RACK_ENV=test \ + RAILS_ENV=test \ + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ + KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ + KNAPSACK_PRO_PROJECT_DIR=~/projects/rails-app \ + KNAPSACK_PRO_CI_NODE_TOTAL=2 \ + KNAPSACK_PRO_CI_NODE_INDEX=0 \ + KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ + bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" + ``` #### What happens when Knapsack Pro API is not available/not reachable temporarily? ##### for knapack_pro regular mode @@ -1586,18 +1668,20 @@ You need to create API token per each tag. In this example we need 3 different API tokens. You need to run below commands for each CI node. - # run only tests with tagA - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]" +```bash +# run only tests with tagA +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]" - # run only tests with tagB - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]" +# run only tests with tagB +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]" - # run other tests without tag A & B - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]" +# run other tests without tag A & B +KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]" +``` #### How to make knapsack_pro works for forked repositories of my project? Imagine one of the scenarios, for this example I use the Travis-CI. @@ -1609,11 +1693,11 @@ This way forked repositories have working test suite but without optimal test suite split across CI nodes. Create the file `bin/knapsack_pro_rspec` with executable chmod in your main project repository. Below example is for rspec. You can change `$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `$KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` if you use cucumber etc. -``` +```bash #!/bin/bash if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \ bundle exec rake knapsack_pro:rspec # use Regular Mode here always @@ -1634,41 +1718,47 @@ ##### How to use junit formatter with knapsack_pro regular mode? You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter). Here you can find example how to generate `rspec.xml` file with junit format and at the same time show normal documentation format output for RSpec. - # Regular Mode - bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" +```bash +# Regular Mode +bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" +``` ##### How to use junit formatter with knapsack_pro queue mode? You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter). - # Queue Mode - bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" +```bash +# Queue Mode +bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" +``` The xml report will contain all tests executed across intermediate test subset runs based on work queue. You need to add after subset queue hook to rename `rspec.xml` to `rspec_final_results.xml` thanks to that the final results file will contain only single xml tag with all tests executed on the CI node. This is related to the way how queue mode works. Detailed explanation is in the [issue](https://github.com/KnapsackPro/knapsack_pro-ruby/issues/40). - # spec_helper.rb or rails_helper.rb +```ruby +# spec_helper.rb or rails_helper.rb - # TODO This must be the same path as value for rspec --out argument - # Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead. - TMP_RSPEC_XML_REPORT = 'tmp/rspec.xml' - # move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT - FINAL_RSPEC_XML_REPORT = 'tmp/rspec_final_results.xml' +# TODO This must be the same path as value for rspec --out argument +# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead. +TMP_RSPEC_XML_REPORT = 'tmp/rspec.xml' +# move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT +FINAL_RSPEC_XML_REPORT = 'tmp/rspec_final_results.xml' - KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| - if File.exist?(TMP_RSPEC_XML_REPORT) - FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT) - end - end +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + if File.exist?(TMP_RSPEC_XML_REPORT) + FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT) + end +end +``` ###### How to use junit formatter with knapsack_pro queue mode when CI nodes use common local drive? Note if you use a CI provider or your own CI solution that uses common local drive for all parallel CI nodes then above solution needs to be adjusted to produce report file with CI node index number in the file name to avoid file conflicts. Example file name with CI node index number: `tmp/rspec_final_results_N.xml`. -``` +```bash # Queue Mode # must be exported to read correctly the value in below knapsack_pro command export KNAPSACK_PRO_CI_NODE_INDEX=0 # if your CI provider exposes CI node index under other environment variable name then you could use it instead @@ -1676,11 +1766,11 @@ bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]" ``` In below code we use CI node index number in `TMP_RSPEC_XML_REPORT` and `FINAL_RSPEC_XML_REPORT`: -``` +```ruby # spec_helper.rb or rails_helper.rb # TODO This must be the same path as value for rspec --out argument # Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead. TMP_RSPEC_XML_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml" @@ -1697,10 +1787,71 @@ ###### Why `tmp/rspec_final_results.xml` is corrupted when I use junit formatter with knapsack_pro queue mode? The `tmp/rspec_final_results.xml` might be corrupted due syntax error in your test suite. First check if your test suite is green. Another reason might be that you did not configure the junit formatter as shown in the example for Queue Mode. Please check above 2 questions & answers explaing that. +#### How to use JSON formatter for RSpec? + +##### How to use RSpec JSON formatter with knapsack_pro Queue Mode? + +You need to specify `format` and `out` argument (it's important to provide both). + +```bash +# Queue Mode +bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec.json]" +``` + +The JSON report will contain all tests executed across intermediate test subset runs based on work queue. You need to add after subset queue hook to rename `rspec.json` to `rspec_final_results.json` thanks to that the final results file will contain valid json with all tests executed on the CI node. This is related to the way how Queue Mode works. Detailed explanation is in the [issue](https://github.com/KnapsackPro/knapsack_pro-ruby/issues/40). + +```ruby +# spec_helper.rb or rails_helper.rb + +# TODO This must be the same path as value for rspec --out argument +# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.json may not work. Please use full path instead. +TMP_RSPEC_JSON_REPORT = 'tmp/rspec.json' +# move results to FINAL_RSPEC_JSON_REPORT so the results won't accumulate with duplicated JSON in TMP_RSPEC_JSON_REPORT +FINAL_RSPEC_JSON_REPORT = 'tmp/rspec_final_results.json' + +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + if File.exist?(TMP_RSPEC_JSON_REPORT) + FileUtils.mv(TMP_RSPEC_JSON_REPORT, FINAL_RSPEC_JSON_REPORT) + end +end +``` + +###### How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive? + +Note if you use a CI provider or your own CI solution that uses common local drive for all parallel CI nodes then above solution needs to be adjusted to produce report file with CI node index number in the file name to avoid file conflicts. Example file name with CI node index number: `tmp/rspec_final_results_N.json`. + +``` +# Queue Mode + +# must be exported to read correctly the value in below knapsack_pro command +export KNAPSACK_PRO_CI_NODE_INDEX=0 +# if your CI provider exposes CI node index under other environment variable name then you could use it instead + +bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.json]" +``` + +In below code we use CI node index number in `TMP_RSPEC_JSON_REPORT` and `FINAL_RSPEC_JSON_REPORT`: + +```ruby +# spec_helper.rb or rails_helper.rb + +# TODO This must be the same path as value for rspec --out argument +# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.json may not work. Please use full path instead. +TMP_RSPEC_JSON_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json" +# move results to FINAL_RSPEC_JSON_REPORT so the results won't accumulate with duplicated JSON in TMP_RSPEC_JSON_REPORT +FINAL_RSPEC_JSON_REPORT = "tmp/rspec_final_results_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json" + +KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| + if File.exist?(TMP_RSPEC_JSON_REPORT) + FileUtils.mv(TMP_RSPEC_JSON_REPORT, FINAL_RSPEC_JSON_REPORT) + end +end +``` + #### How many API keys I need? Basically you need as many API keys as you have steps in your build. Here is example: @@ -1789,11 +1940,11 @@ Let's consider this example. We have 2 CI node. On each CI node we want to run 2 concurrent knapsack_pro commands by parallel_tests gem (`PARALLEL_TESTS_CONCURRENCY=2`). This means we would have 4 parallel knapack_pro commands in total across all CI nodes. So from knapsack_pro perspective you will have 4 nodes in total. Create in your project directory an executable file `bin/parallel_tests`: -``` +```bash #!/bin/bash # This file should be in bin/parallel_tests # updates CI node total based on parallel_tests concurrency KNAPSACK_PRO_CI_NODE_TOTAL=$(( $PARALLEL_TESTS_CONCURRENCY * $KNAPSACK_PRO_CI_NODE_TOTAL )) @@ -1815,33 +1966,33 @@ Now you need to set parallel_tests command per CI node: * CI node 0 (first CI node): - ``` + ```bash export PARALLEL_TESTS_CONCURRENCY=2; # this must be export RAILS_ENV=test \ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \ KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \ KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \ bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests' ``` * CI node 1 (second CI node): - ``` + ```bash export PARALLEL_TESTS_CONCURRENCY=2; # this must be export RAILS_ENV=test \ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \ KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \ KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \ bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests' ``` Please note you need to update `$YOUR_CI_NODE_TOTAL` and `$YOUR_CI_NODE_INDEX` to the ENVs provided by your CI provider. For instance in case of CircleCI it would be `$CIRCLE_NODE_TOTAL` and `$CIRCLE_NODE_INDEX`. Below is an example for CircleCI configuration: -``` +```yaml # circle.yml for CircleCI 1.0 # KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx can be set in CircleCI ENV settings test: override: - export PARALLEL_TESTS_CONCURRENCY=2; RAILS_ENV=test KNAPSACK_PRO_CI_NODE_TOTAL=$CIRCLE_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$CIRCLE_NODE_INDEX bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests': @@ -1873,19 +2024,19 @@ You can use [only failures option in RSpec](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rerun failed tests. Please add to your RSpec configuration: -``` +```ruby RSpec.configure do |c| c.example_status_persistence_file_path = "tmp/rspec_examples.txt" end ``` Then you can execute rspec with only failed tests after main knapsack_pro command finish. -``` +```bash # Run knapack_pro in Queue Mode and it will save failed tests in tmp/rspec_examples.txt bundle exec rake knapsack_pro:queue:rspec # run only failed tests from tmp/rspec_examples.txt bundle exec rspec --only-failures @@ -1954,20 +2105,20 @@ Run all specs from multiple directories except `spec/features` directory which is not listed below. If you would like to run additional directory please add it after comma in `KNAPSACK_PRO_TEST_FILE_PATTERN`. Ensure the list of directories match your spec directory structure. -``` +```bash KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_NON_FEATURE_TESTS \ KNAPSACK_PRO_TEST_DIR=spec \ KNAPSACK_PRO_TEST_FILE_PATTERN="{spec/*_spec.rb,spec/controllers/**/*_spec.rb,spec/mailers/**/*_spec.rb,spec/models/**/*_spec.rb,spec/presenters/**/*_spec.rb,spec/requests/**/*_spec.rb,spec/routing/**/*_spec.rb,spec/services/**/*_spec.rb,spec/workers/**/*_spec.rb,spec/jobs/**/*_spec.rb}" \ bundle exec rake knapsack_pro:queue:rspec ``` When you would like to run tests only from `spec/features` directory then run: -``` +```bash KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS \ KNAPSACK_PRO_TEST_DIR=spec \ KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \ bundle exec rake knapsack_pro:queue:rspec ``` @@ -1976,18 +2127,18 @@ #### How to exclude tests from running them? For instance you would like to run all tests except tests in `features` directory then you could do: -``` +```bash KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \ bundle exec rake knapsack_pro:queue:rspec ``` You can define at the same time the pattern for tests you would like to run and the exclude pattern. For instance run all controller tests except admin controller tests. -``` +```bash KNAPSACK_PRO_TEST_FILE_PATTERN="spec/controllers/**{,/*/**}/*_spec.rb" \ KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/controllers/admin/**{,/*/**}/*_spec.rb" \ bundle exec rake knapsack_pro:queue:rspec ``` @@ -2019,10 +2170,10 @@ #### How to use simplecov in Queue Mode? If you would like to make [simplecov](https://github.com/colszowka/simplecov) gem work with knapack_pro Queue Mode to correctly track code coverage for parallel CI nodes please do: -``` +```ruby # spec_helper.rb or rails_helper.rb require 'knapsack_pro' require 'simplecov' SimpleCov.start