README.md in knapsack_pro-0.30.0 vs README.md in knapsack_pro-0.31.0

- old
+ new

@@ -43,11 +43,11 @@ The knapsack_pro has also [queue mode](#queue-mode) to get most optimal test suite split. # Requirements -* >= Ruby 2.0.0 +`>= Ruby 2.0.0` # Table of Contents <!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> @@ -117,10 +117,14 @@ - [How to use junit formatter?](#how-to-use-junit-formatter) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [How many API keys I need?](#how-many-api-keys-i-need) - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) + - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) + - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) + - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) + - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) <!-- END doctoc generated TOC please keep comment here to allow auto update --> @@ -253,13 +257,15 @@ ```ruby # you can use your own logger require 'logger' KnapsackPro.logger = Logger.new(STDOUT) -KnapsackPro.logger.level = Logger::INFO +KnapsackPro.logger.level = Logger::DEBUG ``` +Debug is default log level and it is recommended as default. [Read more](#how-can-i-change-log-level). + ### Setup your CI server (How to set up 2 of 3) #### Set API key token Set one or a few tokens depend on how many test suites you run on CI server. @@ -347,17 +353,12 @@ bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" Note if you will run queue mode command for the first time it might be slower. The second build should have better optimal test suite split. -If you will encounter problem with stack level too deep then you may want to ensure you load your dependencies only once in `spec/rails_helper.rb` or `spec/spec_helper.rb`. The Queue Mode may load multiple times the `rails_helper.rb` hence the problem. For instance the problem occurs for capybara-screenshot gem. Here is the example how you should load the gem. +If you use capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). - unless ENV['KNAPSACK_PRO_RSPEC_DEPENDENCIES_LOADED'] - ENV['KNAPSACK_PRO_RSPEC_DEPENDENCIES_LOADED'] = 'true' - require 'capybara-screenshot/rspec' - end - ### Additional info about queue mode * You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode. There might be some cached test suite splits for git commits you run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). @@ -547,10 +548,12 @@ If you are using circleci.com you can omit `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX`. Knapsack Pro will use `CIRCLE_NODE_TOTAL` and `CIRCLE_NODE_INDEX` provided by CircleCI. Here is an example for test configuration in your `circleci.yml` file. ```yaml +# CircleCI 1.0 + machine: environment: # Tokens should be set in CircleCI settings to avoid expose tokens in build logs # KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: rspec-token # KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: cucumber-token @@ -573,10 +576,24 @@ # Step for Spinach - bundle exec rake knapsack_pro:spinach: parallel: true # Caution: there are 8 spaces indentation! ``` +Here is another example for CircleCI 2.0 platform. + +```YAML +# CircleCI 2.0 + +# some tests that are not balanced and executed only on first CI node +- run: case $CIRCLE_NODE_INDEX in 0) npm test ;; esac + +# auto-balancing CI build time execution to be flat and optimal (as fast as possible). +# Queue Mode does dynamic tests allocation so the previous not balanced run command won't +# create a bottleneck on the CI node +- run: bundle exec rake knapsack_pro:queue:rspec +``` + Please remember to add additional containers for your project in CircleCI settings. #### Info for Travis users You can parallelize your builds across virtual machines with [travis matrix feature](http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines). Edit `.travis.yml` @@ -660,10 +677,15 @@ # Step for Spinach bundle exec rake knapsack_pro:spinach Please remember to set up token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as global environment. +Here you can find article [how to set up a new pipeline for your project in Buildkite and configure Knapsack Pro](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) and 2 example repositories for Ruby/Rails projects: + +* [Buildkite Rails Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) +* [Buildkite Rails Docker Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro) + #### Info for snap-ci.com users Knapsack Pro supports snap-ci.com ENVs `SNAP_WORKER_TOTAL` and `SNAP_WORKER_INDEX`. The only thing you need to do is to configure number of workers for your project in configuration settings in order to enable parallelism. Next thing is to set below commands to be executed in your stage: # Step for RSpec @@ -830,14 +852,19 @@ ### How can I change log level? You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment variable. - KNAPSACK_PRO_LOG_LEVEL=warn bundle exec rake knapsack_pro:rspec + KNAPSACK_PRO_LOG_LEVEL=info bundle exec rake knapsack_pro:rspec -Available values are `debug`, `info`, and `warn`. The default log level is `info`. +Available values are `debug` (default), `info`, `warn`, `error` and `fatal`. +Recommended log levels you can use: + +* `debug` is default log level and it is recommended to log details about requests to Knapsack Pro API. Thanks to that you can debug things or ensure everything works. For instance in [user dashboard](https://knapsackpro.com/dashboard) you can find tips referring to debug logs. +* `info` level shows message like how to retry tests in development or info why something works this way or the other (for instance why tests were not executed on the CI node). You can use `info` level when you really don't want to see all debug messages from default log level. + ### How to split tests based on test level instead of test file level? If you want to split one big test file (test file with long time execution) across multiple CI nodes then you can: #### A. Create multiple small test files @@ -993,21 +1020,169 @@ * If you pass `--tag tagA` or `--tag ~tagA` then you run subset of your whole test suite hence you need separate API key. * If you use regular or queue mode then you need separate API key for each mode. ### What is optimal order of test commands? +__Tip 1:__ + I recommend to run first the test commands in the regular mode and later the commands in the queue mode. * Step 1. `bundle exec rake knapsack_pro:cucumber` (regular mode) * Step 2. `bundle exec rake knapsack_pro:queue:rspec` (queue mode) Thanks to that when for some reason the tests executed for cucumber in regular mode will not be well balanced across CI nodes (for instance when one of CI node has bad performance) then the rspec tests executed later in the queue mode will autobalance your build. +__Tip 2:__ + +When you have short test suite, for instance in javascript then you could distribute tests this way: + +* CI 0 + * Step 1: `npm test` + * Step 2: `bundle exec rake knapsack_pro:queue:rspec` + +* CI 1 + * Step 1: `bundle exec rake knapsack_pro:queue:rspec` + +You will run your javascript tests on single CI node and the knapack_pro will auto-balance CI build with Queue Mode. Thanks to that CI build time execution will be flat and optimal (as fast as possible). + ### Why my tests are executed twice in queue mode? Why CI node runs whole test suite again? This may happen when one of your CI node started work later when all other CI nodes already executed whole test suite. The slow CI node will initialize a new queue hence the tests executed twice. To solve this problem you should set `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`. Please [read this](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + +### How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)? + +Some tools like [Percy.io](https://percy.io/docs/clients/ruby/capybara-rails) requires to set hooks for RSpec `before(:suite)` and `after(:suite)`. +Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will execute multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: + +```ruby +# spec_helper.rb or rails_helper.rb + +# executes before Queue Mode starts work +Percy::Capybara.initialize_build + +# executes after Queue Mode finishes work +at_exit { Percy::Capybara.finalize_build } +``` + +### How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode? + +Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will be executed multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: + +```ruby +# spec_helper.rb or rails_helper.rb + +RSpec.configure do |config| + config.before(:suite) do + unless ENV['KNAPSACK_PRO_RSPEC_BEFORE_SUITE_LOADED'] + ENV['KNAPSACK_PRO_RSPEC_BEFORE_SUITE_LOADED'] = 'true' + + # this will be called only once before the tests started on the CI node + end + end + + at_exit do + # this will be called only once at the end when the CI node finished tests + end +end +``` + +### How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec? + +Please use fixed version of capybara-screenshot. + +``` +# Gemfile +group :test do + gem 'capybara-screenshot', github: 'ArturT/capybara-screenshot', branch: 'fix-reporter_module-loaded-twice' +end +``` + +Here is [fix PR](https://github.com/mattheworiordan/capybara-screenshot/pull/205) to official capybara-screenshot repository and the explanation of the problem. + +### How to run knapsack_pro with parallel_tests gem? + +You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands per CI node. + +Let's consider this example. We have 2 CI node. On each CI node we want to run 2 concurrent knapsack_pro commands by parallel_tests gem (`PARALLEL_TESTS_CONCURRENCY=2`). +This means we would have 4 parallel knapack_pro commands in total across all CI nodes. So from knapsack_pro perspective you will have 4 nodes in total. + +Create in your project directory an executable file `bin/parallel_tests`: + +``` +#!/bin/bash +# This file should be in bin/parallel_tests + +# updates CI node total based on parallel_tests concurrency +KNAPSACK_PRO_CI_NODE_TOTAL=$(( $PARALLEL_TESTS_CONCURRENCY * $KNAPSACK_PRO_CI_NODE_TOTAL )) + +if [ "$TEST_ENV_NUMBER" == "" ]; then + PARALLEL_TESTS_CONCURRENCY_INDEX=0 +else + PARALLEL_TESTS_CONCURRENCY_INDEX=$(( $TEST_ENV_NUMBER - 1 )) +fi + +KNAPSACK_PRO_CI_NODE_INDEX=$(( $PARALLEL_TESTS_CONCURRENCY_INDEX + ($PARALLEL_TESTS_CONCURRENCY * $KNAPSACK_PRO_CI_NODE_INDEX) )) + +# logs info about ENVs to ensure everything works +echo KNAPSACK_PRO_CI_NODE_TOTAL=$KNAPSACK_PRO_CI_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$KNAPSACK_PRO_CI_NODE_INDEX PARALLEL_TESTS_CONCURRENCY=$PARALLEL_TESTS_CONCURRENCY + +# you can customize your knapsack_pro command here to use regular or queue mode +bundle exec rake knapsack_pro:queue:rspec +``` + +Now you need to set parallel_tests command per CI node: + +* CI node 0 (first CI node): + + ``` + export PARALLEL_TESTS_CONCURRENCY=2; # this must be export + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \ + KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \ + KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \ + bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests' + ``` + +* CI node 1 (second CI node): + + ``` + export PARALLEL_TESTS_CONCURRENCY=2; # this must be export + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx \ + KNAPSACK_PRO_CI_NODE_TOTAL=$YOUR_CI_NODE_TOTAL \ + KNAPSACK_PRO_CI_NODE_INDEX=$YOUR_CI_NODE_INDEX \ + bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests' + ``` + +Please note you need to update `$YOUR_CI_NODE_TOTAL` and `$YOUR_CI_NODE_INDEX` to the ENVs provided by your CI provider. For instance in case of CircleCI it would be `$CIRCLE_NODE_TOTAL` and `$CIRCLE_NODE_INDEX`. Below is an example for CircleCI configuration: + +``` +# circle.yml for CircleCI 1.0 +# KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=xxx can be set in CircleCI ENV settings +test: + override: + - export PARALLEL_TESTS_CONCURRENCY=2; KNAPSACK_PRO_CI_NODE_TOTAL=$CIRCLE_NODE_TOTAL KNAPSACK_PRO_CI_NODE_INDEX=$CIRCLE_NODE_INDEX bundle exec parallel_test -n $PARALLEL_TESTS_CONCURRENCY -e './bin/parallel_tests': + parallel: true # Caution: there are 8 spaces indentation! +``` + +In summary, the `bin/parallel_tests` script will calculate a new values for `KNAPSAKC_PRO_*` environment variables and then run knapsack_pro command with them. +To ensure everything works you can check output for each CI node. + +* Expected output for CI node 0 (first CI node): + + ``` + KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=1 PARALLEL_TESTS_CONCURRENCY=2 + KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=0 PARALLEL_TESTS_CONCURRENCY=2 + (tests output here) + ``` + +* Expected output for CI node 1 (second CI node): + + ``` + KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=2 PARALLEL_TESTS_CONCURRENCY=2 + KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=3 PARALLEL_TESTS_CONCURRENCY=2 + (tests output here) + ``` ## Gem tests ### Spec