README.md in knapsack_pro-0.43.0 vs README.md in knapsack_pro-0.44.0

- old
+ new

@@ -3,47 +3,47 @@ [![Circle CI](https://circleci.com/gh/KnapsackPro/knapsack_pro-ruby.svg)](https://circleci.com/gh/KnapsackPro/knapsack_pro-ruby) [![Gem Version](https://badge.fury.io/rb/knapsack_pro.svg)](https://rubygems.org/gems/knapsack_pro) [![Code Climate](https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby/badges/gpa.svg)](https://codeclimate.com/github/KnapsackPro/knapsack_pro-ruby) [![Test Coverage](https://codeclimate.com/github/KnapsackPro/knapsack-pro-ruby/badges/coverage.svg)](https://codeclimate.com/github/KnapsackPro/knapsack-pro-ruby/coverage) -Knapsack Pro gem splits tests across CI nodes and makes sure that tests will run comparable time on each node. It uses [KnapsackPro.com API](http://docs.knapsackpro.com). Original idea came from [knapsack](https://github.com/ArturT/knapsack) gem. +Knapsack Pro gem splits tests across CI nodes and makes sure that tests will run comparable time on each node. It uses [KnapsackPro.com API](http://docs.knapsackpro.com). You can learn more at [https://knapsackpro.com](https://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=learn_more) The knapsack_pro gem supports: * [RSpec](http://rspec.info) * [Cucumber](https://cucumber.io) * [Minitest](http://docs.seattlerb.org/minitest/) * [Spinach](https://github.com/codegram/spinach) * [Turnip](https://github.com/jnicklas/turnip) -__Would you like to try knapsack_pro gem?__ You can [get API token here](http://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=get_api_token). +__Would you like to try knapsack_pro gem?__ You can [get an API token here](http://knapsackpro.com?utm_source=github&utm_medium=readme&utm_campaign=knapsack_pro-ruby_gem&utm_content=get_api_token). ## Is knapsack_pro gem free? * If your __project is open source__ then you can use Knapsack Pro for free. Please let me know via email (arturtrzop@gmail.com) and I will mark your account on KnapsackPro.com as open source. -* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to paid plan is a good way to validate that and a way to get support from happy users. Maybe you will be the next one who will join and support the project. Thanks! +* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to a paid plan is a good way to validate that and a way to get support from happy users. Maybe you will be the next one who will join and support the project. Thanks! -# How knapsack_pro works? +# How does knapsack_pro work? ## Basics Basically it will track your branches, commits and for how many CI nodes you are running tests. -Collected data about test time execution will be send to API where test suite split is done. -Next time when you will run tests you will get proper test files for each CI node in order to achieve comparable time execution on each CI node. +Collected data about test time execution will be sent to the API where the test suite split is done. +The next time you run your tests, each CI node will get an appropriate set of test files in order to achieve comparable time execution on each CI node. ## Details -For instance when you will run tests with `rake knapsack_pro:rspec` then: +For instance when you run tests with `rake knapsack_pro:rspec`: * information about all your existing test files are sent to API http://docs.knapsackpro.com/api/v1/#build_distributions_subset_post -* API returns which files should be executed on particular CI node (example KNAPSACK_PRO_CI_NODE_INDEX=0) -* when API server has info about previous tests runs then it will use it to return more accurate test split results, in other case API returns simple split based on directory names -* knapsack_pro will run test files which got from API -* after tests finished knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs +* the API returns which files should be executed on a particular CI node (example KNAPSACK_PRO_CI_NODE_INDEX=0) +* if the API server has data about previous test runs then it will use this to return more accurate test split results, otherwise the API returns a simple split based on directory names +* knapsack_pro will run the set of test files which it got from API +* once tests are finished, knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs -The knapsack_pro has also [queue mode](#queue-mode) to get optimal test suite split. +The knapsack_pro has also [queue mode](#queue-mode) to get an optimal test suite split. # Requirements `>= Ruby 2.0.0` @@ -65,13 +65,13 @@ - [Setup your CI server (How to set up 2 of 3)](#setup-your-ci-server-how-to-set-up-2-of-3) - [Set API key token](#set-api-key-token) - [Set knapsack_pro command to execute tests](#set-knapsack_pro-command-to-execute-tests) - [Repository adapter (How to set up 3 of 3)](#repository-adapter-how-to-set-up-3-of-3) - [When you NOT set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER` (default)](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) - - [When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) + - [When should you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) - [Queue Mode](#queue-mode) - - [How queue mode works?](#how-queue-mode-works) + - [How does queue mode work?](#how-does-queue-mode-work) - [How to use queue mode?](#how-to-use-queue-mode) - [Additional info about queue mode](#additional-info-about-queue-mode) - [Extra configuration for Queue Mode](#extra-configuration-for-queue-mode) - [KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node)](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node) - [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests) @@ -100,10 +100,11 @@ - [Info for snap-ci.com users](#info-for-snap-cicom-users) - [Info for Jenkins users](#info-for-jenkins-users) - [FAQ](#faq) - [Common problems](#common-problems) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) + - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) - [Queue Mode problems](#queue-mode-problems) - [Why I don't see collected time execution data for my build in user dashboard?](#why-i-dont-see-collected-time-execution-data-for-my-build-in-user-dashboard) - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) @@ -128,27 +129,29 @@ - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [How to retry failed tests (flaky tests)?](#how-to-retry-failed-tests-flaky-tests) + - [How can I run tests from multiple directories?](#how-can-i-run-tests-from-multiple-directories) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) +- [Mentions](#mentions) <!-- END doctoc generated TOC please keep comment here to allow auto update --> ## Update gem -Please check [changelog](CHANGELOG.md) before update gem. Knapsack Pro follows [semantic versioning](http://semver.org). +Please check [changelog](CHANGELOG.md) before updating gem. Knapsack Pro follows [semantic versioning](http://semver.org). ## Installation -Add those lines to your application's Gemfile: +Add these lines to your application's Gemfile: ```ruby group :test, :development do gem 'knapsack_pro' end @@ -157,28 +160,28 @@ And then execute: $ bundle install -Add this lines at the bottom of `Rakefile` if your project has it: +Add this line at the bottom of `Rakefile` if your project has one: ```ruby KnapsackPro.load_tasks if defined?(KnapsackPro) ``` __Please check [online installation guide](http://docs.knapsackpro.com/knapsack_pro-ruby/guide/#questions) to get started.__ It will ask you a few questions and generate instruction steps for your project. -You can read next section only if you want to better understand optional gem configuration and features. +_You only need to read the next section if you want to understand optional gem configuration and features._ ## How to set up -If you use [VCR](https://github.com/vcr/vcr), [WebMock](https://github.com/bblimke/webmock) or [FakeWeb](https://github.com/chrisk/fakeweb) gems then you need to allow them to make requests to Knapsack Pro API. +If you use [VCR](https://github.com/vcr/vcr), [WebMock](https://github.com/bblimke/webmock) or [FakeWeb](https://github.com/chrisk/fakeweb) gems then you need to allow them to make requests to the Knapsack Pro API. For VCR add Knapsack Pro API subdomain to [ignore hosts](https://www.relishapp.com/vcr/vcr/v/2-9-3/docs/configuration/ignore-request): ```ruby -# spec/spec_helper.rb or wherever is your VCR configuration +# spec/spec_helper.rb or wherever your VCR configuration is require 'vcr' VCR.configure do |config| config.hook_into :webmock # or :fakeweb config.ignore_hosts('localhost', '127.0.0.1', '0.0.0.0', 'api.knapsackpro.com') @@ -191,11 +194,11 @@ # add below when you use FakeWeb require 'fakeweb' FakeWeb.allow_net_connect = %r[^https?://api\.knapsackpro\.com] ``` -Ensure you have require false for webmock gem when VCR is hook into it. Thanks to that webmock configuration in `spec_helper.rb` is loaded properly. +Ensure you have `require false` in your Gemfile for webmock gem (see below) when VCR is hooked into it. That ensures that the webmock configuration in `spec_helper.rb` (above) is loaded properly. ```ruby # Gemfile group :test do @@ -205,11 +208,11 @@ end ``` ### Usage (How to set up 1 of 3) -__Tip:__ You can find here example of rails app with already configured knapsack_pro. +__Tip:__ You can find here an example of a rails app with knapsack_pro already configured. https://github.com/KnapsackPro/rails-app-with-knapsack_pro #### Step for RSpec @@ -260,26 +263,26 @@ KnapsackPro::Adapters::SpinachAdapter.bind ``` #### Custom configuration -You can change default Knapsack Pro configuration for RSpec, Cucumber, Minitest or Spinach tests. Here are examples what you can do. Put below configuration instead of `CUSTOM_CONFIG_GOES_HERE`. +You can change the default Knapsack Pro configuration for RSpec, Cucumber, Minitest or Spinach tests. Here are examples what you can do. Put the configuration below in place of `CUSTOM_CONFIG_GOES_HERE` (in the configuration samples above). ```ruby # you can use your own logger require 'logger' KnapsackPro.logger = Logger.new(STDOUT) KnapsackPro.logger.level = Logger::DEBUG ``` -Debug is default log level and it is recommended as default. [Read more](#how-can-i-change-log-level). +Debug is default log level and this is recommended. [Read more](#how-can-i-change-log-level). ### Setup your CI server (How to set up 2 of 3) #### Set API key token -Set one or a few tokens depend on how many test suites you run on CI server. +Set one or more tokens depending on how many test suites you run on CI server. * `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` - as value set token for rspec test suite. Token can be generated when you sign in to [knapsackpro.com](http://www.knapsackpro.com). * `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` - token for cucumber test suite. * `KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST` - token for minitest test suite. * `KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH` - token for minitest test suite. @@ -314,13 +317,13 @@ $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_test.rb" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:minitest # Step for Spinach $ KNAPSACK_PRO_TEST_FILE_PATTERN="directory_with_features/**{,/*/**}/*.feature" KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:spinach -__Tip:__ If you use one of supported CI providers then instead of above steps you should [take a look on this](#supported-ci-providers). +__Tip:__ If you use one of the supported CI providers then instead of the above steps you should [take a look at this](#supported-ci-providers). -__Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported). +__Tip 2:__ If you use one of unsupported CI providers ([here is list of supported CI providers](#supported-ci-providers)) then you should [set KNAPSACK_PRO_REPOSITORY_ADAPTER=git](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported). ### Repository adapter (How to set up 3 of 3) #### When you NOT set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER` (default) @@ -330,62 +333,64 @@ `KNAPSACK_PRO_COMMIT_HASH` - Commit hash. You run tests for this commit. You can also use git as repository adapter to determine branch and commit hash, please see below section. -#### When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported) +#### When should you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported) -`KNAPSACK_PRO_REPOSITORY_ADAPTER` - When it has value `git` then your local version of git on CI server will be used to get info about branch name and commit hash. You need to set also `KNAPSACK_PRO_PROJECT_DIR` with project directory path. +`KNAPSACK_PRO_REPOSITORY_ADAPTER` - When it has the value `git`, your local version of git on CI server will be used to get the branch name and commit hash. You also need to set `KNAPSACK_PRO_PROJECT_DIR` with the project directory path. -`KNAPSACK_PRO_PROJECT_DIR` - Path to the project on CI node for instance `/home/ubuntu/my-app-repository`. It should be main directory of your repository. +`KNAPSACK_PRO_PROJECT_DIR` - Path to the project on the CI node, for instance `/home/ubuntu/my-app-repository`. It should be the top-level directory of your repository. ## Queue Mode -knapsack_pro has built in queue mode designed to solve problem with optimal test suite split in case of random time execution of test files caused by -CI node overload and a random decrease of performance that may affect how long the test files are executed. -The problem with random time execution of test files may be caused by many things like external requests done in tests. +knapsack_pro has a built-in queue mode designed to determine the optimal test suite split even when there is an unpredictably longer time execution of test files on one node (e.g. by +CI node overload and decrease of performance that may affect how long the tests take on that node, or +things like external requests done in individual tests). -### How queue mode works? +### How does queue mode work? -On the Knapsack Pro API side, there is test files queue generated for your CI build. Each of CI node dynamically asks the Knapsack Pro API for test files -that should be executed. Thanks to that each CI node will finish tests at the same time. +On the Knapsack Pro API side, there is test file queue generated for your CI build. Each CI node periodically requests the Knapsack Pro API for test files +that should be executed next. Thanks to that each CI node will finish tests at the same time. ### How to use queue mode? -Please use different API token for queue mode than for regular mode. +Please use a separate API token for queue mode from one used already for regular mode. Use this command to run queue mode: bundle exec rake knapsack_pro:queue:rspec -If above command fails then you may need to explicitly pass an argument to require `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: +If the above command fails then you may need to explicitly pass an argument to require the `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" -Note if you will run queue mode command for the first time it might be slower. -The second build should have better optimal test suite split. +Note: when you run queue mode command for the first time it might be slower. +The second build should have a more optimal test suite split. Please ensure you have explicitly set `RAILS_ENV=test` on your CI nodes. -If you use capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). +If you use the capybara-screenshot gem then please [follow this step](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec). -If you use rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). +If you use the rspec_junit_formatter gem then please [follow this step](#how-to-use-junit-formatter-with-knapsack_pro-queue-mode). +If your test suite is very long and the RSpec output is too long for your CI node then you can set log level `KNAPSACK_PRO_LOG_LEVEL=info` to don't show debug messages in RSpec output. [Read more about log level](#how-can-i-change-log-level). + ### Additional info about queue mode -* You should use different API token for queue mode than for regular mode to avoid problem with test suite split in case you would like to go back to regular mode. -There might be some cached test suite splits for git commits you run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). +* You should use a separate API token for queue mode than for regular mode to avoid problems with test suite split (especially in case you would like to go back to regular mode). +There might be some cached test suite splits for git commits you have run in past for API token you used in queue mode because of the [flag `KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true` for regular mode which is default](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed). -* If you are not using one of [supported CI providers](#supported-ci-providers) then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit. +* If you are not using one of the [supported CI providers](#supported-ci-providers) then please note that the knapsack_pro gem doesn't have a CI build ID in order to generate a queue for each particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit. - To avoid this you should specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. + To avoid this you should specify a unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. * Note that in the Queue Mode by default you cannot retry the failed CI node with exactly the same subset of tests that were run on the CI node in the first place. It's possible in regular mode ([read more](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)). If you want to have similar behavior in Queue Mode you need to explicitly [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). By default the Queue Mode works this way: - * If you retry the failed build and your all CI nodes will start a new build then there will be a new dynamic test suite split across CI nodes. The reason is that the most of the CI providers schedule a new CI build with different ID when you retry CI build. They retry all CI nodes again. In that case you don't have to worry with below edge cases because the CI build ID will be different so a new queue will be initialized on Knapsack Pro API side and all retried CI node will connect to that queue. + * If you retry the failed build and your all CI nodes start this new build then there will be a new dynamic test suite split across CI nodes. The reason is that the most of the CI providers schedule a new CI build with a different ID when you retry CI build. They retry all CI nodes again. In that case you don't have to worry with below edge cases because the CI build ID will be different so a new queue will be initialized on Knapsack Pro API side and all retried CI node will connect to that queue. Edge cases: * Let's say one of the CI nodes failed and you retry just this single CI node while other CI nodes are still running. Let's assume this retried CI node is part of the same CI build ID when you use supported CI provider or `KNAPSACK_PRO_CI_NODE_BUILD_ID` is defined and stays the same. The retried CI node will be connected to the queue consumed by still running CI nodes. You probably would expect the retried CI node to run the tests that were executed there on the first place. To achieve that you need to [enable it](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). @@ -634,11 +639,16 @@ - run: case $CIRCLE_NODE_INDEX in 0) npm test ;; esac # auto-balancing CI build time execution to be flat and optimal (as fast as possible). # Queue Mode does dynamic tests allocation so the previous not balanced run command won't # create a bottleneck on the CI node -- run: bundle exec rake knapsack_pro:queue:rspec +- run: + name: RSpec via knapsack_pro Queue Mode + command: | + # export word is important here! + export RAILS_ENV=test + bundle exec rake "knapsack_pro:queue:rspec[--format documentation]" ``` Please remember to add additional containers for your project in CircleCI settings. #### Info for Travis users @@ -729,10 +739,12 @@ Here you can find article [how to set up a new pipeline for your project in Buildkite and configure Knapsack Pro](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) and 2 example repositories for Ruby/Rails projects: * [Buildkite Rails Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) * [Buildkite Rails Docker Parallel Example with Knapsack Pro](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro) +If you want to use Buildkite retry single agent feature to retry just failed tests on particular agent (CI node) then you should set [`KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + #### Info for Gitlab CI users Gitlab CI does not provide parallel jobs environment variables so you will have to define `KNAPSACK_PRO_CI_NODE_TOTAL` and `KNAPSACK_PRO_CI_NODE_INDEX` for each parallel job running as part of the same `test` stage. Below is relevant part of `.gitlab-ci.yml` configuration for 2 parallel jobs. ``` @@ -855,13 +867,21 @@ ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]} When Knapsack Pro API returns error like above the problem is because you use CI provider not supported by knapack_pro which means knapack_pro gem cannot determine the git commit hash and branch name. To fix this problem you can do: -* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) +* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-should-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) * if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) +#### Why I see `LoadError: cannot load such file -- spec_helper`? + +When your tests fails with: + + LoadError: cannot load such file -- spec_helper + +then problem might be related to the fact you specified complex `KNAPSACK_PRO_TEST_FILE_PATTERN` and knapsack_pro gem cannot detect correct main test directory with spec_helper. You should set `KNAPSACK_PRO_TEST_DIR=spec`. Please [read also example](#how-can-i-run-tests-from-multiple-directories). + #### Queue Mode problems ##### Why I don't see collected time execution data for my build in user dashboard? If you go to [user dashboard](https://knapsackpro.com/dashboard) and open `Build metrics` for your API token and you open build for your last git commit you should see there info about collected time execution data from all CI nodes. If you don't see collected time execution data for CI nodes then please ensure: @@ -1112,11 +1132,11 @@ KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id| # TODO This must be the same path as value for rspec --out argument old_xml_file = 'tmp/rspec.xml' # move results to new_xml_file so the results won't accumulate with duplicated xml tags in old_xml_file new_xml_file = 'tmp/rspec_final_results.xml' - FileUtils.mv(old_xml_file, new_xml_file) if File.exists?(old_xml_file) + FileUtils.mv(old_xml_file, new_xml_file) if File.exist?(old_xml_file) end #### How many API keys I need? Basically you need as many API keys as you have steps in your build. @@ -1304,10 +1324,39 @@ # run only failed tests from tmp/rspec_examples.txt bundle exec rspec --only-failures ``` +#### How can I run tests from multiple directories? + +The test file pattern config option supports any glob pattern handled by [`Dir.glob`](http://ruby-doc.org/core-2.4.1/Dir.html#method-c-glob) and can be configured to pull test files from multiple directories. An example of this when using RSpec would be `""{spec,engines/*/spec}/**/*_spec.rb" "`. For complex cases like this, the test directory can't be extracted and must be specified manually using the `KNAPSACK_PRO_TEST_DIR` environment variable: + +``` +# This is example where you have in engines directory multiple projects +# and each project directory has a spec folder and you would like to run tests for it. +# You want to use the spec_helper from the main spec directory. +# +# Tree: +# * spec +# * engines +# * project_a +# * spec +# * project_b +# * spec +$ KNAPSACK_PRO_TEST_DIR=spec KNAPSACK_PRO_TEST_FILE_PATTERN="{spec,engines/*/spec}/**/*_spec.rb" bundle exec rake knapsack_pro:queue:rspec +``` + +`KNAPSACK_PRO_TEST_DIR` will be your default path for rspec so you should put there your `spec_helper.rb`. Please ensure you will require it in your test files this way if something doesn't work: + +```ruby +# good +require_relative 'spec_helper' + +# bad - won't work +require 'spec_helper' +``` + ### Questions around data usage and security #### What data is sent to your servers? The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float. @@ -1315,11 +1364,11 @@ Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem. #### How is that data secured? The test file paths and/or branch names can be [encrypted](#test-file-names-encryption) on your CI node with a salt and later send to knapsackpro.com API. -You generate the salt locally and only you can decrypt the test file paths or branch names. +You generate the salt locally and __only you__ can decrypt the test file paths or branch names with the salt. Here you can [see how the data are encrypted](lib/knapsack_pro/crypto/digestor.rb). Connection with knapsackpro.com server is via https. Regarding payments we use the BraintreePayments.com and they store credit cards and your private information. @@ -1344,5 +1393,12 @@ 2. Create your feature branch (`git checkout -b my-new-feature`) 3. Commit your changes (`git commit -am 'Add some feature'`) 4. Push to the branch (`git push origin my-new-feature`) 5. You can create example tests in related repository with example of [rails application and knapsack_pro gem usage](https://github.com/KnapsackPro/rails-app-with-knapsack_pro). 6. Create a new Pull Request + +## Mentions + +List of articles where people mentioned Knapsack Pro: + +* [Treat your Build Pipeline as a Product](https://medium.com/mydr-engineering/treat-your-build-pipeline-as-a-product-61a1b24ae538). + * Video - [Your Build Pipeline is a Product](https://www.youtube.com/watch?v=7e8Qk3H6xhg&t=21m14s)