README.md in knapsack_pro-0.32.0 vs README.md in knapsack_pro-0.33.0

- old
+ new

@@ -72,10 +72,11 @@ - [How queue mode works?](#how-queue-mode-works) - [How to use queue mode?](#how-to-use-queue-mode) - [Additional info about queue mode](#additional-info-about-queue-mode) - [Extra configuration for Queue Mode](#extra-configuration-for-queue-mode) - [KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node)](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node) + - [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests)](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests) - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode) - [Extra configuration for CI server](#extra-configuration-for-ci-server) - [Info about ENV variables](#info-about-env-variables) - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed) - [Environment variables for debugging gem](#environment-variables-for-debugging-gem) @@ -96,37 +97,40 @@ - [Info for semaphoreapp.com users](#info-for-semaphoreappcom-users) - [Info for buildkite.com users](#info-for-buildkitecom-users) - [Info for snap-ci.com users](#info-for-snap-cicom-users) - [Info for Jenkins users](#info-for-jenkins-users) - [FAQ](#faq) - - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - - [for knapack_pro regular mode](#for-knapack_pro-regular-mode) - - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) - - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) - - [for knapack_pro regular mode](#for-knapack_pro-regular-mode-1) - - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode-1) - - [How can I change log level?](#how-can-i-change-log-level) - - [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level) - - [A. Create multiple small test files](#a-create-multiple-small-test-files) - - [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file) - - [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project) + - [Common problems](#common-problems) + - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) + - [Queue Mode problems](#queue-mode-problems) + - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) + - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) + - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) + - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) + - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) + - [General questions](#general-questions) + - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) + - [for knapack_pro regular mode](#for-knapack_pro-regular-mode) + - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) + - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) + - [for knapack_pro regular mode](#for-knapack_pro-regular-mode-1) + - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode-1) + - [How can I change log level?](#how-can-i-change-log-level) + - [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level) + - [A. Create multiple small test files](#a-create-multiple-small-test-files) + - [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file) + - [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project) + - [How to use junit formatter?](#how-to-use-junit-formatter) + - [How many API keys I need?](#how-many-api-keys-i-need) + - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) + - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) + - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) + - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [Questions around data usage and security](#questions-around-data-usage-and-security) - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) - [How is that data secured?](#how-is-that-data-secured) - [Who has access to the data?](#who-has-access-to-the-data) - - [Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result?](#why-when-i-use-queue-mode-for-rspec-and-test-fails-then-i-see-multiple-times-info-about-failed-test-in-rspec-result) - - [Why when I use Queue Mode for RSpec then I see multiple times the same pending tests?](#why-when-i-use-queue-mode-for-rspec-then-i-see-multiple-times-the-same-pending-tests) - - [Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again?](#does-in-queue-mode-the-rspec-is-initialized-many-times-that-causes-rails-load-over-and-over-again) - - [How to use junit formatter?](#how-to-use-junit-formatter) - - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - - [How many API keys I need?](#how-many-api-keys-i-need) - - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - - [Why my tests are executed twice in queue mode? Why CI node runs whole test suite again?](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again) - - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) - - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) - - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) <!-- END doctoc generated TOC please keep comment here to allow auto update --> @@ -390,11 +394,11 @@ #### KNAPSACK_PRO_FIXED_QUEUE_SPLIT (remember queue split on retry CI node) * `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=false` (default) - By default, the fixed queue split is off. It means when you will run tests for the same commit hash and a total number of nodes and for the same branch then each time the queue will be generated dynamically and CI nodes will fetch from Knapsack Pro API the test files in a dynamic way. This is default because it gives the most optimal test suite split for the whole test build across all CI nodes even when you retry the same build on CI many times. + By default, the fixed queue split is off. It means when you will run tests for the same commit hash and a total number of nodes and for the same branch, and the CI build ID is different with second tests run then the queue will be generated dynamically and CI nodes will fetch from Knapsack Pro API the test files in a dynamic way. This is default because it gives the most optimal test suite split for the whole test build across all CI nodes. * `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true` You can enable fixed queue split in order to remember the test suite split across CI nodes when you used Queue Mode. @@ -409,10 +413,22 @@ [knapsack_pro] {"queue_name"=>nil, "test_files"=>[{"path"=>"spec/foo_spec.rb", "time_execution"=>1.23}]} To [reproduce tests executed on CI node](#for-knapsack_pro-queue-mode) in development environment please see FAQ. +#### KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS (hide duplicated summary of pending and failed tests) + +* `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=true` (default) + + By default, the knapack_pro will monkey patch [RSpec Formatters](https://www.relishapp.com/rspec/rspec-core/v/2-6/docs/command-line/format-option) in order to + hide the summary of pending and failed tests after each intermediate run of tests fetched from the work queue on Knapsack Pro API. + knapack_pro shows summary of all pending and failed tests at the very end when work queue ended. If you use your custom formatter and you have problem with it then you can disable `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false` monkey patching. + +* `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false` + + It causes to show summary of pending and failed tests after each intermediate tests run from the work queue. The summary will grown cumulatively after each intermediate tests run so it means you will see multiple times summary of the same pending/failed tests. It doesn't mean the test files are executed twice. Test files are executed only once. Only summary report grows cumulatively. + ### Supported test runners in queue mode At this moment the queue mode works for: * RSpec @@ -789,14 +805,71 @@ Above example shows how to run cucumber tests in regular mode and later the rspec tests in queue mode to autobalance build. If you are going to relay on rspec to autobalance build when cucumber tests were not perfectly distributed you should be aware about [possible edge case if your rspec test suite is very short](#why-my-tests-are-executed-twice-in-queue-mode-why-ci-node-runs-whole-test-suite-again). ## FAQ -### How to run tests for particular CI node in your development environment +### Common problems -#### for knapack_pro regular mode +#### Why I see API error commit_hash parameter is required? + ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]} + +When Knapsack Pro API returns error like above the problem is because you use CI provider not supported by knapack_pro which means +knapack_pro gem cannot determine the git commit hash and branch name. To fix this problem you can do: + +* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) +* if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) + +#### Queue Mode problems + +##### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result? + +The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests). + +When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. +This means RSpec will remember failed tests so far and it will present them at the end of each executed test subset if flag `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false`. +You can see the list of all failed test files at the end of knapack_pro queue mode command. + +##### Why when I use Queue Mode for RSpec then I see multiple times the same pending tests? + +The problem may happen when you use old knapsack_pro `< 0.33.0` or if you use custom rspec formatter, or when you set flag [KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false](#knapsack_pro_modify_default_rspec_formatters-hide-duplicated-summary-of-pending-and-failed-tests). + +When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. +This means RSpec will remember pending tests so far and it will present them at the end of each executed test subset if flag `KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false`. +You can see the list of all pending test files at the end of knapack_pro queue mode command. + +##### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again? + +No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once. + +##### Why my tests are executed twice in queue mode? Why CI node runs whole test suite again? + +This may happen when you use not supported CI provider by knapack_pro. It's because of missing value of CI build ID. You can set unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` for each CI build. The problem with test suite run again happens when one of your CI node started work later when all other CI nodes already executed whole test suite. +The slow CI node that started work late will initialize a new queue hence the tests executed twice. + +To solve this problem you can set `KNAPSACK_PRO_CI_NODE_BUILD_ID` as mentioned above or you can set `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`. +Please [read this](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). + +##### How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec? + +Please use fixed version of capybara-screenshot. + +``` +# Gemfile +group :test do + gem 'capybara-screenshot', github: 'ArturT/capybara-screenshot', branch: 'fix-reporter_module-loaded-twice' +end +``` + +Here is [fix PR](https://github.com/mattheworiordan/capybara-screenshot/pull/205) to official capybara-screenshot repository and the explanation of the problem. + +### General questions + +#### How to run tests for particular CI node in your development environment + +##### for knapack_pro regular mode + In your development environment you can debug tests that were run on the particular CI node. For instance to run subset of tests for the first CI node with specified seed you can do. KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=token \ KNAPSACK_PRO_REPOSITORY_ADAPTER=git \ @@ -819,11 +892,11 @@ bundle exec rspec spec/foo_spec.rb spec/bar_spec.rb --default-path spec If you were running your tests with `--order random` on your CI then you can additionaly pass seed param with proper value in above command (`--seed 123`). -#### for knapsack_pro queue mode +##### for knapsack_pro queue mode There are a few ways to reproduce tests executed on CI node in your development environment. * At the end of `knapsack_pro:queue:rspec` results you will find example of command that you can copy and paste to your development machine. It will run all tests executed on the CI node in a single run. I recommend this approach. @@ -862,21 +935,21 @@ KNAPSACK_PRO_CI_NODE_TOTAL=2 \ KNAPSACK_PRO_CI_NODE_INDEX=0 \ KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \ bundle exec rake "knapsack_pro:queue:rspec[--seed 123]" -### What happens when Knapsack Pro API is not available/not reachable temporarily? +#### What happens when Knapsack Pro API is not available/not reachable temporarily? -#### for knapack_pro regular mode +##### for knapack_pro regular mode knapack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds til it switch to fallback behaviour and it will split test files across CI nodes based on popular test directory names. -#### for knapsack_pro queue mode +##### for knapsack_pro queue mode knapack_pro gem will retry requests to Knapsack Pro API multiple times every few seconds til it fails. -### How can I change log level? +#### How can I change log level? You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment variable. KNAPSACK_PRO_LOG_LEVEL=info bundle exec rake knapsack_pro:rspec @@ -885,20 +958,20 @@ Recommended log levels you can use: * `debug` is default log level and it is recommended to log details about requests to Knapsack Pro API. Thanks to that you can debug things or ensure everything works. For instance in [user dashboard](https://knapsackpro.com/dashboard) you can find tips referring to debug logs. * `info` level shows message like how to retry tests in development or info why something works this way or the other (for instance why tests were not executed on the CI node). You can use `info` level when you really don't want to see all debug messages from default log level. -### How to split tests based on test level instead of test file level? +#### How to split tests based on test level instead of test file level? If you want to split one big test file (test file with long time execution) across multiple CI nodes then you can: -#### A. Create multiple small test files +##### A. Create multiple small test files Create multiple small test files instead of one long running test file with many test cases. A lot of small test files will give you better test suite split results. -#### B. Use tags to mark set of tests in particular test file +##### B. Use tags to mark set of tests in particular test file Another way is to use tags to mark subset of tests in particular test file and then split tests based on tags. Here is example of test file with specified tags for describe groups: @@ -931,11 +1004,11 @@ KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]" # run other tests without tag A & B KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]" -### How to make knapsack_pro works for forked repositories of my project? +#### How to make knapsack_pro works for forked repositories of my project? Imagine one of the scenarios, for this example I use the Travis-CI. * We don’t want to have secrets like the `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in `.travis.yml` in the codebase, because that code is also distributed to clients. * Adding it as env variables to Travis itself is tricky: It has to work for pull requests from developer’s forks into our main fork; this conflicts with the way Travis handles secrets. We also need a fallback if the token is not provided (when developers do builds within their own fork). @@ -959,76 +1032,24 @@ ``` Now you can use `bin/knapsack_pro_rspec` command instead of `bundle exec rake knapsack_pro:rspec`. Remember to follow other steps required for your CI provider. -### Questions around data usage and security +#### How to use junit formatter? -#### What data is sent to your servers? - -The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float. - -Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem. - -#### How is that data secured? - -The test file paths can be [encrypted](#test-file-names-encryption) on your CI node with your salt and later send to knapsackpro.com API. -You generate salt locally and only you can decrypt the test file paths. - -Connection with knapsackpro.com server is via https. - -Regarding payments we use the BraintreePayments.com and they store credit cards and your private information. - -#### Who has access to the data? - -I’m the only admin so I can preview data in case you need help with debugging some problem etc. I’m not able to decrypt them without knowing the salt. - -When you sign in to your user dashboard then you can preview data for recent 100 builds on CI. If the test file paths are encrypted then you only see hashes for test file paths. -You need to [decrypt](#how-to-debug-test-file-names) them locally on your machine to find out what each test file hash is. - -### Why when I use Queue Mode for RSpec and test fails then I see multiple times info about failed test in RSpec result? - -RSpec collects information about failed tests and presents it at the end of RSpec result. -When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. -This means RSpec will remember failed tests so far and it will present them at the end of each executed test subset. -You can see the list of all failed test files at the end of knapack_pro queue mode command. - -### Why when I use Queue Mode for RSpec then I see multiple times the same pending tests? - -RSpec collects information about pending tests and presents it at the end of RSpec result. -When you use Queue Mode then knapack_pro does multiple requests to Knapsack Pro API and fetches a few test files to execute. -This means RSpec will remember pending tests so far and it will present them at the end of each executed test subset. -You can see the list of all pending test files at the end of knapack_pro queue mode command. - -### Does in Queue Mode the RSpec is initialized many times that causes Rails load over and over again? - -No. In Queue Mode the RSpec configuration is updated every time when knapsack_pro gem gets a new set of test files from the Knapsack Pro API and it looks in knapsack_pro output like RSpec was loaded many times but in fact, it loads your project environment only once. - -### How to use junit formatter? - You can use junit formatter for rspec thanks to gem [rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter). Here you can find example how to generate `rspec.xml` file with junit format and at the same time show normal documentation format output for RSpec. # Regular Mode bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" # Queue Mode # The xml report will contain all tests executed across intermediate test subset runs based on queue bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" -### Why I see API error commit_hash parameter is required? +#### How many API keys I need? - ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]} - -When Knapsack Pro API returns error like above the problem is because you use CI provider not supported by knapack_pro which means -knapack_pro gem cannot determine the git commit hash and branch name. To fix this problem you can do: - -* if you have git installed on CI node then you can use it to determine git commit hash and branch name. [See this](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) -* if you have no git installed on CI node then you should manually set `KNAPSACK_PRO_BRANCH` and `KNAPSACK_PRO_COMMIT_HASH`. For instance this might be useful when you use Jenkins. [See this](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) - -### How many API keys I need? - Basically you need as many API keys as you have steps in your build. Here is example: * Step 1. API_KEY_A for `bundle exec rake knapsack_pro:cucumber` @@ -1042,11 +1063,11 @@ * If you specified `KNAPSACK_PRO_TEST_FILE_PATTERN` then you run subset of your whole test suite hence you need separate API key because we want to track only tests for this subset. * If you pass `--tag tagA` or `--tag ~tagA` then you run subset of your whole test suite hence you need separate API key. * If you use regular or queue mode then you need separate API key for each mode. -### What is optimal order of test commands? +#### What is optimal order of test commands? __Tip 1:__ I recommend to run first the test commands in the regular mode and later the commands in the queue mode. @@ -1066,18 +1087,12 @@ * CI 1 * Step 1: `bundle exec rake knapsack_pro:queue:rspec` You will run your javascript tests on single CI node and the knapack_pro will auto-balance CI build with Queue Mode. Thanks to that CI build time execution will be flat and optimal (as fast as possible). -### Why my tests are executed twice in queue mode? Why CI node runs whole test suite again? +#### How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)? -This may happen when one of your CI node started work later when all other CI nodes already executed whole test suite. -The slow CI node will initialize a new queue hence the tests executed twice. To solve this problem you should set `KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true`. -Please [read this](#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node). - -### How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)? - Some tools like [Percy.io](https://percy.io/docs/clients/ruby/capybara-rails) requires to set hooks for RSpec `before(:suite)` and `after(:suite)`. Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will execute multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: ```ruby # spec_helper.rb or rails_helper.rb @@ -1087,11 +1102,11 @@ # executes after Queue Mode finishes work at_exit { Percy::Capybara.finalize_build } ``` -### How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode? +#### How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode? Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will be executed multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: ```ruby # spec_helper.rb or rails_helper.rb @@ -1109,25 +1124,12 @@ # this will be called only once at the end when the CI node finished tests end end ``` -### How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec? +#### How to run knapsack_pro with parallel_tests gem? -Please use fixed version of capybara-screenshot. - -``` -# Gemfile -group :test do - gem 'capybara-screenshot', github: 'ArturT/capybara-screenshot', branch: 'fix-reporter_module-loaded-twice' -end -``` - -Here is [fix PR](https://github.com/mattheworiordan/capybara-screenshot/pull/205) to official capybara-screenshot repository and the explanation of the problem. - -### How to run knapsack_pro with parallel_tests gem? - You can run knapsack_pro with [parallel_tests](https://github.com/grosser/parallel_tests) gem to run multiple concurrent knapsack_pro commands per CI node. Let's consider this example. We have 2 CI node. On each CI node we want to run 2 concurrent knapsack_pro commands by parallel_tests gem (`PARALLEL_TESTS_CONCURRENCY=2`). This means we would have 4 parallel knapack_pro commands in total across all CI nodes. So from knapsack_pro perspective you will have 4 nodes in total. @@ -1204,9 +1206,33 @@ ``` KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=2 PARALLEL_TESTS_CONCURRENCY=2 KNAPSACK_PRO_CI_NODE_TOTAL=4 KNAPSACK_PRO_CI_NODE_INDEX=3 PARALLEL_TESTS_CONCURRENCY=2 (tests output here) ``` + +### Questions around data usage and security + +#### What data is sent to your servers? + +The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float. + +Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem. + +#### How is that data secured? + +The test file paths and/or branch names can be [encrypted](#test-file-names-encryption) on your CI node with a salt and later send to knapsackpro.com API. +You generate the salt locally and only you can decrypt the test file paths or branch names. + +Connection with knapsackpro.com server is via https. + +Regarding payments we use the BraintreePayments.com and they store credit cards and your private information. + +#### Who has access to the data? + +I’m the only admin so I can preview data in case you need help with debugging some problem etc. I’m not able to decrypt them without knowing the salt. + +When you sign in to your user dashboard then you can preview data for recent 100 builds on CI. If the test file paths are encrypted then you only see hashes for test file paths. +You need to [decrypt](#how-to-debug-test-file-names) them locally on your machine to find out what each test file hash is. ## Gem tests ### Spec