README.md in knapsack_pro-0.17.0 vs README.md in knapsack_pro-0.18.0

- old
+ new

@@ -19,11 +19,11 @@ ## Is knapsack_pro gem free? * If your __project is open source__ then you can use Knapsack Pro for free. Just add a link back to the page http://knapsackpro.com and please let me know via email (arturtrzop@gmail.com). I will mark your account on KnapsackPro.com as open source. -* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to paid plan is a good way to validate that and a way to get support from happy users. Currently, there is no hard limit when the free trial is ending. I'm more interested in getting feedback from users in order to improve the tool. As I saw in the past people want to support the project and are asking for paid plan. Maybe you will be the next one who will join and support the project. Thanks! +* If your __project is commercial__ then I'd like to get feedback from you and work closely together to validate if the solution I'm building provide a value for the users. Switching to paid plan is a good way to validate that and a way to get support from happy users. Maybe you will be the next one who will join and support the project. Thanks! # How knapsack_pro works? ## Basics @@ -31,18 +31,20 @@ Collected data about test time execution will be send to API where test suite split is done. Next time when you will run tests you will get proper test files for each CI node in order to achieve comparable time execution on each CI node. ## Details -For instance when you will run tests with rake knapsack_pro:rspec then: +For instance when you will run tests with `rake knapsack_pro:rspec` then: * information about all your existing test files are sent to API http://docs.knapsackpro.com/api/v1/#build_distributions_subset_post * API returns which files should be executed on particular CI node (example KNAPSACK_PRO_CI_NODE_INDEX=0) * when API server has info about previous tests runs then it will use it to return more accurate test split results, in other case API returns simple split based on directory names * knapsack_pro will run test files which got from API * after tests finished knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs +The knapsack_pro has also [queue mode](#queue-mode) to get most optimal test suite split. + # Requirements * >= Ruby 2.0.0 # Table of Contents @@ -64,10 +66,15 @@ - [Set API key token](#set-api-key-token) - [Set knapsack_pro command to execute tests](#set-knapsack_pro-command-to-execute-tests) - [Repository adapter (How to set up 3 of 3)](#repository-adapter-how-to-set-up-3-of-3) - [When you NOT set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER` (default)](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default) - [When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported) +- [Queue Mode](#queue-mode) + - [How queue mode works?](#how-queue-mode-works) + - [How to use queue mode?](#how-to-use-queue-mode) + - [Additional info about queue mode](#additional-info-about-queue-mode) + - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode) - [Extra configuration for CI server](#extra-configuration-for-ci-server) - [Info about ENV variables](#info-about-env-variables) - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed) - [Environment variables for debugging gem](#environment-variables-for-debugging-gem) - [Passing arguments to rake task](#passing-arguments-to-rake-task) @@ -87,10 +94,18 @@ - [Info for snap-ci.com users](#info-for-snap-cicom-users) - [FAQ](#faq) - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) - [How can I change log level?](#how-can-i-change-log-level) + - [How to split tests based on test level instead of test file level?](#how-to-split-tests-based-on-test-level-instead-of-test-file-level) + - [A. Create multiple small test files](#a-create-multiple-small-test-files) + - [B. Use tags to mark set of tests in particular test file](#b-use-tags-to-mark-set-of-tests-in-particular-test-file) + - [How to make knapsack_pro works for forked repositories of my project?](#how-to-make-knapsack_pro-works-for-forked-repositories-of-my-project) + - [Questions around data usage and security](#questions-around-data-usage-and-security) + - [What data is sent to your servers?](#what-data-is-sent-to-your-servers) + - [How is that data secured?](#how-is-that-data-secured) + - [Who has access to the data?](#who-has-access-to-the-data) - [Gem tests](#gem-tests) - [Spec](#spec) - [Contributing](#contributing) <!-- END doctoc generated TOC please keep comment here to allow auto update --> @@ -269,10 +284,45 @@ `KNAPSACK_PRO_REPOSITORY_ADAPTER` - When it has value `git` then your local version of git on CI server will be used to get info about branch name and commit hash. You need to set also `KNAPSACK_PRO_PROJECT_DIR` with project directory path. `KNAPSACK_PRO_PROJECT_DIR` - Path to the project on CI node for instance `/home/ubuntu/my-app-repository`. It should be main directory of your repository. +## Queue Mode + +knapsack_pro has built in queue mode designed to solve problem with optimal test suite split in case of random time execution of test files caused by +CI node overload and a random decrease of performance that may affect how long the test files are executed. +The problem with random time execution of test files may be caused by many things like external requests done in tests. + +### How queue mode works? + +On the Knapsack Pro API side, there is test files queue generated for your CI build. Each of CI node dynamically asks the Knapsack Pro API for test files +that should be executed. Thanks to that each CI node will finish tests at the same time. + +### How to use queue mode? + +Please use this command to run queue mode: + + bundle exec rake knapsack_pro:queue:rspec + +If above command fails then you may need to explicitly pass an argument to require `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files: + + bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]" + +Note if you will run queue mode command for the first time it might be slower. +The second build should have better optimal test suite split. + +### Additional info about queue mode + +If you are not using one of supported CI providers then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit. +To avoid this you can specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`. + +### Supported test runners in queue mode + +At this moment the queue mode works for: + +* RSpec + ## Extra configuration for CI server ### Info about ENV variables By default knapsack_pro gem [supports a few CI providers](#supported-ci-providers) so you don't need to set some environment variables. @@ -357,10 +407,11 @@ ### Knapsack Pro binary You can install knapsack_pro globally and use binary. For instance: $ knapsack_pro rspec "--tag custom_tag_name --profile" + $ knapsack_pro queue:rspec "--tag custom_tag_name --profile" $ knapsack_pro cucumber "--name feature" $ knapsack_pro minitest "--verbose --pride" $ knapsack_pro spinach "--arg_name value" This is optional way of using knapsack_pro when you don't want to add it to `Gemfile`. @@ -561,9 +612,107 @@ You can change log level by specifying the `KNAPSACK_PRO_LOG_LEVEL` environment variable. KNAPSACK_PRO_LOG_LEVEL=warn bundle exec rake knapsack_pro:rspec Available values are `debug`, `info`, and `warn`. The default log level is `info`. + +### How to split tests based on test level instead of test file level? + +If you want to split one big test file (test file with long time execution) across multiple CI nodes then you can: + +#### A. Create multiple small test files + +Create multiple small test files instead of one long running test file with many test cases. +A lot of small test files will give you better test suite split results. + +#### B. Use tags to mark set of tests in particular test file + +Another way is to use tags to mark subset of tests in particular test file and then split tests based on tags. + +Here is example of test file with specified tags for describe groups: + +```ruby +# spec/features/something_spec.rb +describe 'Feature' do + describe 'something A', :tagA do + it {} + it 'another test' {} + end + + describe 'something B', :tagB do + it {} + end + + describe 'something else' do + it {} + end +end +``` + +You need to create multiple API tokens for different tags. In this example we need 3 different API tokens. + +You need to run below commands for each CI node. + + # run only tests with tagA + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]" + + # run only tests with tagB + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagB bundle exec rake "knapsack_pro:rspec[--tag tagB]" + + # run other tests without tag A & B + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tests_without_tags_A_and_B bundle exec rake "knapsack_pro:rspec[--tag ~tagA --tag ~tagB]" + +### How to make knapsack_pro works for forked repositories of my project? + +Imagine one of the scenarios, for this example I use the Travis-CI. + +* We don’t want to have secrets like the `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in `.travis.yml` in the codebase, because that code is also distributed to clients. +* Adding it as env variables to Travis itself is tricky: It has to work for pull requests from developer’s forks into our main fork; this conflicts with the way Travis handles secrets. We also need a fallback if the token is not provided (when developers do builds within their own fork). + +The solution for this problem is to set `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` as env variables in Travis for our main project. +This won't be accessible on forked repositories so we will run knapsack_pro in fallback mode there. +This way forked repositories have working test suite but without optimal test suite split across CI nodes. + +Create the file `bin/knapsack_pro_rspec` with executable chmod in your main project repository. +Below example is for rspec. You can change `$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` to `$KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` if you use cucumber etc. + +``` +#!/bin/bash +if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then + KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \ + KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \ + bundle exec rake knapsack_pro:rspec +else + bundle exec rake knapsack_pro:rspec +fi +``` + +Now you can use `bin/knapsack_pro_rspec` command instead of `bundle exec rake knapsack_pro:rspec`. +Remember to follow other steps required for your CI provider. + +### Questions around data usage and security + +#### What data is sent to your servers? + +The knapsack_pro gem sends branch name, commit hash, CI total node number, CI index node number, the test file paths like `spec/models/user_spec.rb` and the time execution of each test file path as a float. + +Here is the [full specification of the API](http://docs.knapsackpro.com/api/v1/) used by knapsack_pro gem. + +#### How is that data secured? + +The test file paths can be [encrypted](#test-file-names-encryption) on your CI node with your salt and later send to knapsackpro.com API. +You generate salt locally and only you can decrypt the test file paths. + +Connection with knapsackpro.com server is via https. + +Regarding payments we use the BraintreePayments.com and they store credit cards and your private information. + +#### Who has access to the data? + +I’m the only admin so I can preview data in case you need help with debugging some problem etc. I’m not able to decrypt them without knowing the salt. + +When you sign in to your user dashboard then you can preview data for recent 100 builds on CI. If the test file paths are encrypted then you only see hashes for test file paths. +You need to [decrypt](#how-to-debug-test-file-names) them locally on your machine to find out what each test file hash is. ## Gem tests ### Spec