README.md in knapsack_pro-1.14.0 vs README.md in knapsack_pro-1.15.0

- old
+ new

@@ -111,10 +111,11 @@ - [Info for AppVeyor users](#info-for-appveyor-users) - [Info for snap-ci.com users](#info-for-snap-cicom-users) - [Info for cirrus-ci.org users](#info-for-cirrus-ciorg-users) - [Info for Jenkins users](#info-for-jenkins-users) - [Info for GitHub Actions users](#info-for-github-actions-users) + - [Info for Codefresh.io users](#info-for-codefreshio-users) - [FAQ](#faq) - [Common problems](#common-problems) - [Why I see API error commit_hash parameter is required?](#why-i-see-api-error-commit_hash-parameter-is-required) - [Why I see `LoadError: cannot load such file -- spec_helper`?](#why-i-see-loaderror-cannot-load-such-file----spec_helper) - [Why my CI build fails when I use Test::Unit even when all tests passed?](#why-my-ci-build-fails-when-i-use-testunit-even-when-all-tests-passed) @@ -138,10 +139,11 @@ - [How to fix capybara-screenshot fail with `SystemStackError: stack level too deep` when using Queue Mode for RSpec?](#how-to-fix-capybara-screenshot-fail-with-systemstackerror-stack-level-too-deep-when-using-queue-mode-for-rspec) - [Parallel tests Cucumber and RSpec with Cucumber failures exit CI node early leaving fewer CI nodes to finish RSpec Queue.](#parallel-tests-cucumber-and-rspec-with-cucumber-failures-exit-ci-node-early-leaving-fewer-ci-nodes-to-finish-rspec-queue) - [Why when I reran the same build (same commit hash, etc) on Codeship then no tests would get executed in Queue Mode?](#why-when-i-reran-the-same-build-same-commit-hash-etc-on-codeship-then-no-tests-would-get-executed-in-queue-mode) - [Why knapsack_pro hangs / freezes / is stale i.e. for Codeship in Queue Mode?](#why-knapsack_pro-hangs--freezes--is-stale-ie-for-codeship-in-queue-mode) - [How to find seed in RSpec output when I use Queue Mode for RSpec?](#how-to-find-seed-in-rspec-output-when-i-use-queue-mode-for-rspec) + - [How to configure puffing-billy gem with Knapsack Pro Queue Mode?](#how-to-configure-puffing-billy-gem-with-knapsack-pro-queue-mode) - [General questions](#general-questions) - [How to run tests for particular CI node in your development environment](#how-to-run-tests-for-particular-ci-node-in-your-development-environment) - [for knapsack_pro regular mode](#for-knapsack_pro-regular-mode) - [for knapsack_pro queue mode](#for-knapsack_pro-queue-mode) - [What happens when Knapsack Pro API is not available/not reachable temporarily?](#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily) @@ -166,10 +168,12 @@ - [How to use RSpec JSON formatter with knapsack_pro Queue Mode?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode) - [How to use RSpec JSON formatter with knapsack_pro Queue Mode when CI nodes use common local drive?](#how-to-use-rspec-json-formatter-with-knapsack_pro-queue-mode-when-ci-nodes-use-common-local-drive) - [How many API keys I need?](#how-many-api-keys-i-need) - [What is optimal order of test commands?](#what-is-optimal-order-of-test-commands) - [How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)?](#how-to-set-beforesuite-and-aftersuite-rspec-hooks-in-queue-mode-percyio-example) + - [percy-capybara gem version < 4 (old)](#percy-capybara-gem-version--4-old) + - [percy-capybara gem version >= 4](#percy-capybara-gem-version--4) - [How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode?](#how-to-call-beforesuite-and-aftersuite-rspec-hooks-only-once-in-queue-mode) - [What hooks are supported in Queue Mode?](#what-hooks-are-supported-in-queue-mode) - [How to run knapsack_pro with parallel_tests gem?](#how-to-run-knapsack_pro-with-parallel_tests-gem) - [parallel_tests with knapsack_pro on parallel CI nodes](#parallel_tests-with-knapsack_pro-on-parallel-ci-nodes) - [parallel_tests with knapsack_pro on single CI machine](#parallel_tests-with-knapsack_pro-on-single-ci-machine) @@ -245,11 +249,11 @@ config.ignore_hosts('localhost', '127.0.0.1', '0.0.0.0', 'api.knapsackpro.com') end # add below when you hook into webmock require 'webmock/rspec' -WebMock.disable_net_connect!(allow: ['api.knapsackpro.com']) +WebMock.disable_net_connect!(allow_localhost: true, allow: ['api.knapsackpro.com']) # add below when you use FakeWeb require 'fakeweb' FakeWeb.allow_net_connect = %r[^https?://api\.knapsackpro\.com] ``` @@ -263,10 +267,26 @@ gem 'webmock', require: false gem 'fakeweb', require: false # example when you use fakeweb end ``` +If you happen to see your tests failing due to WebMock not allowing requests to Knapsack Pro API it means you probably reconfigure WebMock in some of your tests. +For instance, you may use `WebMock.reset!` or it's called automatically in `after(:each)` block, if you `require 'webmock/rspec'` ([more about the issue](https://github.com/bblimke/webmock/issues/484#issuecomment-116193449)). It will remove api.knapsackpro.com from whitelisted domains. Please try below: + +```ruby +RSpec.configure do |config| + config.after(:suite) do + WebMock.disable_net_connect!( + allow_localhost: true, + allow: [ + 'api.knapsackpro.com', + ], + ) + end +end +``` + ### Usage (How to set up 1 of 3) __Tip:__ You can find here an example of a rails app with knapsack_pro already configured. https://github.com/KnapsackPro/rails-app-with-knapsack_pro @@ -1417,10 +1437,11 @@ * You should create as many parallel jobs as you need with [`matrix` property](https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix). If your test suite is slow you should use more parallel jobs. See comment in below config. Below you can find full GitHub Actions config for Ruby on Rails project. ```yaml +# .github/workflows/main.yaml name: Main on: [push] jobs: @@ -1442,10 +1463,11 @@ - 5432/tcp # needed because the postgres container does not provide a healthcheck options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 strategy: + fail-fast: false matrix: # Set N number of parallel jobs you want to run tests on. # Use higher number if you have slow tests to split them on more parallel jobs. # Remember to update ci_node_index below to 0..N-1 ci_node_total: [2] @@ -1499,10 +1521,132 @@ bundle exec rake knapsack_pro:queue:rspec bundle exec rake knapsack_pro:queue:cucumber bundle exec rake knapsack_pro:queue:minitest ``` +#### Info for Codefresh.io users + +knapsack_pro gem supports environment variables provided by Codefresh.io to run your tests. You will have to define a few things in `.codefresh/codefresh.yml` config file. + +* You need to set an API token like `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` in Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to the pipeline) -> Variables tab (see a vertical menu on the right side). Add there new API token depending on the test runner you use: + * `KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC` + * `KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER` + * `KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST` + * `KNAPSACK_PRO_TEST_SUITE_TEST_UNIT` + * `KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH` +* Set where Codefresh YAML file can be found. In Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to pipeline) -> Workflow tab (horizontal menu on the top) -> Path to YAML (set there `./.codefresh/codefresh.yml`). +* Set how many parallel jobs (parallel CI nodes) you want to run with `KNAPSACK_PRO_CI_NODE_TOTAL` environment variable in `.codefresh/codefresh.yml` file. +* Ensure in the `matrix` section you listed all `KNAPSACK_PRO_CI_NODE_INDEX` environment variables with a value from `0` to `KNAPSACK_PRO_CI_NODE_TOTAL-1`. Codefresh will generate a matrix of parallel jobs where each job has a different value for `KNAPSACK_PRO_CI_NODE_INDEX`. Thanks to that Knapsack Pro knows what tests should be run on each parallel job. + +Below you can find Codefresh YAML config and `Test.Dockerfile` used by Codefresh to run Ruby on Rails project with PostgreSQL inside of Docker container. + +```yaml +# .codefresh/codefresh.yml +version: "1.0" + +stages: + - "clone" + - "build" + - "tests" + +steps: + main_clone: + type: "git-clone" + description: "Cloning main repository..." + repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}" + revision: "${{CF_BRANCH}}" + stage: "clone" + BuildTestDockerImage: + title: Building Test Docker image + type: build + arguments: + image_name: '${{CF_ACCOUNT}}/${{CF_REPO_NAME}}-test' + tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}' + dockerfile: Test.Dockerfile + stage: "build" + + run_tests: + stage: "tests" + image: '${{BuildTestDockerImage}}' + working_directory: /src + fail_fast: false + environment: + - RAILS_ENV=test + # set how many parallel jobs you want to run + - KNAPSACK_PRO_CI_NODE_TOTAL=2 + - PGHOST=postgres + - PGUSER=rails-app-with-knapsack_pro + - PGPASSWORD=password + services: + composition: + postgres: + image: postgres:latest + environment: + - POSTGRES_DB=rails-app-with-knapsack_pro_test + - POSTGRES_PASSWORD=password + - POSTGRES_USER=rails-app-with-knapsack_pro + ports: + - 5432 + matrix: + environment: + # please ensure you have here listed N-1 indexes + # where N is KNAPSACK_PRO_CI_NODE_TOTAL + - KNAPSACK_PRO_CI_NODE_INDEX=0 + - KNAPSACK_PRO_CI_NODE_INDEX=1 + commands: + - bin/rails db:prepare + + # run tests in Knapsack Pro Regular Mode + - bundle exec rake knapsack_pro:rspec + - bundle exec rake knapsack_pro:cucumber + - bundle exec rake knapsack_pro:minitest + - bundle exec rake knapsack_pro:test_unit + - bundle exec rake knapsack_pro:spinach + + # you can use Knapsack Pro in Queue Mode once recorded first CI build with Regular Mode + - bundle exec rake knapsack_pro:queue:rspec + - bundle exec rake knapsack_pro:queue:cucumber + - bundle exec rake knapsack_pro:queue:minitest +``` + +Add `Test.Dockerfile` to your project repository. + +```Dockerfile +# Test.Dockerfile +FROM ruby:2.6.5-alpine3.10 + +# Prepare Docker image for Nokogiri +RUN apk add --update \ + build-base \ + libxml2-dev \ + libxslt-dev \ + jq \ + nodejs \ + npm \ + postgresql-dev \ + python3-dev \ + sqlite-dev \ + git \ + && rm -rf /var/cache/apk/* + +# Install AWS CLI +RUN pip3 install awscli + +# Use libxml2, libxslt a packages from alpine for building nokogiri +RUN bundle config build.nokogiri --use-system-libraries + +# Install Codefresh CLI +RUN wget https://github.com/codefresh-io/cli/releases/download/v0.31.1/codefresh-v0.31.1-alpine-x64.tar.gz +RUN tar -xf codefresh-v0.31.1-alpine-x64.tar.gz -C /usr/local/bin/ + +COPY . /src + +WORKDIR /src + +RUN bundle install +``` + ## FAQ ### Common problems #### Why I see API error commit_hash parameter is required? @@ -1894,10 +2038,55 @@ bundle exec rspec --seed 11055 --default-path spec "spec/a_spec.rb" "spec/b_spec.rb" ``` If you don't use RSpec argument `--order random` then you don't need to provide `--seed` number when you want to reproduce tests in development. +##### How to configure puffing-billy gem with Knapsack Pro Queue Mode? + +If you use [puffing-billy](https://github.com/oesmith/puffing-billy) gem you may notice [puffing-billy may crash](https://github.com/oesmith/puffing-billy/issues/253). It happen due to the way how knapsack_pro in Queue Mode uses `RSpec::Core::Runner` ([see](#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail)). + +Here is a patch for puffing-billy to make it work in knapsack_pro Queue Mode: + +```ruby +# rails_helper.rb or spec_helper.rb + +# A patch to `puffing-billy`'s proxy so that it doesn't try to stop +# eventmachine's reactor if it's not running. +module BillyProxyPatch + def stop + return unless EM.reactor_running? + super + end +end +Billy::Proxy.prepend(BillyProxyPatch) + +# A patch to `puffing-billy` to start EM if it has been stopped +Billy.module_eval do + def self.proxy + if @billy_proxy.nil? || !(EventMachine.reactor_running? && EventMachine.reactor_thread.alive?) + proxy = Billy::Proxy.new + proxy.start + @billy_proxy = proxy + else + @billy_proxy + end + end +end + +if ENV["KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC"] + KnapsackPro::Hooks::Queue.before_queue do + # executes before Queue Mode starts work + Billy.proxy.start + end + + KnapsackPro::Hooks::Queue.after_queue do + # executes after Queue Mode finishes work + Billy.proxy.stop + end +end +``` + ### General questions #### How to run tests for particular CI node in your development environment ##### for knapsack_pro regular mode @@ -2296,11 +2485,11 @@ Here is example: * Step 1. API_KEY_A for `bundle exec rake knapsack_pro:cucumber` * Step 2. API_KEY_B for `bundle exec rake knapsack_pro:rspec` -* Step 3. API_KEY_C for `KNAPSACK_PRO_TEST_FILE_PATTERN="specs/features/*_spec.rb" bundle exec rake knapsack_pro:rspec` +* Step 3. API_KEY_C for `KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/*_spec.rb" bundle exec rake knapsack_pro:rspec` * Step 4. API_KEY_D for `bundle exec rake knapsack_pro:rspec[--tag tagA]` * Step 5. API_KEY_E for `bundle exec rake knapsack_pro:rspec[--tag ~tagA]` * Step 6. API_KEY_F for `bundle exec rake knapsack_pro:queue:rspec` Note: @@ -2333,15 +2522,18 @@ You will run your javascript tests on single CI node and the knapsack_pro will auto-balance CI build with Queue Mode. Thanks to that CI build time execution will be flat and optimal (as fast as possible). #### How to set `before(:suite)` and `after(:suite)` RSpec hooks in Queue Mode (Percy.io example)? +##### percy-capybara gem version < 4 (old) + Some tools like [Percy.io](https://percy.io/docs/clients/ruby/capybara-rails) requires to set hooks for RSpec `before(:suite)` and `after(:suite)`. Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will execute multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: ```ruby # spec_helper.rb or rails_helper.rb +# step for percy-capybara gem version < 4 KnapsackPro::Hooks::Queue.before_queue do |queue_id| # executes before Queue Mode starts work Percy::Capybara.initialize_build end @@ -2350,10 +2542,36 @@ # executes after Queue Mode finishes work Percy::Capybara.finalize_build end ``` +##### percy-capybara gem version >= 4 + +If you use [percy-capybara 4.x](https://docs.percy.io/v1/docs/capybara) then you don't need to set RSpec hooks. Insted you need to run knapsack_pro via percy npm command. + +``` +npx percy exec -- rake knapsack_pro:queue:rspec + +# or you can use knapsack_pro binary version instead of rake task +npx percy exec -- knapsack_pro queue:rspec +``` + +Read more about [knapsack_pro binary version](#knapsack-pro-binary). + +Also you need to follow [Percy step for parallelism](https://docs.percy.io/docs/parallel-test-suites#section-manual-configuration-with-environment-variables). + +* `PERCY_PARALLEL_NONCE` - A unique identifier for this build. This can be anything, but it must be the same across parallel build nodes. Usually, this is just the CI build number or a shared timestamp. You can google environment variables for CI provider you use to check what's the env var for build ID. + + You can also find CI build number for your CI provider in [knapsack_pro source code](https://github.com/KnapsackPro/knapsack_pro-ruby/tree/master/lib/knapsack_pro/config/ci). knapsack_pro has built in environment variables integration for various CI providers. See for example [CircleCI](https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/config/ci/circle.rb) - look for method `node_build_id`. + + ```bash + # example for using CircleCI build ID + export PERCY_PARALLEL_NONCE=$CIRCLE_BUILD_NUM + ``` + +* `PERCY_PARALLEL_TOTAL` - The total number of parallel build nodes. + #### How to call `before(:suite)` and `after(:suite)` RSpec hooks only once in Queue Mode? Knapsack Pro Queue Mode runs subset of test files from the work queue many times. This means the RSpec hooks `before(:suite)` and `after(:suite)` will be executed multiple times. If you want to run some code only once before Queue Mode starts work and after it finishes then you should do it this way: ```ruby @@ -2730,10 +2948,10 @@ #!/bin/bash # add this file in bin/knapsack_pro_rspec_and_npm_test and change chmod # $ chmod a+x bin/knapsack_pro_rspec_and_npm_test # 15 is last CI node (index starts from 0, so in total we have 16 parallel Heroku dynos) -if [ "$CI_NODE_TOTAL" == "15" ]; then +if [ "$CI_NODE_INDEX" == "15" ]; then # run npm tests on the last CI node npm test else KNAPSACK_PRO_CI_NODE_TOTAL=$((CI_NODE_TOTAL-1)) bundle exec rake knapsack_pro:queue:rspec fi