README.md in gemika-0.1.3 vs README.md in gemika-0.1.4

- old
+ new

@@ -1,497 +1,994 @@ -# gemika +# Gemika + +## Test a Ruby gem against multiple versions of everything + +Gemika helps you test your gem against multiple versions of Ruby, gem dependencies and database types. + +Here's what Gemika can give your test's development setup (all steps are opt-in): + +- Test one codebase against multiple sets of gem dependency sets (e.g. Rails 4.2, Rails 5.0). +- Test one codebase against multiple Ruby versions (e.g. Ruby 2.1.8, Ruby 2.3.1). +- Test one codebase against multiple database types (currently MySQL or PostgreSQL). +- Compute a matrix of all possible dependency permutations (Ruby, gem set, database type). Manually exclude incompatible dependency permutations (e.g. Rails 5.0 does not work with Ruby 2.1). +- Let developers enter their local credentials for MySQL and PostgreSQL in a `database.yml` file. +- Define default Ruby version, gem dependencies and database for developers who don't care about every possible permutation for everyday work. +- Help configure a [Travis CI](https://travis-ci.org/) build that tests every dependency permutation after each `git push`. +- Share your Ruby / gem dependeny / database permutation between local development and Travis CI. +- Define an [ActiveRecord database migration](http://api.rubyonrails.org/classes/ActiveRecord/Migration.html) that sets up your test database. +- Automatically drop and re-create your test database before each run of your test suite. +- Configure RSpec to wrap each example in a transaction that is rolled back when the example ends. This way each example starts with a blank database. + + +## Requirements + +- Gemika currently assumes you're testing with [RSpec](http://rspec.info/). +- If you use any database-related features, you need `activaterecord` as a development dependency + + +## Example directory structure + +Below you can see the directory of a gem with a completed Gemika testing setup. The next section describes how to get there: + +```shell +gemfiles/Gemfile.set1 # First dependency set. Should include development dependencies and gemika. +gemfiles/Gemfile.set1.lock # Generated by `rake matrix:install` +gemfiles/Gemfile.set2 # Second dependency set. Should include development dependencies and gemika. +gemfiles/Gemfile.set2.lock # Generated by `rake matrix:install` +gemfiles/Gemfile.set3 # Third dependency set. Should include development dependencies and gemika. +gemfiles/Gemfile.set3.lock # Generated by `rake matrix:install` +Gemfile -> gemfiles/Gemfile.set2 # Symlink to default Gemfile for development +Gemfile.lock -> gemfiles/Gemfile.set2.lock # Symlink to default Gemfile.lock for development +.ruby-version # Default Ruby version for development +.gitignore # Should ignore spec/support/database.yml +.travis.yml # Configures all tested Ruby / gemfile combinations, for both local development and Travis CI +my_gem.gemspec # Specification for your gem +Rakefile # Should require 'gemika/tasks' +README.md # README for your gem +lib/my_gem.rb # Library files for your gem +lib/my_gem/my_class.rb # Class delivered by your gem +lib/my_gem/version.rb # Version definition for your gem +spec/spec_helper.rb # Requires 'gemika' and all files in support folder +spec/support/database.rb # Calls `Gemika.rewrite_schema! { ... }` +spec/support/database.yml # Local +spec/support/database.travis.yml # Calls `Gemika.rewrite_schema! { ... }` +spec/my_gem/my_class_spec.rb # Tests for your gem +``` + +For a live example of this setup, check the [makandra/minidusen](https://github.com/makandra/minidusen) repo. + + +## Step-by-step integration + + +### Have a standard gem setup + +Gemika expects a standard gem directory that looks roughly like this: + +``` +my_gem.gemspec # Specification for your gem +Rakefile # Should require 'gemika/tasks' +lib/my_gem.rb # Library files for your gem +``` + +If you don't have a directory yet, you can [ask Bundler to create it for you](http://bundler.io/rubygems.html): + +``` +bundle gem my_gem +``` + +This will create a new directory named `my_gem` with your new gem skeleton. + + +### Install Gemika + +Switch to your favorite Ruby version and install the Gemika gem: + +```shell +gem install gemika +``` + +Future contributors to your gem can install Gemika using the Gemfiles we will create below. + + +### Rake tasks + +Add this to your `Rakefile` to gain tasks `matrix:install`, `matrix:spec`, `matrix:update`. + +```ruby +begin + require 'gemika/tasks' +rescue LoadError + puts 'Run `gem install gemika` for additional tasks' +end +``` + +Check that the tasks appear with `rake -T`: + +```shell +rake matrix:install # Install all Ruby 2.2.4 gemfiles +rake matrix:spec # Run specs for all Ruby 2.2.4 gemfiles +rake matrix:update[gems] # Update all Ruby 2.2.4 gemfiles +``` + +We also recommend to make `matrix:spec` the default task in your `Rakefile`: + +```ruby +task :default => 'matrix:spec' +``` + +### Define multiple dependency sets + +We are now creating one `Gemfile` for each set of gems and database type you'd like to test again. + +First, create a directory for the gemfiles: + +```shell +mkdir gemfiles +``` + +For each dependency set, create a `Gemfile` in the `gemfiles` directory that contains: + +1. The runtime dependencies you'd like to test against (e.g. Rails 5) +2. The development dependencies for that set (e.g. `rspec`) in a version that is compatible with these runtime gependencies. +3. The `gemika` gem +4. Your own gem from path `..` + +For instance, if one dependency set is Rails 3.2 with a MySQL database, we would create `gemfiles/Gemfile.4.2.mysql2` with these contents: + +```ruby +gem 'rails', '~>3.2.22' +gem 'mysql2', '= 0.3.17' +gem 'rspec', '~> 3.4' + +gem 'rake' +gem 'byebug' +gem 'gemika' + +gem 'my_gem', :path => '..' +``` + +If a second dependency is Rails 5.0 with a PostgreSQL database, we would create `gemfiles/Gemfile.5.0.pg` with these contents: + +```ruby +gem 'rails', '~>5.0.0' +gem 'pg', '~>0.18.4' +gem 'rspec', '~>3.5' + +gem 'rake' +gem 'byebug' +gem 'gemika' + +gem 'my_gem', :path => '..' +``` + +In this example, your `gemfiles` directory should now look like this: + +``` +gemfiles/Gemfile.4.2.mysql2 +gemfiles/Gemfile.5.0.pg +``` + +Create lockfiles for each bundle by running: + +```shell +rake matrix:install +``` + +In this example, your `gemfiles` directory should now contain a lockfile for each gemfile: + +``` +gemfiles/Gemfile.4.2.mysql2 +gemfiles/Gemfile.4.2.mysql2.lock +gemfiles/Gemfile.5.0.pg +gemfiles/Gemfile.5.0.pg.lock +``` + +Gemfiles and lockfiles should be committed to your repo. + +Make sure to re-run `rake matrix:install` after each change to your gemfiles, and commit the generated changes. + + +### Define combinations of gemfiles and Ruby versions + +We will now define a test matrix that contains all permutations of gemfiles and tested Ruby versions. + +We store the matrix in a `.travis.yml` file, **even if the project is not using Travis CI**. This allows us to configure the matrix once and us it for both local developent and Travis CI builds. + +Create a `.travis.yml` that lists all gemfiles and Ruby versions you'd like to test against: + +```yaml +rvm: + - 2.1.8 + - 2.2.4 + - 2.3.1 + +gemfile: + - gemfiles/Gemfile.3.2.mysql2 + - gemfiles/Gemfile.4.2.mysql2 + - gemfiles/Gemfile.4.2.pg + - gemfiles/Gemfile.5.0.mysql2 + - gemfiles/Gemfile.5.0.pg +``` + +Don't mind the `rvm` key if you're using a different version manager locally (like rbenv). Things will still work. + + +#### Excluding incompatible matrix rows + +There might be incompatible combinations of gemfiles and Rubies, e.g. Rails 5.0 does not work with Ruby 2.1 or lower. In this case, add an `matrix`/`exclude` key to your `.travis.yml`: + +```yaml +matrix: + exclude: + - rvm: 2.1.8 + gemfile: gemfiles/Gemfile.5.0.mysql2 + - rvm: 2.1.8 + gemfile: gemfiles/Gemfile.5.0.pg +``` + + +### Default Ruby and dependency set + +Your project will be more approachable if you're defining a default Ruby and dependency set. This way a developer can make changes and run code without knowing about the test matrix. + +Create a `.ruby-version` file with the default Ruby version: + +``` +2.2.4 +``` + +Choose a default dependency set and symlink both gemfile and lockfile to your project root: + +``` +ln -s gemfiles/Gemfile.4.2.mysql2 Gemfile +ln -s gemfiles/Gemfile.4.2.mysql2.lock Gemfile.lock +``` + +Commit both `.ruby-version` and symlinks to your repo. + +We recommend to setup Travis CI (see below) to check the entire test matrix after each push, even if a developer only tested with the defaults. + + +### Test databases + +Create a local test database `my_gem_test` in either MySQL, PostgreSQL or both (depending on what you support). If you want to test against multiple database types, you should have created one gemfile for each type above. + +Now create a file `spec/support/database.yml` that contains your local database credentials: + +```yaml +mysql: + database: my_gem_test + host: localhost + username: root + password: secret + +postgresql: + database: minidusen_test + user: + password: +``` + +We don't want to commit our local credentials, so add a line to your `.gitignore`: + +``` +spec/support/database.yml +``` + +What we *will* commit is a `database.sample.yml` as a template for future contributors: + +``` +cp spec/support/database.yml spec/support/database.sample.yml +``` + +Remember to replace any private passwords in `database.sample.yml` with `secret` before committing. + +To have ActiveRecord connect to the database in `database.yml` before your tests, add a file `spec/support/database.rb` with the following content: + +``` +database = Gemika::Database.new +database.connect +``` + +Now require Gemika and this support file from your `spec_helper.rb`. + +``` +require 'gemika' +require 'spec/support/database' +``` + +Protip: Instead of requiring support files indidually, configure your `spec_helper.rb` to automatically `require` all files in the `spec/support` folder: + +```ruby +Dir["#{File.dirname(__FILE__)}/support/*.rb"].sort.each {|f| require f} +``` + +Now you have a great place for code snippets that need to run before specs (factories, VCR configuration, etc.). + + +#### Test database schema + +If your gem is talking to the database, you probably need to create some example tables. + +Magika lets you define an [ActiveRecord database migration](http://api.rubyonrails.org/classes/ActiveRecord/Migration.html) for that. Before your test suite runs, Magika will drop *all* tables in your test database and recreate them using this migration. + +Add your migration to your `spec/support/database.rb` (created and required above): + +```ruby + +database = Gemika::Database.new +database.connect +database.rewrite_schema! do + + create_table :users do |t| + t.string :name + t.string :email + t.string :city + end + + create_table :recipes do |t| + t.string :name + t.integer :category_id + end + + create_table :recipe_ingredients do |t| + t.string :name + t.integer :recipe_id + end + + create_table :recipe_categories do |t| + t.string :name + end + +end +``` + +#### Wrap examples in transactions + +A very useful Rails default is to wrap every test in a transaction that is rolled back when the example ends. This way each example starts with a blank database. + +To get the same behavior in your gem tests, append the following to your `spec_helper.rb`: + +``` +Gemika::RSpec.configure_transactional_examples +``` + +Note that you also need `require 'gemika'` in your `spec_helper.rb`. + +Now every RSpec example is wrapped in a transaction. To disable this behavior for an individual example, use the `transaction: false` option: + +``` +it 'should work', transaction: false do + # code that doesn't work within a transaction, e.g. threads +end +``` + +### Try it out + +Check if you can install development dependencies for each row in the test matrix: + +``` +bundle exec rake matrix:install +``` + +Check if you can run tests for each row in the test matrix: + +```shell +bundle exec rake matrix:spec +``` + +You should see the command output for each row in the test matrix. Gemika will also print a summary at the end: + +![Matrix task output](https://raw.githubusercontent.com/makandra/gemika/master/doc/minidusen_test.png) + +Note that there is no task for running all gemfiles in all Ruby versions. We had something like this in earlier versions of Gemika and it wasn't as practical as we thought. You need to manually switch Ruby versions and re-run `rake matrix:install`. We recommend to setup Travis CI to check the entire test matrix after each push, including all Rubies. + + +## Activate Travis CI + +We recommend to setup Travis CI to check the entire test matrix after each push. Travis CI will also show the build results on Github's pull request page, helping maintainers decide whether a PR is safe to merge. + +If you plan to use Travis CI, also add a `spec/support/database.travis.yml` with [Travis' default database credentials](https://docs.travis-ci.com/user/database-setup/): + +```yaml +mysql: + database: my_gem_test + username: travis + password: + +postgresql: + database: my_gem_test + user: postgres + password: +``` + +Add options to `.travis.yml` to create databases before running tests: + +```yaml +before_script: + - psql -c 'create database mygem_test;' -U postgres + - mysql -e 'create database IF NOT EXISTS mygem_test;' +``` + +Also add the other `.travis.yml` settings required for a Ruby project: + +```yaml +language: ruby + +sudo: false + +cache: bundler + +notifications: + email: + - notifications@test.com + +script: bundle exec rspec spec +``` + +Adjust the `script` option if you're not using RSpec to test your code. + +#### Activate Github integration + +To activate Travis CI for your Github repo: + +- Log into Github +- Open your gem's project page +- Open *Settings* +- Navigate to *Integrations & services* +- Open the *Add service* dropdown +- Select *Travis CI* +- Authenticate via OAuth + +To check if the integration has worked, push a change and check if you can see your build matrix on the [Travis CI dashboard](https://travis-ci.org/). + +#### Build badge + +You might want to a build status badge to your `README.md` like this: + +[![Build Status](https://travis-ci.org/makandra/minidusen.svg?branch=master)](https://travis-ci.org/makandra/minidusen) + +You can add such a badge using this markdown: + +```markdown +[![Build Status](https://travis-ci.org/my_org/my_gem.svg?branch=master)](https://travis-ci.org/my_org/my_gem) +``` + +#### Protect the `master` branch + +If you're super paranoid you can also prevent anyone from pushing to `master` without a green Travis CI build: + +- Open your Github project settings +- Navigate to *Branches* +- Below *Protected branches*, open the *Choose a branch...* dropdown +- Select `master` +- Check *Protect this branch* +- Check *Require status checks to pass before merging* +- Check the status check `continuous-integration/travis-ci` +- Press *Save changes* + + +## Add development instructions to your README + +Your README should contain instructions how to run tests before making a PR. You might also want to educate future contributors about the existence of your test matrix, and how to use it. + +We recommend to add a section like the following to your `README.md`: + +```markdown +## Development + +There are tests in `spec`. We only accept PRs with tests. To run tests: + +- Install Ruby x.y.z +- Create a local test database `my_gem_test` in both MySQL and PostgreSQL +- Copy `spec/support/database.sample.yml` to `spec/support/database.yml` and enter your local credentials for the test databases +- Install development dependencies using `bundle install` +- Run tests using `bundle exec rspec` + +We recommend to test large changes against multiple versions of Ruby and multiple dependency sets. Supported combinations are configured in `.travis.yml`. We provide some rake tasks to help with this: + +- Install development dependencies using `bundle matrix:install` +- Run tests using `bundle matrix:spec` + +Note that we have configured Travis CI to automatically run tests in all supported Ruby versions and dependency sets after each push. We will only merge pull requests after a green Travis build. +``` + +Credits +------- + +Henning Koch from [makandra](http://makandra.com/)