README.md in strong_migrations-0.6.8 vs README.md in strong_migrations-0.7.0

@@ -23,31 +23,60 @@
 ```sh
 bundle install
 rails generate strong_migrations:install
 ```
 
+Strong Migrations sets a long statement timeout for migrations so you can set a [short statement timeout](#app-timeouts) for your application.
+
+## How It Works
+
+When you run a migration that’s potentially dangerous, you’ll see an error message like:
+
+```txt
+=== Dangerous operation detected #strong_migrations ===
+
+Active Record caches attributes, which causes problems
+when removing columns. Be sure to ignore the column:
+
+class User < ApplicationRecord
+  self.ignored_columns = ["name"]
+end
+
+Deploy the code, then wrap this step in a safety_assured { ... } block.
+
+class RemoveColumn < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured { remove_column :users, :name, :string }
+  end
+end
+```
+
+An operation is classified as dangerous if it either:
+
+- Blocks reads or writes for more than a few seconds (after a lock is acquired)
+- Has a good chance of causing application errors
+
 ## Checks
 
 Potentially dangerous operations:
 
 - [removing a column](#removing-a-column)
 - [adding a column with a default value](#adding-a-column-with-a-default-value)
 - [backfilling data](#backfilling-data)
-- [changing the type of a column](#renaming-or-changing-the-type-of-a-column)
-- [renaming a column](#renaming-or-changing-the-type-of-a-column)
+- [changing the type of a column](#changing-the-type-of-a-column)
+- [renaming a column](#renaming-a-column)
 - [renaming a table](#renaming-a-table)
 - [creating a table with the force option](#creating-a-table-with-the-force-option)
-- [using change_column_null with a default value](#using-change_column_null-with-a-default-value)
+- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 - [executing SQL directly](#executing-SQL-directly)
 
 Postgres-specific checks:
 
 - [adding an index non-concurrently](#adding-an-index-non-concurrently)
 - [adding a reference](#adding-a-reference)
 - [adding a foreign key](#adding-a-foreign-key)
 - [adding a json column](#adding-a-json-column)
-- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
 
 Best practices:
 
 - [keeping non-unique indexes to three columns or less](#keeping-non-unique-indexes-to-three-columns-or-less)
 
@@ -55,11 +84,11 @@
 
 ### Removing a column
 
 #### Bad
 
-ActiveRecord caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
+Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
 
 ```ruby
 class RemoveSomeColumnFromUsers < ActiveRecord::Migration[6.0]
   def change
     remove_column :users, :some_column
@@ -67,11 +96,11 @@
 end
 ```
 
 #### Good
 
-1. Tell ActiveRecord to ignore the column from its cache
+1. Tell Active Record to ignore the column from its cache
 
   ```ruby
   class User < ApplicationRecord
     self.ignored_columns = ["some_column"]
   end
@@ -90,24 +119,24 @@
 
 4. Deploy and run migration
 
 ### Adding a column with a default value
 
-Note: This operation is safe in Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+.
-
 #### Bad
 
-Adding a column with a default value to an existing table causes the entire table to be rewritten.
+In earlier versions of Postgres, MySQL, and MariaDB, adding a column with a default value to an existing table causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
 
 ```ruby
 class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
   def change
     add_column :users, :some_column, :text, default: "default_value"
   end
 end
 ```
 
+In Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+, this no longer requires a table rewrite and is safe.
+
 #### Good
 
 Instead, add the column without a default value, then change the default.
 
 ```ruby
@@ -127,11 +156,11 @@
 
 ### Backfilling data
 
 #### Bad
 
-Backfilling in the same transaction that alters a table locks the table for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
+Active Record creates a transaction around each migration, and backfilling in the same transaction that alters a table keeps the table locked for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
 
 ```ruby
 class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
   def change
     add_column :users, :some_column, :text
@@ -157,43 +186,37 @@
     end
   end
 end
 ```
 
-### Renaming or changing the type of a column
+### Changing the type of a column
 
 #### Bad
 
-```ruby
-class RenameSomeColumn < ActiveRecord::Migration[6.0]
-  def change
-    rename_column :users, :some_column, :new_name
-  end
-end
-```
+Changing the type of a column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
 
-or
-
 ```ruby
 class ChangeSomeColumnType < ActiveRecord::Migration[6.0]
   def change
     change_column :users, :some_column, :new_type
   end
 end
 ```
 
-A few changes are safe in Postgres:
+A few changes don’t require a table rewrite (and are safe) in Postgres:
 
-- Changing between `varchar` and `text` columns
+- Increasing the length limit of a `varchar` column (or removing the limit)
+- Changing a `varchar` column to a `text` column
+- Changing a `text` column to a `varchar` column with no length limit
 - Increasing the precision of a `decimal` or `numeric` column
 - Making a `decimal` or `numeric` column unconstrained
 - Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in Postgres 12+
 
 And a few in MySQL and MariaDB:
 
-- Increasing the length of a `varchar` column from under 255 up to 255
-- Increasing the length of a `varchar` column over 255
+- Increasing the length limit of a `varchar` column from under 255 up to 255
+- Increasing the length limit of a `varchar` column from over 255 to the max limit
 
 #### Good
 
 A safer approach is to:
 
@@ -202,14 +225,41 @@
 3. Backfill data from the old column to the new column
 4. Move reads from the old column to the new column
 5. Stop writing to the old column
 6. Drop the old column
 
+### Renaming a column
+
+#### Bad
+
+Renaming a column that’s in use will cause errors in your application.
+
+```ruby
+class RenameSomeColumn < ActiveRecord::Migration[6.0]
+  def change
+    rename_column :users, :some_column, :new_name
+  end
+end
+```
+
+#### Good
+
+A safer approach is to:
+
+1. Create a new column
+2. Write to both columns
+3. Backfill data from the old column to the new column
+4. Move reads from the old column to the new column
+5. Stop writing to the old column
+6. Drop the old column
+
 ### Renaming a table
 
 #### Bad
 
+Renaming a table that’s in use will cause errors in your application.
+
 ```ruby
 class RenameUsersToCustomers < ActiveRecord::Migration[6.0]
   def change
     rename_table :users, :customers
   end
@@ -257,38 +307,60 @@
 end
 ```
 
 If you intend to drop an existing table, run `drop_table` first.
 
-### Using change_column_null with a default value
+### Setting NOT NULL on an existing column
 
 #### Bad
 
-This generates a single `UPDATE` statement to set the default value.
+Setting `NOT NULL` on an existing column blocks reads and writes while the every row is checked.
 
 ```ruby
-class ChangeSomeColumnNull < ActiveRecord::Migration[6.0]
+class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
   def change
-    change_column_null :users, :some_column, false, "default_value"
+    change_column_null :users, :some_column, false
   end
 end
 ```
 
-#### Good
+#### Good - Postgres
 
-Backfill the column [safely](#backfilling-data). Then use:
+Instead, add a check constraint:
 
 ```ruby
-class ChangeSomeColumnNull < ActiveRecord::Migration[6.0]
+class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
   def change
+    safety_assured do
+      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
+    end
+  end
+end
+```
+
+Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column, but it won’t show up in `schema.rb`. In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
+
+```ruby
+class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
+  def change
+    safety_assured do
+      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
+    end
+
+    # in Postgres 12+, you can then safely set NOT NULL on the column
     change_column_null :users, :some_column, false
+    safety_assured do
+      execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
+    end
   end
 end
 ```
 
-Note: In Postgres, `change_column_null` is still [not safe](#setting-not-null-on-an-existing-column) with this method.
+#### Good - MySQL and MariaDB
 
+[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this.
+
 ### Executing SQL directly
 
 Strong Migrations can’t ensure safety for raw SQL statements. Make really sure that what you’re doing is safe, then use:
 
 ```ruby
@@ -301,11 +373,11 @@
 
 ### Adding an index non-concurrently
 
 #### Bad
 
-In Postgres, adding an index non-concurrently locks the table.
+In Postgres, adding an index non-concurrently blocks writes.
 
 ```ruby
 class AddSomeIndexToUsers < ActiveRecord::Migration[6.0]
   def change
     add_index :users, :some_column
@@ -337,11 +409,11 @@
 
 ### Adding a reference
 
 #### Bad
 
-Rails adds an index non-concurrently to references by default, which is problematic for Postgres.
+Rails adds an index non-concurrently to references by default, which blocks writes in Postgres.
 
 ```ruby
 class AddReferenceToUsers < ActiveRecord::Migration[6.0]
   def change
     add_reference :users, :city
@@ -365,11 +437,11 @@
 
 ### Adding a foreign key
 
 #### Bad
 
-In Postgres, new foreign keys are validated by default, which acquires a `ShareRowExclusiveLock` that can be [expensive on large tables](https://travisofthenorth.com/blog/2017/2/2/postgres-adding-foreign-keys-with-zero-downtime).
+In Postgres, adding a foreign key blocks writes on both tables.
 
 ```ruby
 class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
   def change
     add_foreign_key :users, :orders
@@ -387,11 +459,11 @@
 end
 ```
 
 #### Good
 
-Instead, validate it in a separate migration with a more agreeable `RowShareLock`. This approach is documented by Postgres to have “[the least impact on other work](https://www.postgresql.org/docs/current/sql-altertable.html).”
+Add the foreign key without validating existing rows, then validate them in a separate migration.
 
 For Rails 5.2+, use:
 
 ```ruby
 class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
@@ -399,11 +471,11 @@
     add_foreign_key :users, :orders, validate: false
   end
 end
 ```
 
-Then validate it in a separate migration.
+Then:
 
 ```ruby
 class ValidateForeignKeyOnUsers < ActiveRecord::Migration[6.0]
   def change
     validate_foreign_key :users, :orders
@@ -421,11 +493,11 @@
     end
   end
 end
 ```
 
-Then validate it in a separate migration.
+Then:
 
 ```ruby
 class ValidateForeignKeyOnUsers < ActiveRecord::Migration[5.1]
   def change
     safety_assured do
@@ -437,11 +509,11 @@
 
 ### Adding a json column
 
 #### Bad
 
-In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries.
+In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries in your application.
 
 ```ruby
 class AddPropertiesToUsers < ActiveRecord::Migration[6.0]
   def change
     add_column :users, :properties, :json
@@ -459,58 +531,10 @@
     add_column :users, :properties, :jsonb
   end
 end
 ```
 
-### Setting NOT NULL on an existing column
-
-#### Bad
-
-In Postgres, setting `NOT NULL` on an existing column requires an `AccessExclusiveLock`, which is expensive on large tables.
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    change_column_null :users, :some_column, false
-  end
-end
-```
-
-#### Good
-
-Instead, add a constraint:
-
-```ruby
-class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
-    end
-  end
-end
-```
-
-Then validate it in a separate migration.
-
-```ruby
-class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
-  def change
-    safety_assured do
-      execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
-    end
-
-    # in Postgres 12+, you can safely turn this into a traditional column constraint
-    change_column_null :users, :some_column, false
-    safety_assured do
-      execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
-    end
-  end
-end
-```
-
-Note: This is not 100% the same as `NOT NULL` column constraint before Postgres 12. Here’s a [good explanation](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
-
 ### Keeping non-unique indexes to three columns or less
 
 #### Bad
 
 Adding a non-unique index with more than three columns rarely improves performance.
@@ -585,21 +609,29 @@
 StrongMigrations.disable_check(:add_index)
 ```
 
 Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
 
+## Down Migrations / Rollbacks [unreleased]
+
+By default, checks are disabled when migrating down. Enable them with:
+
+```ruby
+StrongMigrations.check_down = true
+```
+
 ## Custom Messages
 
 To customize specific messages, create an initializer with:
 
 ```ruby
 StrongMigrations.error_messages[:add_column_default] = "Your custom instructions"
 ```
 
 Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
 
-## Timeouts
+## Migration Timeouts
 
 It’s extremely important to set a short lock timeout for migrations. This way, if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it. We also recommend setting a long statement timeout so migrations can run for a while.
 
 Create `config/initializers/strong_migrations.rb` with:
 
@@ -615,10 +647,53 @@
 ALTER ROLE myuser SET statement_timeout = '1h';
 ```
 
 Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
 
+## App Timeouts
+
+We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
+
+For Postgres:
+
+```yml
+production:
+  connect_timeout: 5
+  variables:
+    statement_timeout: 15s
+    lock_timeout: 10s
+```
+
+Note: If you use PgBouncer in transaction mode, you must set the statement and lock timeouts on the database user as shown above.
+
+For MySQL:
+
+```yml
+production:
+  connect_timeout: 5
+  read_timeout: 5
+  write_timeout: 5
+  variables:
+    max_execution_time: 15000 # ms
+    lock_wait_timeout: 10 # sec
+
+```
+
+For MariaDB:
+
+```yml
+production:
+  connect_timeout: 5
+  read_timeout: 5
+  write_timeout: 5
+  variables:
+    max_statement_time: 15 # sec
+    lock_wait_timeout: 10 # sec
+```
+
+For HTTP connections, Redis, and other services, check out [this guide](https://github.com/ankane/the-ultimate-guide-to-ruby-timeouts).
+
 ## Existing Migrations
 
 To mark migrations as safe that were created before installing this gem, create an initializer with:
 
 ```ruby
@@ -662,25 +737,23 @@
 
 ```ruby
 task "db:schema:dump": "strong_migrations:alphabetize_columns"
 ```
 
-## Dangerous Tasks
-
-For safety, dangerous database tasks are disabled in production - `db:drop`, `db:reset`, `db:schema:load`, and `db:structure:load`. To get around this, use:
-
-```sh
-SAFETY_ASSURED=1 rails db:drop
-```
-
 ## Permissions
 
 We recommend using a [separate database user](https://ankane.org/postgres-users) for migrations when possible so you don’t need to grant your app user permission to alter tables.
 
+## Smaller Projects
+
+You probably don’t need this gem for smaller projects, as operations that are unsafe at scale can be perfectly safe on smaller, low-traffic tables.
+
 ## Additional Reading
 
 - [Rails Migrations with No Downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
 - [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
+- [An Overview of DDL Algorithms in MySQL](https://mydbops.wordpress.com/2020/03/04/an-overview-of-ddl-algorithms-in-mysql-covers-mysql-8/)
+- [MariaDB InnoDB Online DDL Overview](https://mariadb.com/kb/en/innodb-online-ddl-overview/)
 
 ## Credits
 
 Thanks to Bob Remeika and David Waller for the [original code](https://github.com/foobarfighter/safe-migrations) and [Sean Huber](https://github.com/LendingHome/zero_downtime_migrations) for the bad/good readme format.