README.md in octoshark-0.1.2 vs README.md in octoshark-0.2.0
- old
+ new
@@ -1,10 +1,10 @@
-Octoshark is a simple ActiveRecord connection manager. It provides a connection switching mechanism that can be used in various scenarios like: sharding, master-slave, multi-tenancy, etc. You have control over how connections are configured and used, see below for examples.
+Octoshark is a simple ActiveRecord connection manager. It provides connection switching mechanisms that can be used in various scenarios like master-slave, sharding or multi-tenant architecture.
## Installation
Add this line to your application's Gemfile:
@@ -26,142 +26,146 @@
```
## Usage
-Create a new connection manager with connection 2 pools for Octoshark to manage. This usually goes in the app initializer.
+Octoshark has two connection managers: `ConnectionPoolsManager` for managing connection pools using persistent connections and `ConnectionManager` for managing non-persistent connections. It depends on your application performance and scaling requirements which one to use.
-```ruby
-CONN_MANAGER = Octoshark::ConnectionManager.new({
- db1: { adapter: "sqlite3", database: "db/db1.sqlite" },
- db2: { adapter: "sqlite3", database: "db/db2.sqlite" }
-})
-```
+- If you have a limited number of consumers (application and worker servers), `ConnectionPoolsManager` would be the preferred option. Standard Rails application has a single connection pool, and `ConnectionPoolsManager` just makes it possible for application models to work with multiple connection pools.
-Configure which ActiveRecord model will use the Octoshark connection by overriding the `Model.connection` method.
+- If you have a very big infrastructure with lots of consumers (application and worker servers) and you are hitting max connections limit on database servers, i.e. you need to scale horizontally, `ConnectionManager` is the option to use. Because it uses non-persistent connections it comes up with a performance penalty because connections are re-established over and over again. Some ActiveRecord plugins that depend on having an active database connection all the time might need a change in order to work with non-persistent connections.
+`ConnectionPoolsManager` and `ConnectionManager` can be combined together and many of them can be used at the same time.
+
+Here is how to create connection pools manager:
+
```ruby
-class Post < ActiveRecord::Base
- def self.connection
- CONN_MANAGER.current_connection
- end
-end
+CONN_MANAGER = Octoshark::ConnectionPoolsManager.new({ c1: config1, c2: config2 })
```
-Alternatively, extract it as a module and include in models.
+`config1` and `config2` are standard ActiveRecord database configs:
```ruby
-module ShardingModel
- extend ActiveSupport::Concern
-
- module ClassMethods
- def connection
- OCTOSHARK.current_connection
- end
- end
-end
+config = {
+ adapter: 'mysql2',
+ host: 'localhost',
+ port: 3306,
+ database: 'database',
+ username: 'root',
+ password: 'pass',
+ pool: 1,
+ encoding: 'utf8',
+ reconnect: false
+}
```
-To use a specific database connection:
+To switch a connection using a specific pool:
```ruby
-CONN_MANAGER.with_connection(:db1) do
- # run queries on db1
- Post.first
+CONN_MANAGER.with_connection(:c1) do |connection|
+ connection.execute("SELECT 1")
end
```
Multiple `with_connection` blocks can be nested:
```ruby
-CONN_MANAGER.with_connection(:db1) do
- # run queries on db1
+CONN_MANAGER.with_connection(config1) do
+ # run queries on connection specified with config1
- CONN_MANAGER.with_connection(:db2) do
- # run queries on db2
+ CONN_MANAGER.with_connection(config2) do
+ # run queries on connection specified with config2
end
- # run queries on db1
+ # run queries on connection specified with config1
end
```
-`CONN_MANAGER.current_connection` returns the active connection while in the `with_connection` block or raises `Octoshark::Error::NoCurrentConnection` otherwise.
+If you establish a connection to database server instead of to database, then you can use the second optional argument `database_name` to tell the connection manager to switch the connection to that database within the same connection. This is useful when you want to have fewer connection pools and keep number of active connection per database server under control. This option is only MySQL specific for now, it uses `USE database_name` statement to switch the connection.
+```ruby
+CONN_MANAGER.with_connection(:c1, database_name) do |connection|
+ connection.execute("SELECT 1")
+end
+```
-## Sharding example
+Using non-persistent connections with `Octoshark::ConnectionManager` has the same API:
-Some models are in the core DB, and others in shard DBs. Shard is selected based on a user attribute. For core models use the default ActiveRecord connection and for sharded models define and use Octoshark connections.
-Switch the connection in a controller with an around filter:
-
```ruby
-around_filter :select_shard
+CONN_MANAGER = Octoshark::ConnectionManager.new
+```
-def select_shard(&block)
- CONN_MANAGER.with_connection(current_user.shard, &block)
+Opening a new connection, executing query and closing the connection:
+
+```ruby
+CONN_MANAGER.with_connection(config) do |connection|
+ connection.execute("SELECT 1")
end
```
-Similar approach applies to other application entry-points like background jobs.
+## Using Octoshark with ActiveRecord models
-## Master-Slave example
+To tell an ActiveRecord model to use the Octoshark connection we can override the `Model.connection` method.
-All models are in master and slave databases. For master models use the default ActiveRecord connection and for slave models define and use Octoshark connections.
-
```ruby
-class ActiveRecord::Base
+class Post < ActiveRecord::Base
def self.connection
- # Return the current connection (from with_connection block) or default one
- CONN_MANAGER.current_or_default_connection
+ CONN_MANAGER.current_connection
end
end
```
-`CONN_MANAGER.current_or_default_connection` method returns the current connection while in `with_connection` block or the default ActiveRecord connection when outside.
+Alternatively, we can extract it as a module and include in multiple models.
+```ruby
+module ShardingModel
+ extend ActiveSupport::Concern
-## Multi-tenant example
+ module ClassMethods
+ def connection
+ CONN_MANAGER.current_connection
+ end
+ end
+end
+```
-Some models are in the core DB, and others in user's own dedicated database. For core models use the default ActiveRecord connection and for tenant models can use Octoshark's mechanism to create new temporary connection.
+To use a specific database connection:
-Switch the connection in a controller with an around filter:
-
```ruby
-# in initializer
-CONN_MANAGER = Octoshark::ConnectionManager.new
+CONN_MANAGER.with_connection(:c1) do
+ # run queries on c1
+ Post.first
+end
+```
-# in controller
+This connection switching in Rails applications is usually done from within an `around_filter` for controllers and in a similar way for other application "entry-points" like background jobs:
+
+```ruby
around_filter :select_shard
def select_shard(&block)
- CONN_MANAGER.with_new_connection(name, config, reusable: false, &block)
+ CONN_MANAGER.with_connection(current_user.shard, &block)
end
```
-`CONN_MANAGER.with_new_connection` method creates a temporary connection that will automatically disconnect. If you want to reuse it in subsequent connection switches, set `reusable: true` and it will be added to the connection manager and reused with the next calls. Depends on the use-case and what's preferable. In test environment usually you would want to set it to `reusable` so that database cleaner can clean data with transaction strategy.
+`CONN_MANAGER.current_connection` returns the active connection while the execution is in the `with_connection` block or raises `Octoshark::Error::NoCurrentConnection` outside of the `with_connection` block. In some cases, falling back to the default database connection for the Rails app might be preferable which can be done using `CONN_MANAGER.current_or_default_connection`.
-Alternatively, for better performance (only supported on MySQL), database connection can be switched with `use database` statement. Once connection manager is defined with connectino configs to database servers, selecting a database can be done with:
-```ruby
-CONN_MANAGER.use_database(:db1, 'database') do
- # run queries on database server identified by 'db1' using database 'database'
-end
-```
+## Octoshark::ConnectionPoolsManager.reset_connection_managers!
-## Octoshark.reset_connection_managers!
+When using `Octoshark::ConnectionPoolsManager`, whenever ActiveRecord::Base calls `establish_connection` (usually by an ancestor process that must have subsequently forked), `Octoshark.reset_connection_managers!` is automatically called to re-establish the Octoshark connections. It prevents `ActiveRecord::ConnectionNotEstablished` in the scenarios like:
-Whenever ActiveRecord::Base calls `establish_connection` (usually by an ancestor process that must have subsequently forked), `Octoshark.reset_connection_managers!` is automatically called to re-establish the Octoshark connections. It prevents `ActiveRecord::ConnectionNotEstablished` in the scenarios like:
-
* Unicorn before/after fork
* Spring prefork/serve
* Some rake tasks like `rake db:test:prepare`
-## Database Cleaner
+## Cleaning test databases
-RSpec example on how to clean default and Octoshark data with Database Cleaner:
+When using persistent connections, you can use tools like [DatabaseCleaner](https://github.com/DatabaseCleaner/database_cleaner) or [DatabaseRewinder](https://github.com/amatsuda/database_rewinder) to clean test databases. Here's an example of RSpec config for `DatabaseCleaner`:
```ruby
config.before(:suite) do
setup_database_cleaner
DatabaseCleaner.clean_with(:truncation)
@@ -177,15 +181,83 @@
DatabaseCleaner.clean_with(:transaction)
end
def setup_database_cleaner
DatabaseCleaner[:active_record, {connection: ActiveRecord::Base.connection_pool}]
- Octoshark.connection_managers.each do |manager|
+ Octoshark::ConnectionPoolsManager.connection_managers.each do |manager|
manager.connection_pools.each_pair do |connection_name, connection_pool|
DatabaseCleaner[:active_record, {connection: connection_pool}]
end
end
end
+```
+
+When using non-persistent connections where transaction rollback as a cleaning strategy will not work, we can use a custom solution inspired by `DatabaseRewinder`. It also works with dynamic databases created on the fly in the test suite.
+
+```ruby
+module DatabaseCleaner
+ module InsertRecorder
+ def execute(sql, *)
+ DatabaseCleaner.record_inserted_table(self, sql)
+ super
+ end
+
+ def exec_query(sql, *)
+ DatabaseCleaner.record_inserted_table(self, sql)
+ super
+ end
+ end
+
+ @@tables = []
+
+ def self.tables
+ @@tables
+ end
+
+ def self.record_inserted_table(connection, sql)
+ match = sql.match(/\AINSERT(?:\s+IGNORE)?\s+INTO\s+(?:\.*[`"]?([^.\s`"]+)[`"]?)*/i)
+ tables << match[1] if match && !tables.include?(match[1])
+ end
+
+ def self.clean
+ CoreDB.with_connection do |connection|
+ (
+ connection.tables.reject { |t| t == ActiveRecord::Migrator.schema_migrations_table_name } & tables
+ ).each do |table|
+ connection.disable_referential_integrity do
+ connection.execute "DELETE FROM #{connection.quote_table_name(table)};"
+ end
+ end
+ end
+ @@tables = []
+ end
+end
+
+require 'active_record/connection_adapters/abstract_mysql_adapter'
+ActiveRecord::ConnectionAdapters::AbstractMysqlAdapter.send(:prepend, DatabaseCleaner::InsertRecorder)
+```
+
+
+## Development Setup
+
+Setup database config and create databases:
+
+```bash
+cp spec/support/config.yml.template spec/support/config.yml
+rake db:create
+```
+
+Run specs:
+
+```bash
+bundle exec rspec spec
+```
+
+Install different active record versions defined in `Appraisals` and run specs for all of them:
+
+```bash
+bundle exec appraisal
+bundle exec appraisal rspec spec
```
## Logo