README.textile in methodmissing-scrooge-1.0.4 vs README.textile in methodmissing-scrooge-2.0.0

- old
+ new

@@ -1,535 +1,225 @@ h1. Scrooge -A Framework and ORM agnostic Model / record attribute tracker to ensure production -Ruby applications only fetch the database content needed to minimize wire traffic -and reduce conversion overheads to native Ruby types. +h4. This is a complete rewrite from the initial coverage at "igvita.com":http://www.igvita.com/2009/02/27/activerecord-optimization-with-scrooge/ - read on below -This is mostly an experiment into unobtrusive tracking, respecting development workflows -and understanding Rack internals better. +Many thanks to Stephen Sykes ( "pennysmalls.com":http://pennysmalls.com ) for his time spent on shaping, implementing and troubleshooting this release. +An ActiveRecord attribute tracker to ensure production Ruby applications only fetch the database content needed to minimize wire traffic and reduce conversion overheads to native Ruby types. + h2. Why bother ? * Object conversion and moving unnecessary data is both expensive and tax existing infrastructure in high load setups * Manually extracting and scoping SELECT clauses is not sustainable in a clean and painless manner with iterative development, even less so in large projects. -h2. Suggested Use +h2. What it does -There's 3 basic modes of operation : +<pre> +<code> + Processing HotelsController#show (for 127.0.0.1 at 2009-03-12 14:32:45) [GET] + Parameters: {"action"=>"show", "id"=>"8699-radisson-hotel-waterfront-cape-town", "controller"=>"hotels"} + Hotel Load Scrooged (0.3ms) SELECT `hotels`.id FROM `hotels` WHERE (`hotels`.`id` = 8699) + Rendering template within layouts/application + Rendering hotels/show + Hotel Load (0.2ms) SELECT `hotels`.location_id,`hotels`.hotel_name,`hotels`.location,`hotels`.from_price,`hotels`.star_rating,`hotels`.apt,`hotels`.latitude,`hotels`.longitude,`hotels`.distance,`hotels`.narrative,`hotels`.telephone,`hotels`.important_notes,`hotels`.nearest_tube,`hotels`.nearest_rail,`hotels`.created_at,`hotels`.updated_at FROM `hotels` WHERE (`hotels`.`id` = 8699) + Image Load Scrooged (0.2ms) SELECT `images`.id FROM `images` WHERE (`images`.hotel_id = 8699) LIMIT 1 + Image Load (0.2ms) SELECT `images`.hotel_id,`images`.title,`images`.url,`images`.width,`images`.height,`images`.thumbnail_url,`images`.thumbnail_width,`images`.thumbnail_height,`images`.has_thumbnail,`images`.created_at,`images`.updated_at FROM `images` WHERE (`images`.`id` = 488) + Rendered shared/_header (0.0ms) + Rendered shared/_navigation (0.2ms) + Image Load Scrooged (0.2ms) SELECT `images`.id FROM `images` WHERE (`images`.hotel_id = 8699) + CACHE (0.0ms) SELECT `images`.hotel_id,`images`.title,`images`.url,`images`.width,`images`.height,`images`.thumbnail_url,`images`.thumbnail_width,`images`.thumbnail_height,`images`.has_thumbnail,`images`.created_at,`images`.updated_at FROM `images` WHERE (`images`.`id` = 488) + Address Columns (44.8ms) SHOW FIELDS FROM `addresses` + Address Load Scrooged (0.5ms) SELECT `addresses`.id FROM `addresses` WHERE (`addresses`.hotel_id = 8699) LIMIT 1 + Rendered hotels/_show_sidebar (49.4ms) + Rendered shared/_footer (0.1ms) + Completed in 56ms (View: 8, DB: 46) | 200 OK [http://localhost/hotels/8699-radisson-hotel-waterfront-cape-town] -* Track : Track attribute access to dump a representative scope profile. -* Scope : Scope the process and related resources to a previously persisted scope profile. + Processing HotelsController#show (for 127.0.0.1 at 2009-03-12 14:32:48) [GET] + Parameters: {"action"=>"show", "id"=>"8699-radisson-hotel-waterfront-cape-town", "controller"=>"hotels"} + Hotel Load Scrooged (0.3ms) SELECT `hotels`.narrative,`hotels`.from_price,`hotels`.star_rating,`hotels`.hotel_name,`hotels`.id FROM `hotels` WHERE (`hotels`.`id` = 8699) + Rendering template within layouts/application + Rendering hotels/show + Image Load Scrooged (0.3ms) SELECT `images`.url,`images`.id,`images`.height,`images`.width FROM `images` WHERE (`images`.hotel_id = 8699) LIMIT 1 + Rendered shared/_header (0.0ms) + Rendered shared/_navigation (0.2ms) + Image Load Scrooged (0.3ms) SELECT `images`.thumbnail_width,`images`.id,`images`.thumbnail_height,`images`.thumbnail_url FROM `images` WHERE (`images`.hotel_id = 8699) + Address Load Scrooged (0.2ms) SELECT `addresses`.id FROM `addresses` WHERE (`addresses`.hotel_id = 8699) LIMIT 1 + Rendered hotels/_show_sidebar (1.3ms) + Rendered shared/_footer (0.0ms) + Completed in 7ms (View: 5, DB: 1) | 200 OK [http://localhost/hotels/8699-radisson-hotel-waterfront-cape-town] +</code> +</pre> -* Track then scope : A multi-stage strategy that tracks attribute access for a given warmup period, aggregates the results - across multiple processes and enforce a scoping policy representative of the tracking phase. +h2. Suggested Use -h2. Resources +Install, and you're off to the races! -A resource is : +h2. Installation -* A controller and action endpoint ( inferred through framework specific routing ) -* A content type / format - a PDF representation may have different Model attribute requirements than a vanilla ERB view. -* Request method - typically popular public facing GET requests -* Public or Private as a logged in / authenticated representation likely have different attribute requirements. +h4. As a Rails plugin ( Recommended ) -All Model to attribute mappings is tracked on a per Resource basis.Multiple Models per Resource is supported. + ./script/plugin install git://github.com/methodmissing/scrooge.git -h2. Strategies +h4. From Git -h4. Tracking + git pull git://github.com/methodmissing/scrooge.git -In tracking mode Scrooge installs filters ( either through Rack middleware or framework specific hooks ) that track attribute access on a per Resource basis. +h4. As a Gem -A Kernel#at_exit callback dumps and timestamps this profile ( or scope ) to eg. *framework_configuration_directory/config/scopes/1234147851/scope.yml* + sudo gem install methodmissing-scrooge -s http://gems.github.com -This typically works well with functional or integration testing and can yield a substantial birds eye view of attribute -use.The accuracy is directly proportional to test coverage and the quality of the test suite. +h2. Stability -Example log output : +The whole ActiveRecord test suite passes with scrooge, except for 9 failures related to callsite augmentation (note the SQL reload snippets below).Thoughts on handling or circumventing this much appreciated. <pre> <code> - Processing HotelsController#index (for 0.0.0.0 at 2009-02-09 02:55:55) [GET] - Parameters: {"action"=>"index", "controller"=>"hotels"} - Hotel Load (0.3ms) SELECT * FROM `hotels` LIMIT 0, 15 - Rendering template within layouts/application - Rendering hotels/index - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 491) LIMIT 1 - [Scrooge] read attribute updated_at - Rendered hotels/_hotel (2.7ms) - Rendered shared/_header (0.1ms) - Rendered shared/_navigation (0.3ms) - Missing template hotels/_index_sidebar.erb in view path app/views - Rendered shared/_sidebar (0.1ms) - Rendered shared/_footer (0.1ms) - Completed in 91ms (View: 90, DB: 1) | 200 OK [http://test.host/hotels] - SQL (0.3ms) ROLLBACK - SQL (0.1ms) BEGIN + 2) Failure: +test_finding_with_includes_on_belongs_to_association_with_same_include_includes_only_once(EagerAssociationTest) + [/opt/local/lib/ruby/gems/1.8/gems/activerecord-2.3.1/test/cases/../../lib/active_record/test_case.rb:31:in `assert_queries' + /opt/local/lib/ruby/gems/1.8/gems/activerecord-2.3.1/test/cases/associations/eager_test.rb:139:in `test_finding_with_includes_on_belongs_to_association_with_same_include_includes_only_once' + /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.3.1/lib/active_support/testing/setup_and_teardown.rb:57:in `__send__' + /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.3.1/lib/active_support/testing/setup_and_teardown.rb:57:in `run']: +5 instead of 3 queries were executed. +Queries: +SELECT `posts`.id,`posts`.type FROM `posts` WHERE (`posts`.`id` = 1) +SELECT `posts`.author_id,`posts`.title,`posts`.body,`posts`.comments_count,`posts`.taggings_count FROM `posts` WHERE (`posts`.`id` = 1) +SELECT `authors`.name,`authors`.id FROM `authors` WHERE (`authors`.`id` = 1) +SELECT `authors`.author_address_id,`authors`.author_address_extra_id FROM `authors` WHERE (`authors`.`id` = 1) +SELECT `author_addresses`.id FROM `author_addresses` WHERE (`author_addresses`.`id` = 1) . +<3> expected but was +<5>. </code> </pre> -An example scope / profile, saved to disk : +To run tests in your environment : -<pre> -<code> ---- -- hotels_show_get: - :action: show - :controller: hotels - :method: :get - :is_public: true - :format: "*/*" - :models: - - Address: - - line1 - - line2 - - created_at - - postcode - - updated_at - - country_id - - county - - location_id - - town - - hotel_id - - Hotel: - - important_notes - - location_id -- locations_index_get: - :action: index - :controller: locations - :method: :get - :is_public: true - :format: "*/*" - :models: - - Location: - - name - - created_at - - code - - updated_at - - level - - id -- countries_index_get: - :action: index - :controller: countries - :method: :get - :is_public: true - :format: "*/*" - :models: - - Country: - - name - - created_at - - code - - updated_at - - id - - location_id - - continent_id -- hotels_index_get: - :action: index - :controller: hotels - :method: :get - :is_public: true - :format: "*/*" - :models: - - Hotel: - - from_price - - narrative - - star_rating - - latitude - - created_at - - hotel_name - - updated_at - - important_notes - - id - - apt - - location_id - - nearest_tube - - longitude - - telephone - - nearest_rail - - location_name - - distance - - Image: - - thumbnail_width - - created_at - - title - - updated_at - - url - - thumbnail_height - - height - - thumbnail_url - - has_thumbnail - - hotel_id - - width -</code> -</pre> +* Configure to run the ActiveRecord test suite as per the "docs":http://github.com/rails/rails/blob/8a17fd1a65ab8e2fa6b36d79603fde0e6ffd083f/activerecord/RUNNING_UNIT_TESTS +* 'rake test' from within the scrooge root directory +* It'll attempt to find the path to the ActiveRecord test cases through rubygems +* Known to work with both 2.2.2, 2.3.0 and the upcoming 2.3.1 -h4. Scope +h2. Initial Benchmarks -A previously persisted scope / profile can be restored from disk and injected to the applicable Resources.Database content retrieved will match that of the given scope timestamp. +Passenger, Rails 2.2.2, remote DB : -This is typically pushed to production where a hybrid ( track then scope strategy) mode of operation is frowned upon and adjusted for each major release or deployment. - -Example log output : - <pre> <code> - Processing HotelsController#index (for 0.0.0.0 at 2009-02-09 02:59:41) [GET] - Parameters: {"action"=>"index", "controller"=>"hotels"} - Hotel Load (0.4ms) SELECT hotels.narrative, hotels.from_price, hotels.created_at, hotels.latitude, hotels.star_rating, hotels.hotel_name, hotels.updated_at, hotels.important_notes, hotels.apt, hotels.id, hotels.nearest_tube, hotels.location_id, hotels.nearest_rail, hotels.telephone, hotels.longitude, hotels.distance, hotels.location_name FROM `hotels` LIMIT 0, 15 - Rendering template within layouts/application - Rendering hotels/index - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 491) LIMIT 1 - Rendered hotels/_hotel (2.8ms) - Rendered shared/_header (0.1ms) - Rendered shared/_navigation (0.3ms) - Missing template hotels/_index_sidebar.erb in view path app/views - Rendered shared/_sidebar (0.1ms) - Rendered shared/_footer (0.1ms) - Completed in 90ms (View: 5, DB: 1) | 200 OK [http://test.host/hotels] - SQL (0.1ms) ROLLBACK - SQL (0.1ms) BEGIN -</code> -</pre> +Without scrooge: -h4. Track then scope +Concurrency Level: 1 +Time taken for tests: 68.279156 seconds +Complete requests: 150 +Failed requests: 0 +Write errors: 0 +Total transferred: 13741201 bytes +HTML transferred: 13679100 bytes +Requests per second: 2.20 [#/sec] (mean) +Time per request: 455.194 [ms] (mean) +Time per request: 455.194 [ms] (mean, across all concurrent requests) +Transfer rate: 196.53 [Kbytes/sec] received -Multi-stage and self configuring strategy that tracks attribute access for a given warmup period, synchronize the results across n-1 processes, aggregate the results to be representative of the whole cluster ( or seamless fallback to a single process ), remove the tracking filters and install functionality that scopes database access to that of the tracking phase. +With scrooge: -Recommended for production use. - -Example log output : - -<pre> -<code> -Processing HotelsController#index (for 127.0.0.1 at 2009-02-16 00:00:58) [GET] - Parameters: {"action"=>"index", "controller"=>"hotels"} - Hotel Load (0.5ms) SELECT * FROM `hotels` LIMIT 0, 15 - Hotel Columns (7.7ms) SHOW FIELDS FROM `hotels` - SQL (3.9ms) SELECT count(*) AS count_all FROM `hotels` -Rendering template within layouts/application -Rendering hotels/index - Image Load (0.5ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11381) LIMIT 1 - Image Columns (3.6ms) SHOW FIELDS FROM `images` -Rendered hotels/_hotel (200.2ms) - Image Load (0.4ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11382) LIMIT 1 -Rendered hotels/_hotel (2.4ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11697) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12693) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12738) LIMIT 1 -Rendered hotels/_hotel (1.6ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12886) LIMIT 1 -Rendered hotels/_hotel (1.9ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13007) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13074) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13077) LIMIT 1 -Rendered hotels/_hotel (1.6ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13078) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.3ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13079) LIMIT 1 -Rendered hotels/_hotel (2.4ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13080) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13082) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13085) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13105) LIMIT 1 -Rendered hotels/_hotel (1.6ms) -Rendered shared/_header (0.4ms) -Rendered shared/_navigation (0.8ms) -Missing template hotels/_index_sidebar.erb in view path app/views -Rendered shared/_sidebar (0.4ms) -Rendered shared/_footer (0.3ms) -Completed in 270ms (View: 243, DB: 20) | 200 OK [http://localhost/hotels] -[Scrooge] Execute stage :synchronize ... -[Scrooge] Uninstalling tracking middleware ... -[Scrooge] Stop tracking ... -[Scrooge] Synchronize results with other processes ... -Cache write: 17619400_63223_756033 -Cache read: scrooge_tracker_aggregation -Cache write: scrooge_tracker_aggregation -[Scrooge] Execute stage :aggregate ... -[Scrooge] Aggregate results from other processes ... - - -Processing HotelsController#index (for 127.0.0.1 at 2009-02-16 00:01:37) [GET] - Parameters: {"action"=>"index", "controller"=>"hotels"} - Hotel Load (0.5ms) SELECT * FROM `hotels` LIMIT 0, 15 - SQL (0.2ms) SELECT count(*) AS count_all FROM `hotels` -Rendering template within layouts/application -Rendering hotels/index - Image Load (0.3ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11381) LIMIT 1 -Rendered hotels/_hotel (2.0ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11382) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 11697) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12693) LIMIT 1 -Rendered hotels/_hotel (1.6ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12738) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 12886) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13007) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13074) LIMIT 1 -Rendered hotels/_hotel (1.4ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13077) LIMIT 1 -Rendered hotels/_hotel (1.6ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13078) LIMIT 1 -Rendered hotels/_hotel (1.6ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13079) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13080) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13082) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13085) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT * FROM `images` WHERE (`images`.hotel_id = 13105) LIMIT 1 -Rendered hotels/_hotel (81.1ms) -Rendered shared/_header (0.1ms) -Rendered shared/_navigation (0.3ms) -Missing template hotels/_index_sidebar.erb in view path app/views -Rendered shared/_sidebar (0.1ms) -Rendered shared/_footer (0.1ms) -Completed in 113ms (View: 107, DB: 4) | 200 OK [http://localhost/hotels] -Cache read: scrooge_tracker_aggregation -Cache read: 17619400_63223_756033 -[Scrooge] Execute stage :scope ... -[Scrooge] Scope ... - - -Processing HotelsController#index (for 127.0.0.1 at 2009-02-16 00:01:53) [GET] - Parameters: {"action"=>"index", "controller"=>"hotels"} - Hotel Load (0.7ms) SELECT hotels.narrative, hotels.from_price, hotels.created_at, hotels.latitude, hotels.star_rating, hotels.hotel_name, hotels.updated_at, hotels.important_notes, hotels.apt, hotels.id, hotels.nearest_tube, hotels.location_id, hotels.nearest_rail, hotels.telephone, hotels.longitude, hotels.distance, hotels.location_name FROM `hotels` LIMIT 0, 15 - SQL (0.2ms) SELECT count(*) AS count_all FROM `hotels` -Rendering template within layouts/application -Rendering hotels/index - Image Load (0.3ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 11381) LIMIT 1 -Rendered hotels/_hotel (2.0ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 11382) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 11697) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 12693) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 12738) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 12886) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13007) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13074) LIMIT 1 -Rendered hotels/_hotel (1.4ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13077) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13078) LIMIT 1 -Rendered hotels/_hotel (1.8ms) - Image Load (0.3ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13079) LIMIT 1 -Rendered hotels/_hotel (1.9ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13080) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13082) LIMIT 1 -Rendered hotels/_hotel (1.5ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13085) LIMIT 1 -Rendered hotels/_hotel (1.7ms) - Image Load (0.2ms) SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 13105) LIMIT 1 -Rendered hotels/_hotel (1.7ms) -Rendered shared/_header (0.1ms) -Rendered shared/_navigation (0.2ms) -Missing template hotels/_index_sidebar.erb in view path app/views -Rendered shared/_sidebar (0.0ms) -Rendered shared/_footer (0.0ms) -Completed in 34ms (View: 27, DB: 4) | 200 OK [http://localhost/hotels] +Concurrency Level: 1 +Time taken for tests: 58.162039 seconds +Complete requests: 150 +Failed requests: 0 +Write errors: 0 +Total transferred: 13747200 bytes +HTML transferred: 13685100 bytes +Requests per second: 2.58 [#/sec] (mean) +Time per request: 387.747 [ms] (mean) +Time per request: 387.747 [ms] (mean, across all concurrent requests) +Transfer rate: 230.82 [Kbytes/sec] received </code> </pre> -h2. Installation +h2. How it works -h4. As a Rails plugin ( Recommended ) +h4. Callsites - ./script/plugin install git://github.com/methodmissing/scrooge.git +Ruby allows introspection of the call tree through -h4. From Git - - git pull git://github.com/methodmissing/scrooge.git - - -h4. As a Gem - - sudo gem install methodmissing-scrooge -s http://gems.github.com - -h2. Configuration - -Scrooge installs ( see recommended installation above ) a configuration file with the following format within *framework_configuration_directory/scrooge.yml ( RAILS_ROOT/config/scrooge.yml for a Rails setup ) : - <pre> <code> - production: - orm: :active_record - storage: :memory - strategy: :track_then_scope - warmup: 600 # warmup / track for 10 minutes - scope: - on_missing_attribute: :reload # or :raise - logged_in_session: :user_id # session key that represents the logged in user - enabled: true - development: - orm: :active_record - storage: :memory - strategy: :track - warmup: 600 # warmup / track for 10 minutes - scope: - on_missing_attribute: :reload # or :raise - logged_in_session: :user_id # session key that represents the logged in user - enabled: true - test: - orm: :active_record - storage: :memory - strategy: :track - warmup: 600 # warmup / track for 10 minutes - scope: - on_missing_attribute: :reload # or :raise - logged_in_session: :user_id # session key that represents the logged in user - enabled: true -</code> -</pre> + Kernel#caller +</code> +</pre> -h4. ORM +Scrooge analyzes the last 10 calltree elements that triggered -Scrooge is ORM agnostic and ships with an ActiveRecord layer. - <pre> <code> - orm: :active_record + ActiveRecord::Base.find_by_sql +</code> </pre> -</code> -h4. Storage backend +Lets refer to that as a callsite, or signature. -Tracking results can be persisted to a given backend or storage option.Ships with a memory store, but can be extended to file system, memcached etc. as all Tracker components is designed to be Marshal friendly.A stub for future functionality. +Thus given SQL such as <pre> <code> - storage: :memory +"SELECT * FROM `images` WHERE (`images`.hotel_id = 11697) LIMIT 1" +</code> </pre> -</code> -h4. Verbose +Called from our application helper -Log all tracking interactions to the framework logger when enabled.Disabled for production. - <pre> <code> - verbose: false +["/Users/lourens/projects/superbreak_app/vendor/plugins/scrooge/rails/../lib/scrooge.rb:27:in `find_by_sql'", "/Users/lourens/projects/superbreak_app/vendor/rails/activerecord/lib/active_record/base.rb:1557:in `find_every'", "/Users/lourens/projects/superbreak_app/vendor/rails/activerecord/lib/active_record/base.rb:1514:in `find_initial'", "/Users/lourens/projects/superbreak_app/vendor/rails/activerecord/lib/active_record/base.rb:613:in `find'", "/Users/lourens/projects/superbreak_app/vendor/rails/activerecord/lib/active_record/associations/association_collection.rb:60:in `find'", "/Users/lourens/projects/superbreak_app/vendor/rails/activerecord/lib/active_record/associations/association_collection.rb:67:in `first'", "/Users/lourens/projects/superbreak_app/app/helpers/application_helper.rb:60:in `hotel_image'", "/Users/lourens/projects/superbreak_app/app/views/hotels/_hotel.html.erb:4:in `_run_erb_app47views47hotels47_hotel46html46erb_locals_hotel_hotel_counter_object'", "/Users/lourens/projects/superbreak_app/vendor/rails/actionpack/lib/action_view/renderable.rb:36:in `send'", "/Users/lourens/projects/superbreak_app/vendor/rails/actionpack/lib/action_view/renderable.rb:36:in `render'", "/Users/lourens/projects/superbreak_app/vendor/rails/actionpack/lib/action_view/renderable_partial.rb:20:in `render'"] +</code> </pre> -</code> -h4. Strategy - -One of :track, :scope or :track_then_scope .Only the :track_then_scope strategy respects the :warmup configuration option. - -h4. Warmup - -The designated warmup period for the :track_then_scope strategy, given in seconds.Typically 600 to 3600. - +We can generate a unique callsite identifier with the following calculation : <pre> <code> - warmup: 600 +(The above calltree << "SELECT * FROM `images` ).hash " # cut off conditions etc. +</code> </pre> -</code> +Callsites are tracked on a per model ( table name ) basis. + h4. Scope -A scope is a reference to a timestamped Scrooge run where access to Model attributes is tracked on a per Resource basis. +Only SQL statements that meet the following criteria is considered for optimization : -<pre> -<code> - scope: 1234567891 -</pre> -</code> +* A SELECT statement -If not scope is given in the configuration, ENV['scope'] would also be considered to facilitate configuration through Capistrano etc. +* Not an INNER JOIN -h4. Handling Missing Attributes +* The Model has a primary key defined -When the contents for a given Model attribute has not been retrieved from the database, most ORM frameworks raise an error by default.This is configurable to reloading the model with all it's columns or raise instead. +h4. How it tracks -<pre> -<code> - on_missing_attribute: :reload # or :raise -</pre> -</code> +The ActiveRecord attributes Hash is replaced with a proxy that automatically augments the callsite with any attributes referenced through the Hash lookup keys. -h4. Private and Public Resources +h4. Storage -An authenticated or logged in resource likely has different model requirements than it's public counterpart. +There's a slight memory hit for each model as the callsites is stored as a class level Hash, which is relatively lightweight and looks like this : <pre> <code> - logged_in_session: :user_id # session key that represents the logged in user -</pre> +{-113952497=>#<Set: {"User", "Password"}>} </code> - -h4. Status - -Scrooge can be disabled with : - -<pre> -<code> - enabled: false </pre> -</code> -h2. Rails specific rake tasks. +h4. Tracking and scoping ? -Ships with tasks to assist in inspecting results. +The tracking and scoping phases is superseded by this implementation - none of those hindrances anymore. -<pre> -<code> -methodmissing:superbreak_app lourens$ rake scrooge:list -(in /Users/lourens/projects/superbreak_app) -- 1234735663 -- 1234735722 -- 1234735744 -- 1234735790 -- 1234738880 -methodmissing:superbreak_app lourens$ rake scope=1234735790 scrooge:inspect -(in /Users/lourens/projects/superbreak_app) -#<GET :hotels/show (*/*) - - #<Hotel :important_notes, :location_id> - - #<Address :line1, :created_at, :line2, :postcode, :updated_at, :country_id, :county, :location_id, :town, :hotel_id> +h2. Todo -#<GET :countries/index (*/*) - - #<Country :name, :created_at, :code, :updated_at, :id, :location_id, :continent_id> +* Deeper coverage for Scrooge::AttributesProxy ( pending conversion to a subclass of Hash instead ) -#<GET :locations/index (*/*) - - #<Location :name, :created_at, :code, :updated_at, :level, :id> +* Extract Scrooge::Callsite -#<GET :hotels/index (*/*) - - #<Image :created_at, :thumbnail_width, :title, :updated_at, :url, :thumbnail_height, :height, :thumbnail_url, :has_thumbnail, :width, :hotel_id> - - #<Hotel :narrative, :from_price, :created_at, :latitude, :star_rating, :hotel_name, :updated_at, :important_notes, :apt, :id, :nearest_tube, :location_id, :nearest_rail, :telephone, :longitude, :distance, :location_name> -</code> -</pre> +* Track both columns AND association invocations off Scrooge::Callsite -h2. Notes +* Have invoking Model#attributes not associate all columns with the callsite -This is an initial release, has not yet been battle tested in production and is pending Ruby 1.9.1 compatibility. +* Avoid possible missing attribute exceptions for destroyed objects -Developed on and for Rails 2.3, known to work with Rails 2.2.2 - -h2. Known pending issues - -* Do not track columns that isn't explicitly defined ( AR counter caching checks etc. ) : -<code> -<pre> -ActiveRecord::StatementInvalid (Mysql::Error: Unknown column 'orders.brochures_count' in 'field list': SELECT orders.brochures_count, orders.id FROM `orders` WHERE (`orders`.`id` = 1688139) LIMIT 1) -</code> -</pre> - -* alias_attribute compatibility \ No newline at end of file +(c) 2009 Lourens Naudé (methodmissing) and Stephen Sykes (sdsykes)