PERFORMANCE.md in pupa-0.1.7 vs PERFORMANCE.md in pupa-0.1.8

- old
+ new

@@ -1,22 +1,20 @@ -# Pupa.rb: A Data Scraping Framework +# Pupa.rb: Performance -## Performance - Pupa.rb offers several ways to significantly improve performance. In an example case, reducing disk I/O and skipping validation as described below reduced the time to scrape 10,000 documents from 100 cached HTTP responses from 100 seconds down to 5 seconds. Like fast tests, fast scrapers make development smoother. The `import` action's performance is currently limited by the database when a dependency graph is used to determine the evaluation order. If a dependency graph cannot be used because you don't know a related object's ID, [several optimizations](https://github.com/opennorth/pupa-ruby/issues/12) can be implemented to improve performance. -### Reducing HTTP requests +## Reducing HTTP requests HTTP requests consume the most time. To avoid repeat HTTP requests while developing a scraper, cache all HTTP responses. Pupa.rb will by default use a `web_cache` directory in the same directory as your script. You can change the directory by setting the `--cache_dir` switch on the command line, for example: ruby cat.rb --cache_dir /tmp/my_cache_dir -### Parallelizing HTTP requests +## Parallelizing HTTP requests To enable parallel requests, use the `typhoeus` gem. Unless you are using an old version of Typhoeus (< 0.5), both Faraday and Typhoeus define a Faraday adapter, but you must use the one defined by Typhoeus, like so: ```ruby require 'pupa' @@ -56,15 +54,15 @@ # If error... end end ``` -### Reducing disk I/O +## Reducing disk I/O After HTTP requests, disk I/O is the slowest operation. Two types of files are written to disk: HTTP responses are written to the cache directory, and JSON documents are written to the output directory. Writing to memory is much faster than writing to disk. -#### RAM file systems +### RAM file systems A simple solution is to create a file system in RAM, like `tmpfs` on Linux for example, and to use it as your `output_dir` and `cache_dir`. On OS X, you must create a RAM disk. To create a 128MB RAM disk, for example, run: ramdisk=$(hdiutil attach -nomount ram://$((128 * 2048)) | tr -d ' \t') diskutil erasevolume HFS+ 'ramdisk' $ramdisk @@ -78,19 +76,19 @@ diskutil unmount $ramdisk hdiutil detach $ramdisk Using a RAM disk will significantly improve performance; however, the data will be lost between reboots unless you move the data to a hard disk. Using Memcached (for caching) and Redis (for storage) is moderately faster than using a RAM disk, and Redis will not lose your output data between reboots. -#### Memcached +### Memcached You may cache HTTP responses in [Memcached](http://memcached.org/). First, require the `dalli` gem. Then: ruby cat.rb --cache_dir memcached://localhost:11211 The data in Memcached will be lost between reboots. -#### Redis +### Redis You may dump JSON documents in [Redis](http://redis.io/). First, require the `redis-store` gem. Then: ruby cat.rb --output_dir redis://localhost:6379/0 @@ -100,20 +98,20 @@ Requiring the `hiredis` gem will slightly improve performance. Note that Pupa.rb flushes the Redis database before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa.rb and other applications. You can select a different database than the default `0` for use with Pupa.rb by passing an argument like `redis://localhost:6379/15`, where `15` is the database number. -### Skipping validation +## Skipping validation The `json-schema` gem is slow compared to, for example, [JSV](https://github.com/garycourt/JSV). Setting the `--no-validate` switch and running JSON Schema validations separately can further reduce a scraper's running time. The [pupa-validate](https://npmjs.org/package/pupa-validate) npm package can be used to validate JSON documents using the faster JSV. In an example case, using JSV instead of the `json-schema` gem reduced by half the time to validate 10,000 documents. -### Ruby version +## Ruby version Pupa.rb requires Ruby 2.x. If you have already made all the above optimizations, you may notice a significant improvement by using Ruby 2.1, which has better garbage collection than Ruby 2.0. -### Profiling +## Profiling You can profile your code using [perftools.rb](https://github.com/tmm1/perftools.rb). First, install the gem: gem install perftools.rb