README.rdoc in anemone-0.2.3 vs README.rdoc in anemone-0.3.0

- old
+ new

@@ -13,9 +13,11 @@ * Allows exclusion of URLs based on regular expressions * Choose the links to follow on each page with focus_crawl() * HTTPS support * Records response time for each page * CLI program can list all pages in a domain, calculate page depths, and more +* Obey robots.txt +* In-memory or persistent storage of pages during crawl, using TokyoCabinet or PStore == Examples See the scripts under the <tt>lib/anemone/cli</tt> directory for examples of several useful Anemone tasks. == Requirements