README.md in spidr-0.5.0 vs README.md in spidr-0.6.0
- old
+ new
@@ -3,10 +3,11 @@
* [Homepage](https://github.com/postmodern/spidr#readme)
* [Source](https://github.com/postmodern/spidr)
* [Issues](https://github.com/postmodern/spidr/issues)
* [Mailing List](http://groups.google.com/group/spidr)
* [IRC](http://webchat.freenode.net/?channels=spidr&uio=d4)
+* [![Build Status](https://travis-ci.org/postmodern/spidr.svg)](https://travis-ci.org/postmodern/spidr)
## Description
Spidr is a versatile Ruby web spidering library that can spider a site,
multiple domains, certain links or infinitely. Spidr is designed to be fast
@@ -26,11 +27,12 @@
* URL scheme.
* Host name
* Port number
* Full link
* URL extension
-* Provides call-backs for:
+ * Optional `/robots.txt` support.
+* Provides callbacks for:
* Every visited Page.
* Every visited URL.
* Every visited URL that matches a specified pattern.
* Every origin and destination URI of a link.
* Every URL that failed to be visited.
@@ -179,10 +181,10 @@
end
end
## Requirements
-* [ruby] >= 1.9.1
+* [ruby] >= 2.0.0
* [nokogiri] ~> 1.3
## Install
$ gem install spidr