Sha256: 2e7ba5fd077766ed0db6c68047c964cc5746209eca166d8f2ceaceb429229b24

Contents?: true

Size: 202 Bytes

Versions: 49

Compression:

Stored size: 202 Bytes

Contents

# robots.txt for http://www.example.com/

User-agent: cybermapper
Disallow:

User-agent: google
Disallow: /

User-agent: *
Disallow: /forms
Disallow: /gallery
Disallow: /more.html
Allow: /catalogs/about

Version data entries

49 entries across 49 versions & 1 rubygems

Version Path
cobweb-1.2.1 spec/samples/sample_site/robots.txt
cobweb-1.2.0 spec/samples/sample_site/robots.txt
cobweb-1.1.0 spec/samples/sample_site/robots.txt
cobweb-1.0.29 spec/samples/sample_site/robots.txt
cobweb-1.0.28 spec/samples/sample_site/robots.txt
cobweb-1.0.27 spec/samples/sample_site/robots.txt
cobweb-1.0.26 spec/samples/sample_site/robots.txt
cobweb-1.0.25 spec/samples/sample_site/robots.txt
cobweb-1.0.24 spec/samples/sample_site/robots.txt
cobweb-1.0.23 spec/samples/sample_site/robots.txt
cobweb-1.0.22 spec/samples/sample_site/robots.txt
cobweb-1.0.21 spec/samples/sample_site/robots.txt
cobweb-1.0.20 spec/samples/sample_site/robots.txt
cobweb-1.0.19 spec/samples/sample_site/robots.txt
cobweb-1.0.18 spec/samples/sample_site/robots.txt
cobweb-1.0.17 spec/samples/sample_site/robots.txt
cobweb-1.0.16 spec/samples/sample_site/robots.txt
cobweb-1.0.15 spec/samples/sample_site/robots.txt
cobweb-1.0.12 spec/samples/sample_site/robots.txt
cobweb-1.0.11 spec/samples/sample_site/robots.txt