Contents

class Robots
  
  def initialize(url, file_name="robots.txt")
    uri = URI.parse(url)
    [uri.scheme, "://", uri.host, ":", uri.port, "/", file_name].join
    Cobweb.new(:cache => 6000).get([uri.scheme, "://", uri.host, ":", uri.port, "/", file_name].join)
    
  end
end

Version data entries

32 entries across 32 versions & 1 rubygems

Version	Path
cobweb-0.0.54	lib/robots.rb
cobweb-0.0.53	lib/robots.rb
cobweb-0.0.52	lib/robots.rb
cobweb-0.0.50	lib/robots.rb
cobweb-0.0.49	lib/robots.rb
cobweb-0.0.48	lib/robots.rb
cobweb-0.0.47	lib/robots.rb
cobweb-0.0.46	lib/robots.rb
cobweb-0.0.45	lib/robots.rb
cobweb-0.0.44	lib/robots.rb
cobweb-0.0.43	lib/robots.rb
cobweb-0.0.42	lib/robots.rb
cobweb-0.0.41	lib/robots.rb
cobweb-0.0.40	lib/robots.rb
cobweb-0.0.39	lib/robots.rb
cobweb-0.0.38	lib/robots.rb
cobweb-0.0.37	lib/robots.rb
cobweb-0.0.36	lib/robots.rb
cobweb-0.0.34	lib/robots.rb
cobweb-0.0.33	lib/robots.rb