Contents

#! /usr/bin/env ruby
#
# Vendors the USA.gov-maintained list of US domains into domains.txt
# Source: https://github.com/GSA-OCSIT/govt-urls
#
# Usage: script/vendor-us
#
# Will automatically fetch latest version of the list and merge
# You can check for changes and commit via `git status`
#
# It's also probably a good idea to run `script/ci-build` for good measure

require './lib/gman/importer'

blacklist = %w(usagovQUASI usagovFEDgov)
source = 'https://raw.githubusercontent.com/GSA/govt-urls/master/government-urls-hierarchical-list.txt'

data = open(source).read
data = data.split('_' * 74)
data = data.last.strip
data = data.split(/\r?\n/).reject(&:empty?)

domains = {}
group = ''
data.each do |row|
  if row =~ /^\w/
    group = row
    domains[group] = []
  else
    domains[group].push row.sub("\.\t", '').strip
  end
end

domains.reject! { |g, _| blacklist.include?(g) }
Gman.import(domains)

Version data entries

2 entries across 2 versions & 1 rubygems

Version	Path
gman-6.0.1	script/vendor-us
gman-6.0.0	script/vendor-us