[^a]fish
[+:,\.\;\/\\-]bot
^$
^IDA$
^ruby$
^voyager\/
acme\.spider
alexa
Alexandria(\s|\+)prototype(\s|\+)project
AllenTrack
almaden
appie
Arachmo
architext
archive\.org_bot
arks
asterias
atomz
autoemailspider
awbot
baiduspider
bbot
BDFetch
biadu
biglotron
bjaaland
blaiz\-bee
bloglines
blogpulse
boitho\.com\-dc
bookmark\-manager
bot
Brutus\/AET
bspider
bwh3_user_agent
celestial
cfnetwork|checkbot
checkprivacy
China\sLocal\sBrowse\s2\.6
cloakDetect
Code\sSample\sWeb\sClient
combine
commons\-httpclient
contentmatch
ContentSmartz
core
CoverScout
crawl
crawler 
cursor
custo
DataCha0s\/2\.0
daumoa
Demo\sBot
docomo
Download\+Master
DSurf
dtSearchSpider
dumbot
easydl
EmailSiphon
EmailWolf
exabot
fast-webcrawler
favorg
FDM(\s|\+)1
feedburner
FeedFetcher
feedfetcher\-google
ferret
Fetch(\s|\+)API(\s|\+)Request
findlinks
Fulltext
Funnelback
gaisbot
GetRight
geturl
gigabot
girafabot
gnodspider
Goldfire(\s|\+)Server
google
grub
gulliver
harvest
heritrix
hl_ftien_spider
holmes
htdig
htmlparser
HttpComponents\/1.1
HTTPFetcher
httpget\?5\.2\.2
httpget\-5\.2\.2
httrack
ia_archiver
ichiro
iktomi
ilse
internetseer
intute
iSiloX
Jakarta\+Commons\-HttpClient
java
jeeves
jobo
kyluka
larbin
libcurl
libwww
libwww\-perl
lilina
linkbot
linkcheck
linkchecker
LinkLint-checkonly
linkscan
linkwalker
livejournal\.com
lmspider
LOCKSS
lwp
LWP\:\:Simple
lwp\-request
lwp\-tivial
lwp\-trivial
lwp-request
lycos[_+]
mail.ru
MarcEdit.5.2.Web.Client
mediapartners\-google
Mediapartners-Google
megite
Microsoft(\s|\+)URL(\s|\+)Control
milbot
mimas
mj12bot
mnogosearch
moget
mojeekbot
momspider
motor
msiecrawler
msnbot
MuscatFerre
myweb
NABOT
nagios
NaverBot
netcraft
netluchs
ng\/2\.
Ning
no_user_agent
nomad
nutch
ocelli
Offline(\s|\+)Navigator
onetszukaj
OurBrowser
parsijoo
pear.php.net
perman
PHP\/
pioneer
playmusic\.com
playstarmusic\.com
powermarks
psbot
PycURL
python
qihoobot
rambler
Readpaper
redalert|robozilla
RePEc.link.checker
robot
robots
RPT\-HTTPClient\/0.3-3E
rss
scan4mail
scientificcommons
scirus
scooter
seekbot
seznambot
shoutcast
slurp
sogou
speedy
spider
spiderman
spiderview
Strider
sunrise
superbot
surveybot
T\-H\-U\-N\-D\-E\-R\-S\-T\-O\-N\-E
tailrank
technoratibot
Teleport(\s|\+)Pro
Teoma
titan
turnitinbot
twiceler
ucsd
ultraseek
URL2File
urlaliasbuilder
urllib
validator
virus[_+]detector
voila
w3c\-checklink
Wanadoo
Web(\s|\+)Downloader
WebCloner
webcollage
WebCopier
Webinator
weblayers
Webmetrics
webmirror
webreaper
WebStripper
WebZIP
Wget
wordpress
worm
www.gnip.com 
WWW\-Mechanize
xenu
Xenu(\s|\+)Link(\s|\+)Sleuth
y!j
yacy
yahoo
yandex
yodaobot
zealbot
zeus
zyborg