Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Googlebot-Image/1.0
Googlebot-News
Googlebot-Video/1.0
AdsBot-Google (+http://www.google.com/adsbot.html)
AdsBot-Google-Mobile-Apps
Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=728742641706423)
Mediapartners-Google
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)
Mozilla/5.0 (compatible; Google-InspectionTool/1.0)
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36
GoogleOther
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
WGETbot/1.0 (+http://wget.alanreed.org)
Wget/1.14 (linux-gnu)
Wget/1.20.3 (linux-gnu)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
Python-urllib/1.17
Python-urllib/2.5
Python-urllib/2.6
Python-urllib/2.7
Python-urllib/3.1
Python-urllib/3.2
Python-urllib/3.3
Python-urllib/3.4
Python-urllib/3.5
Python-urllib/3.6
Python-urllib/3.7
python-requests/2.9.2
python-requests/2.11.1
python-requests/2.18.4
python-requests/2.19.1
python-requests/2.20.0
python-requests/2.21.0
python-requests/2.22.0
Python/3.9 aiohttp/3.7.3
Python/3.8 aiohttp/3.7.2
Python/3.7 aiohttp/3.6.2a2
python-httpx/0.16.1
python-httpx/0.13.0.dev1
2Bone_LinkChecker/1.0 libwww-perl/6.03
2Bone_LinkChkr/1.0 libwww-perl/6.03
amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
httpunit/1.x
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.16 (KHTML, like Gecko; compatible; Friendly_Crawler/2.0) Chrome/120.0.6099.217 Safari/605.1.15/Nutch-1.20-SNAPSHOT
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
istellabot-nutch/Nutch-1.10
Go-http-client/1.1
Go-http-client/2.0
phpcrawl
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
librabot/1.0 (+http://search.msn.com/msnbot.htm)
librabot/2.0 (+http://search.msn.com/msnbot.htm)
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm).
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8
FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)
BIGLOTRON (Beta 2;GNU/Linux)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Gigabot/1.0
Gigabot/2.0 (http://www.gigablast.com/spider.html)
GigablastOpenSource/1.0
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0;  http://www.exabot.com/go/robot)
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
ia_archiver-web.archive.org
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
panscient.com
yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.8-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.9-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.11-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4  Firefox/3.5.3
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
MLBot (www.metadatalabs.com/mlbot)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots
Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots yabs01)
Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/4.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexTurbo/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAdNet/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexForDomain/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMarket/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMarket/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMobileScreenShotBot/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexOntoDB/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexOntoDBAPI/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPartner/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexRCA/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSpravBot/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexTracker/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVerticals/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots)
Linguee Bot (http://www.linguee.com/bot)
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
PostRank/2.0 (postrank.com)
PostRank/2.0 (postrank.com; 1 subscribers)
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
Turnitin (https://bit.ly/2UvnfoQ)
Mozilla/5.0 (compatible;  Page2RSS/0.7; +http://page2rss.com/)
Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
linkdex.com/v2.0
linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
dotbot
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/6.1; News; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
fuelbot
CrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot)
Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)
Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
woobot
ZoominfoBot (zoominfobot at zoominfo dot com)
Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot
Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)
Mozilla/5.0 (compatible; eright/1.0; +bot@eright.com)
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
semanticbot
semanticbot (info@semanticaudience.com)
Aboundex/0.2 (http://www.aboundex.com/crawler/)
Aboundex/0.3 (http://www.aboundex.com/crawler/)
CipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/www.example.com)
Summify (Summify/1.0.1; +http://summify.com)
CCBot/2.0 (http://commoncrawl.org/faq/)
CCBot/2.0 (https://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
ec2linkfinder
Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
Wotbox/2.01 (+http://www.wotbox.com/bot/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/2.0 (ichiro@nttr.co.jp)
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
'Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)'
drupact/0.7; http://www.arocom.de/drupact
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
coccoc
coccoc/1.0 ()
coccoc/1.0 (http://help.coccoc.com/)
coccoc/1.0 (http://help.coccoc.vn/)
www.integromedb.org/Crawler
it2media-domain-crawler/1.0 on crawler-prod.it2media.de
it2media-domain-crawler/2.0
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;  http://www.changedetection.com/bot.html )
WeSEE:Search
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )  Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)  Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
psbot-image (+http://www.picsearch.com/bot.html)
psbot-page (+http://www.picsearch.com/bot.html)
psbot/0.1 (+http://www.picsearch.com/bot.html)
CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
Mozilla/5.0 (compatible; fr-crawler/1.1)
binlar_2.6.3 binlar2.6.3@unspecified.mail
binlar_2.6.3 binlar_2.6.3@unspecified.mail
binlar_2.6.3 larbin2.6.3@unspecified.mail
binlar_2.6.3 phanendra_kalapala@McAfee.com
binlar_2.6.3 test@mgmt.mic
SimpleCrawler/0.1
Twitterbot/0.1
Twitterbot/1.0
cXensebot/1.1a
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
SMTBot (similartech.com/smtbot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)                 Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
A6-Indexer
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
Facebot/1.0
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com
Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
ltx71 - (http://ltx71.com/)
nerdybot
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
crawler4j (http://code.google.com/p/crawler4j/)
crawler4j (https://github.com/yasserg/crawler4j/)
Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
SEMrushBot
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey Site Explorer; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html))
Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
AddThis.com robot tech.support@clearspring.com
Screaming Frog SEO Spider/5.1
MetaURI API/2.0 +metauri.com
Scrapy/1.0.3 (+http://scrapy.org)
LivelapBot/0.2 (http://site.livelap.com/crawler)
Livelapbot/0.1
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
CapsuleChecker (http://www.capsulink.com/)
Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
netEstate NE Crawler (+http://www.sengine.info/)
netEstate NE Crawler (+http://www.website-datenbank.de/)
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
www.deadlinkchecker.com XMLHTTP/1.0
www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
Slack-ImgProxy (+https://api.slack.com/robots)
Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
Slack-ImgProxy 149 (+https://api.slack.com/robots)
Embedly +support@embed.ly
Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
iskanie (+http://www.iskanie.com)
SafeDNSBot (https://www.safedns.com/searchbot)
Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
Slackbot-LinkExpanding (+https://api.slack.com/robots)
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Slackbot 1.0 (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
datagnionbot (+http://www.datagnion.com/bot.html)
Google-Adwords-Instant (+http://www.google.com/adsbot.html)
Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
adbeat_bot
WhatsApp
WhatsApp/0.3.4479 N
WhatsApp/0.3.4679 N
WhatsApp/0.3.4941 N
WhatsApp/2.12.15/i
WhatsApp/2.12.16/i
WhatsApp/2.12.17/i
WhatsApp/2.12.449 A
WhatsApp/2.12.453 A
WhatsApp/2.12.510 A
WhatsApp/2.12.540 A
WhatsApp/2.12.548 A
WhatsApp/2.12.555 A
WhatsApp/2.12.556 A
WhatsApp/2.16.1/i
WhatsApp/2.16.13 A
WhatsApp/2.16.2/i
WhatsApp/2.16.42 A
WhatsApp/2.16.57 A
WhatsApp/2.19.92 i
WhatsApp/2.19.175 A
WhatsApp/2.19.244 A
WhatsApp/2.19.258 A
WhatsApp/2.19.308 A
WhatsApp/2.19.330 A
Mozilla/5.0 (compatible;contxbot/1.0)
Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Pinterest/0.2 (+http://www.pinterest.com/bot.html)
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;  WOW64;  Trident/6.0;  BingPreview/1.0b)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  WOW64;  Trident/5.0;  BingPreview/1.0b)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/
Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)
Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
DomainStatsBot/1.0 (http://domainstats.io/our-bot)
mindUpBot (datenbutler.de)
Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de
Xenu Link Sleuth/1.3.8
Pcore-HTTP/v0.40.3
Pcore-HTTP/v0.44.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot
Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/600.1.3 (KHTML, like Gecko) Version/8.0 Mobile/12A4345d Safari/600.1.4 moatbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +http://kosm.io/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.1 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.5 Chrome/78.0.3904.130 Electron/7.3.15 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) renderer/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; http://www.pingdom.com/)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)
PiplBot (+http://www.pipl.com/bot/)
Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
TelegramBot (like TwitterBot)
Mozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)
Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
Barkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) http://www.exensa.com/crawl - admin@exensa.com -- based on BuBiNG
Barkrowler/0.7 (+http://www.exensa.com/crawl)
BarkRowler/0.7 (+http://www.exensa.com/crawling)
Barkrowler/0.9 (+http://www.exensa.com/crawl)
Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)
TinEye/1.1 (http://tineye.com/crawler.html)
SocialRankIOBot; http://socialrank.io/about
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0
Ocarinabot
Mozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot)
Mozilla/5.0 (compatible; Primalbot; +https://www.primal.com;)
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
@LinkArchiver twitter bot
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
Mozilla/5.0 (compatible; PaperLiBot/2.1; https://support.paper.li/entries/20023257-what-is-paper-li)
Digg Deeper/v1 (http://digg.com/about)
dcrawl/1.0
Mozilla/5.0 (compatible; Snacktory; +https://github.com/karussell/snacktory)
Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)
Fyrebot/1.0
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
Mediatoolkitbot (complaints@mediatoolkit.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0
Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
NING/1.0
okhttp/2.5.0
okhttp/2.7.5
okhttp/3.2.0
okhttp/3.5.0
okhttp/4.1.0
Nuzzel
omgili/0.5 +http://omgili.com
PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
YisouSpider
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)
Jamie's Spider (http://jamiembrown.com/)
AHC/2.0
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)
Apache-HttpClient/4.2.3 (java 1.5)
Apache-HttpClient/4.2.5 (java 1.5)
Apache-HttpClient/4.3.1 (java 1.5)
Apache-HttpClient/4.3.3 (java 1.5)
Apache-HttpClient/4.3.5 (java 1.5)
Apache-HttpClient/4.4.1 (Java/1.8.0_65)
Apache-HttpClient/4.5.2 (Java/1.8.0_65)
Apache-HttpClient/4.5.2 (Java/1.8.0_151)
Apache-HttpClient/4.5.2 (Java/1.8.0_161)
Apache-HttpClient/4.5.2 (Java/1.8.0_181)
Apache-HttpClient/4.5.3 (Java/1.8.0_121)
Apache-HttpClient/4.5.3-SNAPSHOT (Java/1.8.0_152)
Apache-HttpClient/4.5.7 (Java/11.0.3)
Apache-HttpClient/4.5.10 (Java/1.8.0_201)
AppEngine-Google; (+http://code.google.com/appengine; appid: example)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
Jetty/9.3.z-SNAPSHOT
Upflow/1.0
Thinklab (thinklab.com)
Traackr.com
Ruby, Twurly v1.1 (http://twurly.org)
http.rb/2.2.2 (Mastodon/1.5.1; +https://example-masto-instance.org/)
http_get
Mozilla/5.0 (compatible; DnyzBot/1.0)
Mozilla/5.0 (compatible; botify; http://botify.com)
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/11A465 Twitter for iPhone BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
check_http/v2.2.1 (nagios-plugins 2.2.1)
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
EZID (EZID link checker; https://ezid.cdlib.org/)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
ArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)
LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
BLP_bbot/0.1
Mozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot)
Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
Companybook-Crawler (+https://www.companybooknetworking.com/)
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
MeltwaterNews www.meltwater.com
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com)
newspaper/0.1.0.7
newspaper/0.2.5
newspaper/0.2.6
newspaper/0.2.8
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
sentry/8.22.0 (https://sentry.io)
Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
OutclicksBot/2 +https://www.outclicks.net/agent/VjzDygCuk4ubNmg40ZMbFqT0sIh7UfOKk8s8ZMiupUR
OutclicksBot/2 +https://www.outclicks.net/agent/gIYbZ38dfAuhZkrFVl7sJBFOUhOVct6J1SvxgmBZgCe
OutclicksBot/2 +https://www.outclicks.net/agent/PryJzTl8POCRHfvEUlRN5FKtZoWDQOBEvFJ2wh6KH5J
OutclicksBot/2 +https://www.outclicks.net/agent/p2i4sNUh7eylJF1S6SGgRs5mP40ExlYvsr9GBxVQG6h
Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)
Hatena Antenna/0.3
Hatena::Russia::Crawler/0.01
Hatena-Favicon/2 (http://www.hatena.ne.jp/faq/)
Hatena::Scissors/0.01
HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
Hatena::Fetcher/0.01 (master) Furl/3.13
Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
MauiBot (crawler.feedback+wc@gmail.com)
Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)
SBL-BOT (http://sbl.net)
IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)
Mozilla/5.0 (compatible; adscanner/)
Netvibes (crawler/bot; http://www.netvibes.com
Netvibes (crawler; http://www.netvibes.com)
Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
Baidu-YunGuanCe-Bot(ce.baidu.com)
Baidu-YunGuanCe-SLABot(ce.baidu.com)
Baidu-YunGuanCe-ScanBot(ce.baidu.com)
Baidu-YunGuanCe-PerfBot(ce.baidu.com)
Baidu-YunGuanCe-VSBot(ce.baidu.com)
bitlybot/3.0 (+http://bit.ly/)
bitlybot/2.0
bitlybot
blogmuraBot (+http://www.blogmura.com)
Bot.AraTurka.com/0.0.1
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
Mozilla/5.0 (compatible; BoxcarBot/1.1; +awesome@boxcar.io)
BTWebClient/180B(9704)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)
ContextAd Bot 1.0
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.)
Disqus/1.0
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
FeedlyBot/1.0 (http://feedly.com)
Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
g2reader-bot/1.0 (+http://www.g2reader.com/)
G2 Web Services/1.0 (built with StormCrawler Archetype 1.8; https://www.g2webservices.com/; developers@g2llc.com)
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
K7MLWCBot/1.0 (+http://www.k7computing.com)
Kemvibot/1.0 (http://kemvi.com, marco@kemvi.com)
Landau-Media-Spider/1.0(http://bots.landaumedia.de/bot.html)
linkapediabot (+http://www.linkapedia.com)
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost
Mozilla/5.0 (compatible; ZuperlistBot/1.0)
Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.3; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.7; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.10; +https://miniflux.net)
Mozilla/5.0 (compatibl$; Miniflux/2.0.x-dev; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/2.0.11; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/2.0.12; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/ae1dc1a; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/3b6e44c; +https://miniflux.app)
Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)
Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
Mozilla/5.0 zgrab/0.x
Mozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)
AdsTxtCrawler/1.0
Datafeedwatch/2.1.x
Zabbix
TangibleeBot/1.0.0.0 (http://tangiblee.com/bot)
google-xrawler
axios/0.18.0
axios/0.19.0
Amazon CloudFront
Pulsepoint XT3 web scraper
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +https://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: AAAAAAAAAAAAAAAA)
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: AAAAAAAAAAAAAAAA)
Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)
Cloudflare-SSLDetector
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
WordupInfoSearch/1.0
Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)
Jersey/2.25.1 (HttpUrlConnection 1.8.0_141)
ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
VelenPublicWebCrawler (velen.io)
MoodleBot/1.0
jpg-newsbot/2.0; (+https://vipnytt.no/bots/)
Mozilla/5.0 (Java) outbrain
W3C_Validator/1.3
Validator.nu/LV
W3C-checklink/2.90 libwww-perl/5.64
W3C-checklink/3.6.2.3 libwww-perl/5.64
W3C-checklink/4.2 [4.20] libwww-perl/5.803
W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
W3C-checklink/4.3 [4.42] libwww-perl/5.805
W3C-checklink/4.3 [4.42] libwww-perl/5.808
W3C-checklink/4.3 [4.42] libwww-perl/5.820
W3C-checklink/4.5 [4.154] libwww-perl/5.823
W3C-checklink/4.5 [4.160] libwww-perl/5.823
W3C-mobileOK/DDC-1.0
W3C_I18n-Checker/1.0
FeedValidator/1.3
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
W3C_Unicorn/1.0
Mozilla/5.0 (Google-PhysicalWeb)
Blackboard Safeassign
Mozilla/5.0 (compatible; ICBot/0.1; +https://ideasandcode.xyz
Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 1 subscribers)
Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
Mozilla/5.0 (compatible; Rivva; http://rivva.de)
Experibot-v2 http://goo.gl/ZAr8wX
Experibot-v3 http://goo.gl/ZAr8wX
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22 +awesomecrawler
Mozilla/5.0 (compatible; Dataprovider.com)
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/
Mozilla/5.0 (compatible; theoldreader.com)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.24; +http://software.schmorp.de/pkg/AnyEvent)
Uptimebot.org - Free website monitoring
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
2ip.ru CMS Detector (https://2ip.ru/cms/)
Clickagy Intelligence Bot v2
Caliperbot/1.0 (+http://www.conductor.com/caliperbot)
MBCrawler/1.0 (https://monitorbacklinks.com)
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
B2B Bot
Mozilla/5.0 (compatible; AddSearchBot/0.9; +http://www.addsearch.com/bot; info@addsearch.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
Mozilla/5.0 (X11; Linux x86_64; HubSpot Single Page link check; web-crawlers+links@hubspot.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Mozilla/5.0 (compatible; HubSpot Crawler; web-crawlers@hubspot.com)
HubSpot Connect 2.0 (http://dev.hubspot.com/) - BizOpsCompanies-Tq2-BizCoDomainValidationAudit
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4695.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/74.0.3729.169 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0
serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
MixnodeCache/1.8(+https://cache.mixnode.com/)
curl
curl/7.29.0
curl/7.47.0
curl/7.54.0
curl/7.55.1
curl/7.64.0
curl/7.64.1
curl/7.65.3
Mozilla/5.0 (compatible; SimpleScraper)
RSSingBot (http://www.rssing.com)
Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
venus/fedoraplanet
Friendica 'The Tazmans Flax-lily' 2019.01-1293; https://hoyer.xyz
NextCloud-News/1.0
Tiny Tiny RSS/1.15.3 (http://tt-rss.org/)
Tiny Tiny RSS/17.12 (a2d1fa5) (http://tt-rss.org/)
Tiny Tiny RSS/19.2 (b68db2d) (http://tt-rss.org/)
Tiny Tiny RSS/19.8 (http://tt-rss.org/)
Mozilla/5.0 (compatible; RegionStuttgartBot/1.0; +http://it.region-stuttgart.de/competenzatlas/unternehmen-suchen/)
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3754.1902 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4454.1745 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2988.1545 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4141.1682 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.3478.1649 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.5267.1259 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.7990.1979 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2268.1523 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2576.1836 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9681.1227 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.6023.1635 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.4944.1981 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.3613.1739 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4022.1033 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.3248.1547 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.5527.1507 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.5216.1326 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.9038.1080 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
Mozilla/5.0 (compatible; +http://tweetedtimes.com)
Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
Gwene/1.0 (The gwene.org rss-to-news gateway) Googlebot
SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko)
SearchAtlas.com SEO Crawler
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:1162088860
wp.com feedbot/1.0 (+https://wp.com)
UT-Dorkbot/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)
Mozilla/5.0 (compatible; Eyeotabot/1.0; +http://www.eyeota.com)
Mozilla/5.0 (compatible; officestorebot/1.0; +https://aka.ms/officestorebot)
Neticle Crawler v1.0 ( https://neticle.com/bot/en/ )
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X)
Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321
AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)
AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com)
RyteBot/1.0.0 (+https://bot.ryte.com/)
FreeWebMonitoring SiteChecker/0.2 (+https://www.freewebmonitoring.com/bot.html)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)
NAVER Blog Rssbot
Mozilla/5.0 (compatible; zenback bot; powered by logly +http://corp.logly.co.jp/)
SentiBot www.sentibot.eu (compatible with Googlebot)
Mozilla/5.0 (compatible; Domains Project/1.0.3; +https://github.com/tb0hdan/domains)
Pandalytics/1.0 (https://domainsbot.com/pandalytics/)
Mozilla/5.0 (compatible; VKRobot/1.0)
bidswitchbot/1.0
tigerbot
Mozilla/5.0 (compatible; NIXStatsbot/1.1; +http://www.nixstats.com/bot.html)
RSSMicro.com RSS/Atom Feed Robot
Curebot/1.0
curebot-feed-fetcher
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 (compatible; PagePeeker/3.0; +https://pagepeeker.com/robots/)
Mozilla/5.0 (compatible; Vigil/1.0; +http://vigil-app.com/bot.html)
rssbot/1.4.3 (+https://t.me/RustRssBot)
Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot)
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
http://seewithkids.com/bot
NINJA bot
Cutbot; 1.5; http://cutbot.net/
BublupBot (+https://www.bublup.com/bublup-bot.html)
BrandONbot (http://brandonmedia.net)
Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co)
Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321
Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)
Mozilla/5.0 (compatible; Dubbotbot/0.2; +http://dubbot.com)
Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm)
infoobot/0.1 (https://www.infoo.nl/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 (Refindbot/1.0)
Mozilla/5.0 (compatible; BlogTraffic/1.4 Feed-Fetcher; +http://www.blogtraffic.de/rss-bot.html)
SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)
Mozilla/5.0 (compatible; Cincraw/1.0; +http://cincrawdata.net/bot/)
Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0; Dragonbot; http://www.dragonmetrics.com
Mozilla/5.0 (compatible; VoluumDSP-content-bot/2.0; +dsp-dev@codewise.com)
FreshRSS/1.11.2 (Linux; https://freshrss.org) like Googlebot
Mozilla/5.0 (compatible; BitBot/v1.19.0; +https://bitbot.dev)
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.2.24 curl/7.61.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.19 curl/7.66.0
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.23 curl/7.66.0
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.7 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.9 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.10 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.11 curl/7.69.1
Google-Certificates-Bridge
Mozilla/5.0 (compatible; Go-http-client/1.1; +centurybot9@gmail.com)
Viber
e.ventures Investment Crawler (eventures.vc)
Mozilla/5.0 (compatible; evc-batch/2.0)
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 PTST/211202.211915
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0 PTST/211202.211915
Testomatobot/1.0 (Linux x86_64; +https://www.testomato.com/testomatobot) minicrawler/5.2.2
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cookiebot/1.0; +http://cookiebot.com/) Chrome/97.0.4692.71 Safari/537.36
Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)
IonCrawl (https://www.ionos.de/terms-gtc/faq-crawler-en/)
Uptime-Kuma/1.23.16
Uptime-Kuma/1.23.15
Uptime-Kuma/1.23.14
Uptime-Kuma/1.23.13
Uptime-Kuma/1.23.12
Uptime-Kuma/1.23.11
Uptime-Kuma/1.23.10
Uptime-Kuma/1.18.0
Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)
Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
FreshpingBot/1.0 (+https://freshping.io/)
Feedbin feed-id:2005098 - 2 subscribers
CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/)
Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots
Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
RuxitSynthetic/1.0
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
Valve/Steam HTTP Client 1.0 (SteamChatURLLookup)
OdklBot/1.0 (share@odnoklassniki.ru)
Mozilla/5.0 (compatible; OdklBot/1.0 like Linux; klass@odnoklassniki.ru)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0
LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US
Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler)
Mozilla/5.0 (compatible; BitSightBot/1.0)
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1 (compatible; woorankreview/2.0; +https://www.woorank.com/)
Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)
Google-Safety
Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
Mozilla/5.0 (compatible; Linespider/1.1; +https://lin.ee/4dwXkTH)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Linespider/1.1; +https://lin.ee/4dwXkTH) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; WellKnownBot/0.1; +https://well-known.dev/about/#bot)
E. Orliac, G. Fourestey/2.3 (A Patent Crawler; http://scitas.epfl.ch/; etienne.orliac@epfl.ch, gilles.fourestey@epfl.ch)
Mozilla/5.0 (compatible; StractBot/0.1; open source search engine; +https://trystract.com/webmasters)
search.marginalia.nu
YouBot (+http://www.you.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36
Mozilla/5.0 (compatible; Neevabot/1.0; +https://neeva.com/neevabot)
BrightEdge Crawler/1.0 (crawler@brightedge.com)
SiteCheckerBotCrawler/1.0 (+http://sitechecker.pro)
Mozilla/5.0 (compatible; TombaPublicWebCrawler/1.0; +https://tombascraper.com)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 (compatible; CrawlyProjectCrawler/0.1.3; crawlyproject@digitaldragon.dev +https://crawlyproject.digitaldragon.dev/)
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
CISPA Webcrawler (https://vuln-notify-checker.cispa.saarland)
MTRobot/0.2 (Metrics Tools Analytics Crawler; https://metrics-tools.de/robot.html; crawler@metrics-tools.de)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; HyScore/1.0; +https://hyscore.io/crawler/)
Mozilla/5.0 (Linux) (compatible; AlexandriaOrgBot/1.0; +https://www.alexandria.org/bot.html)
2ip bot/1.1 (+http://2ip.io)
Mozilla/5.0 (compatible; Yellowbrandprotectionbot/1.0; +https://www.yellowbp.com/bot.html)
SEOlizer/1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)
Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html)
INETDEX-BOT/1.5 (Mozilla/5.0; https://inetdex.com/bot.html)
Synapse (bot; +https://github.com/matrix-org/synapse)
Mozilla/5.0 (compatible; t3versionsBot/1.0; +https://www.t3versions.com/bot)
deepnoc - https://deepnoc.com/bot
Mozilla/5.0 (compatible; Cocolyzebot/1.0; https://cocolyze.com/bot)
Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/bot)
Mozilla/5.0 (compatible; ReverseEngineeringBot/0.1; +https://torus.company/bot.html)
Mozilla/5.0 (compatible; Semanticbot/1.0; +http://sempi.tech/bot.html)
Iframely/1.3.1 (+https://iframely.com/docs/about) Atlassian
MetaInspector/5.6.0 (+https://github.com/jaimeiniesta/metainspector)
node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
l9explore/1.2.2
lkxscan/v0.1.0 (+https://leakix.net) l9explore/v1.0.0 (+https://github.com/LeakIX/l9explore)
python-opengraph-jaywink/0.2.0 (+https://github.com/jaywink/python-opengraph)
OpenGraphCheck/2.1 (+https://opengraphcheck.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google-PageRenderer Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/
SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)
Mozilla/5.0 (compatible; MaCoCu; +https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/)
NewsBlur Feed Fetcher - 1 subscriber - http://www.newsblur.com/site/0000000/webpage (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15)
Mozilla/5.0 (compatible; inoreader.com; 1 subscribers)
NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com
PageThing http://pagething.com curl www
WordPress/X.X.X; https://example.com
PhxBot/0.1 (phxbot@protonmail.com)
Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)
Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)
BW/1.1; bit.ly/3eZNDnO
BW/1.1; rb.gy/oupwis
Mozilla/5.0 (compatible; GeedoBot; +http://www.geedo.com/bot.html)
Audisto Crawler (mobile; +https://audisto.com/bot)
Audisto Crawler (desktop; +https://audisto.com/bot)
Audisto Crawler (mobile; essential; +https://audisto.com/bot)
Audisto Crawler (desktop; essential; +https://audisto.com/bot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
claudebot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; Monsidobot/2.2; +http://monsido.com/bot.html; info@monsido.com)
GroupMeBot/1.0
Vercelbot (+https://vercel.com)
facebookcatalog/1.0
meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalads/1.1
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalagent/1.1
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalfetcher/1.1
AcademicBotRTU (https://academicbot.rtu.lv; mailto:caps@rtu.lv)
Mozilla/5.0 (compatible; KeybaseBot; +https://keybase.io)
Lemmy/0.19.8; +https://leminal.space
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 CookieHubScan/3.0
Hydrozen.io/1.0
HTTP Banner Detection (https://security.ipip.net)
SummalyBot/5.1.0
MicrosoftPreview/2.0; +https://aka.ms/MicrosoftPreview
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)
Mozilla/5.0 (compatible; OnCrawl/1.0; +http://www.oncrawl.com)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; OnCrawl/1.0; +http://www.oncrawl.com)
got (https://github.com/sindresorhus/got)
Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)
Mozilla/5.0 (compatible; SBIntuitionsBot/0.1; +https://www.sbintuitions.co.jp/bot/)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6439.0 Mobile Safari/537.36 +https://sitebulb.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/87.0.4280.88 YextBot/Java Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.168 Safari/537.36 DatadogSynthetics
Mozilla/5.0 AppleWebKit/537.36 Chrome/139.0.7258.127 Safari/537.36 Google-Ads-Conversions
Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
Mozilla/5.0 (Linux; Android 14; Pixel 8 Pro) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1