Top 165 Web Crawlers (Total 256) |
|
Name
|
Hits
|
1. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
4212998 |
2. |
Web Crawler - Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler) |
388697 |
3. |
Web Crawler - Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) |
235887 |
4. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
89869 |
5. |
Web Crawler - Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) |
84503 |
6. |
Web Crawler - meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
77212 |
7. |
Web Crawler - Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) |
57403 |
8. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) |
25933 |
9. |
Web Crawler - Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/) |
24699 |
10. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
21479 |
11. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ |
19725 |
12. |
Web Crawler - netEstate NE Crawler (+http://www.website-datenbank.de/) |
14797 |
13. |
Web Crawler - msnbot/1.1 (+http://search.msn.com/msnbot.htm) |
12779 |
14. |
Web Crawler - Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) |
12142 |
15. |
Web Crawler - MauiBot (crawler.feedback+wc@gmail.com) |
8886 |
16. |
Web Crawler - Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) |
7933 |
17. |
Web Crawler - DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.com) |
7753 |
18. |
Web Crawler - netEstate NE Crawler (+http://www.sengine.info/) |
7435 |
19. |
Web Crawler - msnbot/1.0 (+http://search.msn.com/msnbot.htm) |
6558 |
20. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
5718 |
21. |
Web Crawler - Aboundex/0.3 (http://www.aboundex.com/crawler/) |
5323 |
22. |
Web Crawler - Mozilla/5.0 (compatible; fr-crawler/1.1) |
5099 |
23. |
Web Crawler - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
4941 |
24. |
Web Crawler - msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) |
3351 |
25. |
Web Crawler - TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html) |
3156 |
26. |
Web Crawler - TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) |
2967 |
27. |
Web Crawler - Mozilla/5.0 (compatible; ca-crawler/1.0) |
2025 |
28. |
Web Crawler - Aboundex/0.2 (http://www.aboundex.com/crawler/) |
1885 |
29. |
Web Crawler - Googlebot-Image/1.0 |
1470 |
30. |
Web Crawler - ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) |
1234 |
31. |
Web Crawler - spiderman |
953 |
32. |
Web Crawler - msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
739 |
33. |
Web Crawler - Cityreview Robot (+http://www.cityreview.org/crawler/) |
656 |
34. |
Web Crawler - CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.com |
594 |
35. |
Web Crawler - ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com) |
542 |
36. |
Web Crawler - msnbot/0.01 (+http://search.msn.com/msnbot.htm) |
530 |
37. |
Web Crawler - Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/) |
496 |
38. |
Web Crawler - MauiBot (crawler.feedback+dc@gmail.com) |
464 |
39. |
Web Crawler - WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
463 |
40. |
Web Crawler - Barkrowler experimental crawler - 0.4.7 |
433 |
41. |
Web Crawler - Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) |
400 |
42. |
Web Crawler - ia_archiver |
392 |
43. |
Web Crawler - WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
361 |
44. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/) |
350 |
45. |
Web Crawler - NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) |
348 |
46. |
Web Crawler - crawler (crawler.feedback@gmail.com) |
313 |
47. |
Web Crawler - Virusdie crawler/3.0 |
252 |
48. |
Web Crawler - WebAlta Crawler/1.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
245 |
49. |
Web Crawler - LSSRocketCrawler/1.0 LightspeedSystems |
231 |
50. |
Web Crawler - DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html) |
177 |
51. |
Web Crawler - Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/) |
163 |
52. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) |
148 |
53. |
Web Crawler - crawler4j (http://code.google.com/p/crawler4j/) |
137 |
54. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io) |
126 |
55. |
Web Crawler - Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) |
114 |
56. |
Web Crawler - Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com) |
109 |
57. |
Web Crawler - MBCrawler/1.0 (https://monitorbacklinks.com) |
98 |
58. |
Web Crawler - Googlebot/2.1 (+http://www.googlebot.com/bot.html) |
86 |
59. |
Web Crawler - Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler) |
72 |
60. |
Web Crawler - Googlebot/2.1 (+http://www.google.com/bot.html) |
61 |
61. |
Web Crawler - renlifangbot/1.0 (+http://search.msn.com/msnbot.htm) |
57 |
62. |
Web Crawler - kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) |
57 |
63. |
Web Crawler - librabot/1.0 (+http://search.msn.com/msnbot.htm) |
52 |
64. |
Web Crawler - CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com |
48 |
65. |
Web Crawler - User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
46 |
66. |
Web Crawler - DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.org |
46 |
67. |
Web Crawler - Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) |
38 |
68. |
Web Crawler - WebTarantula.com Crawler |
32 |
69. |
Web Crawler - Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4 |
30 |
70. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) |
30 |
71. |
Web Crawler - Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com) |
26 |
72. |
Web Crawler - BacklinkCrawler (http://www.backlinktest.com/crawler.html) |
25 |
73. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) |
25 |
74. |
Web Crawler - GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com) |
25 |
75. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
24 |
76. |
Web Crawler - ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm) |
23 |
77. |
Web Crawler - CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com |
20 |
78. |
Web Crawler - www.integromedb.org/Crawler |
20 |
79. |
Web Crawler - CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com |
19 |
80. |
Web Crawler - VelenPublicWebCrawler (velen.io) |
19 |
81. |
Web Crawler - Mozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) |
18 |
82. |
Web Crawler - MSR-ISRCCrawler |
18 |
83. |
Web Crawler - CRAZYWEBCRAWLER 0.9.9, http://www.crazywebcrawler.com |
18 |
84. |
Web Crawler - Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) |
17 |
85. |
Web Crawler - Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/) |
17 |
86. |
Web Crawler - AdnormCrawler www.adnorm.com/crawler |
16 |
87. |
Web Crawler - Googlebot (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
16 |
88. |
Web Crawler - GoogleBot 1.0 |
15 |
89. |
Web Crawler - Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. |
15 |
90. |
Web Crawler - TurnitinBot (https://turnitin.com/robot/crawlerinfo.html) |
15 |
91. |
Web Crawler - CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com |
14 |
92. |
Web Crawler - W3C_Validator/1.654 |
13 |
93. |
Web Crawler - EmailWolf 1.00 |
12 |
94. |
Web Crawler - msnbot/0.11 ( http://search.msn.com/msnbot.htm) |
12 |
95. |
Web Crawler - Googlebot/2.1 ( http://www.googlebot.com/bot.html) |
12 |
96. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 |
11 |
97. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.org) |
11 |
98. |
Web Crawler - GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com) |
11 |
99. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
11 |
100. |
Web Crawler - DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.net |
11 |
101. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.net) |
10 |
102. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.com) |
10 |
103. |
Web Crawler - Friendly_Crawler/Nutch-1.20-SNAPSHOT |
10 |
104. |
Web Crawler - Googlebot-News |
9 |
105. |
Web Crawler - WebZIP/3.5 (http://www.spidersoft.com) |
9 |
106. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) |
9 |
107. |
Web Crawler - DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.com |
9 |
108. |
Web Crawler - A .NET Web Crawler |
9 |
109. |
Web Crawler - msnbot/2.0b |
9 |
110. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) |
9 |
111. |
Web Crawler - Mozilla/5.0 (compatible; TestCrawler) |
9 |
112. |
Web Crawler - Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) |
8 |
113. |
Web Crawler - Comodo-Webinspector-Crawler 2.1 |
8 |
114. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com) |
8 |
115. |
Web Crawler - WebCrawler |
8 |
116. |
Web Crawler - Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; ) |
8 |
117. |
Web Crawler - msnbot/1.1 ( http://search.msn.com/msnbot.htm) |
8 |
118. |
Web Crawler - msnbot/1.0 ( http://search.msn.com/msnbot.htm) |
8 |
119. |
Web Crawler - Googlebot-Video/1.0 |
7 |
120. |
Web Crawler - Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) |
7 |
121. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 |
7 |
122. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/forum/index.php) |
6 |
123. |
Web Crawler - Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) |
6 |
124. |
Web Crawler - ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com) |
6 |
125. |
Web Crawler - blogtop.us crawler - http://blogtop.us/ |
6 |
126. |
Web Crawler - yyqnnlrcewrmaitmsimi |
6 |
127. |
Web Crawler - Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io) |
6 |
128. |
Web Crawler - Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) |
6 |
129. |
Web Crawler - msnbot-media/1.1 ( http://search.msn.com/msnbot.htm) |
6 |
130. |
Web Crawler - FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler) |
6 |
131. |
Web Crawler - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) |
6 |
132. |
Web Crawler - http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html |
5 |
133. |
Web Crawler - CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) |
5 |
134. |
Web Crawler - Python-urllib/2.7 (+http://example.com/crawler.html) |
5 |
135. |
Web Crawler - SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) |
5 |
136. |
Web Crawler - Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 |
5 |
137. |
Web Crawler - Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) |
5 |
138. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) |
5 |
139. |
Web Crawler - Attributor/Nutch-1.0-dev (Test crawler; http://www.attributor.com; info at attributor com) |
4 |
140. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler) |
4 |
141. |
Web Crawler - DomainCrawler/1.0 |
4 |
142. |
Web Crawler - abnmwnanoifid5hkxn hdmwrmaDDirls5ngpnk |
4 |
143. |
Web Crawler - Mozilla/5.0 (compatible; NOT Googlebot/2.1; +http://www.google.com/bot.html) |
4 |
144. |
Web Crawler - SheenBot/SheenBot-1.0.2 (Sheen web crawler) |
4 |
145. |
Web Crawler - SheenBot/SheenBot-1.0.1 (Sheen web crawler) |
4 |
146. |
Web Crawler - UXCrawlerBot |
4 |
147. |
Web Crawler - Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) |
4 |
148. |
Web Crawler - Jigsaw/2.2.5 W3C_CSS_Validator_JFouffa/2.0 |
4 |
149. |
Web Crawler - Quick-Crawler (+https://www.scrapinghub.com/) |
4 |
150. |
Web Crawler - UCMore Crawler App |
3 |
151. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html) |
3 |
152. |
Web Crawler - Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot) |
3 |
153. |
Web Crawler - Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com) |
3 |
154. |
Web Crawler - Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) |
3 |
155. |
Web Crawler - HouxouCrawler/Nutch-0.9 (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.houxou.com/crawler; crawler at houxou dot com) |
3 |
156. |
Web Crawler - DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org) |
3 |
157. |
Web Crawler - Jayde Crawler. http://www.jayde.com |
3 |
158. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) |
3 |
159. |
Web Crawler - ahvemnkucouiqrmntldg9ylrmauqxl |
3 |
160. |
Web Crawler - Jayde crawler |
3 |
161. |
Web Crawler - CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) |
3 |
162. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com) |
3 |
163. |
Web Crawler - W3C_Validator/1.3 http://validator.w3.org/services |
3 |
164. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html |
3 |
165. |
Web Crawler - SheenBot/SheenBot-1.0.3 (Sheen web crawler) |
3 |