Robots.txt

748 technologies in this category

Every Robots.txt technology tracked by Ful.io, ranked by how many websites use each one. Explore the tools in this category and see which platforms lead the market.

LLMS Text and Petal Bot Disallow together are used by about 30.10% of all Robots.txt websites Ful.io tracks across 748 technologies. LLMS Text alone runs roughly 20.22% and Petal Bot Disallow 9.88%.

2,665,355
websites tracked
across 748 Robots.txt technologies
Robots.txt websites by countrymarket share · top 8 countries
🇺🇸United States
42%
🇩🇪Germany
9.4%
🇬🇧United Kingdom
4.9%
🇫🇷France
2.8%
🇳🇱Netherlands
2.4%
🇨🇦Canada
2.4%
🇦🇺Australia
2.3%
🇪🇸Spain
2%
0%13%25%38%50%
Robots.txt category sharetop 10 technologies
LLMS Text · 20%Petal Bot Disallow · 9.9%MJ12 Bot Disallow · 3.1%aHrefs Bot Disallow · 2.7%Semrush Bot Disallow · 2.5%GPTBot Disallow · 2.3%DotBot Disallow · 2.3%Common Crawl Bot Disallow · 2.2%ByteDance Bytespider Disallow · 2.2%Amazon Bot Disallow · 1.9%Others (738) · 51%20%LLMS TEXT
LLMS Text20%
Petal Bot Disallow9.9%
MJ12 Bot Disallow3.1%
aHrefs Bot Disallow2.7%
Semrush Bot Disallow2.5%
GPTBot Disallow2.3%
DotBot Disallow2.3%
Common Crawl Bot Disallow2.2%
ByteDance Bytespider Disallow2.2%
Amazon Bot Disallow1.9%
Others (738)51%

All Robots.txt technologies

TechnologyMarket shareWebsites
LLMS Text539,027
Petal Bot Disallow263,336
MJ12 Bot Disallow83,199
aHrefs Bot Disallow72,697
Semrush Bot Disallow66,891
GPTBot Disallow62,347
DotBot Disallow60,108
Common Crawl Bot Disallow59,939
ByteDance Bytespider Disallow58,163
Amazon Bot Disallow51,138
Anthropic Claude Bot Disallow47,704
Meta External Agent Disallow41,900
Google-Extended Disallow41,233
Baidu Spider Disallow37,537
Yandex Bot Disallow35,493
Applebot-Extended Disallow34,836
Anthropic AI Disallow26,695
Nutch-based Bot Disallow24,824
Rogerbot Disallow24,372
omgilibot Disallow22,533
Facebook Bot Disallow20,492
Cohere AI Disallow20,474
ChatGPT-User Disallow19,347
Robots Disallow19,294
Diffbot Disallow18,953
Barkrowler Disallow18,191
Turnitin Bot Disallow17,589
Data For Seo Bot Disallow16,833
Perplexity Bot Disallow16,574
Anthropic Claude Web Bot Disallow16,243
ExaBot Disallow15,968
YouBot Disallow14,493
Timpibot Disallow12,729
serpstatbot Disallow12,184
Scrapy Disallow12,080
Seznam Bot Disallow11,780
ZoominfoBot Disallow11,674
CloudflareBrowserRenderingCrawler Disallow11,530
EmailWolf Disallow11,498
AI2Bot Disallow11,118
VelenPublicWebCrawler Disallow11,094
Gigabot Disallow10,959
Magpie-Crawler Disallow10,701
LinkWalker Disallow10,514
OAI-SearchBot Disallow10,242
aiHitBot Disallow10,129
Ai2Bot-Dolma Disallow9,790
GoogleOther Disallow9,713
img2dataset Disallow9,627
ICC-Crawler Disallow9,537
Webzio-Extended Disallow8,851
Microsoft BingBot Disallow8,566
Awario RSS Bot Disallow8,375
Site Audit Bot Disallow8,223
Sogou Spider Disallow8,205
Soso Spider Disallow7,807
Meta-ExternalFetcher Disallow7,639
Applebot Disallow7,636
TikTokSpider Disallow7,557
Ezooms Disallow7,330
FriendlyCrawler Disallow7,084
PanguBot Disallow7,022
Yeti Naverbot Disallow7,021
DuckAssistBot Disallow6,795
archive.org bot Disallow6,790
Survey Bot Disallow6,625
Yahoo! Slurp Disallow6,505
Split Signal Bot Disallow6,425
Cotoyogi Disallow6,407
Mail.Ru Bot Disallow6,291
Google-CloudVertexBot Disallow6,185
Trendiction Bot Disallow6,179
Gaisbot Disallow6,084
BacklinkCrawler Disallow6,063
SEOkicks Disallow6,032
360Spider Disallow5,846
Wotbox Disallow5,811
Heritrix Disallow5,696
Perplexity-User Disallow5,524
Anthropic Claude Search Bot Disallow5,456
Panscient Disallow5,379
BUbiNG Disallow5,306
Adsbot Disallow5,300
Youdao Bot Disallow5,249
Discobot Disallow5,209
Kangaroo Bot Disallow5,045
ScoutJet Disallow4,972
LinkpadBot Disallow4,833
GoogleOther-Image Disallow4,791
MyCentralAIScraperBot Disallow4,784
Spinn3r Disallow4,770
Anthropic Claude User Bot Disallow4,758
SBIntuitionsBot Disallow4,749
GoogleOther-Video Disallow4,690
Linkdex Bot Disallow4,476
LTX71 Disallow4,466
QualifiedBot Disallow4,293
oBot Disallow4,268
Cliqzbot Disallow4,226
Sidetrade indexer bot Disallow4,068
Octopus Disallow4,042
Meanpath Bot Disallow4,033
Panscient.com Disallow4,023
Screaming Frog SEO Spider Disallow4,004
Tweetmeme Bot Disallow3,942
Crawlspace Disallow3,901
DomainStats Bot Disallow3,853
DomainCrawler Disallow3,752
FirecrawlAgent Disallow3,708
Ahrefs Site Audit Disallow3,681
Zao Disallow3,646
ISSCyberRiskCrawler Disallow3,575
PiplBot Disallow3,540
MegaIndex Disallow3,516
SMTBot Disallow3,458
Aboundexbot Disallow3,452
Factset_spyderbot Disallow3,418
ichiro Disallow3,285
Turnitin Disallow3,267
Grapeshot Disallow3,250
Dataprovider Disallow3,247
Riddler Disallow3,235
Mediatoolkit Bot Disallow3,191
LCC Disallow3,106
YaK Disallow3,027
BDCbot Disallow3,015
WARDBot Disallow3,011
Vagabondo Disallow2,971
Shop Wiki Disallow2,921
Sputnik Bot Disallow2,906
Orange Bot Disallow2,812
AddSearchBot Disallow2,776
SiteSucker Disallow2,750
ADMantX Disallow2,747
Netvibes Disallow2,746
Brightbot 1.0 Disallow2,726
ZumBot Disallow2,687
Abonti Disallow2,669
Adscanner Disallow2,664
Quora Bot Disallow2,620
SBIder Disallow2,620
Screener Bot Disallow2,549
OpenLinkProfiler Disallow2,547
netEstate Imprint Crawler Disallow2,530
Zgrab Disallow2,526
Swiftbot Disallow2,503
AspiegelBot Disallow2,483
Datenbank Crawler Disallow2,444
Masscan Disallow2,420
BuiltWith Disallow2,395
IPS Agent Disallow2,390
UptimeRobot Disallow2,366
bedrockbot Disallow2,364
Nmap Disallow2,360
EchoboxBot Disallow2,358
Bright Bot Disallow2,348
Domain Appender Disallow2,345
BoardReader Disallow2,344
Mojeek Bot Disallow2,342
Buck Disallow2,332
Neevabot Disallow2,288
BDFetch Disallow2,263
evc-batch Disallow2,259
NetTrack Disallow2,191
CATExplorador Disallow2,183
TraceMyFile Disallow2,157
Google-NotebookLM Disallow2,155
Siteimprove Disallow2,154
Seekport Disallow2,138
Who.is Bot Disallow2,138
Seoscanners.net Disallow2,135
Pandalytics Disallow2,107
The Knowledge AI Disallow2,051
CensysInspect Disallow2,031
Bravebot Disallow2,022
Domains Project Disallow1,992
netEstate Disallow1,952
Amazon-Kendra Disallow1,937
Zoom Bot Disallow1,905
Arquivo.pt Disallow1,895
Xovi Bot Disallow1,723
SurdotlyBot Disallow1,722
CareerBot Disallow1,703
Jobboerse Bot Disallow1,681
Cocolyzebot Disallow1,658
SiteChecker BotCrawler Disallow1,636
Easou Spider Disallow1,620
Twitterbot Disallow1,606
Googlebot Disallow1,557
Domain Re-Animator Bot Disallow1,555
Coc Coc Bot Disallow1,534
Awario Disallow1,524
CheckMark Network Disallow1,511
MTRobot Disallow1,473
Seobility Disallow1,472
DomCop Bot Disallow1,393
Robozilla Disallow1,390
TinEye Disallow1,325
WikiDo Disallow1,323
Speedy Disallow1,305
SISTRIX Crawler Disallow1,290
SpiderLing Disallow1,290
Blekkobot Disallow1,284
GeedoProductSearch Disallow1,284
Acoon Disallow1,176
DuckDuckBot Disallow1,146
MetaJobBot Disallow1,111
Safe DNS Bot Disallow1,101
Ask Jeeves Disallow1,097
Gigablast Disallow1,018
Scooter Disallow1,018
Synapse Disallow892
BitlyBot Disallow858
Pinterest Disallow838
YaCy Disallow834
AdsTxtCrawlerTP Disallow824
Plukkie Disallow802
PaperLiBot Disallow764
Openindex Spider Disallow720
Internet Measurement Disallow712
HubSpot Disallow611
Geedo Bot Disallow598
LinkedIn Bot Disallow589
Mixrank Bot Disallow589
Semantic Scholar Bot Disallow579
Backlinks Extended Bot Disallow550
Nimbostratus Bot Disallow549
Bloglines Disallow542
Botify Disallow498
CriteoBot Disallow472
SemrushBot-SWA Disallow466
Amazon AdBot Disallow461
NetResearchServer Disallow432
CrystalSemantics Bot Disallow409
Charlotte Disallow396
Research Scan Disallow386
Daum Disallow378
CloudServerMarket Spider Disallow375
Paqlebot Disallow359
Expanse Disallow351
Findxbot Disallow338
RavenCrawler Disallow330
Zookabot Disallow325
Cincraw Disallow325
AddThis.com Disallow310
ContentKing Disallow310
Googlebot News Disallow277
Linespider Disallow269
Nice Crawler Disallow262
FreeWebMonitoring Disallow261
Qwantbot Disallow258
PagePeeker Disallow253
Sitebulb Disallow251
Telegram Bot Disallow249
Slack Bot Disallow247
Gluten Free Crawler Disallow242
Cookiebot Disallow241
Feedly Disallow239
LeakIX Disallow231
Webwiki Disallow228
Pompos Disallow227
WeSEE:Search Disallow220
SabsimBot Disallow220
BrandVerity Disallow213
UniversalFeedParser Disallow204
W3C CSS Validator Disallow200
Simple Pie Disallow199
MuckRack Disallow195
Grammarly Disallow192
Discord Bot Disallow189
Detectify Disallow189
SEOENG Bot Disallow188
ArchiveBot Disallow183
l9explore Disallow181
Flipboard Disallow179
Feedspot Disallow164
URL Append Bot Disallow163
Embedly Disallow154
Backlink-Check.de Disallow147
WellKnownBot Disallow145
Dazoobot Disallow144
OWLer Disallow139
Monsidobot Disallow139
MeltwaterNews Disallow125
AdsTxtCrawler Disallow124
MetaInspector Disallow123
IDG Disallow123
Feedbin Disallow120
Odin Disallow116
Bountii Bot Disallow109
Serendeputy Bot Disallow109
datagnionbot Disallow108
KStandBot Disallow108
TigerBot Disallow96
Clickagy Disallow94
FeedBurner Disallow92
Dubbotbot Disallow91
Inetdex Bot Disallow90
2ip Disallow89
KomodiaBot Disallow87
Newslitbot Disallow86
Outbrain Disallow85
Datanyze Disallow83
eCairn-Grabber Disallow83
BrightEdge Disallow83
Taboolabot Disallow81
Blackbox Exporter Disallow79
NetSystemsResearch Disallow77
TweetedTimes Bot Disallow75
InfoTigerBot Disallow75
nbertaupete95 Disallow74
MuscatFerret Disallow73
VORTEX Disallow72
Lycos Disallow66
ImageSift Disallow65
TestCrawler Disallow62
Wappalyzer Disallow61
Yourls Disallow61
Gowikibot Disallow60
BLEXBot Crawler Disallow57
WooRank Disallow56
SiteScore Bot Disallow55
Apache Bench Disallow51
Microsoft Preview Disallow51
Weborama Disallow48
CloudFlare Always Online Disallow46
NetLyzer FastProbe Disallow46
MakeMerryBot Disallow46
Browsershots Disallow45
WebDataStats Disallow45
inoreader Disallow44
LumtelBot Disallow43
Yahoo! Japan Disallow41
Iframely Disallow41
EMail Exractor Disallow40
Skype URI Preview Disallow40
Konturbot Disallow40
Cloudflare Health Checks Disallow39
OpenWebSpider Disallow39
Pingdom Bot Disallow39
uMBot Disallow39
Better Uptime Bot Disallow38
Cloudflare Traffic Manager Disallow38
Gobuster Disallow38
MagpieRSS Disallow37
Sprinklr Disallow37
Birdcrawlerbot Disallow37
Crawlson Disallow37
COMODO DCV Disallow37
Cloudflare SSL Detector Disallow35
Google Store Bot Disallow35
StatusCake Disallow35
HuggingFaceBot Disallow35
Cloudflare Custom Hostname Verification Disallow34
Google Favicon Disallow33
NodePing Disallow33
Reddit Bot Disallow33
GTmetrix Disallow33
Fever Disallow32
Twingly Recon Disallow32
MaCoCu Disallow32
Accompany Bot Disallow32
EyeMonit Disallow32
WebMon Disallow32
Alexa Site Audit Disallow31
Fresh RSS Disallow31
Nuzzel Disallow31
leak.info Disallow31
Google Docs Disallow30
colly Disallow30
Adbeat Disallow28
deepnoc Disallow28
Disqus Disallow28
Twurly Disallow28
SISTRIX Optimizer Disallow27
VK Robot Disallow27
WhatCMS Disallow27
Statista Disallow27
MicroAdBot Disallow26
Prerender Disallow26
DuplexWeb-Google Disallow25
vuhuv Bot Disallow25
Blogtrottr Disallow24
CyberFind Crawler Disallow24
sfFeedReader Disallow24
Bloglovin Disallow22
Snap URL Preview Service Disallow22
Comscore Disallow22
Ruxit Synthetic Disallow22
W3C I18N Checker Disallow20
Yahoo! Link Preview Disallow20
Inktomi Slurp Disallow20
ModatScanner Disallow20
Superfeedr Bot Disallow19
Hatena Bookmark Disallow19
Uptime-Kuma Disallow19
Arachni Disallow18
Tiny Tiny RSS Disallow18
PageThing Disallow18
httpx Disallow18
DepSpid Disallow18
NLCrawler Disallow17
WeViKaBot Disallow16
Google BIG Bench Disallow15
Google-Safety Disallow15
Quora Link Preview Disallow15
WebCEO Disallow15
NewsGator Disallow14
VeryHip Disallow14
Morningscore Bot Disallow14
NewsBlur Disallow13
SeolytBot Disallow13
Google Apps Script Disallow13
semaltbot Disallow13
Ghost Inspector Disallow12
PRTG Network Monitor Disallow11
parse.ly Disallow11
SiteOne Crawler Disallow11
Peer39 Disallow11
Google PageSpeed Insights Disallow10
Google Structured Data Testing Tool Disallow10
Catchpoint Disallow10
Think Chaos Disallow10
Montastic Monitor Disallow10
Senuto Disallow10
Oh Dear Disallow10
Showing all 426 of 748 technologies