Robots.txt

748 technologies in this category

Every Robots.txt technology tracked by Ful.io, ranked by how many websites use each one. Explore the tools in this category and see which platforms lead the market.

LLMS Text and Petal Bot Disallow together are used by about 26.76% of all Robots.txt websites Ful.io tracks across 748 technologies. LLMS Text alone runs roughly 18.97% and Petal Bot Disallow 7.80%.

722,875
websites tracked
across 748 Robots.txt technologies
Robots.txt websites by countrymarket share · top 8 countries
🇺🇸United States
31%
🇬🇧United Kingdom
9%
🇩🇪Germany
6%
🇫🇷France
4.5%
🇨🇦Canada
3.7%
🇪🇸Spain
3%
🇦🇺Australia
2.9%
🇳🇱Netherlands
2.6%
0%10%20%30%40%
Robots.txt category sharetop 10 technologies
LLMS Text · 19%Petal Bot Disallow · 7.8%GPTBot Disallow · 3.3%ByteDance Bytespider Disallow · 3.2%MJ12 Bot Disallow · 3.1%Common Crawl Bot Disallow · 3%Amazon Bot Disallow · 2.9%Anthropic Claude Bot Disallow · 2.8%aHrefs Bot Disallow · 2.6%Semrush Bot Disallow · 2.6%Others (738) · 50%19%LLMS TEXT
LLMS Text19%
Petal Bot Disallow7.8%
GPTBot Disallow3.3%
ByteDance Bytespider Disallow3.2%
MJ12 Bot Disallow3.1%
Common Crawl Bot Disallow3%
Amazon Bot Disallow2.9%
Anthropic Claude Bot Disallow2.8%
aHrefs Bot Disallow2.6%
Semrush Bot Disallow2.6%
Others (738)50%

All Robots.txt technologies

TechnologyMarket shareWebsites
LLMS Text137,101
Petal Bot Disallow56,360
GPTBot Disallow24,002
ByteDance Bytespider Disallow23,163
MJ12 Bot Disallow22,339
Common Crawl Bot Disallow21,907
Amazon Bot Disallow21,106
Anthropic Claude Bot Disallow20,348
aHrefs Bot Disallow19,104
Semrush Bot Disallow18,502
Meta External Agent Disallow18,305
Google-Extended Disallow17,511
Applebot-Extended Disallow16,188
DotBot Disallow14,887
CloudflareBrowserRenderingCrawler Disallow10,280
Baidu Spider Disallow10,049
Yandex Bot Disallow9,665
Nutch-based Bot Disallow8,440
Anthropic AI Disallow6,853
Diffbot Disallow5,439
omgilibot Disallow5,369
ChatGPT-User Disallow5,314
Facebook Bot Disallow5,109
Cohere AI Disallow5,063
Data For Seo Bot Disallow4,661
Perplexity Bot Disallow4,543
Anthropic Claude Web Bot Disallow4,500
Barkrowler Disallow4,437
Turnitin Bot Disallow4,315
Robots Disallow4,151
Rogerbot Disallow4,131
ExaBot Disallow3,847
Scrapy Disallow3,485
YouBot Disallow3,477
Timpibot Disallow3,371
EmailWolf Disallow3,209
serpstatbot Disallow3,071
LinkWalker Disallow2,962
OAI-SearchBot Disallow2,944
Seznam Bot Disallow2,868
ZoominfoBot Disallow2,829
Gigabot Disallow2,818
VelenPublicWebCrawler Disallow2,758
Magpie-Crawler Disallow2,727
Microsoft BingBot Disallow2,584
AI2Bot Disallow2,509
Sogou Spider Disallow2,313
Applebot Disallow2,279
Webzio-Extended Disallow2,196
Ai2Bot-Dolma Disallow2,137
img2dataset Disallow2,030
Meta-ExternalFetcher Disallow2,023
Yeti Naverbot Disallow2,011
PanguBot Disallow2,004
GoogleOther Disallow1,916
ICC-Crawler Disallow1,915
Ezooms Disallow1,878
aiHitBot Disallow1,838
Gaisbot Disallow1,818
Soso Spider Disallow1,795
Awario RSS Bot Disallow1,777
FriendlyCrawler Disallow1,712
archive.org bot Disallow1,693
Yahoo! Slurp Disallow1,647
TikTokSpider Disallow1,625
Site Audit Bot Disallow1,618
360Spider Disallow1,607
Cotoyogi Disallow1,548
Zao Disallow1,443
Heritrix Disallow1,406
DuckAssistBot Disallow1,361
Kangaroo Bot Disallow1,360
Survey Bot Disallow1,350
SEOkicks Disallow1,347
Youdao Bot Disallow1,326
Anthropic Claude Search Bot Disallow1,325
Adsbot Disallow1,312
Split Signal Bot Disallow1,273
Perplexity-User Disallow1,272
Discobot Disallow1,252
Trendiction Bot Disallow1,252
Wotbox Disallow1,249
BacklinkCrawler Disallow1,243
Google-CloudVertexBot Disallow1,216
SBIntuitionsBot Disallow1,193
Anthropic Claude User Bot Disallow1,167
BUbiNG Disallow1,150
Crawlspace Disallow1,130
Mail.Ru Bot Disallow1,129
Panscient.com Disallow1,074
Sidetrade indexer bot Disallow1,038
MegaIndex Disallow1,026
Screaming Frog SEO Spider Disallow1,014
oBot Disallow1,006
DomainCrawler Disallow972
MyCentralAIScraperBot Disallow958
Factset_spyderbot Disallow944
ISSCyberRiskCrawler Disallow943
Linkdex Bot Disallow930
Tweetmeme Bot Disallow911
Octopus Disallow909
LTX71 Disallow899
ichiro Disallow899
Cliqzbot Disallow895
LinkpadBot Disallow882
LCC Disallow875
Mojeek Bot Disallow864
Grapeshot Disallow848
ScoutJet Disallow847
Panscient Disallow837
Riddler Disallow834
QualifiedBot Disallow819
PiplBot Disallow812
Brightbot 1.0 Disallow783
GoogleOther-Image Disallow779
GoogleOther-Video Disallow769
Shop Wiki Disallow765
Meanpath Bot Disallow751
Seekport Disallow726
YaK Disallow721
AspiegelBot Disallow718
Aboundexbot Disallow716
SMTBot Disallow715
Ahrefs Site Audit Disallow710
BDCbot Disallow702
FirecrawlAgent Disallow694
Mediatoolkit Bot Disallow679
netEstate Disallow648
Spinn3r Disallow643
Dataprovider Disallow611
Buck Disallow604
Xovi Bot Disallow602
SBIder Disallow594
AddSearchBot Disallow585
Turnitin Disallow580
netEstate Imprint Crawler Disallow576
Bright Bot Disallow559
Amazon-Kendra Disallow552
SiteSucker Disallow544
Sputnik Bot Disallow539
DomainStats Bot Disallow534
Orange Bot Disallow526
Datenbank Crawler Disallow523
Speedy Disallow521
Vagabondo Disallow517
WARDBot Disallow514
Netvibes Disallow512
ADMantX Disallow506
GeedoProductSearch Disallow503
CensysInspect Disallow492
EchoboxBot Disallow492
bedrockbot Disallow491
The Knowledge AI Disallow486
Twitterbot Disallow480
Robozilla Disallow478
Abonti Disallow475
ZumBot Disallow454
Adscanner Disallow451
Screener Bot Disallow446
UptimeRobot Disallow440
SISTRIX Crawler Disallow439
Easou Spider Disallow428
Domains Project Disallow428
Zgrab Disallow423
Masscan Disallow422
Bravebot Disallow418
Neevabot Disallow414
Google-NotebookLM Disallow412
Swiftbot Disallow411
BuiltWith Disallow408
Zoom Bot Disallow407
Googlebot Disallow404
Coc Coc Bot Disallow403
Scooter Disallow399
MetaJobBot Disallow394
IPS Agent Disallow385
Nmap Disallow382
BoardReader Disallow375
OpenLinkProfiler Disallow374
Siteimprove Disallow362
Quora Bot Disallow359
Pandalytics Disallow359
Awario Disallow356
Cocolyzebot Disallow348
Domain Appender Disallow348
evc-batch Disallow339
BDFetch Disallow338
NetTrack Disallow337
TraceMyFile Disallow333
CATExplorador Disallow331
Who.is Bot Disallow324
Arquivo.pt Disallow323
SiteChecker BotCrawler Disallow314
WikiDo Disallow312
MTRobot Disallow306
Pinterest Disallow304
BitlyBot Disallow300
Seobility Disallow297
TinEye Disallow295
Blekkobot Disallow285
CheckMark Network Disallow283
Geedo Bot Disallow282
DuckDuckBot Disallow279
HubSpot Disallow279
LinkedIn Bot Disallow275
DomCop Bot Disallow274
Ask Jeeves Disallow272
YaCy Disallow267
Seoscanners.net Disallow239
Nimbostratus Bot Disallow224
SurdotlyBot Disallow220
PaperLiBot Disallow217
Openindex Spider Disallow212
Mixrank Bot Disallow209
Internet Measurement Disallow200
Safe DNS Bot Disallow198
AdsTxtCrawlerTP Disallow192
Amazon AdBot Disallow189
Research Scan Disallow174
CareerBot Disallow166
CriteoBot Disallow157
Plukkie Disallow156
Jobboerse Bot Disallow156
FreeWebMonitoring Disallow155
Backlinks Extended Bot Disallow154
AddThis.com Disallow145
SemrushBot-SWA Disallow141
Bloglines Disallow138
Semantic Scholar Bot Disallow131
W3C CSS Validator Disallow128
Expanse Disallow128
Domain Re-Animator Bot Disallow124
CrystalSemantics Bot Disallow124
ContentKing Disallow110
Zookabot Disallow108
Linespider Disallow105
Gigablast Disallow103
Googlebot News Disallow101
Sitebulb Disallow101
Cincraw Disallow94
MuckRack Disallow94
Synapse Disallow92
Paqlebot Disallow91
Telegram Bot Disallow90
Slack Bot Disallow88
RavenCrawler Disallow88
Qwantbot Disallow86
Charlotte Disallow86
Detectify Disallow83
Botify Disallow82
Daum Disallow81
Feedly Disallow81
Nice Crawler Disallow81
BrandVerity Disallow80
SpiderLing Disallow77
Acoon Disallow76
WeSEE:Search Disallow76
UniversalFeedParser Disallow69
Findxbot Disallow64
NetResearchServer Disallow62
Grammarly Disallow59
ArchiveBot Disallow58
Monsidobot Disallow57
SEOENG Bot Disallow56
WellKnownBot Disallow54
Discord Bot Disallow53
Dazoobot Disallow52
MetaInspector Disallow52
CloudServerMarket Spider Disallow52
Feedspot Disallow50
l9explore Disallow50
2ip Disallow49
PagePeeker Disallow45
Simple Pie Disallow44
Dubbotbot Disallow43
Backlink-Check.de Disallow41
Feedbin Disallow40
FeedBurner Disallow40
AdsTxtCrawler Disallow40
OWLer Disallow40
Newslitbot Disallow39
Bountii Bot Disallow36
Gluten Free Crawler Disallow35
LeakIX Disallow35
Blackbox Exporter Disallow34
Serendeputy Bot Disallow34
nbertaupete95 Disallow33
Pompos Disallow33
SabsimBot Disallow33
KStandBot Disallow33
Flipboard Disallow31
Wappalyzer Disallow31
Embedly Disallow31
WooRank Disallow31
BrightEdge Disallow31
SiteScore Bot Disallow31
Outbrain Disallow30
Clickagy Disallow29
URL Append Bot Disallow27
Taboolabot Disallow26
Weborama Disallow24
Cookiebot Disallow24
MeltwaterNews Disallow24
InfoTigerBot Disallow23
NetSystemsResearch Disallow22
Inetdex Bot Disallow22
Webwiki Disallow22
EyeMonit Disallow22
MakeMerryBot Disallow22
Lycos Disallow21
leak.info Disallow21
IDG Disallow20
TestCrawler Disallow20
NodePing Disallow19
Apache Bench Disallow18
MuscatFerret Disallow18
Google BIG Bench Disallow17
BLEXBot Crawler Disallow16
Datanyze Disallow16
Pingdom Bot Disallow16
WebDataStats Disallow16
datagnionbot Disallow16
Birdcrawlerbot Disallow16
ImageSift Disallow16
Odin Disallow16
Browsershots Disallow15
Gowikibot Disallow15
Cloudflare Health Checks Disallow14
Better Uptime Bot Disallow14
Yahoo! Japan Disallow14
KomodiaBot Disallow14
Blogtrottr Disallow13
Gobuster Disallow13
SISTRIX Optimizer Disallow13
Adbeat Disallow13
TigerBot Disallow13
CloudFlare Always Online Disallow12
Google Favicon Disallow12
DuplexWeb-Google Disallow12
Skype URI Preview Disallow12
Sprinklr Disallow12
CyberFind Crawler Disallow12
Ruxit Synthetic Disallow12
Cloudflare SSL Detector Disallow11
Cloudflare Custom Hostname Verification Disallow11
Cloudflare Traffic Manager Disallow11
Nuzzel Disallow11
Twingly Recon Disallow11
VK Robot Disallow11
GTmetrix Disallow11
WhatCMS Disallow11
deepnoc Disallow11
Iframely Disallow11
Uptime-Kuma Disallow11
sfFeedReader Disallow11
HuggingFaceBot Disallow11
Crawlson Disallow10
Microsoft Preview Disallow10
Showing all 358 of 748 technologies