Robots.txt

748 technologies in this category

Every Robots.txt technology tracked by Ful.io, ranked by how many websites use each one. Explore the tools in this category and see which platforms lead the market.

LLMS Text and GPTBot Disallow together are used by about 21.08% of all Robots.txt websites Ful.io tracks across 748 technologies. LLMS Text alone runs roughly 15.20% and GPTBot Disallow 5.89%.

3,823
websites tracked
across 748 Robots.txt technologies
Robots.txt websites by countrymarket share · top 8 countries
🇺🇸United States
18%
🇮🇳India
8%
🇨🇦Canada
5%
🇩🇪Germany
4.2%
🇬🇧United Kingdom
3.5%
🇦🇺Australia
2.7%
🇧🇷Brazil
2.6%
🇸🇦Saudi Arabia
2.4%
0%5%10%15%20%
Robots.txt category sharetop 10 technologies
LLMS Text · 15%GPTBot Disallow · 5.9%ByteDance Bytespider Disallow · 5.8%Common Crawl Bot Disallow · 5.8%Anthropic Claude Bot Disallow · 5.3%Amazon Bot Disallow · 5.2%Meta External Agent Disallow · 5%Google-Extended Disallow · 5%Applebot-Extended Disallow · 4.9%CloudflareBrowserRenderingCrawler Disallow · 4.4%Others (738) · 37%15%LLMS TEXT
LLMS Text15%
GPTBot Disallow5.9%
ByteDance Bytespider Disallow5.8%
Common Crawl Bot Disallow5.8%
Anthropic Claude Bot Disallow5.3%
Amazon Bot Disallow5.2%
Meta External Agent Disallow5%
Google-Extended Disallow5%
Applebot-Extended Disallow4.9%
CloudflareBrowserRenderingCrawler Disallow4.4%
Others (738)37%

All Robots.txt technologies

TechnologyMarket shareWebsites
LLMS Text581
GPTBot Disallow225
ByteDance Bytespider Disallow223
Common Crawl Bot Disallow222
Anthropic Claude Bot Disallow204
Amazon Bot Disallow197
Meta External Agent Disallow193
Google-Extended Disallow192
Applebot-Extended Disallow189
CloudflareBrowserRenderingCrawler Disallow168
Petal Bot Disallow102
MJ12 Bot Disallow61
Semrush Bot Disallow54
DotBot Disallow50
aHrefs Bot Disallow48
Google BIG Bench Disallow43
Anthropic AI Disallow34
Diffbot Disallow32
Cohere AI Disallow31
Nutch-based Bot Disallow30
Robots Disallow30
omgilibot Disallow29
ChatGPT-User Disallow27
Anthropic Claude Web Bot Disallow26
Scrapy Disallow24
Perplexity Bot Disallow24
Yandex Bot Disallow23
Facebook Bot Disallow23
Timpibot Disallow20
Baidu Spider Disallow18
Data For Seo Bot Disallow18
YouBot Disallow18
Gigabot Disallow16
AI2Bot Disallow15
Magpie-Crawler Disallow13
Webzio-Extended Disallow13
Turnitin Bot Disallow12
Anthropic Claude User Bot Disallow12
VelenPublicWebCrawler Disallow12
PanguBot Disallow11
Applebot Disallow10
Rogerbot Disallow10
ZoominfoBot Disallow10
OAI-SearchBot Disallow10
Kangaroo Bot Disallow10
Perplexity-User Disallow10
Showing all 46 of 748 technologies