Robots.txt

748 technologies in this category

Every Robots.txt technology tracked by Ful.io, ranked by how many websites use each one. Explore the tools in this category and see which platforms lead the market.

LLMS Text and Google BIG Bench Disallow together are used by about 28.36% of all Robots.txt websites Ful.io tracks across 748 technologies. LLMS Text alone runs roughly 19.33% and Google BIG Bench Disallow 9.03%.

476

websites tracked

across 748 Robots.txt technologies

Robots.txt websites by countrymarket share · top 8 countries

🇺🇸United States

22%

🇩🇪Germany

18%

🇧🇷Brazil

15%

🇮🇳India

7.1%

🇫🇷France

6.5%

🇬🇧United Kingdom

5.7%

🇦🇺Australia

4.4%

🇨🇾Cyprus

1.9%

0%6.3%13%19%25%

Robots.txt websites by industrydetected industries

SaaS & Cloud38%

Education25%

Shopping20%

Construction12%

Medical & Healthcare1.7%

Travel1.2%

All Robots.txt technologies

Technology	Websites	Market share
LLMS Text	92	19%	Create report
Google BIG Bench Disallow	43	9%	Create report
Common Crawl Bot Disallow	20	4.2%	Create report
GPTBot Disallow	20	4.2%	Create report
ByteDance Bytespider Disallow	19	4%	Create report
Anthropic Claude Bot Disallow	18	3.8%	Create report
Google-Extended Disallow	16	3.4%	Create report
Meta External Agent Disallow	15	3.2%	Create report
Amazon Bot Disallow	13	2.7%	Create report
Applebot-Extended Disallow	13	2.7%	Create report
CloudflareBrowserRenderingCrawler Disallow	12	2.5%	Create report
Petal Bot Disallow	10	2.1%	Create report

Showing all 12 of 748 technologies