Robots.txt

748 technologies in this category

Every Robots.txt technology tracked by Ful.io, ranked by how many websites use each one. Explore the tools in this category and see which platforms lead the market.

LLMS Text and GPTBot Disallow together are used by about 22.08% of all Robots.txt websites Ful.io tracks across 748 technologies. LLMS Text alone runs roughly 15.93% and GPTBot Disallow 6.15%.

3,057

websites tracked

across 748 Robots.txt technologies

Robots.txt websites by countrymarket share · top 8 countries

🇺🇸United States

15%

🇮🇳India

8.9%

🇩🇪Germany

4.2%

🇬🇧United Kingdom

3.9%

🇦🇺Australia

3.4%

🇧🇷Brazil

3.3%

🇨🇦Canada

2.8%

🇸🇦Saudi Arabia

2.7%

0%5%10%15%20%

Robots.txt websites by industrydetected industries

SaaS & Cloud10%

Shopping9.7%

Technology & Computing9.5%

Education7.9%

Wholesale & Distribution7.2%

Medical & Healthcare5.8%

Software Development5.3%

News & Weather4.2%

Other industries40%

All Robots.txt technologies

Technology	Websites	Market share
LLMS Text	487	16%	Create report
GPTBot Disallow	188	6.1%	Create report
ByteDance Bytespider Disallow	184	6%	Create report
Common Crawl Bot Disallow	182	6%	Create report
Anthropic Claude Bot Disallow	171	5.6%	Create report
Google-Extended Disallow	163	5.3%	Create report
Amazon Bot Disallow	162	5.3%	Create report
Meta External Agent Disallow	160	5.2%	Create report
Applebot-Extended Disallow	156	5.1%	Create report
CloudflareBrowserRenderingCrawler Disallow	144	4.7%	Create report
Petal Bot Disallow	84	2.7%	Create report
MJ12 Bot Disallow	50	1.6%	Create report
Google BIG Bench Disallow	43	1.4%	Create report
Semrush Bot Disallow	41	1.3%	Create report
DotBot Disallow	40	1.3%	Create report
aHrefs Bot Disallow	37	1.2%	Create report
Robots Disallow	27	0.9%	Create report
Diffbot Disallow	23	0.8%	Create report
Anthropic AI Disallow	23	0.8%	Create report
Cohere AI Disallow	21	0.7%	Create report
omgilibot Disallow	19	0.6%	Create report
ChatGPT-User Disallow	18	0.6%	Create report
Perplexity Bot Disallow	18	0.6%	Create report
Yandex Bot Disallow	17	0.6%	Create report
Scrapy Disallow	17	0.6%	Create report
Nutch-based Bot Disallow	17	0.6%	Create report
Anthropic Claude Web Bot Disallow	16	0.5%	Create report
Baidu Spider Disallow	15	0.5%	Create report
Facebook Bot Disallow	15	0.5%	Create report
Data For Seo Bot Disallow	15	0.5%	Create report
Gigabot Disallow	14	0.5%	Create report
Timpibot Disallow	13	0.4%	Create report
Turnitin Bot Disallow	11	0.4%	Create report
Magpie-Crawler Disallow	10	0.3%	Create report
Rogerbot Disallow	10	0.3%	Create report
YouBot Disallow	10	0.3%	Create report

Showing all 36 of 748 technologies