Common Crawl Bot Disallow

Common Crawl Bot Disallow tracking and usage

Websites using Common Crawl Bot Disallow by countrymarket share
10%
🇺🇸US
8.1%
🇮🇳IN
4.1%
🇨🇦CA
4.1%
🇦🇺AU
3.2%
🇬🇧GB
2.7%
🇩🇪DE
2.7%
🇵🇰PK
2.3%
🇮🇩ID
Websites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
SaaS & Cloud
27%
Technology & Computing
21%
Shopping
19%
Hardware & Electronics
13%
News & Weather
10%
Education
10%
0%7.5%15%23%30%
Download report

We track 222 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 23 websites using Common Crawl Bot Disallow in United States.

Filter by country
🇺🇸 United States23 sites
🇮🇳 India18 sites
🇨🇦 Canada9 sites
🇦🇺 Australia9 sites
🇬🇧 United Kingdom7 sites
🇩🇪 Germany6 sites
🇵🇰 Pakistan6 sites
🇮🇩 Indonesia5 sites
🇸🇦 Saudi Arabia5 sites
🇻🇳 Viet Nam4 sites
🇵🇱 Poland3 sites
🇷🇴 Romania3 sites
🇧🇷 Brazil3 sites
🇰🇿 Kazakhstan2 sites
🇲🇩 Moldova, Republic of2 sites
🇯🇵 Japan2 sites
🇹🇭 Thailand2 sites
🇩🇰 Denmark2 sites
🇧🇲 Bermuda1 sites
🇰🇪 Kenya1 sites
🇸🇬 Singapore1 sites
🇧🇪 Belgium1 sites
🇵🇭 Philippines1 sites
🇹🇷 Turkey1 sites
🇳🇬 Nigeria1 sites
🇦🇫 Afghanistan1 sites
🇮🇸 Iceland1 sites
🇵🇪 Peru1 sites
🇲🇸 Montserrat1 sites
🇲🇳 Mongolia1 sites
🇳🇱 Netherlands1 sites
🇫🇷 France1 sites
🇬🇷 Greece1 sites
🇦🇹 Austria1 sites
🇨🇳 China1 sites
🇰🇷 Korea, Republic of1 sites
🇲🇦 Morocco1 sites
🇨🇿 Czechia1 sites
🇮🇷 Iran, Islamic Republic of1 sites
🇨🇾 Cyprus1 sites
🇧🇩 Bangladesh1 sites
🇪🇪 Estonia1 sites

About Common Crawl Bot Disallow

This website has found the Disallow rules for Common Crawl Bot.

Categories
Robots.txtWeb Standards
Official websitecommoncrawl.org/faq↗

Websites using Common Crawl Bot Disallow

showing 25 of 222
WebsiteCountryTrafficIndustry
tumblr.com🇺🇸USTop 500Technology & Computing
globo.com🇧🇷BRTop 500Technology & Computing
eventbrite.com🇺🇸USTop 500Software Development
amazon.de🇩🇪DETop 500Logistics & Shipping
bbc.co.uk🇬🇧GBTop 500News & Weather
theguardian.com🇬🇧GBTop 500Print & Publishing
bbc.com🇬🇧GBTop 500News & Weather
soundcloud.com🇩🇪DETop 500Entertainment
nytimes.com🇺🇸USTop 500News & Weather
cnn.com🇺🇸USTop 500News & Weather
cookielaw.org🇺🇸USTop 500Software Development
forter.com🇺🇸USTop 500SaaS & Cloud
t-online.de🇩🇪DETop 500News & Weather
weather.com🇺🇸USTop 500Movies & TV
npr.org🇺🇸USTop 500Movies & TV
launchpad.net🇬🇧GBTop 500Mental Health
behance.net🇺🇸USTop 500Technology & Computing
businessinsider.com🇺🇸USTop 500News & Weather
calendly.com🇺🇸USTop 500Software Development
nature.com🇬🇧GBTop 500Print & Publishing
wired.com🇺🇸USTop 1KTechnology & Computing
sagepub.com🇺🇸USTop 1KPrint & Publishing
bild.de🇩🇪DETop 1KNews & Weather
amazon.ca🇨🇦CATop 1KLogistics & Shipping
theverge.com🇺🇸USTop 1KNews & Weather
Showing 25 of 222 websitesView detailed list →

Alternatives

same category
HTML5 DocType43.32MsitesFacebook Redirect16.79MsitesInstagram10.91MsitesOrganization Schema6.74MsitesLinkedIn5.78MsitesEvents Page4.13MsitesMembers Page3.56MsitesGoogle AJAX Feed API3.06Msites