Common Crawl Bot Disallow tracking and usage
Websites using Common Crawl Bot Disallow by countrymarket share
10%
🇺🇸US8.1%
🇮🇳IN4.1%
🇨🇦CA4.1%
🇦🇺AU3.2%
🇬🇧GB2.7%
🇩🇪DE2.7%
🇵🇰PK2.3%
🇮🇩IDWebsites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
SaaS & Cloud
Technology & Computing
Shopping
Hardware & Electronics
News & Weather
Education
0%7.5%15%23%30%
Download report
We track 222 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 23 websites using Common Crawl Bot Disallow in United States.
Filter by country
🇺🇸 United States23 sites
🇮🇳 India18 sites
🇨🇦 Canada9 sites
🇦🇺 Australia9 sites
🇬🇧 United Kingdom7 sites
🇩🇪 Germany6 sites
🇵🇰 Pakistan6 sites
🇮🇩 Indonesia5 sites
🇸🇦 Saudi Arabia5 sites
🇻🇳 Viet Nam4 sites
🇵🇱 Poland3 sites
🇷🇴 Romania3 sites
🇧🇷 Brazil3 sites
🇰🇿 Kazakhstan2 sites
🇲🇩 Moldova, Republic of2 sites
🇯🇵 Japan2 sites
🇹🇭 Thailand2 sites
🇩🇰 Denmark2 sites
🇧🇲 Bermuda1 sites
🇰🇪 Kenya1 sites
🇸🇬 Singapore1 sites
🇧🇪 Belgium1 sites
🇵🇭 Philippines1 sites
🇹🇷 Turkey1 sites
🇳🇬 Nigeria1 sites
🇦🇫 Afghanistan1 sites
🇮🇸 Iceland1 sites
🇵🇪 Peru1 sites
🇲🇸 Montserrat1 sites
🇲🇳 Mongolia1 sites
🇳🇱 Netherlands1 sites
🇫🇷 France1 sites
🇬🇷 Greece1 sites
🇦🇹 Austria1 sites
🇨🇳 China1 sites
🇰🇷 Korea, Republic of1 sites
🇲🇦 Morocco1 sites
🇨🇿 Czechia1 sites
🇮🇷 Iran, Islamic Republic of1 sites
🇨🇾 Cyprus1 sites
🇧🇩 Bangladesh1 sites
🇪🇪 Estonia1 sites
About Common Crawl Bot Disallow
This website has found the Disallow rules for Common Crawl Bot.
Category
Official website
commoncrawl.org/faq↗
Websites using Common Crawl Bot Disallow
showing 25 of 222| Website | Country | Traffic | Industry |
|---|---|---|---|
| 🇬🇷GR | Top 5K | Technology & Computing | |
| 🇦🇺AU | Top 10K | Telecommunications | |
| 🇩🇪DE | Top 50K | News & Weather | |
| 🇮🇳IN | Top 50K | Education | |
| 🇺🇸US | Top 50K | Shopping | |
| 🇸🇦SA | Top 50K | Shopping | |
| 🇦🇹AT | Top 500K | Hardware & Electronics | |
| 🇸🇦SA | Top 500K | Technology & Computing | |
| 🇪🇪EE | Top 500K | Technology & Computing | |
| 🇸🇦SA | Top 500K | Technology & Computing | |
| 🇺🇸US | Top 1M | Society | |
| 🇺🇸US | Top 1M | - | |
| 🇵🇰PK | Top 1M | Business & Finance | |
| 🇨🇾CY | Top 1M | Education | |
| 🇵🇱PL | Top 1M | Automotive | |
| 🇩🇪DE | - | - | |
| 🇸🇬SG | - | SaaS & Cloud | |
| 🇧🇲BM | - | Insurance | |
| 🇺🇸US | - | Technology & Computing | |
| 🇺🇸US | - | Entertainment | |
| 🇮🇳IN | - | Shopping | |
| 🇦🇺AU | - | Design & Creative | |
| - | - | Technology & Computing | |
| 🇦🇺AU | - | Automotive | |
| 🇳🇱NL | - | Design & Creative |
Showing 25 of 222 websitesView detailed list →
