This website has found the Disallow rules for Common Crawl Bot.
222
websites using Common Crawl Bot Disallow
Create a technology report →Export the list of all websites using Common Crawl Bot Disallow with their listed metadata
Websites using Common Crawl Bot Disallow by countrymarket share · top 8 countries
🇺🇸United States
🇮🇳India
🇨🇦Canada
🇦🇺Australia
🇬🇧United Kingdom
🇩🇪Germany
🇵🇰Pakistan
🇮🇩Indonesia
0%3%6%9%12%
Websites using Common Crawl Bot Disallow by industrydetected industries
SaaS & Cloud27%
Technology & Computing21%
Shopping19%
Hardware & Electronics13%
News & Weather10%
Education10%
Websites using Common Crawl Bot Disallow
showing 25 of 222| Website | Country | Industry | Size | Founded | |
|---|---|---|---|---|---|
| 🇬🇷GR | Technology & Computing | 51-200 | 2007 | ||
| 🇦🇺AU | Telecommunications | 10001+ | 1901 | ||
| 🇩🇪DE | News & Weather | 1001-5000 | 1945 | ||
| 🇮🇳IN | Education | 11-50 | 2017 | ||
| 🇺🇸US | Shopping | 10001+ | 1983 | ||
| 🇸🇦SA | Shopping | 1001-5000 | 1979 | ||
| 🇦🇹AT | Hardware & Electronics | 5001-10000 | 1950 | ||
| 🇸🇦SA | Technology & Computing | 51-200 | 2016 | ||
| 🇪🇪EE | Technology & Computing | 1-10 | - | ||
| 🇸🇦SA | Technology & Computing | 11-50 | 2017 | ||
| 🇺🇸US | Society | 10001+ | 1926 | ||
| 🇺🇸US | - | 1-10 | - | ||
| 🇵🇰PK | Business & Finance | 1001-5000 | 2005 | ||
| 🇨🇾CY | Education | 201-500 | 2005 | ||
| 🇵🇱PL | Automotive | 1-10 | 2018 | ||
| 🇩🇪DE | - | 1-10 | - | ||
| 🇸🇬SG | SaaS & Cloud | 1-10 | - | ||
| 🇧🇲BM | Insurance | 10001+ | - | ||
| 🇺🇸US | Technology & Computing | 10001+ | 1988 | ||
| 🇺🇸US | Entertainment | 1001-5000 | - | ||
| 🇮🇳IN | Shopping | 51-200 | 2000 | ||
| 🇦🇺AU | Design & Creative | 11-50 | 1981 | ||
| - | Technology & Computing | 1-10 | - | ||
| 🇦🇺AU | Automotive | 201-500 | 1955 | ||
| 🇳🇱NL | Design & Creative | 1-10 | 2017 |
Showing 25 of 222 websitesGet the full list →