Common Crawl Bot Disallow tracking and usage
Websites using Common Crawl Bot Disallow by countrymarket share
38%
🇺🇸US5.9%
🇩🇪DE5%
🇬🇧GB2.9%
🇫🇷FR2.6%
🇨🇦CA2.4%
🇳🇱NL2.1%
🇦🇺AU1.9%
🇪🇸ESWebsites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
Shopping
News & Weather
Technology & Computing
SaaS & Cloud
Education
Religion & Spirituality
0%6.3%13%19%25%
Download report
We track 52,661 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 20,212 websites using Common Crawl Bot Disallow in United States.
Filter by country
🇺🇸 United States20,212 sites
🇩🇪 Germany3,085 sites
🇬🇧 United Kingdom2,649 sites
🇫🇷 France1,535 sites
🇨🇦 Canada1,374 sites
🇳🇱 Netherlands1,276 sites
🇦🇺 Australia1,091 sites
🇪🇸 Spain1,017 sites
🇮🇳 India944 sites
🇧🇷 Brazil766 sites
🇮🇹 Italy680 sites
🇵🇱 Poland655 sites
🇨🇭 Switzerland625 sites
🇸🇪 Sweden493 sites
🇮🇩 Indonesia371 sites
🇩🇰 Denmark324 sites
🇳🇴 Norway318 sites
🇧🇪 Belgium279 sites
🇹🇷 Turkey274 sites
🇫🇮 Finland235 sites
🇷🇺 Russian Federation225 sites
🇨🇿 Czechia223 sites
🇯🇵 Japan213 sites
🇦🇹 Austria199 sites
🇻🇳 Viet Nam197 sites
🇿🇦 South Africa197 sites
🇦🇷 Argentina196 sites
🇬🇷 Greece177 sites
🇳🇿 New Zealand171 sites
🇨🇳 China165 sites
🇵🇹 Portugal156 sites
🇮🇪 Ireland150 sites
🇲🇽 Mexico150 sites
🇨🇱 Chile142 sites
🇰🇷 Korea, Republic of131 sites
🇦🇪 United Arab Emirates130 sites
🇸🇬 Singapore130 sites
🇷🇴 Romania127 sites
🇺🇦 Ukraine120 sites
🇮🇱 Israel113 sites
🇭🇺 Hungary100 sites
🇨🇴 Colombia97 sites
🇵🇰 Pakistan91 sites
🇲🇾 Malaysia79 sites
🇹🇭 Thailand73 sites
🇳🇬 Nigeria70 sites
🇧🇩 Bangladesh69 sites
🇱🇹 Lithuania65 sites
🇵🇪 Peru65 sites
🇸🇦 Saudi Arabia63 sites
🇭🇰 Hong Kong62 sites
🇸🇰 Slovakia59 sites
🇪🇬 Egypt55 sites
🇧🇬 Bulgaria54 sites
🇹🇼 Taiwan, Province of China54 sites
🇰🇿 Kazakhstan48 sites
🇮🇷 Iran, Islamic Republic of47 sites
🇪🇪 Estonia45 sites
🇵🇭 Philippines43 sites
🇸🇮 Slovenia42 sites
🇨🇾 Cyprus41 sites
🇭🇷 Croatia39 sites
🇷🇸 Serbia34 sites
🇱🇻 Latvia32 sites
🇲🇦 Morocco27 sites
🇪🇨 Ecuador24 sites
🇰🇪 Kenya24 sites
🇷🇼 Rwanda24 sites
🇧🇾 Belarus23 sites
🇮🇸 Iceland23 sites
🇱🇰 Sri Lanka23 sites
🇲🇹 Malta20 sites
🇦🇿 Azerbaijan19 sites
🇨🇷 Costa Rica19 sites
🇱🇺 Luxembourg19 sites
🇹🇳 Tunisia19 sites
🇧🇦 Bosnia and Herzegovina16 sites
🇻🇪 Venezuela, Bolivarian Republic of16 sites
🇩🇴 Dominican Republic15 sites
🇬🇭 Ghana14 sites
🇵🇷 Puerto Rico13 sites
🇱🇧 Lebanon13 sites
🇺🇬 Uganda13 sites
🇬🇹 Guatemala13 sites
🇲🇰 North Macedonia12 sites
🇺🇾 Uruguay12 sites
🇺🇿 Uzbekistan12 sites
🇦🇲 Armenia12 sites
🇩🇿 Algeria11 sites
🇯🇴 Jordan11 sites
🇲🇩 Moldova, Republic of11 sites
🇵🇦 Panama11 sites
🇸🇳 Senegal11 sites
🇶🇦 Qatar11 sites
🇻🇨 Saint Vincent and the Grenadines11 sites
🇸🇻 El Salvador10 sites
🇦🇱 Albania9 sites
🇪🇹 Ethiopia9 sites
🇲🇨 Monaco9 sites
🇬🇪 Georgia8 sites
🇳🇦 Namibia8 sites
🇳🇵 Nepal8 sites
🇴🇲 Oman8 sites
🇨🇮 Côte d'Ivoire8 sites
🇹🇿 Tanzania, United Republic of8 sites
🇨🇲 Cameroon7 sites
🇵🇾 Paraguay7 sites
🇧🇸 Bahamas6 sites
🇲🇺 Mauritius6 sites
🇲🇬 Madagascar6 sites
🇦🇬 Antigua and Barbuda5 sites
🇰🇬 Kyrgyzstan5 sites
🇱🇮 Liechtenstein5 sites
🇯🇲 Jamaica5 sites
🇧🇴 Bolivia, Plurinational State of5 sites
🇦🇩 Andorra4 sites
🇧🇯 Benin4 sites
🇭🇹 Haiti4 sites
🇰🇭 Cambodia4 sites
🇰🇼 Kuwait4 sites
🇲🇪 Montenegro4 sites
🇹🇯 Tajikistan4 sites
🇭🇳 Honduras4 sites
🇧🇿 Belize4 sites
🇧🇲 Bermuda4 sites
🇨🇽 Christmas Island4 sites
🇮🇶 Iraq4 sites
🇻🇬 Virgin Islands, British4 sites
🇰🇾 Cayman Islands4 sites
🇦🇸 American Samoa3 sites
🇬🇦 Gabon3 sites
🇸🇴 Somalia3 sites
🇹🇹 Trinidad and Tobago3 sites
🇨🇼 Curaçao3 sites
🇱🇾 Libya3 sites
🇦🇼 Aruba2 sites
🇧🇭 Bahrain2 sites
🇲🇲 Myanmar2 sites
🇽🇰 Kosovo2 sites
🇵🇫 French Polynesia2 sites
🇬🇾 Guyana2 sites
🇾🇹 Mayotte2 sites
🇬🇬 Guernsey2 sites
🇬🇩 Grenada2 sites
🇯🇪 Jersey2 sites
🇲🇱 Mali2 sites
🇵🇲 Saint Pierre and Miquelon2 sites
🇸🇱 Sierra Leone2 sites
🇧🇼 Botswana2 sites
🇸🇷 Suriname2 sites
🇷🇪 Réunion2 sites
🇲🇵 Northern Mariana Islands2 sites
🇹🇬 Togo2 sites
🇻🇮 Virgin Islands, U.S.2 sites
🇲🇳 Mongolia2 sites
🇲🇶 Martinique1 sites
🇳🇨 New Caledonia1 sites
🇸🇾 Syrian Arab Republic1 sites
🇹🇩 Chad1 sites
🇻🇦 Holy See (Vatican City State)1 sites
🇧🇳 Brunei Darussalam1 sites
🇬🇱 Greenland1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇸🇨 Seychelles1 sites
🇿🇼 Zimbabwe1 sites
🇱🇷 Liberia1 sites
🇲🇴 Macao1 sites
🇫🇯 Fiji1 sites
🇲🇻 Maldives1 sites
🇻🇺 Vanuatu1 sites
🇰🇵 Korea, Democratic People's Republic of1 sites
🇬🇸 South Georgia and the South Sandwich Islands1 sites
🇸🇹 Sao Tome and Principe1 sites
🇧🇮 Burundi1 sites
🇵🇬 Papua New Guinea1 sites
🇮🇴 British Indian Ocean Territory1 sites
🇱🇨 Saint Lucia1 sites
🇩🇲 Dominica1 sites
🇸🇲 San Marino1 sites
🇬🇲 Gambia1 sites
🇮🇲 Isle of Man1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇦🇫 Afghanistan1 sites
🇨🇻 Cabo Verde1 sites
🇨🇺 Cuba1 sites
🇬🇫 French Guiana1 sites
🇬🇺 Guam1 sites
🇲🇸 Montserrat1 sites
🇰🇲 Comoros1 sites
🇬🇵 Guadeloupe1 sites
🇲🇭 Marshall Islands1 sites
About Common Crawl Bot Disallow
This website has found the Disallow rules for Common Crawl Bot.
Categories
Official website
commoncrawl.org/faq↗
Websites using Common Crawl Bot Disallow
showing 25 of 52,661| Website | Country | Traffic | Industry |
|---|---|---|---|
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇧🇷BR | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | Software Development | |
 amazon.de | 🇩🇪DE | Top 500 | Logistics & Shipping |
| 🇬🇧GB | Top 500 | News & Weather | |
| 🇬🇧GB | Top 500 | Print & Publishing | |
| 🇬🇧GB | Top 500 | News & Weather | |
| 🇩🇪DE | Top 500 | Entertainment | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Software Development | |
| 🇺🇸US | Top 500 | SaaS & Cloud | |
| 🇩🇪DE | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Movies & TV | |
| 🇺🇸US | Top 500 | Movies & TV | |
| 🇬🇧GB | Top 500 | Mental Health | |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Software Development | |
| 🇬🇧GB | Top 500 | Print & Publishing | |
| 🇺🇸US | Top 1K | Technology & Computing | |
| 🇺🇸US | Top 1K | Print & Publishing | |
| 🇩🇪DE | Top 1K | News & Weather | |
 amazon.ca | 🇨🇦CA | Top 1K | Logistics & Shipping |
| 🇺🇸US | Top 1K | News & Weather |
Showing 25 of 52,661 websitesView detailed list →
