Common Crawl Bot Disallow tracking and usage
Websites using Common Crawl Bot Disallow by countrymarket share
35%
🇺🇸US6.4%
🇩🇪DE5.6%
🇬🇧GB3.1%
🇫🇷FR2.8%
🇳🇱NL2.7%
🇨🇦CA2.2%
🇦🇺AU2%
🇪🇸ESWebsites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
Shopping
News & Weather
Technology & Computing
SaaS & Cloud
Education
Religion & Spirituality
0%6.3%13%19%25%
Download report
We track 59,939 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 21,062 websites using Common Crawl Bot Disallow in United States.
Filter by country
🇺🇸 United States21,062 sites
🇩🇪 Germany3,846 sites
🇬🇧 United Kingdom3,336 sites
🇫🇷 France1,884 sites
🇳🇱 Netherlands1,667 sites
🇨🇦 Canada1,621 sites
🇦🇺 Australia1,320 sites
🇪🇸 Spain1,189 sites
🇮🇳 India1,105 sites
🇵🇱 Poland946 sites
🇮🇹 Italy916 sites
🇧🇷 Brazil857 sites
🇨🇭 Switzerland838 sites
🇸🇪 Sweden642 sites
🇩🇰 Denmark571 sites
🇷🇺 Russian Federation437 sites
🇳🇴 Norway411 sites
🇮🇩 Indonesia394 sites
🇨🇿 Czechia390 sites
🇯🇵 Japan379 sites
🇧🇪 Belgium337 sites
🇫🇮 Finland337 sites
🇹🇷 Turkey285 sites
🇦🇹 Austria269 sites
🇬🇷 Greece266 sites
🇿🇦 South Africa240 sites
🇻🇳 Viet Nam228 sites
🇦🇷 Argentina215 sites
🇺🇦 Ukraine210 sites
🇳🇿 New Zealand209 sites
🇰🇷 Korea, Republic of206 sites
🇵🇹 Portugal188 sites
🇨🇳 China187 sites
🇮🇪 Ireland180 sites
🇲🇽 Mexico174 sites
🇨🇱 Chile162 sites
🇷🇴 Romania157 sites
🇦🇪 United Arab Emirates145 sites
🇭🇺 Hungary143 sites
🇸🇬 Singapore141 sites
🇮🇱 Israel122 sites
🇰🇿 Kazakhstan115 sites
🇵🇰 Pakistan107 sites
🇨🇴 Colombia99 sites
🇲🇾 Malaysia95 sites
🇹🇭 Thailand84 sites
🇸🇰 Slovakia79 sites
🇱🇹 Lithuania78 sites
🇳🇬 Nigeria74 sites
🇭🇰 Hong Kong72 sites
🇧🇩 Bangladesh71 sites
🇵🇪 Peru70 sites
🇹🇼 Taiwan, Province of China70 sites
🇸🇦 Saudi Arabia67 sites
🇧🇬 Bulgaria65 sites
🇵🇭 Philippines61 sites
🇪🇪 Estonia60 sites
🇪🇬 Egypt59 sites
🇸🇮 Slovenia56 sites
🇮🇷 Iran, Islamic Republic of51 sites
🇭🇷 Croatia47 sites
🇨🇾 Cyprus45 sites
🇷🇼 Rwanda44 sites
🇷🇸 Serbia40 sites
🇧🇾 Belarus38 sites
🇱🇻 Latvia38 sites
🇮🇸 Iceland29 sites
🇰🇪 Kenya28 sites
🇲🇦 Morocco27 sites
🇪🇨 Ecuador25 sites
🇱🇺 Luxembourg25 sites
🇱🇰 Sri Lanka23 sites
🇦🇿 Azerbaijan21 sites
🇨🇷 Costa Rica21 sites
🇲🇹 Malta21 sites
🇹🇳 Tunisia19 sites
🇻🇪 Venezuela, Bolivarian Republic of19 sites
🇩🇴 Dominican Republic18 sites
🇧🇦 Bosnia and Herzegovina17 sites
🇬🇭 Ghana16 sites
🇬🇹 Guatemala16 sites
🇻🇨 Saint Vincent and the Grenadines16 sites
🇦🇱 Albania13 sites
🇲🇰 North Macedonia13 sites
🇵🇷 Puerto Rico13 sites
🇺🇾 Uruguay13 sites
🇺🇿 Uzbekistan13 sites
🇱🇧 Lebanon13 sites
🇺🇬 Uganda13 sites
🇶🇦 Qatar13 sites
🇯🇴 Jordan12 sites
🇲🇩 Moldova, Republic of12 sites
🇵🇦 Panama12 sites
🇦🇲 Armenia12 sites
🇸🇳 Senegal12 sites
🇬🇩 Grenada12 sites
🇩🇿 Algeria11 sites
🇸🇻 El Salvador11 sites
🇬🇪 Georgia10 sites
🇲🇨 Monaco10 sites
🇳🇵 Nepal10 sites
🇴🇲 Oman10 sites
🇸🇽 Sint Maarten (Dutch part)10 sites
🇨🇲 Cameroon9 sites
🇪🇹 Ethiopia9 sites
🇸🇴 Somalia9 sites
🇱🇮 Liechtenstein9 sites
🇻🇬 Virgin Islands, British9 sites
🇳🇦 Namibia8 sites
🇨🇮 Côte d'Ivoire8 sites
🇰🇬 Kyrgyzstan8 sites
🇹🇿 Tanzania, United Republic of8 sites
🇯🇲 Jamaica8 sites
🇵🇾 Paraguay7 sites
🇧🇴 Bolivia, Plurinational State of7 sites
🇦🇸 American Samoa6 sites
🇧🇸 Bahamas6 sites
🇦🇬 Antigua and Barbuda6 sites
🇲🇺 Mauritius6 sites
🇧🇿 Belize6 sites
🇲🇬 Madagascar6 sites
🇦🇩 Andorra5 sites
🇭🇹 Haiti5 sites
🇨🇽 Christmas Island5 sites
🇰🇾 Cayman Islands5 sites
🇷🇪 Réunion5 sites
🇧🇯 Benin4 sites
🇬🇦 Gabon4 sites
🇰🇭 Cambodia4 sites
🇰🇼 Kuwait4 sites
🇲🇪 Montenegro4 sites
🇹🇯 Tajikistan4 sites
🇭🇳 Honduras4 sites
🇸🇨 Seychelles4 sites
🇧🇲 Bermuda4 sites
🇮🇶 Iraq4 sites
🇧🇭 Bahrain3 sites
🇹🇹 Trinidad and Tobago3 sites
🇨🇼 Curaçao3 sites
🇱🇾 Libya3 sites
🇬🇸 South Georgia and the South Sandwich Islands3 sites
🇸🇷 Suriname3 sites
🇲🇳 Mongolia3 sites
🇦🇽 Åland Islands3 sites
🇵🇼 Palau3 sites
🇦🇼 Aruba2 sites
🇲🇲 Myanmar2 sites
🇸🇾 Syrian Arab Republic2 sites
🇽🇰 Kosovo2 sites
🇵🇫 French Polynesia2 sites
🇬🇾 Guyana2 sites
🇾🇹 Mayotte2 sites
🇿🇼 Zimbabwe2 sites
🇬🇬 Guernsey2 sites
🇯🇪 Jersey2 sites
🇲🇱 Mali2 sites
🇫🇯 Fiji2 sites
🇵🇲 Saint Pierre and Miquelon2 sites
🇻🇺 Vanuatu2 sites
🇸🇱 Sierra Leone2 sites
🇧🇼 Botswana2 sites
🇲🇵 Northern Mariana Islands2 sites
🇹🇬 Togo2 sites
🇻🇮 Virgin Islands, U.S.2 sites
🇮🇲 Isle of Man2 sites
🇰🇳 Saint Kitts and Nevis2 sites
🇲🇸 Montserrat2 sites
🇲🇶 Martinique1 sites
🇳🇨 New Caledonia1 sites
🇹🇩 Chad1 sites
🇻🇦 Holy See (Vatican City State)1 sites
🇧🇳 Brunei Darussalam1 sites
🇬🇱 Greenland1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇱🇷 Liberia1 sites
🇨🇩 Congo, The Democratic Republic of the1 sites
🇲🇴 Macao1 sites
🇿🇲 Zambia1 sites
🇲🇻 Maldives1 sites
🇰🇵 Korea, Democratic People's Republic of1 sites
🇸🇹 Sao Tome and Principe1 sites
🇧🇮 Burundi1 sites
🇵🇬 Papua New Guinea1 sites
🇮🇴 British Indian Ocean Territory1 sites
🇱🇨 Saint Lucia1 sites
🇩🇲 Dominica1 sites
🇸🇲 San Marino1 sites
🇬🇲 Gambia1 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇦🇫 Afghanistan1 sites
🇨🇻 Cabo Verde1 sites
🇨🇺 Cuba1 sites
🇬🇫 French Guiana1 sites
🇬🇺 Guam1 sites
🇰🇲 Comoros1 sites
🇩🇯 Djibouti1 sites
🇬🇵 Guadeloupe1 sites
🇱🇸 Lesotho1 sites
🇲🇭 Marshall Islands1 sites
🇲🇿 Mozambique1 sites
🇵🇸 Palestine, State of1 sites
🇸🇧 Solomon Islands1 sites
🇼🇸 Samoa1 sites
🇲🇼 Malawi1 sites
🇸🇩 Sudan1 sites
About Common Crawl Bot Disallow
This website has found the Disallow rules for Common Crawl Bot.
Categories
Official website
commoncrawl.org/faq↗
Websites using Common Crawl Bot Disallow
showing 25 of 59,939| Website | Country | Traffic | Industry |
|---|---|---|---|
 amazon.co.jp | 🇯🇵JP | Top 500 | Shopping |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇩🇪DE | Top 500 | Print & Publishing | |
| 🇮🇳IN | Top 500 | Religion & Spirituality | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | Print & Publishing | |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Music | |
| 🇫🇷FR | Top 500 | News & Weather | |
| 🇫🇷FR | Top 500 | Law & Government | |
 amazon.co.uk | 🇬🇧GB | Top 500 | Entertainment |
| 🇺🇸US | Top 500 | Print & Publishing | |
 amazon.fr | 🇫🇷FR | Top 500 | Logistics & Shipping |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇧🇷BR | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | Software Development | |
 amazon.de | 🇩🇪DE | Top 500 | Logistics & Shipping |
| 🇬🇧GB | Top 500 | News & Weather |
Showing 25 of 59,939 websitesView detailed list →
