Common Crawl Bot Disallow tracking and usage
Websites using Common Crawl Bot Disallow by countrymarket share
29%
🇺🇸US7.4%
🇬🇧GB4.2%
🇩🇪DE3.8%
🇫🇷FR3.5%
🇨🇦CA3.1%
🇮🇳IN2.7%
🇦🇺AU2.7%
🇳🇱NLWebsites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
Shopping
Technology & Computing
News & Weather
Education
SaaS & Cloud
Entertainment
0%7.5%15%23%30%
Download report
We track 21,907 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 6,330 websites using Common Crawl Bot Disallow in United States.
Filter by country
🇺🇸 United States6,330 sites
🇬🇧 United Kingdom1,618 sites
🇩🇪 Germany921 sites
🇫🇷 France832 sites
🇨🇦 Canada769 sites
🇮🇳 India689 sites
🇦🇺 Australia595 sites
🇳🇱 Netherlands587 sites
🇪🇸 Spain529 sites
🇧🇷 Brazil518 sites
🇮🇹 Italy404 sites
🇵🇱 Poland344 sites
🇨🇭 Switzerland309 sites
🇮🇩 Indonesia301 sites
🇸🇪 Sweden256 sites
🇹🇷 Turkey191 sites
🇧🇪 Belgium156 sites
🇷🇺 Russian Federation153 sites
🇦🇷 Argentina142 sites
🇫🇮 Finland128 sites
🇨🇿 Czechia126 sites
🇿🇦 South Africa122 sites
🇩🇰 Denmark120 sites
🇻🇳 Viet Nam113 sites
🇨🇱 Chile109 sites
🇲🇽 Mexico107 sites
🇦🇹 Austria105 sites
🇯🇵 Japan105 sites
🇳🇿 New Zealand105 sites
🇬🇷 Greece102 sites
🇨🇳 China101 sites
🇳🇴 Norway101 sites
🇮🇱 Israel95 sites
🇮🇪 Ireland94 sites
🇰🇷 Korea, Republic of87 sites
🇦🇪 United Arab Emirates83 sites
🇵🇹 Portugal83 sites
🇸🇬 Singapore82 sites
🇷🇴 Romania79 sites
🇨🇴 Colombia72 sites
🇺🇦 Ukraine70 sites
🇵🇰 Pakistan64 sites
🇭🇺 Hungary59 sites
🇲🇾 Malaysia57 sites
🇧🇩 Bangladesh54 sites
🇳🇬 Nigeria53 sites
🇸🇦 Saudi Arabia48 sites
🇵🇪 Peru46 sites
🇹🇭 Thailand43 sites
🇪🇬 Egypt43 sites
🇭🇰 Hong Kong39 sites
🇱🇹 Lithuania37 sites
🇸🇰 Slovakia36 sites
🇹🇼 Taiwan, Province of China35 sites
🇧🇬 Bulgaria34 sites
🇵🇭 Philippines32 sites
🇪🇪 Estonia29 sites
🇰🇿 Kazakhstan29 sites
🇱🇻 Latvia29 sites
🇭🇷 Croatia28 sites
🇷🇸 Serbia28 sites
🇮🇷 Iran, Islamic Republic of27 sites
🇨🇾 Cyprus26 sites
🇱🇰 Sri Lanka20 sites
🇸🇮 Slovenia19 sites
🇪🇨 Ecuador18 sites
🇲🇦 Morocco18 sites
🇹🇳 Tunisia17 sites
🇧🇾 Belarus15 sites
🇨🇷 Costa Rica14 sites
🇩🇴 Dominican Republic14 sites
🇰🇪 Kenya14 sites
🇦🇿 Azerbaijan13 sites
🇻🇪 Venezuela, Bolivarian Republic of12 sites
🇱🇧 Lebanon11 sites
🇮🇸 Iceland10 sites
🇲🇹 Malta10 sites
🇸🇳 Senegal10 sites
🇶🇦 Qatar10 sites
🇦🇱 Albania9 sites
🇬🇭 Ghana9 sites
🇲🇩 Moldova, Republic of9 sites
🇲🇰 North Macedonia9 sites
🇺🇾 Uruguay9 sites
🇦🇲 Armenia9 sites
🇧🇦 Bosnia and Herzegovina9 sites
🇸🇻 El Salvador8 sites
🇺🇿 Uzbekistan8 sites
🇨🇲 Cameroon7 sites
🇳🇦 Namibia7 sites
🇳🇵 Nepal7 sites
🇴🇲 Oman7 sites
🇬🇹 Guatemala7 sites
🇵🇷 Puerto Rico6 sites
🇺🇬 Uganda6 sites
🇪🇹 Ethiopia6 sites
🇩🇿 Algeria6 sites
🇻🇨 Saint Vincent and the Grenadines6 sites
🇲🇬 Madagascar6 sites
🇹🇿 Tanzania, United Republic of6 sites
🇱🇺 Luxembourg5 sites
🇬🇪 Georgia5 sites
🇲🇺 Mauritius5 sites
🇯🇲 Jamaica5 sites
🇵🇾 Paraguay5 sites
🇵🇦 Panama5 sites
🇲🇨 Monaco5 sites
🇧🇴 Bolivia, Plurinational State of5 sites
🇰🇼 Kuwait4 sites
🇲🇪 Montenegro4 sites
🇨🇽 Christmas Island4 sites
🇯🇴 Jordan4 sites
🇮🇶 Iraq4 sites
🇧🇯 Benin3 sites
🇸🇴 Somalia3 sites
🇧🇿 Belize3 sites
🇱🇾 Libya3 sites
🇨🇮 Côte d'Ivoire3 sites
🇧🇲 Bermuda3 sites
🇱🇮 Liechtenstein3 sites
🇦🇩 Andorra2 sites
🇬🇦 Gabon2 sites
🇲🇲 Myanmar2 sites
🇹🇯 Tajikistan2 sites
🇽🇰 Kosovo2 sites
🇭🇹 Haiti2 sites
🇦🇬 Antigua and Barbuda2 sites
🇭🇳 Honduras2 sites
🇵🇲 Saint Pierre and Miquelon2 sites
🇸🇱 Sierra Leone2 sites
🇰🇬 Kyrgyzstan2 sites
🇨🇼 Curaçao2 sites
🇰🇭 Cambodia2 sites
🇧🇮 Burundi2 sites
🇻🇬 Virgin Islands, British2 sites
🇸🇷 Suriname2 sites
🇷🇪 Réunion2 sites
🇲🇳 Mongolia2 sites
🇧🇭 Bahrain1 sites
🇳🇨 New Caledonia1 sites
🇸🇾 Syrian Arab Republic1 sites
🇹🇩 Chad1 sites
🇻🇦 Holy See (Vatican City State)1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇸🇨 Seychelles1 sites
🇬🇾 Guyana1 sites
🇧🇸 Bahamas1 sites
🇱🇷 Liberia1 sites
🇬🇬 Guernsey1 sites
🇲🇴 Macao1 sites
🇫🇯 Fiji1 sites
🇲🇻 Maldives1 sites
🇻🇺 Vanuatu1 sites
🇯🇪 Jersey1 sites
🇧🇼 Botswana1 sites
🇰🇵 Korea, Democratic People's Republic of1 sites
🇸🇹 Sao Tome and Principe1 sites
🇦🇸 American Samoa1 sites
🇵🇬 Papua New Guinea1 sites
🇷🇼 Rwanda1 sites
🇮🇴 British Indian Ocean Territory1 sites
🇱🇨 Saint Lucia1 sites
🇰🇾 Cayman Islands1 sites
🇲🇵 Northern Mariana Islands1 sites
🇹🇬 Togo1 sites
🇩🇲 Dominica1 sites
🇸🇲 San Marino1 sites
🇻🇮 Virgin Islands, U.S.1 sites
🇬🇲 Gambia1 sites
🇬🇩 Grenada1 sites
🇮🇲 Isle of Man1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇾🇹 Mayotte1 sites
🇦🇫 Afghanistan1 sites
🇨🇻 Cabo Verde1 sites
🇲🇸 Montserrat1 sites
🇲🇱 Mali1 sites
About Common Crawl Bot Disallow
This website has found the Disallow rules for Common Crawl Bot.
Categories
Official website
commoncrawl.org/faq↗
Websites using Common Crawl Bot Disallow
showing 25 of 21,907| Website | Country | Traffic | Industry |
|---|---|---|---|
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇧🇷BR | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | Software Development | |
 amazon.de | 🇩🇪DE | Top 500 | Logistics & Shipping |
| 🇬🇧GB | Top 500 | News & Weather | |
| 🇬🇧GB | Top 500 | Print & Publishing | |
| 🇬🇧GB | Top 500 | News & Weather | |
| 🇩🇪DE | Top 500 | Entertainment | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Software Development | |
| 🇺🇸US | Top 500 | SaaS & Cloud | |
| 🇩🇪DE | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Movies & TV | |
| 🇺🇸US | Top 500 | Movies & TV | |
| 🇬🇧GB | Top 500 | Mental Health | |
| 🇺🇸US | Top 500 | Technology & Computing | |
| 🇺🇸US | Top 500 | News & Weather | |
| 🇺🇸US | Top 500 | Software Development | |
| 🇬🇧GB | Top 500 | Print & Publishing | |
| 🇺🇸US | Top 1K | Technology & Computing | |
| 🇺🇸US | Top 1K | Print & Publishing | |
| 🇩🇪DE | Top 1K | News & Weather | |
 amazon.ca | 🇨🇦CA | Top 1K | Logistics & Shipping |
| 🇺🇸US | Top 1K | News & Weather |
Showing 25 of 21,907 websitesView detailed list →
