Common Crawl Bot Disallow

Common Crawl Bot Disallow tracking and usage

Websites using Common Crawl Bot Disallow by countrymarket share
22%
🇺🇸US
6.5%
🇬🇧GB
4.5%
🇩🇪DE
3.4%
🇫🇷FR
3.3%
🇦🇺AU
3.3%
🇨🇦CA
3.1%
🇳🇱NL
2.8%
🇮🇳IN
Websites using Common Crawl Bot Disallow by industrymarket share · top 6 industries
Shopping
28%
Technology & Computing
19%
SaaS & Cloud
17%
Education
16%
News & Weather
12%
Medical & Healthcare
7.9%
0%7.5%15%23%30%
Download report

We track 116,221 websites using Common Crawl Bot Disallow, along with their country, traffic, industry and platforms. There are 25,832 websites using Common Crawl Bot Disallow in United States.

Filter by country
🇺🇸 United States25,832 sites
🇬🇧 United Kingdom7,532 sites
🇩🇪 Germany5,186 sites
🇫🇷 France3,925 sites
🇦🇺 Australia3,852 sites
🇨🇦 Canada3,822 sites
🇳🇱 Netherlands3,587 sites
🇮🇳 India3,227 sites
🇧🇷 Brazil2,960 sites
🇪🇸 Spain2,789 sites
🇮🇹 Italy2,145 sites
🇵🇱 Poland1,923 sites
🇨🇭 Switzerland1,666 sites
🇹🇷 Turkey1,553 sites
🇩🇰 Denmark1,412 sites
🇸🇪 Sweden1,273 sites
🇮🇩 Indonesia1,113 sites
🇷🇺 Russian Federation809 sites
🇧🇪 Belgium766 sites
🇨🇿 Czechia722 sites
🇳🇴 Norway693 sites
🇯🇵 Japan659 sites
🇬🇷 Greece616 sites
🇦🇷 Argentina601 sites
🇲🇽 Mexico570 sites
🇫🇮 Finland568 sites
🇿🇦 South Africa565 sites
🇳🇿 New Zealand562 sites
🇻🇳 Viet Nam557 sites
🇦🇹 Austria543 sites
🇦🇪 United Arab Emirates534 sites
🇵🇹 Portugal498 sites
🇨🇳 China475 sites
🇨🇱 Chile458 sites
🇮🇪 Ireland413 sites
🇺🇦 Ukraine392 sites
🇷🇴 Romania384 sites
🇸🇬 Singapore364 sites
🇵🇰 Pakistan361 sites
🇰🇷 Korea, Republic of332 sites
🇭🇺 Hungary309 sites
🇨🇴 Colombia308 sites
🇲🇾 Malaysia295 sites
🇮🇱 Israel280 sites
🇸🇦 Saudi Arabia250 sites
🇳🇬 Nigeria222 sites
🇹🇭 Thailand216 sites
🇵🇪 Peru211 sites
🇧🇩 Bangladesh200 sites
🇪🇬 Egypt188 sites
🇭🇰 Hong Kong179 sites
🇹🇼 Taiwan, Province of China168 sites
🇰🇿 Kazakhstan163 sites
🇵🇭 Philippines162 sites
🇧🇬 Bulgaria157 sites
🇱🇹 Lithuania155 sites
🇸🇰 Slovakia155 sites
🇮🇷 Iran, Islamic Republic of142 sites
🇪🇪 Estonia116 sites
🇸🇮 Slovenia113 sites
🇭🇷 Croatia112 sites
🇷🇸 Serbia107 sites
🇰🇪 Kenya106 sites
🇨🇾 Cyprus105 sites
🇱🇰 Sri Lanka89 sites
🇲🇦 Morocco88 sites
🇱🇻 Latvia81 sites
🇷🇼 Rwanda79 sites
🇪🇨 Ecuador74 sites
🇮🇸 Iceland70 sites
🇬🇭 Ghana66 sites
🇩🇴 Dominican Republic57 sites
🇺🇾 Uruguay56 sites
🇦🇿 Azerbaijan55 sites
🇲🇹 Malta55 sites
🇵🇦 Panama54 sites
🇧🇾 Belarus51 sites
🇨🇷 Costa Rica51 sites
🇱🇺 Luxembourg49 sites
🇻🇪 Venezuela, Bolivarian Republic of49 sites
🇯🇴 Jordan48 sites
🇳🇵 Nepal48 sites
🇬🇪 Georgia47 sites
🇬🇹 Guatemala46 sites
🇱🇧 Lebanon43 sites
🇶🇦 Qatar43 sites
🇹🇳 Tunisia42 sites
🇦🇲 Armenia39 sites
🇦🇱 Albania38 sites
🇰🇼 Kuwait38 sites
🇧🇦 Bosnia and Herzegovina36 sites
🇺🇬 Uganda34 sites
🇴🇲 Oman33 sites
🇮🇶 Iraq31 sites
🇻🇨 Saint Vincent and the Grenadines31 sites
🇲🇰 North Macedonia28 sites
🇺🇿 Uzbekistan28 sites
🇲🇺 Mauritius27 sites
🇵🇾 Paraguay27 sites
🇲🇩 Moldova, Republic of26 sites
🇵🇷 Puerto Rico26 sites
🇹🇿 Tanzania, United Republic of26 sites
🇸🇻 El Salvador24 sites
🇲🇨 Monaco23 sites
🇸🇳 Senegal23 sites
🇧🇭 Bahrain22 sites
🇨🇲 Cameroon22 sites
🇩🇿 Algeria22 sites
🇨🇮 Côte d'Ivoire22 sites
🇱🇮 Liechtenstein21 sites
🇧🇴 Bolivia, Plurinational State of21 sites
🇪🇹 Ethiopia18 sites
🇰🇭 Cambodia18 sites
🇸🇴 Somalia16 sites
🇽🇰 Kosovo16 sites
🇨🇼 Curaçao16 sites
🇬🇩 Grenada16 sites
🇷🇪 Réunion16 sites
🇯🇲 Jamaica15 sites
🇲🇬 Madagascar15 sites
🇧🇿 Belize14 sites
🇰🇬 Kyrgyzstan14 sites
🇦🇴 Angola14 sites
🇯🇪 Jersey13 sites
🇳🇦 Namibia12 sites
🇭🇳 Honduras12 sites
🇻🇬 Virgin Islands, British12 sites
🇦🇸 American Samoa11 sites
🇲🇲 Myanmar11 sites
🇲🇪 Montenegro11 sites
🇿🇼 Zimbabwe11 sites
🇦🇽 Åland Islands11 sites
🇸🇽 Sint Maarten (Dutch part)11 sites
🇦🇩 Andorra9 sites
🇹🇯 Tajikistan9 sites
🇹🇹 Trinidad and Tobago9 sites
🇦🇬 Antigua and Barbuda9 sites
🇸🇨 Seychelles9 sites
🇨🇽 Christmas Island9 sites
🇲🇻 Maldives9 sites
🇰🇾 Cayman Islands9 sites
🇵🇸 Palestine, State of9 sites
🇧🇯 Benin8 sites
🇧🇸 Bahamas8 sites
🇸🇾 Syrian Arab Republic8 sites
🇿🇲 Zambia8 sites
🇲🇳 Mongolia8 sites
🇮🇲 Isle of Man8 sites
🇭🇹 Haiti7 sites
🇹🇬 Togo7 sites
🇵🇼 Palau7 sites
🇧🇲 Bermuda6 sites
🇱🇾 Libya6 sites
🇸🇷 Suriname6 sites
🇸🇩 Sudan6 sites
🇨🇻 Cabo Verde6 sites
🇬🇦 Gabon5 sites
🇲🇶 Martinique5 sites
🇧🇳 Brunei Darussalam5 sites
🇨🇩 Congo, The Democratic Republic of the5 sites
🇫🇯 Fiji5 sites
🇬🇮 Gibraltar5 sites
🇦🇼 Aruba4 sites
🇳🇨 New Caledonia4 sites
🇵🇫 French Polynesia4 sites
🇾🇹 Mayotte4 sites
🇬🇬 Guernsey4 sites
🇬🇸 South Georgia and the South Sandwich Islands4 sites
🇲🇱 Mali4 sites
🇵🇲 Saint Pierre and Miquelon4 sites
🇸🇱 Sierra Leone4 sites
🇧🇼 Botswana4 sites
🇻🇮 Virgin Islands, U.S.4 sites
🇦🇫 Afghanistan4 sites
🇲🇸 Montserrat4 sites
🇬🇵 Guadeloupe3 sites
🇩🇲 Dominica3 sites
🇾🇪 Yemen3 sites
🇨🇺 Cuba3 sites
🇩🇯 Djibouti3 sites
🇧🇮 Burundi3 sites
🇰🇳 Saint Kitts and Nevis3 sites
🇸🇹 Sao Tome and Principe3 sites
🇬🇫 French Guiana3 sites
🇬🇲 Gambia3 sites
🇲🇿 Mozambique3 sites
🇬🇺 Guam3 sites
🇹🇱 Timor-Leste3 sites
🇹🇫 French Southern Territories3 sites
🇬🇳 Guinea3 sites
🇹🇩 Chad2 sites
🇬🇱 Greenland2 sites
🇬🇾 Guyana2 sites
🇲🇴 Macao2 sites
🇳🇮 Nicaragua2 sites
🇻🇺 Vanuatu2 sites
🇧🇫 Burkina Faso2 sites
🇧🇧 Barbados2 sites
🇲🇭 Marshall Islands2 sites
🇹🇨 Turks and Caicos Islands2 sites
🇧🇶 Bonaire, Sint Eustatius and Saba2 sites
🇮🇴 British Indian Ocean Territory2 sites
🇲🇼 Malawi2 sites
🇲🇵 Northern Mariana Islands2 sites
🇰🇲 Comoros2 sites
🇻🇦 Holy See (Vatican City State)1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇱🇷 Liberia1 sites
🇧🇱 Saint Barthélemy1 sites
🇸🇿 Eswatini1 sites
🇦🇮 Anguilla1 sites
🇰🇵 Korea, Democratic People's Republic of1 sites
🇸🇸 South Sudan1 sites
🇨🇰 Cook Islands1 sites
🇵🇬 Papua New Guinea1 sites
🇱🇨 Saint Lucia1 sites
🇸🇲 San Marino1 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇱🇸 Lesotho1 sites
🇸🇧 Solomon Islands1 sites
🇼🇸 Samoa1 sites
🇼🇫 Wallis and Futuna1 sites
🇹🇲 Turkmenistan1 sites

About Common Crawl Bot Disallow

This website has found the Disallow rules for Common Crawl Bot.

Categories
Robots.txtWeb Standards
Official websitecommoncrawl.org/faq↗

Websites using Common Crawl Bot Disallow

showing 25 of 116,221
WebsiteCountryTrafficIndustry
amazon.co.jp🇯🇵JPTop 500Shopping
nbcnews.com🇺🇸USTop 500News & Weather
apnews.com🇺🇸USTop 500News & Weather
washingtonpost.com🇺🇸USTop 500News & Weather
spiegel.de🇩🇪DETop 500Print & Publishing
lnkd.in🇮🇳INTop 500Religion & Spirituality
cnbc.com🇺🇸USTop 500News & Weather
fastcompany.com🇺🇸USTop 500Technology & Computing
fortune.com🇺🇸USTop 500News & Weather
goodreads.com🇺🇸USTop 500Technology & Computing
hbr.org🇺🇸USTop 500Print & Publishing
usnews.com🇺🇸USTop 500Technology & Computing
scientificamerican.com🇺🇸USTop 500News & Weather
bandcamp.com🇺🇸USTop 500Music
lemonde.fr🇫🇷FRTop 500News & Weather
unesco.org🇫🇷FRTop 500Law & Government
amazon.co.uk🇬🇧GBTop 500Entertainment
inc.com🇺🇸USTop 500Print & Publishing
amazon.fr🇫🇷FRTop 500Logistics & Shipping
tumblr.com🇺🇸USTop 500Technology & Computing
bloomberg.com🇺🇸USTop 500News & Weather
globo.com🇧🇷BRTop 500Technology & Computing
eventbrite.com🇺🇸USTop 500Software Development
amazon.de🇩🇪DETop 500Logistics & Shipping
bbc.co.uk🇬🇧GBTop 500News & Weather
Showing 25 of 116,221 websitesView detailed list →

Alternatives

same category
HTML5 DocType45.32MsitesFacebook Redirect17.91MsitesInstagram11.7MsitesOrganization Schema7.24MsitesLinkedIn6.25MsitesUTF-86.01MsitesEvents Page4.51MsitesAPI or Developer Page3.76Msites