ICC-Crawler Disallow

ICC-Crawler Disallow tracking and usage

Websites using ICC-Crawler Disallow by countrymarket share
15%
🇺🇸US
7.6%
🇬🇧GB
4.2%
🇧🇷BR
4.1%
🇮🇳IN
3.9%
🇨🇦CA
3.4%
🇩🇪DE
3.1%
🇦🇺AU
2.9%
🇫🇷FR
Websites using ICC-Crawler Disallow by industrymarket share · top 6 industries
Technology & Computing
24%
Shopping
22%
SaaS & Cloud
15%
News & Weather
15%
Education
14%
Construction
10%
0%6.3%13%19%25%
Download report

We track 39,513 websites using ICC-Crawler Disallow, along with their country, traffic, industry and platforms. There are 5,862 websites using ICC-Crawler Disallow in United States.

Filter by country
🇺🇸 United States5,862 sites
🇬🇧 United Kingdom2,998 sites
🇧🇷 Brazil1,642 sites
🇮🇳 India1,614 sites
🇨🇦 Canada1,541 sites
🇩🇪 Germany1,360 sites
🇦🇺 Australia1,231 sites
🇫🇷 France1,150 sites
🇪🇸 Spain1,149 sites
🇳🇱 Netherlands1,065 sites
🇨🇭 Switzerland779 sites
🇹🇷 Turkey730 sites
🇮🇹 Italy716 sites
🇸🇪 Sweden700 sites
🇮🇩 Indonesia655 sites
🇵🇱 Poland622 sites
🇯🇵 Japan361 sites
🇳🇴 Norway336 sites
🇲🇽 Mexico315 sites
🇦🇷 Argentina306 sites
🇦🇪 United Arab Emirates299 sites
🇧🇪 Belgium269 sites
🇵🇹 Portugal239 sites
🇿🇦 South Africa238 sites
🇨🇱 Chile226 sites
🇻🇳 Viet Nam222 sites
🇩🇰 Denmark217 sites
🇳🇿 New Zealand217 sites
🇮🇪 Ireland204 sites
🇬🇷 Greece203 sites
🇫🇮 Finland202 sites
🇵🇰 Pakistan193 sites
🇨🇳 China188 sites
🇦🇹 Austria185 sites
🇸🇬 Singapore163 sites
🇲🇾 Malaysia161 sites
🇨🇴 Colombia157 sites
🇸🇦 Saudi Arabia151 sites
🇨🇿 Czechia146 sites
🇷🇴 Romania144 sites
🇳🇬 Nigeria132 sites
🇮🇱 Israel128 sites
🇧🇩 Bangladesh117 sites
🇪🇬 Egypt112 sites
🇵🇪 Peru112 sites
🇺🇦 Ukraine101 sites
🇹🇭 Thailand93 sites
🇭🇰 Hong Kong83 sites
🇭🇺 Hungary77 sites
🇹🇼 Taiwan, Province of China77 sites
🇷🇺 Russian Federation76 sites
🇵🇭 Philippines72 sites
🇱🇹 Lithuania69 sites
🇸🇰 Slovakia68 sites
🇰🇪 Kenya65 sites
🇱🇰 Sri Lanka63 sites
🇲🇦 Morocco56 sites
🇰🇷 Korea, Republic of53 sites
🇨🇾 Cyprus53 sites
🇷🇸 Serbia49 sites
🇭🇷 Croatia46 sites
🇬🇭 Ghana46 sites
🇧🇬 Bulgaria45 sites
🇸🇮 Slovenia43 sites
🇲🇨 Monaco43 sites
🇮🇷 Iran, Islamic Republic of41 sites
🇪🇨 Ecuador40 sites
🇺🇾 Uruguay36 sites
🇪🇪 Estonia35 sites
🇵🇦 Panama35 sites
🇱🇻 Latvia31 sites
🇳🇵 Nepal31 sites
🇩🇴 Dominican Republic31 sites
🇰🇼 Kuwait29 sites
🇶🇦 Qatar27 sites
🇬🇪 Georgia27 sites
🇯🇴 Jordan26 sites
🇨🇷 Costa Rica26 sites
🇦🇿 Azerbaijan26 sites
🇱🇧 Lebanon26 sites
🇬🇹 Guatemala25 sites
🇮🇶 Iraq23 sites
🇦🇲 Armenia22 sites
🇮🇸 Iceland21 sites
🇴🇲 Oman21 sites
🇱🇺 Luxembourg20 sites
🇻🇪 Venezuela, Bolivarian Republic of20 sites
🇺🇬 Uganda19 sites
🇲🇹 Malta19 sites
🇦🇱 Albania19 sites
🇰🇿 Kazakhstan18 sites
🇧🇭 Bahrain18 sites
🇵🇾 Paraguay18 sites
🇹🇳 Tunisia18 sites
🇵🇷 Puerto Rico17 sites
🇸🇻 El Salvador17 sites
🇲🇺 Mauritius15 sites
🇰🇭 Cambodia14 sites
🇹🇿 Tanzania, United Republic of13 sites
🇨🇮 Côte d'Ivoire13 sites
🇽🇰 Kosovo13 sites
🇧🇦 Bosnia and Herzegovina12 sites
🇺🇿 Uzbekistan12 sites
🇲🇩 Moldova, Republic of12 sites
🇦🇴 Angola12 sites
🇧🇾 Belarus11 sites
🇧🇴 Bolivia, Plurinational State of11 sites
🇷🇼 Rwanda11 sites
🇸🇳 Senegal10 sites
🇨🇲 Cameroon10 sites
🇦🇽 Åland Islands10 sites
🇲🇰 North Macedonia9 sites
🇩🇿 Algeria9 sites
🇯🇪 Jersey9 sites
🇷🇪 Réunion9 sites
🇵🇸 Palestine, State of9 sites
🇨🇼 Curaçao8 sites
🇲🇬 Madagascar8 sites
🇿🇼 Zimbabwe8 sites
🇲🇻 Maldives8 sites
🇹🇹 Trinidad and Tobago7 sites
🇿🇲 Zambia7 sites
🇪🇹 Ethiopia7 sites
🇲🇲 Myanmar7 sites
🇻🇨 Saint Vincent and the Grenadines7 sites
🇭🇳 Honduras6 sites
🇧🇿 Belize6 sites
🇯🇲 Jamaica6 sites
🇱🇮 Liechtenstein6 sites
🇧🇳 Brunei Darussalam5 sites
🇸🇾 Syrian Arab Republic5 sites
🇲🇪 Montenegro5 sites
🇨🇻 Cabo Verde5 sites
🇦🇩 Andorra4 sites
🇧🇸 Bahamas4 sites
🇳🇦 Namibia4 sites
🇫🇯 Fiji4 sites
🇲🇳 Mongolia4 sites
🇦🇬 Antigua and Barbuda4 sites
🇨🇩 Congo, The Democratic Republic of the4 sites
🇦🇸 American Samoa4 sites
🇬🇮 Gibraltar4 sites
🇸🇩 Sudan4 sites
🇸🇴 Somalia4 sites
🇹🇯 Tajikistan3 sites
🇧🇯 Benin3 sites
🇾🇪 Yemen3 sites
🇸🇷 Suriname3 sites
🇹🇬 Togo3 sites
🇻🇮 Virgin Islands, U.S.3 sites
🇰🇾 Cayman Islands3 sites
🇸🇨 Seychelles3 sites
🇹🇫 French Southern Territories3 sites
🇬🇳 Guinea3 sites
🇱🇾 Libya3 sites
🇬🇩 Grenada2 sites
🇩🇲 Dominica2 sites
🇵🇫 French Polynesia2 sites
🇧🇧 Barbados2 sites
🇹🇨 Turks and Caicos Islands2 sites
🇧🇮 Burundi2 sites
🇧🇶 Bonaire, Sint Eustatius and Saba2 sites
🇧🇼 Botswana2 sites
🇲🇱 Mali2 sites
🇦🇫 Afghanistan2 sites
🇮🇴 British Indian Ocean Territory2 sites
🇬🇲 Gambia2 sites
🇨🇽 Christmas Island2 sites
🇲🇸 Montserrat2 sites
🇭🇹 Haiti1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇬🇬 Guernsey1 sites
🇬🇸 South Georgia and the South Sandwich Islands1 sites
🇧🇲 Bermuda1 sites
🇧🇫 Burkina Faso1 sites
🇸🇿 Eswatini1 sites
🇲🇭 Marshall Islands1 sites
🇦🇮 Anguilla1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇰🇬 Kyrgyzstan1 sites
🇨🇺 Cuba1 sites
🇲🇼 Malawi1 sites
🇳🇨 New Caledonia1 sites
🇸🇸 South Sudan1 sites
🇨🇰 Cook Islands1 sites
🇦🇼 Aruba1 sites
🇲🇿 Mozambique1 sites
🇲🇶 Martinique1 sites
🇬🇺 Guam1 sites
🇲🇵 Northern Mariana Islands1 sites
🇻🇺 Vanuatu1 sites
🇳🇮 Nicaragua1 sites
🇸🇧 Solomon Islands1 sites
🇼🇸 Samoa1 sites
🇮🇲 Isle of Man1 sites
🇵🇲 Saint Pierre and Miquelon1 sites
🇼🇫 Wallis and Futuna1 sites
🇹🇩 Chad1 sites
🇸🇹 Sao Tome and Principe1 sites
🇹🇲 Turkmenistan1 sites

About ICC-Crawler Disallow

This website has found the Disallow rules for ICC-Crawler.

Categories
Robots.txtWeb Standards
Official websitewww.nict.go.jp/en/univ-com/plan/crawl.html↗

Websites using ICC-Crawler Disallow

showing 25 of 39,513
WebsiteCountryTrafficIndustry
amazon.co.jp🇯🇵JPTop 500Shopping
usnews.com🇺🇸USTop 500Technology & Computing
lemonde.fr🇫🇷FRTop 500News & Weather
amazon.co.uk🇬🇧GBTop 500Entertainment
amazon.fr🇫🇷FRTop 500Logistics & Shipping
amazon.de🇩🇪DETop 500Logistics & Shipping
ea.com🇺🇸USTop 500Entertainment
cnn.com🇺🇸USTop 500News & Weather
launchpad.net🇬🇧GBTop 500Mental Health
canva.com🇦🇺AUTop 500Software Development
lesechos.fr🇫🇷FRTop 1KNews & Weather
slate.com🇺🇸USTop 1KPrint & Publishing
amazon.it🇮🇹ITTop 1KShopping
welt.de🇩🇪DETop 1KNews & Weather
ouest-france.fr🇫🇷FRTop 1KNews & Weather
unep.org🇰🇪KETop 1KLaw & Government
amazon.es🇪🇸ESTop 1KTechnology & Computing
amazon.com.au🇦🇺AUTop 1KShopping
wikihow.com🇺🇸USTop 1KTechnology & Computing
bild.de🇩🇪DETop 1KNews & Weather
amazon.ca🇨🇦CATop 1KLogistics & Shipping
theverge.com🇺🇸USTop 1KNews & Weather
amazon.in🇮🇳INTop 1KShopping
dailymail.com🇺🇸USTop 1KPrint & Publishing
nintendo.com🇺🇸USTop 1KGaming
Showing 25 of 39,513 websitesView detailed list →

Alternatives

same category
HTML5 DocType45.32MsitesFacebook Redirect17.91MsitesInstagram11.7MsitesOrganization Schema7.24MsitesLinkedIn6.25MsitesUTF-86.01MsitesEvents Page4.51MsitesAPI or Developer Page3.76Msites