Robots Disallow

Robots Disallow tracking and usage

Websites using Robots Disallow by countrymarket share
23%
🇺🇸US
16%
🇩🇪DE
3.6%
🇬🇧GB
3.4%
🇷🇺RU
2.4%
🇳🇱NL
1.9%
🇮🇳IN
1.8%
🇨🇿CZ
1.8%
🇫🇷FR
Websites using Robots Disallow by industrymarket share · top 6 industries
Shopping
28%
Technology & Computing
23%
Education
16%
SaaS & Cloud
13%
Law & Government
11%
Marketing & Advertising
9.9%
0%7.5%15%23%30%
Download report

We track 19,294 websites using Robots Disallow, along with their country, traffic, industry and platforms. There are 4,389 websites using Robots Disallow in United States.

Filter by country
🇺🇸 United States4,389 sites
🇩🇪 Germany3,067 sites
🇬🇧 United Kingdom692 sites
🇷🇺 Russian Federation664 sites
🇳🇱 Netherlands464 sites
🇮🇳 India358 sites
🇨🇿 Czechia350 sites
🇫🇷 France349 sites
🇯🇵 Japan343 sites
🇨🇦 Canada285 sites
🇧🇷 Brazil258 sites
🇪🇸 Spain243 sites
🇦🇺 Australia233 sites
🇮🇹 Italy231 sites
🇨🇳 China190 sites
🇸🇪 Sweden187 sites
🇨🇭 Switzerland171 sites
🇰🇷 Korea, Republic of156 sites
🇧🇪 Belgium126 sites
🇳🇴 Norway122 sites
🇵🇱 Poland117 sites
🇮🇩 Indonesia104 sites
🇲🇽 Mexico102 sites
🇫🇮 Finland82 sites
🇩🇰 Denmark77 sites
🇺🇦 Ukraine72 sites
🇦🇹 Austria69 sites
🇹🇷 Turkey66 sites
🇵🇭 Philippines54 sites
🇳🇿 New Zealand53 sites
🇭🇺 Hungary51 sites
🇦🇪 United Arab Emirates47 sites
🇿🇦 South Africa43 sites
🇮🇪 Ireland42 sites
🇵🇹 Portugal41 sites
🇮🇷 Iran, Islamic Republic of40 sites
🇷🇴 Romania39 sites
🇸🇬 Singapore37 sites
🇨🇴 Colombia35 sites
🇭🇰 Hong Kong34 sites
🇹🇼 Taiwan, Province of China33 sites
🇬🇷 Greece29 sites
🇨🇱 Chile28 sites
🇦🇷 Argentina27 sites
🇲🇾 Malaysia27 sites
🇵🇰 Pakistan27 sites
🇱🇹 Lithuania24 sites
🇵🇪 Peru23 sites
🇧🇾 Belarus22 sites
🇸🇰 Slovakia21 sites
🇻🇳 Viet Nam21 sites
🇮🇱 Israel20 sites
🇸🇦 Saudi Arabia19 sites
🇹🇭 Thailand18 sites
🇧🇬 Bulgaria17 sites
🇮🇸 Iceland17 sites
🇪🇬 Egypt14 sites
🇳🇬 Nigeria14 sites
🇪🇪 Estonia13 sites
🇧🇩 Bangladesh12 sites
🇱🇺 Luxembourg12 sites
🇨🇾 Cyprus11 sites
🇱🇧 Lebanon10 sites
🇱🇮 Liechtenstein10 sites
🇲🇦 Morocco9 sites
🇨🇲 Cameroon9 sites
🇺🇾 Uruguay8 sites
🇵🇦 Panama8 sites
🇰🇪 Kenya8 sites
🇸🇮 Slovenia8 sites
🇦🇲 Armenia7 sites
🇦🇿 Azerbaijan7 sites
🇱🇻 Latvia7 sites
🇱🇰 Sri Lanka7 sites
🇰🇿 Kazakhstan7 sites
🇻🇪 Venezuela, Bolivarian Republic of6 sites
🇬🇭 Ghana6 sites
🇲🇩 Moldova, Republic of6 sites
🇭🇷 Croatia6 sites
🇦🇱 Albania5 sites
🇨🇷 Costa Rica5 sites
🇰🇬 Kyrgyzstan5 sites
🇯🇴 Jordan5 sites
🇻🇨 Saint Vincent and the Grenadines5 sites
🇷🇸 Serbia5 sites
🇬🇹 Guatemala5 sites
🇪🇨 Ecuador4 sites
🇲🇹 Malta4 sites
🇶🇦 Qatar4 sites
🇬🇪 Georgia4 sites
🇸🇨 Seychelles4 sites
🇰🇼 Kuwait4 sites
🇩🇴 Dominican Republic4 sites
🇺🇿 Uzbekistan4 sites
🇳🇵 Nepal4 sites
🇧🇿 Belize4 sites
🇮🇲 Isle of Man4 sites
🇬🇱 Greenland4 sites
🇲🇸 Montserrat4 sites
🇩🇿 Algeria3 sites
🇯🇲 Jamaica3 sites
🇻🇬 Virgin Islands, British3 sites
🇸🇷 Suriname3 sites
🇺🇬 Uganda3 sites
🇿🇼 Zimbabwe3 sites
🇭🇳 Honduras3 sites
🇸🇳 Senegal3 sites
🇸🇴 Somalia3 sites
🇲🇺 Mauritius3 sites
🇵🇼 Palau3 sites
🇷🇪 Réunion3 sites
🇲🇰 North Macedonia3 sites
🇦🇼 Aruba2 sites
🇧🇭 Bahrain2 sites
🇮🇶 Iraq2 sites
🇹🇹 Trinidad and Tobago2 sites
🇧🇴 Bolivia, Plurinational State of2 sites
🇱🇾 Libya2 sites
🇵🇾 Paraguay2 sites
🇧🇫 Burkina Faso2 sites
🇨🇩 Congo, The Democratic Republic of the2 sites
🇭🇹 Haiti2 sites
🇦🇴 Angola2 sites
🇹🇳 Tunisia2 sites
🇷🇼 Rwanda2 sites
🇧🇦 Bosnia and Herzegovina2 sites
🇲🇿 Mozambique2 sites
🇴🇲 Oman2 sites
🇵🇷 Puerto Rico2 sites
🇵🇸 Palestine, State of2 sites
🇰🇭 Cambodia2 sites
🇲🇨 Monaco2 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇩🇯 Djibouti1 sites
🇦🇸 American Samoa1 sites
🇱🇨 Saint Lucia1 sites
🇸🇻 El Salvador1 sites
🇸🇾 Syrian Arab Republic1 sites
🇹🇬 Togo1 sites
🇿🇲 Zambia1 sites
🇬🇦 Gabon1 sites
🇦🇬 Antigua and Barbuda1 sites
🇸🇩 Sudan1 sites
🇸🇿 Eswatini1 sites
🇦🇩 Andorra1 sites
🇬🇩 Grenada1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇧🇧 Barbados1 sites
🇧🇼 Botswana1 sites
🇹🇿 Tanzania, United Republic of1 sites
🇹🇲 Turkmenistan1 sites
🇧🇲 Bermuda1 sites
🇨🇮 Côte d'Ivoire1 sites
🇨🇽 Christmas Island1 sites
🇲🇪 Montenegro1 sites
🇲🇳 Mongolia1 sites
🇲🇵 Northern Mariana Islands1 sites
🇹🇨 Turks and Caicos Islands1 sites
🇨🇬 Congo1 sites
🇲🇻 Maldives1 sites
🇾🇹 Mayotte1 sites
🇳🇦 Namibia1 sites
🇳🇪 Niger1 sites
🇬🇸 South Georgia and the South Sandwich Islands1 sites
🇸🇱 Sierra Leone1 sites
🇵🇲 Saint Pierre and Miquelon1 sites
🇹🇯 Tajikistan1 sites
🇻🇺 Vanuatu1 sites
🇩🇲 Dominica1 sites
🇫🇯 Fiji1 sites
🇵🇫 French Polynesia1 sites

About Robots Disallow

Sites blocking the Robots crawler in robots.txt.

Categories
Robots.txtWeb Standards

Websites using Robots Disallow

showing 25 of 19,294
WebsiteCountryTrafficIndustry
urldefense.com🇺🇸USTop 1KShopping
secureserver.net🇺🇸USTop 1KTechnology & Computing
sharethrough.com🇺🇸USTop 1KTechnology & Computing
1drv.ms🇲🇸MSTop 5KSociety
cargocollective.com🇨🇿CZTop 5KScience
lovdata.no🇳🇴NOTop 5KLegal Services
litmus.com🇺🇸USTop 5KSoftware Development
sogou.com🇨🇳CNTop 5KTechnology & Computing
amsterdam.nl🇳🇱NLTop 5KLaw & Government
qcloud.com🇨🇳CNTop 5KSoftware Development
cervantes.es🇪🇸ESTop 5KLaw & Government
co.com-Top 5KTechnology & Computing
fortinet.com🇺🇸USTop 5KCybersecurity
vtb.ru🇷🇺RUTop 5K-
arte.tv🇫🇷FRTop 5KMovies & TV
cnki.net🇨🇳CNTop 10KSoftware Development
arcor.de🇩🇪DETop 10KTelecommunications
pinboard.in🇮🇳INTop 10KTechnology & Computing
swin.edu.au🇦🇺AUTop 10KShopping
wezhan.cn🇨🇳CNTop 10K-
yclients.com🇨🇾CYTop 10KTechnology & Computing
lexisnexis.com🇺🇸USTop 10KTechnology & Computing
helsinki.fi🇫🇮FITop 10KEducation
spokeo.com🇺🇸USTop 10KTechnology & Computing
nixle.com🇺🇸USTop 50K-
Showing 25 of 19,294 websitesView detailed list →

Alternatives

same category
HTML5 DocType43.94MsitesFacebook Redirect17.24MsitesInstagram11.18MsitesOrganization Schema6.91MsitesLinkedIn5.92MsitesEvents Page4.24MsitesUTF-83.82MsitesMembers Page3.51Msites