Robots Disallow

Robots Disallow tracking and usage

Websites using Robots Disallow by countrymarket share
19%
🇺🇸US
9.2%
🇩🇪DE
7.1%
🇬🇧GB
4.9%
🇷🇺RU
3.9%
🇮🇳IN
3.8%
🇳🇱NL
2.9%
🇧🇷BR
2.5%
🇫🇷FR
Websites using Robots Disallow by industrymarket share · top 6 industries
Technology & Computing
28%
Shopping
23%
Education
19%
Marketing & Advertising
11%
Software Development
10%
Construction
8.8%
0%7.5%15%23%30%
Download report

We track 4,151 websites using Robots Disallow, along with their country, traffic, industry and platforms. There are 809 websites using Robots Disallow in United States.

Filter by country
🇺🇸 United States809 sites
🇩🇪 Germany382 sites
🇬🇧 United Kingdom295 sites
🇷🇺 Russian Federation202 sites
🇮🇳 India161 sites
🇳🇱 Netherlands156 sites
🇧🇷 Brazil122 sites
🇫🇷 France104 sites
🇪🇸 Spain96 sites
🇨🇦 Canada89 sites
🇨🇳 China89 sites
🇦🇺 Australia85 sites
🇮🇹 Italy84 sites
🇧🇪 Belgium53 sites
🇳🇴 Norway50 sites
🇲🇽 Mexico41 sites
🇸🇪 Sweden40 sites
🇨🇭 Switzerland36 sites
🇫🇮 Finland33 sites
🇰🇷 Korea, Republic of29 sites
🇵🇱 Poland29 sites
🇹🇷 Turkey29 sites
🇮🇩 Indonesia28 sites
🇿🇦 South Africa26 sites
🇨🇿 Czechia25 sites
🇮🇷 Iran, Islamic Republic of25 sites
🇩🇰 Denmark24 sites
🇦🇪 United Arab Emirates23 sites
🇯🇵 Japan22 sites
🇳🇿 New Zealand22 sites
🇵🇭 Philippines20 sites
🇨🇴 Colombia18 sites
🇵🇹 Portugal18 sites
🇷🇴 Romania18 sites
🇮🇪 Ireland18 sites
🇹🇼 Taiwan, Province of China16 sites
🇦🇹 Austria16 sites
🇸🇬 Singapore15 sites
🇭🇺 Hungary14 sites
🇦🇷 Argentina13 sites
🇨🇱 Chile12 sites
🇸🇦 Saudi Arabia12 sites
🇭🇰 Hong Kong12 sites
🇵🇰 Pakistan12 sites
🇵🇪 Peru10 sites
🇺🇦 Ukraine10 sites
🇻🇳 Viet Nam9 sites
🇮🇱 Israel8 sites
🇳🇬 Nigeria8 sites
🇬🇷 Greece7 sites
🇲🇾 Malaysia7 sites
🇨🇾 Cyprus7 sites
🇹🇭 Thailand6 sites
🇧🇬 Bulgaria6 sites
🇪🇬 Egypt6 sites
🇸🇰 Slovakia5 sites
🇨🇲 Cameroon5 sites
🇧🇩 Bangladesh5 sites
🇪🇪 Estonia5 sites
🇺🇾 Uruguay4 sites
🇱🇰 Sri Lanka4 sites
🇱🇹 Lithuania4 sites
🇰🇪 Kenya3 sites
🇻🇪 Venezuela, Bolivarian Republic of3 sites
🇱🇧 Lebanon3 sites
🇭🇷 Croatia3 sites
🇰🇼 Kuwait3 sites
🇶🇦 Qatar3 sites
🇨🇷 Costa Rica3 sites
🇮🇸 Iceland3 sites
🇦🇿 Azerbaijan2 sites
🇧🇭 Bahrain2 sites
🇪🇨 Ecuador2 sites
🇯🇲 Jamaica2 sites
🇯🇴 Jordan2 sites
🇲🇹 Malta2 sites
🇱🇻 Latvia2 sites
🇰🇿 Kazakhstan2 sites
🇻🇨 Saint Vincent and the Grenadines2 sites
🇳🇵 Nepal2 sites
🇧🇿 Belize2 sites
🇲🇦 Morocco2 sites
🇱🇾 Libya2 sites
🇦🇱 Albania2 sites
🇬🇹 Guatemala2 sites
🇸🇴 Somalia2 sites
🇲🇺 Mauritius2 sites
🇨🇩 Congo, The Democratic Republic of the2 sites
🇬🇭 Ghana2 sites
🇲🇩 Moldova, Republic of2 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇻🇬 Virgin Islands, British1 sites
🇰🇬 Kyrgyzstan1 sites
🇿🇲 Zambia1 sites
🇬🇦 Gabon1 sites
🇸🇩 Sudan1 sites
🇮🇲 Isle of Man1 sites
🇸🇿 Eswatini1 sites
🇿🇼 Zimbabwe1 sites
🇭🇳 Honduras1 sites
🇷🇸 Serbia1 sites
🇸🇳 Senegal1 sites
🇹🇹 Trinidad and Tobago1 sites
🇸🇷 Suriname1 sites
🇧🇴 Bolivia, Plurinational State of1 sites
🇵🇾 Paraguay1 sites
🇦🇩 Andorra1 sites
🇺🇬 Uganda1 sites
🇧🇫 Burkina Faso1 sites
🇬🇩 Grenada1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇧🇧 Barbados1 sites
🇧🇼 Botswana1 sites
🇵🇦 Panama1 sites
🇹🇿 Tanzania, United Republic of1 sites
🇭🇹 Haiti1 sites
🇱🇺 Luxembourg1 sites
🇸🇮 Slovenia1 sites
🇦🇴 Angola1 sites
🇹🇲 Turkmenistan1 sites
🇦🇲 Armenia1 sites
🇬🇪 Georgia1 sites
🇹🇳 Tunisia1 sites
🇷🇼 Rwanda1 sites
🇱🇮 Liechtenstein1 sites
🇩🇿 Algeria1 sites
🇲🇸 Montserrat1 sites

About Robots Disallow

Sites blocking the Robots crawler in robots.txt.

Categories
Robots.txtWeb Standards

Websites using Robots Disallow

showing 25 of 4,151
WebsiteCountryTrafficIndustry
sharethrough.com🇺🇸USTop 1KTechnology & Computing
fortinet.com🇺🇸USTop 5KCybersecurity
vtb.ru🇷🇺RUTop 5K-
arte.tv🇫🇷FRTop 5KMovies & TV
yclients.com🇨🇾CYTop 10KTechnology & Computing
lexisnexis.com🇺🇸USTop 10KTechnology & Computing
helsinki.fi🇫🇮FITop 10KEducation
spokeo.com🇺🇸USTop 10KTechnology & Computing
yoshki.com🇬🇧GBTop 50KTechnology & Computing
snapcomms.com🇳🇿NZTop 50KSoftware Development
display.io🇺🇸USTop 50KMarketing & Advertising
surfly.com🇳🇱NLTop 50KTechnology & Computing
spideroak.com🇺🇸USTop 50KSoftware Development
radioshack.com🇺🇸USTop 50KShopping
tekrom.com🇹🇷TRTop 50KTechnology & Computing
unimed.ac.id🇮🇩IDTop 50KEducation
ula.ve🇻🇪VETop 50KEducation
pulseway.com🇮🇪IETop 50KSoftware Development
21cnjy.com🇨🇳CNTop 50KSoftware Development
gitconnected.com🇺🇸USTop 50KTechnology & Computing
niftygateway.com🇺🇸USTop 50KEntertainment
powerdms.com🇺🇸USTop 50KSoftware Development
tib.eu🇩🇪DETop 50KBooks & Literature
tn.edu.tw🇹🇼TWTop 50KEducation
loadbee.com🇩🇪DETop 50KTechnology & Computing
Showing 25 of 4,151 websitesView detailed list →

Alternatives

same category
HTML5 DocType43.42MsitesFacebook Redirect16.88MsitesInstagram10.95MsitesOrganization Schema6.77MsitesLinkedIn5.8MsitesEvents Page4.12MsitesMembers Page3.53MsitesGoogle AJAX Feed API3.04Msites