Robots Disallow tracking and usage
Websites using Robots Disallow by countrymarket share
28%
🇺🇸US15%
🇩🇪DE3.5%
🇬🇧GB1.9%
🇳🇱NL1.7%
🇷🇺RU1.7%
🇮🇳IN1.6%
🇫🇷FR1.4%
🇧🇷BRWebsites using Robots Disallow by industrymarket share · top 6 industries
Shopping
Technology & Computing
Education
SaaS & Cloud
Law & Government
Software Development
0%7.5%15%23%30%
Download report
We track 15,302 websites using Robots Disallow, along with their country, traffic, industry and platforms. There are 4,213 websites using Robots Disallow in United States.
Filter by country
🇺🇸 United States4,213 sites
🇩🇪 Germany2,252 sites
🇬🇧 United Kingdom542 sites
🇳🇱 Netherlands295 sites
🇷🇺 Russian Federation267 sites
🇮🇳 India261 sites
🇫🇷 France247 sites
🇧🇷 Brazil215 sites
🇪🇸 Spain188 sites
🇨🇦 Canada186 sites
🇮🇹 Italy159 sites
🇨🇳 China153 sites
🇦🇺 Australia147 sites
🇨🇿 Czechia137 sites
🇸🇪 Sweden94 sites
🇧🇪 Belgium93 sites
🇳🇴 Norway87 sites
🇲🇽 Mexico82 sites
🇨🇭 Switzerland78 sites
🇵🇱 Poland66 sites
🇮🇩 Indonesia64 sites
🇰🇷 Korea, Republic of62 sites
🇫🇮 Finland57 sites
🇯🇵 Japan53 sites
🇹🇷 Turkey51 sites
🇳🇿 New Zealand43 sites
🇩🇰 Denmark42 sites
🇦🇪 United Arab Emirates40 sites
🇵🇹 Portugal39 sites
🇦🇹 Austria35 sites
🇿🇦 South Africa35 sites
🇨🇴 Colombia33 sites
🇮🇪 Ireland29 sites
🇷🇴 Romania28 sites
🇸🇬 Singapore28 sites
🇮🇷 Iran, Islamic Republic of28 sites
🇵🇭 Philippines28 sites
🇦🇷 Argentina23 sites
🇵🇰 Pakistan23 sites
🇨🇱 Chile22 sites
🇭🇰 Hong Kong22 sites
🇭🇺 Hungary22 sites
🇺🇦 Ukraine22 sites
🇹🇼 Taiwan, Province of China22 sites
🇬🇷 Greece20 sites
🇵🇪 Peru19 sites
🇸🇦 Saudi Arabia19 sites
🇻🇳 Viet Nam17 sites
🇮🇱 Israel16 sites
🇲🇾 Malaysia16 sites
🇧🇬 Bulgaria14 sites
🇪🇬 Egypt13 sites
🇱🇹 Lithuania12 sites
🇳🇬 Nigeria12 sites
🇸🇰 Slovakia11 sites
🇹🇭 Thailand10 sites
🇨🇾 Cyprus10 sites
🇪🇪 Estonia9 sites
🇧🇩 Bangladesh9 sites
🇵🇦 Panama8 sites
🇲🇦 Morocco8 sites
🇮🇸 Iceland7 sites
🇰🇪 Kenya7 sites
🇨🇲 Cameroon7 sites
🇦🇲 Armenia6 sites
🇱🇧 Lebanon6 sites
🇱🇺 Luxembourg6 sites
🇦🇿 Azerbaijan5 sites
🇱🇻 Latvia5 sites
🇻🇪 Venezuela, Bolivarian Republic of5 sites
🇱🇰 Sri Lanka5 sites
🇬🇭 Ghana5 sites
🇨🇷 Costa Rica4 sites
🇪🇨 Ecuador4 sites
🇺🇾 Uruguay4 sites
🇯🇴 Jordan4 sites
🇲🇹 Malta4 sites
🇶🇦 Qatar4 sites
🇸🇮 Slovenia4 sites
🇲🇩 Moldova, Republic of4 sites
🇰🇼 Kuwait4 sites
🇭🇷 Croatia4 sites
🇷🇸 Serbia4 sites
🇧🇾 Belarus4 sites
🇦🇱 Albania3 sites
🇩🇿 Algeria3 sites
🇬🇪 Georgia3 sites
🇻🇨 Saint Vincent and the Grenadines3 sites
🇧🇿 Belize3 sites
🇲🇺 Mauritius3 sites
🇧🇭 Bahrain2 sites
🇮🇶 Iraq2 sites
🇯🇲 Jamaica2 sites
🇻🇬 Virgin Islands, British2 sites
🇸🇷 Suriname2 sites
🇹🇹 Trinidad and Tobago2 sites
🇺🇬 Uganda2 sites
🇧🇴 Bolivia, Plurinational State of2 sites
🇸🇨 Seychelles2 sites
🇿🇼 Zimbabwe2 sites
🇩🇴 Dominican Republic2 sites
🇱🇮 Liechtenstein2 sites
🇺🇿 Uzbekistan2 sites
🇰🇿 Kazakhstan2 sites
🇳🇵 Nepal2 sites
🇮🇲 Isle of Man2 sites
🇭🇳 Honduras2 sites
🇱🇾 Libya2 sites
🇬🇹 Guatemala2 sites
🇵🇾 Paraguay2 sites
🇸🇴 Somalia2 sites
🇨🇩 Congo, The Democratic Republic of the2 sites
🇦🇴 Angola2 sites
🇦🇼 Aruba1 sites
🇱🇦 Lao People's Democratic Republic1 sites
🇩🇯 Djibouti1 sites
🇰🇬 Kyrgyzstan1 sites
🇦🇸 American Samoa1 sites
🇱🇨 Saint Lucia1 sites
🇸🇻 El Salvador1 sites
🇸🇾 Syrian Arab Republic1 sites
🇹🇬 Togo1 sites
🇿🇲 Zambia1 sites
🇬🇦 Gabon1 sites
🇦🇬 Antigua and Barbuda1 sites
🇸🇩 Sudan1 sites
🇸🇿 Eswatini1 sites
🇸🇳 Senegal1 sites
🇦🇩 Andorra1 sites
🇧🇫 Burkina Faso1 sites
🇬🇩 Grenada1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇧🇧 Barbados1 sites
🇧🇼 Botswana1 sites
🇹🇿 Tanzania, United Republic of1 sites
🇭🇹 Haiti1 sites
🇹🇲 Turkmenistan1 sites
🇹🇳 Tunisia1 sites
🇷🇼 Rwanda1 sites
🇲🇪 Montenegro1 sites
🇲🇵 Northern Mariana Islands1 sites
🇲🇿 Mozambique1 sites
🇲🇸 Montserrat1 sites
🇷🇪 Réunion1 sites
🇲🇨 Monaco1 sites
🇲🇰 North Macedonia1 sites
🇳🇦 Namibia1 sites
🇴🇲 Oman1 sites
🇵🇷 Puerto Rico1 sites
About Robots Disallow
Sites blocking the Robots crawler in robots.txt.
Categories
Websites using Robots Disallow
showing 25 of 15,302| Website | Country | Traffic | Industry |
|---|---|---|---|
| 🇺🇸US | Top 1K | Technology & Computing | |
| 🇺🇸US | Top 5K | Cybersecurity | |
| 🇷🇺RU | Top 5K | - | |
| 🇫🇷FR | Top 5K | Movies & TV | |
| 🇨🇾CY | Top 10K | Technology & Computing | |
| 🇺🇸US | Top 10K | Technology & Computing | |
| 🇫🇮FI | Top 10K | Education | |
| 🇺🇸US | Top 10K | Technology & Computing | |
| 🇬🇧GB | Top 50K | Technology & Computing | |
| 🇳🇿NZ | Top 50K | Software Development | |
| 🇺🇸US | Top 50K | Marketing & Advertising | |
| 🇳🇱NL | Top 50K | Technology & Computing | |
| 🇺🇸US | Top 50K | Software Development | |
| 🇺🇸US | Top 50K | Shopping | |
| 🇹🇷TR | Top 50K | Technology & Computing | |
| 🇮🇩ID | Top 50K | Education | |
| 🇻🇪VE | Top 50K | Education | |
| 🇮🇪IE | Top 50K | Software Development | |
| 🇨🇳CN | Top 50K | Software Development | |
| 🇺🇸US | Top 50K | Technology & Computing | |
| 🇺🇸US | Top 50K | Entertainment | |
| 🇺🇸US | Top 50K | Software Development | |
| 🇩🇪DE | Top 50K | Books & Literature | |
| 🇹🇼TW | Top 50K | Education | |
| 🇩🇪DE | Top 50K | Technology & Computing |
Showing 25 of 15,302 websitesView detailed list →
