Robots Disallow tracking and usage
Websites using Robots Disallow by countrymarket share
17%
🇺🇸US12%
🇩🇪DE3.9%
🇬🇧GB3.6%
🇷🇺RU2.6%
🇳🇱NL2.6%
🇮🇳IN2.2%
🇧🇷BR1.9%
🇫🇷FRWebsites using Robots Disallow by industrymarket share · top 6 industries
Shopping
Technology & Computing
Education
SaaS & Cloud
Marketing & Advertising
Software Development
0%7.5%15%23%30%
Download report
We track 41,041 websites using Robots Disallow, along with their country, traffic, industry and platforms. There are 6,780 websites using Robots Disallow in United States.
Filter by country
🇺🇸 United States6,780 sites
🇩🇪 Germany4,821 sites
🇬🇧 United Kingdom1,597 sites
🇷🇺 Russian Federation1,489 sites
🇳🇱 Netherlands1,074 sites
🇮🇳 India1,057 sites
🇧🇷 Brazil907 sites
🇫🇷 France789 sites
🇯🇵 Japan786 sites
🇨🇿 Czechia763 sites
🇦🇺 Australia759 sites
🇨🇦 Canada709 sites
🇨🇳 China591 sites
🇪🇸 Spain549 sites
🇮🇹 Italy478 sites
🇸🇪 Sweden427 sites
🇨🇭 Switzerland398 sites
🇳🇴 Norway288 sites
🇰🇷 Korea, Republic of284 sites
🇧🇪 Belgium277 sites
🇵🇱 Poland265 sites
🇲🇽 Mexico243 sites
🇮🇩 Indonesia229 sites
🇵🇭 Philippines216 sites
🇦🇪 United Arab Emirates193 sites
🇹🇷 Turkey178 sites
🇦🇹 Austria164 sites
🇫🇮 Finland164 sites
🇺🇦 Ukraine150 sites
🇩🇰 Denmark147 sites
🇳🇿 New Zealand129 sites
🇵🇹 Portugal113 sites
🇮🇷 Iran, Islamic Republic of107 sites
🇭🇺 Hungary102 sites
🇸🇬 Singapore94 sites
🇿🇦 South Africa91 sites
🇵🇰 Pakistan89 sites
🇦🇷 Argentina85 sites
🇨🇱 Chile78 sites
🇨🇴 Colombia76 sites
🇭🇰 Hong Kong74 sites
🇮🇪 Ireland74 sites
🇷🇴 Romania70 sites
🇬🇷 Greece68 sites
🇲🇾 Malaysia63 sites
🇹🇼 Taiwan, Province of China61 sites
🇸🇦 Saudi Arabia60 sites
🇵🇪 Peru53 sites
🇪🇬 Egypt52 sites
🇻🇳 Viet Nam48 sites
🇧🇩 Bangladesh46 sites
🇮🇱 Israel43 sites
🇳🇬 Nigeria40 sites
🇸🇰 Slovakia39 sites
🇧🇬 Bulgaria37 sites
🇧🇾 Belarus37 sites
🇱🇹 Lithuania36 sites
🇪🇪 Estonia35 sites
🇹🇭 Thailand33 sites
🇱🇧 Lebanon28 sites
🇰🇪 Kenya26 sites
🇰🇿 Kazakhstan24 sites
🇲🇦 Morocco23 sites
🇨🇾 Cyprus23 sites
🇮🇸 Iceland22 sites
🇱🇰 Sri Lanka22 sites
🇺🇾 Uruguay20 sites
🇷🇸 Serbia20 sites
🇭🇷 Croatia20 sites
🇱🇻 Latvia19 sites
🇱🇺 Luxembourg19 sites
🇱🇮 Liechtenstein19 sites
🇬🇭 Ghana18 sites
🇶🇦 Qatar17 sites
🇸🇮 Slovenia17 sites
🇦🇿 Azerbaijan15 sites
🇨🇷 Costa Rica15 sites
🇪🇨 Ecuador15 sites
🇦🇱 Albania14 sites
🇵🇦 Panama14 sites
🇰🇼 Kuwait14 sites
🇺🇿 Uzbekistan14 sites
🇴🇲 Oman14 sites
🇻🇪 Venezuela, Bolivarian Republic of13 sites
🇨🇲 Cameroon13 sites
🇲🇩 Moldova, Republic of12 sites
🇬🇪 Georgia12 sites
🇵🇷 Puerto Rico12 sites
🇳🇵 Nepal12 sites
🇸🇴 Somalia12 sites
🇻🇨 Saint Vincent and the Grenadines12 sites
🇸🇳 Senegal12 sites
🇩🇴 Dominican Republic11 sites
🇦🇲 Armenia10 sites
🇧🇭 Bahrain10 sites
🇬🇹 Guatemala10 sites
🇩🇿 Algeria9 sites
🇹🇿 Tanzania, United Republic of9 sites
🇮🇶 Iraq8 sites
🇰🇬 Kyrgyzstan8 sites
🇯🇴 Jordan8 sites
🇺🇬 Uganda8 sites
🇹🇳 Tunisia8 sites
🇧🇿 Belize8 sites
🇲🇹 Malta7 sites
🇱🇨 Saint Lucia7 sites
🇲🇳 Mongolia7 sites
🇮🇲 Isle of Man7 sites
🇱🇾 Libya7 sites
🇸🇨 Seychelles6 sites
🇧🇦 Bosnia and Herzegovina6 sites
🇬🇦 Gabon6 sites
🇷🇼 Rwanda6 sites
🇲🇨 Monaco6 sites
🇯🇲 Jamaica5 sites
🇹🇹 Trinidad and Tobago5 sites
🇦🇸 American Samoa5 sites
🇿🇼 Zimbabwe5 sites
🇵🇾 Paraguay5 sites
🇵🇸 Palestine, State of5 sites
🇹🇯 Tajikistan5 sites
🇰🇭 Cambodia5 sites
🇲🇺 Mauritius5 sites
🇭🇳 Honduras5 sites
🇨🇩 Congo, The Democratic Republic of the5 sites
🇬🇱 Greenland5 sites
🇷🇪 Réunion5 sites
🇵🇲 Saint Pierre and Miquelon5 sites
🇦🇼 Aruba4 sites
🇧🇴 Bolivia, Plurinational State of4 sites
🇲🇲 Myanmar4 sites
🇧🇧 Barbados4 sites
🇪🇹 Ethiopia4 sites
🇨🇮 Côte d'Ivoire4 sites
🇲🇸 Montserrat4 sites
🇵🇼 Palau4 sites
🇲🇻 Maldives4 sites
🇲🇰 North Macedonia4 sites
🇧🇯 Benin4 sites
🇻🇬 Virgin Islands, British3 sites
🇸🇷 Suriname3 sites
🇹🇬 Togo3 sites
🇬🇮 Gibraltar3 sites
🇲🇬 Madagascar3 sites
🇽🇰 Kosovo3 sites
🇳🇦 Namibia3 sites
🇦🇬 Antigua and Barbuda3 sites
🇧🇫 Burkina Faso3 sites
🇭🇹 Haiti3 sites
🇦🇴 Angola3 sites
🇨🇽 Christmas Island3 sites
🇲🇿 Mozambique3 sites
🇬🇸 South Georgia and the South Sandwich Islands3 sites
🇱🇦 Lao People's Democratic Republic2 sites
🇸🇻 El Salvador2 sites
🇸🇾 Syrian Arab Republic2 sites
🇿🇲 Zambia2 sites
🇧🇳 Brunei Darussalam2 sites
🇻🇮 Virgin Islands, U.S.2 sites
🇦🇫 Afghanistan2 sites
🇯🇪 Jersey2 sites
🇫🇯 Fiji2 sites
🇸🇩 Sudan2 sites
🇧🇼 Botswana2 sites
🇹🇨 Turks and Caicos Islands2 sites
🇨🇬 Congo2 sites
🇲🇱 Mali2 sites
🇩🇯 Djibouti1 sites
🇲🇼 Malawi1 sites
🇨🇼 Curaçao1 sites
🇲🇶 Martinique1 sites
🇸🇿 Eswatini1 sites
🇦🇩 Andorra1 sites
🇬🇩 Grenada1 sites
🇰🇳 Saint Kitts and Nevis1 sites
🇹🇲 Turkmenistan1 sites
🇧🇲 Bermuda1 sites
🇲🇪 Montenegro1 sites
🇲🇵 Northern Mariana Islands1 sites
🇾🇹 Mayotte1 sites
🇳🇪 Niger1 sites
🇸🇱 Sierra Leone1 sites
🇻🇺 Vanuatu1 sites
🇩🇲 Dominica1 sites
🇬🇾 Guyana1 sites
🇵🇫 French Polynesia1 sites
🇸🇧 Solomon Islands1 sites
🇹🇫 French Southern Territories1 sites
🇬🇵 Guadeloupe1 sites
🇲🇴 Macao1 sites
🇸🇽 Sint Maarten (Dutch part)1 sites
🇧🇮 Burundi1 sites
🇧🇸 Bahamas1 sites
🇰🇲 Comoros1 sites
🇬🇲 Gambia1 sites
🇬🇼 Guinea-Bissau1 sites
🇱🇷 Liberia1 sites
🇹🇩 Chad1 sites
🇾🇪 Yemen1 sites
About Robots Disallow
Sites blocking the Robots crawler in robots.txt.
Categories
Websites using Robots Disallow
showing 25 of 41,041| Website | Country | Traffic | Industry |
|---|---|---|---|
| 🇺🇸US | Top 1K | Shopping | |
| 🇺🇸US | Top 1K | Technology & Computing | |
| 🇺🇸US | Top 1K | Technology & Computing | |
| 🇲🇸MS | Top 5K | Society | |
| 🇨🇿CZ | Top 5K | Science | |
| 🇳🇴NO | Top 5K | Legal Services | |
| 🇺🇸US | Top 5K | Software Development | |
| 🇨🇳CN | Top 5K | Technology & Computing | |
| 🇳🇱NL | Top 5K | Law & Government | |
| 🇨🇳CN | Top 5K | Software Development | |
| 🇪🇸ES | Top 5K | Law & Government | |
| - | Top 5K | Technology & Computing | |
| 🇺🇸US | Top 5K | Cybersecurity | |
| 🇷🇺RU | Top 5K | - | |
| 🇫🇷FR | Top 5K | Movies & TV | |
| 🇨🇳CN | Top 10K | Software Development | |
| 🇩🇪DE | Top 10K | Telecommunications | |
| 🇮🇳IN | Top 10K | Technology & Computing | |
| 🇦🇺AU | Top 10K | Shopping | |
| 🇨🇳CN | Top 10K | - | |
| 🇨🇾CY | Top 10K | Technology & Computing | |
| 🇺🇸US | Top 10K | Technology & Computing | |
| 🇫🇮FI | Top 10K | Education | |
| 🇺🇸US | Top 10K | Technology & Computing | |
| 🇺🇸US | Top 50K | - |
Showing 25 of 41,041 websitesView detailed list →
