Nutch-based Bot Disallow

Nutch-based Bot Disallow tracking and usage

Websites using Nutch-based Bot Disallow by countrymarket share
30%
🇺🇸US
7.1%
🇩🇪DE
5.5%
🇦🇺AU
5%
🇬🇧GB
4.7%
🇧🇷BR
4%
🇦🇷AR
4%
🇫🇷FR
2.9%
🇨🇦CA
Websites using Nutch-based Bot Disallow by industrymarket share · top 6 industries
Shopping
59%
Fashion & Beauty
18%
Food & Drink
6.6%
SaaS & Cloud
6.2%
Technology & Computing
5.8%
Health & Fitness
4.4%
0%15%30%45%60%
Download report

We track 36,254 websites using Nutch-based Bot Disallow, along with their country, traffic, industry and platforms. There are 11,049 websites using Nutch-based Bot Disallow in United States.

Filter by country
🇺🇸 United States11,049 sites
🇩🇪 Germany2,570 sites
🇦🇺 Australia1,988 sites
🇬🇧 United Kingdom1,817 sites
🇧🇷 Brazil1,701 sites
🇦🇷 Argentina1,446 sites
🇫🇷 France1,445 sites
🇨🇦 Canada1,040 sites
🇮🇳 India982 sites
🇪🇸 Spain891 sites
🇳🇱 Netherlands710 sites
🇮🇹 Italy646 sites
🇻🇳 Viet Nam640 sites
🇹🇷 Turkey598 sites
🇵🇱 Poland596 sites
🇷🇺 Russian Federation413 sites
🇨🇭 Switzerland295 sites
🇳🇿 New Zealand184 sites
🇸🇪 Sweden179 sites
🇸🇬 Singapore157 sites
🇧🇪 Belgium150 sites
🇩🇰 Denmark144 sites
🇲🇾 Malaysia142 sites
🇲🇽 Mexico133 sites
🇦🇪 United Arab Emirates131 sites
🇯🇵 Japan128 sites
🇦🇹 Austria112 sites
🇫🇮 Finland111 sites
🇨🇳 China104 sites
🇨🇿 Czechia97 sites
🇸🇦 Saudi Arabia95 sites
🇵🇹 Portugal94 sites
🇨🇴 Colombia91 sites
🇮🇪 Ireland73 sites
🇨🇱 Chile72 sites
🇵🇰 Pakistan72 sites
🇿🇦 South Africa71 sites
🇮🇩 Indonesia68 sites
🇷🇼 Rwanda67 sites
🇷🇴 Romania64 sites
🇭🇺 Hungary64 sites
🇬🇷 Greece61 sites
🇭🇰 Hong Kong60 sites
🇳🇴 Norway60 sites
🇪🇪 Estonia59 sites
🇺🇦 Ukraine56 sites
🇮🇱 Israel52 sites
🇹🇭 Thailand41 sites
🇹🇼 Taiwan, Province of China32 sites
🇱🇻 Latvia28 sites
🇰🇷 Korea, Republic of27 sites
🇵🇭 Philippines26 sites
🇮🇷 Iran, Islamic Republic of26 sites
🇧🇬 Bulgaria24 sites
🇱🇹 Lithuania24 sites
🇸🇰 Slovakia21 sites
🇸🇮 Slovenia14 sites
🇧🇾 Belarus12 sites
🇵🇪 Peru11 sites
🇶🇦 Qatar11 sites
🇱🇧 Lebanon11 sites
🇭🇷 Croatia11 sites
🇷🇸 Serbia10 sites
🇪🇬 Egypt10 sites
🇧🇩 Bangladesh10 sites
🇲🇦 Morocco9 sites
🇲🇹 Malta9 sites
🇮🇸 Iceland8 sites
🇰🇿 Kazakhstan7 sites
🇰🇼 Kuwait7 sites
🇩🇴 Dominican Republic7 sites
🇰🇪 Kenya7 sites
🇬🇪 Georgia7 sites
🇺🇾 Uruguay7 sites
🇧🇦 Bosnia and Herzegovina6 sites
🇵🇷 Puerto Rico6 sites
🇴🇲 Oman6 sites
🇻🇨 Saint Vincent and the Grenadines6 sites
🇪🇨 Ecuador5 sites
🇱🇺 Luxembourg5 sites
🇵🇦 Panama5 sites
🇱🇰 Sri Lanka4 sites
🇦🇿 Azerbaijan4 sites
🇧🇭 Bahrain4 sites
🇲🇰 North Macedonia4 sites
🇦🇩 Andorra3 sites
🇯🇴 Jordan3 sites
🇲🇩 Moldova, Republic of3 sites
🇦🇲 Armenia3 sites
🇻🇪 Venezuela, Bolivarian Republic of3 sites
🇨🇾 Cyprus3 sites
🇵🇾 Paraguay3 sites
🇦🇬 Antigua and Barbuda3 sites
🇱🇮 Liechtenstein3 sites
🇮🇶 Iraq2 sites
🇯🇲 Jamaica2 sites
🇭🇳 Honduras2 sites
🇸🇴 Somalia2 sites
🇲🇨 Monaco2 sites
🇹🇳 Tunisia2 sites
🇳🇬 Nigeria2 sites
🇳🇦 Namibia2 sites
🇻🇮 Virgin Islands, U.S.2 sites
🇸🇳 Senegal2 sites
🇧🇲 Bermuda2 sites
🇬🇹 Guatemala2 sites
🇻🇬 Virgin Islands, British2 sites
🇧🇿 Belize2 sites
🇨🇼 Curaçao2 sites
🇨🇽 Christmas Island2 sites
🇲🇲 Myanmar2 sites
🇮🇲 Isle of Man1 sites
🇹🇹 Trinidad and Tobago1 sites
🇷🇪 Réunion1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇨🇷 Costa Rica1 sites
🇰🇾 Cayman Islands1 sites
🇨🇩 Congo, The Democratic Republic of the1 sites
🇦🇫 Afghanistan1 sites
🇯🇪 Jersey1 sites
🇵🇸 Palestine, State of1 sites
🇿🇲 Zambia1 sites
🇧🇧 Barbados1 sites
🇬🇬 Guernsey1 sites
🇧🇸 Bahamas1 sites
🇱🇾 Libya1 sites
🇩🇿 Algeria1 sites
🇹🇬 Togo1 sites
🇦🇸 American Samoa1 sites
🇧🇴 Bolivia, Plurinational State of1 sites
🇩🇲 Dominica1 sites
🇬🇱 Greenland1 sites
🇰🇬 Kyrgyzstan1 sites
🇱🇷 Liberia1 sites
🇵🇼 Palau1 sites
🇸🇲 San Marino1 sites
🇸🇽 Sint Maarten (Dutch part)1 sites
🇧🇯 Benin1 sites
🇨🇲 Cameroon1 sites
🇳🇵 Nepal1 sites
🇲🇿 Mozambique1 sites
🇹🇨 Turks and Caicos Islands1 sites
🇨🇮 Côte d'Ivoire1 sites
🇰🇭 Cambodia1 sites
🇹🇿 Tanzania, United Republic of1 sites

About Nutch-based Bot Disallow

This website has found the Disallow rules for Nutch-based Bot.

Categories
Robots.txtWeb Standards
Official websitenutch.apache.org↗

Websites using Nutch-based Bot Disallow

showing 25 of 36,254
WebsiteCountryTrafficIndustry
slate.com🇺🇸USTop 1KPrint & Publishing
sagepub.com🇺🇸USTop 1KPrint & Publishing
theverge.com🇺🇸USTop 1KNews & Weather
figma.com🇺🇸USTop 1KDesign & Creative
python.org🇺🇸USTop 1KSoftware Development
repubblica.it🇮🇹ITTop 1KNews & Weather
elpais.com🇪🇸ESTop 1KNews & Weather
nintendo.com🇺🇸USTop 1KGaming
independent.co.uk🇬🇧GBTop 1KEntertainment
aol.com🇺🇸USTop 1KSoftware Development
cnet.com🇺🇸USTop 1KEntertainment
rfi.fr🇫🇷FRTop 5KNews & Weather
sbs.com.au🇦🇺AUTop 5KMovies & TV
yomiuri.co.jp🇯🇵JPTop 5KNews & Weather
dejure.org🇩🇪DETop 5KTechnology & Computing
news24.com-Top 5KMovies & TV
lastampa.it🇮🇹ITTop 5KNews & Weather
20minutes.fr🇫🇷FRTop 5KNews & Weather
screenrant.com🇨🇦CATop 5KEntertainment
thrillist.com🇺🇸USTop 5KNews & Weather
bhf.org.uk🇬🇧GBTop 5KNon-profit & Charity
dukeupress.edu🇺🇸USTop 5KEducation
sbnation.com🇺🇸USTop 5KNews & Weather
thecut.com🇺🇸USTop 5KNews & Weather
elmundo.es🇪🇸ESTop 5KNews & Weather
Showing 25 of 36,254 websitesView detailed list →

Alternatives

same category
HTML5 DocType45.32MsitesFacebook Redirect17.91MsitesInstagram11.7MsitesOrganization Schema7.24MsitesLinkedIn6.25MsitesUTF-86.01MsitesEvents Page4.51MsitesAPI or Developer Page3.76Msites