Nutch-based Bot Disallow

Nutch-based Bot Disallow tracking and usage

Websites using Nutch-based Bot Disallow by countrymarket share
44%
🇺🇸US
8.8%
🇩🇪DE
4.4%
🇬🇧GB
3.2%
🇫🇷FR
3.1%
🇦🇺AU
3%
🇦🇷AR
3%
🇧🇷BR
2.7%
🇮🇳IN
Websites using Nutch-based Bot Disallow by industrymarket share · top 6 industries
Shopping
57%
Fashion & Beauty
20%
Food & Drink
6.8%
Technology & Computing
6.1%
Health & Fitness
5.1%
Medical & Healthcare
5%
0%15%30%45%60%
Download report

We track 22,836 websites using Nutch-based Bot Disallow, along with their country, traffic, industry and platforms. There are 10,004 websites using Nutch-based Bot Disallow in United States.

Filter by country
🇺🇸 United States10,004 sites
🇩🇪 Germany2,013 sites
🇬🇧 United Kingdom1,010 sites
🇫🇷 France733 sites
🇦🇺 Australia715 sites
🇦🇷 Argentina690 sites
🇧🇷 Brazil679 sites
🇮🇳 India619 sites
🇨🇦 Canada568 sites
🇪🇸 Spain450 sites
🇻🇳 Viet Nam380 sites
🇹🇷 Turkey339 sites
🇮🇹 Italy316 sites
🇳🇱 Netherlands300 sites
🇵🇱 Poland242 sites
🇨🇭 Switzerland122 sites
🇳🇿 New Zealand99 sites
🇲🇾 Malaysia94 sites
🇷🇺 Russian Federation91 sites
🇧🇪 Belgium84 sites
🇸🇬 Singapore78 sites
🇸🇪 Sweden76 sites
🇲🇽 Mexico74 sites
🇦🇪 United Arab Emirates73 sites
🇨🇳 China72 sites
🇩🇰 Denmark72 sites
🇯🇵 Japan51 sites
🇦🇹 Austria46 sites
🇨🇱 Chile45 sites
🇨🇴 Colombia44 sites
🇮🇪 Ireland42 sites
🇵🇰 Pakistan40 sites
🇷🇴 Romania37 sites
🇫🇮 Finland36 sites
🇮🇱 Israel36 sites
🇿🇦 South Africa36 sites
🇵🇹 Portugal32 sites
🇭🇰 Hong Kong31 sites
🇳🇴 Norway29 sites
🇸🇦 Saudi Arabia29 sites
🇬🇷 Greece26 sites
🇨🇿 Czechia24 sites
🇹🇭 Thailand23 sites
🇷🇼 Rwanda23 sites
🇮🇩 Indonesia20 sites
🇭🇺 Hungary19 sites
🇹🇼 Taiwan, Province of China19 sites
🇵🇭 Philippines15 sites
🇪🇪 Estonia13 sites
🇱🇹 Lithuania12 sites
🇺🇦 Ukraine12 sites
🇧🇬 Bulgaria10 sites
🇱🇻 Latvia10 sites
🇵🇪 Peru8 sites
🇪🇬 Egypt8 sites
🇸🇰 Slovakia7 sites
🇱🇧 Lebanon7 sites
🇵🇷 Puerto Rico6 sites
🇰🇷 Korea, Republic of6 sites
🇧🇩 Bangladesh6 sites
🇶🇦 Qatar5 sites
🇰🇼 Kuwait5 sites
🇮🇷 Iran, Islamic Republic of5 sites
🇲🇦 Morocco4 sites
🇲🇹 Malta4 sites
🇵🇦 Panama4 sites
🇷🇸 Serbia4 sites
🇦🇿 Azerbaijan4 sites
🇮🇸 Iceland4 sites
🇸🇮 Slovenia4 sites
🇩🇴 Dominican Republic4 sites
🇪🇨 Ecuador3 sites
🇰🇿 Kazakhstan3 sites
🇱🇺 Luxembourg3 sites
🇭🇷 Croatia3 sites
🇰🇪 Kenya3 sites
🇻🇨 Saint Vincent and the Grenadines3 sites
🇺🇾 Uruguay3 sites
🇯🇴 Jordan2 sites
🇧🇭 Bahrain2 sites
🇴🇲 Oman2 sites
🇸🇴 Somalia2 sites
🇸🇳 Senegal2 sites
🇧🇾 Belarus2 sites
🇦🇩 Andorra1 sites
🇧🇦 Bosnia and Herzegovina1 sites
🇮🇲 Isle of Man1 sites
🇱🇰 Sri Lanka1 sites
🇹🇹 Trinidad and Tobago1 sites
🇦🇲 Armenia1 sites
🇷🇪 Réunion1 sites
🇸🇯 Svalbard and Jan Mayen1 sites
🇨🇷 Costa Rica1 sites
🇰🇾 Cayman Islands1 sites
🇨🇩 Congo, The Democratic Republic of the1 sites
🇮🇶 Iraq1 sites
🇯🇲 Jamaica1 sites
🇲🇩 Moldova, Republic of1 sites
🇻🇪 Venezuela, Bolivarian Republic of1 sites
🇦🇫 Afghanistan1 sites
🇨🇾 Cyprus1 sites
🇭🇳 Honduras1 sites
🇹🇳 Tunisia1 sites
🇯🇪 Jersey1 sites
🇻🇮 Virgin Islands, U.S.1 sites
🇧🇲 Bermuda1 sites
🇬🇪 Georgia1 sites
🇦🇬 Antigua and Barbuda1 sites
🇲🇨 Monaco1 sites
🇧🇸 Bahamas1 sites
🇱🇾 Libya1 sites
🇩🇿 Algeria1 sites
🇳🇬 Nigeria1 sites
🇵🇾 Paraguay1 sites
🇬🇹 Guatemala1 sites
🇹🇬 Togo1 sites
🇻🇬 Virgin Islands, British1 sites
🇩🇲 Dominica1 sites
🇱🇷 Liberia1 sites
🇱🇮 Liechtenstein1 sites
🇲🇰 North Macedonia1 sites

About Nutch-based Bot Disallow

This website has found the Disallow rules for Nutch-based Bot.

Categories
Robots.txtWeb Standards
Official websitenutch.apache.org↗

Websites using Nutch-based Bot Disallow

showing 25 of 22,836
WebsiteCountryTrafficIndustry
sagepub.com🇺🇸USTop 1KPrint & Publishing
theverge.com🇺🇸USTop 1KNews & Weather
figma.com🇺🇸USTop 1KDesign & Creative
python.org🇺🇸USTop 1KSoftware Development
repubblica.it🇮🇹ITTop 1KNews & Weather
elpais.com🇪🇸ESTop 1KNews & Weather
nintendo.com🇺🇸USTop 1KGaming
independent.co.uk🇬🇧GBTop 1KEntertainment
aol.com🇺🇸USTop 1KSoftware Development
cnet.com🇺🇸USTop 1KEntertainment
wallapop.com🇪🇸ESTop 5KTechnology & Computing
gazeta.pl🇵🇱PLTop 5KMarketing & Advertising
alltrails.com🇺🇸USTop 5KTechnology & Computing
adweek.com🇺🇸USTop 5KNews & Weather
marca.com🇪🇸ESTop 5KNews & Weather
vox.com🇺🇸USTop 5KNews & Weather
nanit.com🇺🇸USTop 5KHardware & Electronics
aol.de🇩🇪DETop 5KTechnology & Computing
mos.ru🇷🇺RUTop 5KLaw & Government
everydayhealth.com🇺🇸USTop 5KTechnology & Computing
ilmeteo.it🇮🇹ITTop 5KTechnology & Computing
express.co.uk🇬🇧GBTop 5KNews & Weather
pcmag.com🇺🇸USTop 5KPrint & Publishing
timeout.com🇬🇧GBTop 5KSoftware Development
ign.com🇺🇸USTop 5KNews & Weather
Showing 25 of 22,836 websitesView detailed list →

Alternatives

same category
HTML5 DocType43.67MsitesFacebook Redirect17.12MsitesInstagram11.09MsitesOrganization Schema6.85MsitesLinkedIn5.86MsitesEvents Page4.18MsitesUTF-83.51MsitesMembers Page3.5Msites