Scrapy Disallow

Scrapy Disallow tracking and usage

Websites using Scrapy Disallow by countrymarket share
21%
🇺🇸US
17%
🇨🇦CA
8.3%
🇮🇳IN
4.2%
🇬🇧GB
4.2%
🇦🇺AU
4.2%
🇻🇳VN
4.2%
🇩🇪DE
4.2%
🇧🇲BM
Websites using Scrapy Disallow by industrymarket share · top 6 industries
News & Weather
25%
Entertainment
25%
Software Development
25%
Technology & Computing
8.3%
Medical & Healthcare
8.3%
Telecommunications
8.3%
0%6.3%13%19%25%
Download report

We track 24 websites using Scrapy Disallow, along with their country, traffic, industry and platforms. There are 5 websites using Scrapy Disallow in United States.

Filter by country
🇺🇸 United States5 sites
🇨🇦 Canada4 sites
🇮🇳 India2 sites
🇬🇧 United Kingdom1 sites
🇦🇺 Australia1 sites
🇻🇳 Viet Nam1 sites
🇩🇪 Germany1 sites
🇧🇲 Bermuda1 sites
🇰🇭 Cambodia1 sites
🇧🇷 Brazil1 sites
🇸🇦 Saudi Arabia1 sites

About Scrapy Disallow

This website has found the Disallow rules for Scrapy.

Category
Robots.txt
Official websitescrapy.org↗

Websites using Scrapy Disallow

showing 17 of 24
WebsiteCountryTrafficIndustry
telstra.com.au🇦🇺AUTop 10KTelecommunications
babbel.com🇩🇪DETop 50KEducation
yourparkingspace.co.uk🇬🇧GBTop 500KTechnology & Computing
drlogy.com🇮🇳INTop 500KMedical & Healthcare
secoda.co🇨🇦CATop 1MSoftware Development
web.facebook.com🇰🇭KH-Marketing & Advertising
axaxl.com🇧🇲BM-Insurance
eonline.com🇺🇸US-Entertainment
vagaro.com🇺🇸US-Software Development
aramcoimports.com🇺🇸US-Wholesale & Distribution
adventuresoverland.com🇮🇳IN-Travel
vietnam.vn🇻🇳VNTop 5K-
fragrantica.com.br🇧🇷BRTop 50K-
law.com🇺🇸USTop 50KNews & Weather
inkwellcomic.xyz---
amazon.sa🇸🇦SA--
yutsialbert.com---
Showing 17 of 24 websitesView detailed list →

Alternatives

same category
LLMS Text581sitesGPTBot Disallow225sitesByteDance Bytespider Disallow223sitesCommon Crawl Bot Disallow222sitesAnthropic Claude Bot Disallow204sitesAmazon Bot Disallow197sitesMeta External Agent Disallow193sitesGoogle-Extended Disallow192sites