Read-only technical reference - reviewed July 2026

Anti-Bot Protection Registry

A technical reference of the anti-bot protection observed on 94 popular websites across 15 verticals, and what it takes to collect their public data legitimately.

This registry documents the anti-bot and anti-scraping technologies observed on 94 popular websites across 15 verticals - from CDN-level challenges and TLS fingerprinting to behavioral analysis and IP reputation scoring. Each entry describes the defense at the mechanism level and, at a high level, the class of IP infrastructure that legitimate large-scale data collection tends to require. It is a read-only technical reference, not a bypass guide: always respect robots.txt, a site's Terms of Service, and applicable law, and collect only data you are permitted to collect.

94 sites15 verticals10 protection technologies57 hard targetsOpen dataset

Browse the registry

Search by site or domain, filter by category, technology or difficulty, and sort any column. Every row links to a full breakdown.

Difficulty
94 of 94 sites
Allegro
allegro.pl
Online Marketplaces
DataDome
Hard
Capterra
capterra.com
Reviews & Ratings
Cloudflare (Bot/Challenge)
Hard
Cars.com
cars.com
Automotive
Cloudflare (Bot/Challenge)
Hard
Carvana
carvana.com
Automotive
Cloudflare (Bot/Challenge)
Hard
Cdiscount
cdiscount.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
Chewy
chewy.com
Retail & E-commerce
KasadaAkamai (CDN/WAF)
Hard
Currys
currys.co.uk
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
Depop
depop.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
END. Clothing
endclothing.com
Sneakers & Limited Drops
Akamai Bot Manager
Hard
Etsy
etsy.com
Online Marketplaces
DataDome
Hard
Expedia
expedia.com
Travel & Hospitality
Akamai Bot Manager
Hard
Fnac
fnac.com
Retail & E-commerce
DataDomeAkamai (CDN/WAF)
Hard
G2
g2.com
Reviews & Ratings
DataDomeCloudflare (Bot/Challenge)
Hard
Gap
gap.com
Retail & E-commerce
Akamai Bot Manager
Hard
Glassdoor
glassdoor.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
GOAT
goat.com
Sneakers & Limited Drops
Cloudflare (Bot/Challenge)
Hard
Home Depot
homedepot.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
Hotels.com
hotels.com
Travel & Hospitality
Akamai Bot Manager
Hard
Idealista
idealista.com
Real Estate & Property Listings
DataDome
Hard
Indeed
indeed.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
Leboncoin
leboncoin.fr
Classifieds
DataDome
Hard
LinkedIn
linkedin.com
Social Platforms
PerimeterX / HUMANCloudflare (CDN/WAF)reCAPTCHA
Hard
MediaMarkt
mediamarkt.de
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
Mercari
mercari.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
Nike
nike.com
Retail & E-commerce
KasadaAkamai (CDN/WAF)
Hard
Otto
otto.de
Online Marketplaces
Kasada
Hard
Overstock
overstock.com
Retail & E-commerce
Akamai Bot Manager
Hard
Petco
petco.com
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
Priceline
priceline.com
Travel & Hospitality
PerimeterX / HUMANCloudflare (CDN/WAF)
Hard
QVC
qvc.com
Retail & E-commerce
Akamai Bot Manager
Hard
realestate.com.au
realestate.com.au
Real Estate & Property Listings
KasadaAkamai (CDN/WAF)
Hard
Realtor.com
realtor.com
Real Estate & Property Listings
Kasada
Hard
Resy
resy.com
Reservations & Dining
Imperva (Incapsula)reCAPTCHA
Hard
Sam's Club
samsclub.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
SeatGeek
seatgeek.com
Tickets & Live Events
DataDome
Hard
Sephora
sephora.com
Retail & E-commerce
Akamai Bot Manager
Hard
Skillshare
skillshare.com
Online Learning & Other
Cloudflare (Bot/Challenge)
Hard
Skyscanner
skyscanner.net
Travel & Hospitality
PerimeterX / HUMAN
Hard
Southwest Airlines
southwest.com
Travel & Hospitality
Akamai Bot Manager
Hard
SSENSE
ssense.com
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
StockX
stockx.com
Sneakers & Limited Drops
PerimeterX / HUMANCloudflare (Bot/Challenge)
Hard
Target
target.com
Retail & E-commerce
PerimeterX / HUMAN
Hard
Trivago
trivago.com
Travel & Hospitality
Akamai Bot Manager
Hard
Trulia
trulia.com
Real Estate & Property Listings
PerimeterX / HUMAN
Hard
Uber Eats
ubereats.com
Food Delivery
Cloudflare (Bot/Challenge)
Hard
Udemy
udemy.com
Online Learning & Other
Cloudflare (Bot/Challenge)
Hard
Uniqlo
uniqlo.com
Retail & E-commerce
Akamai Bot Manager
Hard
Vivid Seats
vividseats.com
Tickets & Live Events
Akamai Bot Manager
Hard
Vrbo
vrbo.com
Travel & Hospitality
Akamai Bot Manager
Hard
Walgreens
walgreens.com
Retail & E-commerce
Akamai Bot Manager
Hard
Walmart
walmart.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
Walmart Canada
walmart.ca
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
Wayfair
wayfair.com
Retail & E-commerce
PerimeterX / HUMANCloudflare (CDN/WAF)
Hard
Yelp
yelp.com
Reviews & Ratings
DataDome
Hard
Zara
zara.com
Retail & E-commerce
Akamai Bot Manager
Hard
ZipRecruiter
ziprecruiter.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
Zoopla
zoopla.co.uk
Real Estate & Property Listings
Cloudflare (Bot/Challenge)
Hard
Adidas
adidas.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Agoda
agoda.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
Airbnb
airbnb.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
AliExpress
aliexpress.com
Online Marketplaces
Akamai (CDN/WAF)
Moderate
Amazon
amazon.com
Online Marketplaces
AWS WAF
Moderate
Apartments.com
apartments.com
Real Estate & Property Listings
Akamai (CDN/WAF)
Moderate
Argos
argos.co.uk
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
ASOS
asos.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Autotrader
autotrader.com
Automotive
Akamai (CDN/WAF)
Moderate
AXS
axs.com
Tickets & Live Events
Cloudflare (CDN/WAF)
Moderate
Carrefour
carrefour.fr
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
Costco
costco.com
Retail & E-commerce
Akamai (CDN/WAF)Queue-it
Moderate
Craigslist
craigslist.org
Classifieds
hCaptcha
Moderate
Crunchbase
crunchbase.com
Data & Business Intelligence
Cloudflare (CDN/WAF)
Moderate
CVS
cvs.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Domain
domain.com.au
Real Estate & Property Listings
Akamai (CDN/WAF)
Moderate
DoorDash
doordash.com
Food Delivery
Cloudflare (CDN/WAF)
Moderate
eBay
ebay.com
Online Marketplaces
Akamai (CDN/WAF)
Moderate
Funda
funda.nl
Real Estate & Property Listings
Akamai (CDN/WAF)reCAPTCHA
Moderate
Grailed
grailed.com
Online Marketplaces
Cloudflare (CDN/WAF)
Moderate
Groupon
groupon.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
H&M
hm.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Kohl's
kohls.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Live Nation
livenation.com
Tickets & Live Events
Queue-it
Moderate
Lowe's
lowes.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Lululemon
lululemon.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Macy's
macys.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
Newegg
newegg.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
Rakuten
rakuten.co.jp
Online Marketplaces
Akamai (CDN/WAF)
Moderate
SHEIN
shein.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
Snapchat
snapchat.com
Social Platforms
hCaptcha
Moderate
Temu
temu.com
Online Marketplaces
Cloudflare (CDN/WAF)
Moderate
Tesco
tesco.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
TikTok
tiktok.com
Social Platforms
Akamai (CDN/WAF)hCaptcha
Moderate
Ulta Beauty
ulta.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
United Airlines
united.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
X (Twitter)
x.com
Social Platforms
Cloudflare (CDN/WAF)
Moderate
94 rowsReviewed July 2026 · click any row for the full breakdown
New: the Anti-Bot Census

The whole registry as an open dataset - vendor market share, difficulty by vertical, and downloadable CSV / NDJSON / JSON. Free to cite.

View the census

Complete directory by category

Every site in the registry, grouped by vertical.

Retail & E-commerce

36 sites

Large retailers run enterprise bot management that scores IP reputation before any page or challenge renders, and pricing and stock endpoints are among the most heavily defended surfaces on the web - datacenter ranges are blocked wholesale and overused residential pools increasingly arrive pre-flagged. Legitimate price and availability monitoring typically needs carrier-grade mobile or provenance-verified residential exits, stable per-region sessions, and collection scoped to what robots.txt and each site's terms permit.

Online Marketplaces

12 sites

Marketplaces layer rate limiting, TLS and browser fingerprinting, and login walls around listing and seller data, and many of the largest players are regional, so traffic from the wrong country stands out immediately. Collection here usually calls for geo-matched mobile or residential IPs, session persistence, and respect for each platform's terms and applicable data-protection law.

Travel & Hospitality

10 sites

Travel sites have fought automated fare and rate collection longer than almost any other vertical, because every search triggers costly backend lookups - expect strict search metering, IP-reputation gating before challenges, and prices that vary by geography and session. Legitimate fare and rate monitoring generally requires geo-accurate carrier IPs, conservative request pacing, and adherence to each provider's terms.

Real Estate & Property Listings

8 sites

Property portals treat listing data as licensed core IP, back it with enterprise bot management, and the vertical has a notable history of legal enforcement against scrapers. Beyond clean mobile or residential IP provenance, teams need legal review of listing-data licensing and terms before collecting at scale.

Social Platforms

4 sites

Social platforms combine login walls, device and browser fingerprinting, behavioral modeling, and per-account plus per-IP throttling, and their detection baselines are tuned to real mobile-carrier CGNAT traffic because that is where most legitimate users come from. Since much of the data is personal data, compliance with platform terms and privacy law matters as much as IP quality.

Tickets & Live Events

4 sites

Ticketing platforms face legal mandates against purchase bots in some jurisdictions and defend inventory with queues, challenges, purchase limits, and highly sensitive IP-reputation scoring, so even benign monitoring traffic is treated adversarially. The legitimate use case is market and price research, done with clean carrier-grade IPs, low volumes, and strict adherence to platform terms and applicable law.

Automotive

3 sites

Vehicle listing sites defend inventory and pricing data with bot management and rate limits, and much of the underlying data carries licensing restrictions from dealers and data providers. Teams typically need geo-matched residential or mobile exits for regional inventory views plus a clear legal basis for how listing data is used.

Jobs & Recruitment

3 sites

Job boards mix public listings with account-gated data, apply heavy fingerprinting and rate limiting, and the vertical has produced some of the most cited scraping litigation, so the legal boundaries are actively contested. Legitimate labor-market research needs clean IP provenance, restraint around logged-in surfaces, and careful terms and privacy review.

Reviews & Ratings

3 sites

Review platforms treat their review corpus as the core asset and defend it with strong bot detection, markup obfuscation, and restrictive reuse terms, while the reviews themselves are user-generated personal data. Collection typically requires provenance-verified IPs and a compliance posture that covers both platform terms and data-protection rules.

Sneakers & Limited Drops

3 sites

Limited-release retail attracts the densest bot pressure on the web, so defenses run at maximum sensitivity - queues, raffles, aggressive device and browser fingerprinting, and IP ranges ever observed near a drop tend to stay pre-flagged. The legitimate use case is resale and market price monitoring, which still demands clean carrier-grade mobile IPs and full respect for platform terms.

Classifieds

2 sites

Classifieds sites block unfamiliar IP ranges aggressively, the category has a long litigation history against automated access, and per-country sites expect local-looking traffic. Legitimate listing research calls for geo-local mobile or residential exits, modest volumes, and strict attention to terms and applicable law.

Food Delivery

2 sites

Delivery platforms are mobile-app-first, pairing device attestation on apps with bot management on the web, and menus, prices, and availability change with the exact delivery location. Meaningful collection needs city-accurate carrier IPs and location-consistent sessions, alongside compliance with platform terms.

Online Learning & Other

2 sites

Learning platforms gate much of their catalog and pricing behind accounts and regional storefronts, applying standard rate limiting and bot scoring rather than the extreme defenses of transactional verticals. Course and pricing research still benefits from geo-matched residential or mobile IPs and terms-compliant, logged-out collection wherever possible.

Data & Business Intelligence

1 sites

Business-intelligence platforms sell the very data being requested, so unlicensed scraping competes directly with their core business - expect strict terms, login walls, tight rate limits, and strong bot detection. Teams should evaluate licensed API access first and keep any permitted collection on clean-provenance IPs at low volume.

Reservations & Dining

1 sites

Reservation platforms defend table availability because automated hoarding and resale of bookings is a recognized abuse pattern, so they apply account gating, device checks, and sensitive rate limiting. Legitimate availability and market research needs city-accurate mobile IPs, low request rates, and strict adherence to platform terms.

How this is measured. Each entry is derived from public HTTP response signatures (cookies, headers and challenge assets) observed on the site's landing response, reviewed July 2026. We list a technology only when a canonical signal was present. Providers change their stack over time.

Responsible use. This registry describes defenses that exist, not how to defeat them. Respect each site's robots.txt, Terms of Service and applicable law, and collect only what you are permitted to.

Have a hard target on this list?

Hand it to Data Works and get clean, QA-checked data delivered, or run it yourself on real 4G/5G carrier IPs from $4/GB.