Anti-Bot Protection Registry reviewed July 2026

A technical reference of the anti-bot protection observed on 94 popular websites across 15 verticals, and what it takes to collect their public data legitimately.

Open dataset
94
Sites indexed
15
Categories
10
Technologies
57
Hard targets

Registry - search, filter and sort every indexed site

#
1Allegro
allegro.pl
Online Marketplaces
DataDome
Hard
2Capterra
capterra.com
Reviews & Ratings
Cloudflare (Bot/Challenge)
Hard
3Cars.com
cars.com
Automotive
Cloudflare (Bot/Challenge)
Hard
4Carvana
carvana.com
Automotive
Cloudflare (Bot/Challenge)
Hard
5Cdiscount
cdiscount.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
6Chewy
chewy.com
Retail & E-commerce
KasadaAkamai (CDN/WAF)
Hard
7Currys
currys.co.uk
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
8Depop
depop.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
9END. Clothing
endclothing.com
Sneakers & Limited Drops
Akamai Bot Manager
Hard
10Etsy
etsy.com
Online Marketplaces
DataDome
Hard
11Expedia
expedia.com
Travel & Hospitality
Akamai Bot Manager
Hard
12Fnac
fnac.com
Retail & E-commerce
DataDomeAkamai (CDN/WAF)
Hard
13G2
g2.com
Reviews & Ratings
DataDomeCloudflare (Bot/Challenge)
Hard
14Gap
gap.com
Retail & E-commerce
Akamai Bot Manager
Hard
15Glassdoor
glassdoor.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
16GOAT
goat.com
Sneakers & Limited Drops
Cloudflare (Bot/Challenge)
Hard
17Home Depot
homedepot.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
18Hotels.com
hotels.com
Travel & Hospitality
Akamai Bot Manager
Hard
19Idealista
idealista.com
Real Estate & Property Listings
DataDome
Hard
20Indeed
indeed.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
21Leboncoin
leboncoin.fr
Classifieds
DataDome
Hard
22LinkedIn
linkedin.com
Social Platforms
PerimeterX / HUMANCloudflare (CDN/WAF)reCAPTCHA
Hard
23MediaMarkt
mediamarkt.de
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
24Mercari
mercari.com
Online Marketplaces
Cloudflare (Bot/Challenge)
Hard
25Nike
nike.com
Retail & E-commerce
KasadaAkamai (CDN/WAF)
Hard
26Otto
otto.de
Online Marketplaces
Kasada
Hard
27Overstock
overstock.com
Retail & E-commerce
Akamai Bot Manager
Hard
28Petco
petco.com
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
29Priceline
priceline.com
Travel & Hospitality
PerimeterX / HUMANCloudflare (CDN/WAF)
Hard
30QVC
qvc.com
Retail & E-commerce
Akamai Bot Manager
Hard
31realestate.com.au
realestate.com.au
Real Estate & Property Listings
KasadaAkamai (CDN/WAF)
Hard
32Realtor.com
realtor.com
Real Estate & Property Listings
Kasada
Hard
33Resy
resy.com
Reservations & Dining
Imperva (Incapsula)reCAPTCHA
Hard
34Sam's Club
samsclub.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
35SeatGeek
seatgeek.com
Tickets & Live Events
DataDome
Hard
36Sephora
sephora.com
Retail & E-commerce
Akamai Bot Manager
Hard
37Skillshare
skillshare.com
Online Learning & Other
Cloudflare (Bot/Challenge)
Hard
38Skyscanner
skyscanner.net
Travel & Hospitality
PerimeterX / HUMAN
Hard
39Southwest Airlines
southwest.com
Travel & Hospitality
Akamai Bot Manager
Hard
40SSENSE
ssense.com
Retail & E-commerce
Cloudflare (Bot/Challenge)
Hard
41StockX
stockx.com
Sneakers & Limited Drops
PerimeterX / HUMANCloudflare (Bot/Challenge)
Hard
42Target
target.com
Retail & E-commerce
PerimeterX / HUMAN
Hard
43Trivago
trivago.com
Travel & Hospitality
Akamai Bot Manager
Hard
44Trulia
trulia.com
Real Estate & Property Listings
PerimeterX / HUMAN
Hard
45Uber Eats
ubereats.com
Food Delivery
Cloudflare (Bot/Challenge)
Hard
46Udemy
udemy.com
Online Learning & Other
Cloudflare (Bot/Challenge)
Hard
47Uniqlo
uniqlo.com
Retail & E-commerce
Akamai Bot Manager
Hard
48Vivid Seats
vividseats.com
Tickets & Live Events
Akamai Bot Manager
Hard
49Vrbo
vrbo.com
Travel & Hospitality
Akamai Bot Manager
Hard
50Walgreens
walgreens.com
Retail & E-commerce
Akamai Bot Manager
Hard
51Walmart
walmart.com
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
52Walmart Canada
walmart.ca
Retail & E-commerce
PerimeterX / HUMANAkamai (CDN/WAF)
Hard
53Wayfair
wayfair.com
Retail & E-commerce
PerimeterX / HUMANCloudflare (CDN/WAF)
Hard
54Yelp
yelp.com
Reviews & Ratings
DataDome
Hard
55Zara
zara.com
Retail & E-commerce
Akamai Bot Manager
Hard
56ZipRecruiter
ziprecruiter.com
Jobs & Recruitment
Cloudflare (Bot/Challenge)
Hard
57Zoopla
zoopla.co.uk
Real Estate & Property Listings
Cloudflare (Bot/Challenge)
Hard
58Adidas
adidas.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
59Agoda
agoda.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
60Airbnb
airbnb.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
61AliExpress
aliexpress.com
Online Marketplaces
Akamai (CDN/WAF)
Moderate
62Amazon
amazon.com
Online Marketplaces
AWS WAF
Moderate
63Apartments.com
apartments.com
Real Estate & Property Listings
Akamai (CDN/WAF)
Moderate
64Argos
argos.co.uk
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
65ASOS
asos.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
66Autotrader
autotrader.com
Automotive
Akamai (CDN/WAF)
Moderate
67AXS
axs.com
Tickets & Live Events
Cloudflare (CDN/WAF)
Moderate
68Carrefour
carrefour.fr
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
69Costco
costco.com
Retail & E-commerce
Akamai (CDN/WAF)Queue-it
Moderate
70Craigslist
craigslist.org
Classifieds
hCaptcha
Moderate
71Crunchbase
crunchbase.com
Data & Business Intelligence
Cloudflare (CDN/WAF)
Moderate
72CVS
cvs.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
73Domain
domain.com.au
Real Estate & Property Listings
Akamai (CDN/WAF)
Moderate
74DoorDash
doordash.com
Food Delivery
Cloudflare (CDN/WAF)
Moderate
75eBay
ebay.com
Online Marketplaces
Akamai (CDN/WAF)
Moderate
76Funda
funda.nl
Real Estate & Property Listings
Akamai (CDN/WAF)reCAPTCHA
Moderate
77Grailed
grailed.com
Online Marketplaces
Cloudflare (CDN/WAF)
Moderate
78Groupon
groupon.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
79H&M
hm.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
80Kohl's
kohls.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
81Live Nation
livenation.com
Tickets & Live Events
Queue-it
Moderate
82Lowe's
lowes.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
83Lululemon
lululemon.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
84Macy's
macys.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
85Newegg
newegg.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
86Rakuten
rakuten.co.jp
Online Marketplaces
Akamai (CDN/WAF)
Moderate
87SHEIN
shein.com
Retail & E-commerce
Cloudflare (CDN/WAF)
Moderate
88Snapchat
snapchat.com
Social Platforms
hCaptcha
Moderate
89Temu
temu.com
Online Marketplaces
Cloudflare (CDN/WAF)
Moderate
90Tesco
tesco.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
91TikTok
tiktok.com
Social Platforms
Akamai (CDN/WAF)hCaptcha
Moderate
92Ulta Beauty
ulta.com
Retail & E-commerce
Akamai (CDN/WAF)
Moderate
93United Airlines
united.com
Travel & Hospitality
Akamai (CDN/WAF)
Moderate
94X (Twitter)
x.com
Social Platforms
Cloudflare (CDN/WAF)
Moderate
Showing 1-94 of 94 sites
Page 1 of 1
The Anti-Bot Census - open dataset

The whole registry as an open dataset: vendor market share, difficulty by category, downloadable CSV / NDJSON / JSON. Free to cite.

View the census

Complete directory

Every site in the registry, grouped by category.

Large retailers run enterprise bot management that scores IP reputation before any page or challenge renders, and pricing and stock endpoints are among the most heavily defended surfaces on the web - datacenter ranges are blocked wholesale and overused residential pools increasingly arrive pre-flagged. Legitimate price and availability monitoring typically needs carrier-grade mobile or provenance-verified residential exits, stable per-region sessions, and collection scoped to what robots.txt and each site's terms permit.

Marketplaces layer rate limiting, TLS and browser fingerprinting, and login walls around listing and seller data, and many of the largest players are regional, so traffic from the wrong country stands out immediately. Collection here usually calls for geo-matched mobile or residential IPs, session persistence, and respect for each platform's terms and applicable data-protection law.

Travel sites have fought automated fare and rate collection longer than almost any other vertical, because every search triggers costly backend lookups - expect strict search metering, IP-reputation gating before challenges, and prices that vary by geography and session. Legitimate fare and rate monitoring generally requires geo-accurate carrier IPs, conservative request pacing, and adherence to each provider's terms.

Property portals treat listing data as licensed core IP, back it with enterprise bot management, and the vertical has a notable history of legal enforcement against scrapers. Beyond clean mobile or residential IP provenance, teams need legal review of listing-data licensing and terms before collecting at scale.

Social platforms combine login walls, device and browser fingerprinting, behavioral modeling, and per-account plus per-IP throttling, and their detection baselines are tuned to real mobile-carrier CGNAT traffic because that is where most legitimate users come from. Since much of the data is personal data, compliance with platform terms and privacy law matters as much as IP quality.

Ticketing platforms face legal mandates against purchase bots in some jurisdictions and defend inventory with queues, challenges, purchase limits, and highly sensitive IP-reputation scoring, so even benign monitoring traffic is treated adversarially. The legitimate use case is market and price research, done with clean carrier-grade IPs, low volumes, and strict adherence to platform terms and applicable law.

Automotive

3 sites

Vehicle listing sites defend inventory and pricing data with bot management and rate limits, and much of the underlying data carries licensing restrictions from dealers and data providers. Teams typically need geo-matched residential or mobile exits for regional inventory views plus a clear legal basis for how listing data is used.

Job boards mix public listings with account-gated data, apply heavy fingerprinting and rate limiting, and the vertical has produced some of the most cited scraping litigation, so the legal boundaries are actively contested. Legitimate labor-market research needs clean IP provenance, restraint around logged-in surfaces, and careful terms and privacy review.

Review platforms treat their review corpus as the core asset and defend it with strong bot detection, markup obfuscation, and restrictive reuse terms, while the reviews themselves are user-generated personal data. Collection typically requires provenance-verified IPs and a compliance posture that covers both platform terms and data-protection rules.

Limited-release retail attracts the densest bot pressure on the web, so defenses run at maximum sensitivity - queues, raffles, aggressive device and browser fingerprinting, and IP ranges ever observed near a drop tend to stay pre-flagged. The legitimate use case is resale and market price monitoring, which still demands clean carrier-grade mobile IPs and full respect for platform terms.

Classifieds sites block unfamiliar IP ranges aggressively, the category has a long litigation history against automated access, and per-country sites expect local-looking traffic. Legitimate listing research calls for geo-local mobile or residential exits, modest volumes, and strict attention to terms and applicable law.

Delivery platforms are mobile-app-first, pairing device attestation on apps with bot management on the web, and menus, prices, and availability change with the exact delivery location. Meaningful collection needs city-accurate carrier IPs and location-consistent sessions, alongside compliance with platform terms.

Learning platforms gate much of their catalog and pricing behind accounts and regional storefronts, applying standard rate limiting and bot scoring rather than the extreme defenses of transactional verticals. Course and pricing research still benefits from geo-matched residential or mobile IPs and terms-compliant, logged-out collection wherever possible.

Business-intelligence platforms sell the very data being requested, so unlicensed scraping competes directly with their core business - expect strict terms, login walls, tight rate limits, and strong bot detection. Teams should evaluate licensed API access first and keep any permitted collection on clean-provenance IPs at low volume.

Reservation platforms defend table availability because automated hoarding and resale of bookings is a recognized abuse pattern, so they apply account gating, device checks, and sensitive rate limiting. Legitimate availability and market research needs city-accurate mobile IPs, low request rates, and strict adherence to platform terms.

About this registry

What is the anti-bot protection registry?

A read-only reference of the bot-protection technology observed on 94 popular websites across 15 categories, with a difficulty rating for collecting each site's public data. It answers questions like "does this site use Cloudflare or DataDome" and "how hard is this site to scrape". Reviewed July 2026.

How is the protection detected?

Each entry is derived from public HTTP response signatures - cookies, headers and challenge assets observed on the site's landing response. We list a technology only when a canonical signal was present, so entries are evidence-backed rather than guessed. Providers change their stacks over time.

What does the difficulty rating mean?

A three-level measurement of how hard large-scale public-data collection tends to be. Light is mostly open, Moderate typically needs residential or mobile IPs plus a real browser fingerprint, and Hard usually requires real mobile carrier IPs and careful session handling. The rating reflects the network layer, not a bypass method.

Is this a guide to bypass bot protection?

No. The registry describes defenses that exist; it does not explain how to defeat them. Respect each site's robots.txt, Terms of Service and applicable law, and collect only what you are permitted to.

How often is the registry updated?

The dataset is periodically re-reviewed against live signatures; the current review date is shown as "July 2026". The full registry is also published as an open, citable census dataset.

Have a hard target on this list?

Hand it to Data Works and get clean, QA-checked data delivered, or run it yourself on real 4G/5G carrier IPs from $4/GB.