A technical reference of the anti-bot protection observed on 95 popular websites across 15 verticals, and what it takes to collect their public data legitimately.
This registry documents the anti-bot and anti-scraping technologies observed on 95 popular websites across 15 verticals - from CDN-level challenges and TLS fingerprinting to behavioral analysis and IP reputation scoring. Each entry describes the defense at the mechanism level and, at a high level, the class of IP infrastructure that legitimate large-scale data collection tends to require. It is a read-only technical reference, not a bypass guide: always respect robots.txt, a site's Terms of Service, and applicable law, and collect only data you are permitted to collect.
Search by site or domain, filter by category, technology or difficulty, and sort. Every entry links to a full breakdown.
Enterprise bot & online-fraud protection
CDN with bot management & challenge layer
Enterprise bot management at the CDN edge
Enterprise bot defense & fraud prevention
Bot mitigation with client interrogation
WAF with advanced bot protection
Web application firewall with bot control
Virtual waiting room / traffic gating
CAPTCHA / risk-scoring challenge layer
CAPTCHA / challenge layer
Every site in the registry, grouped by vertical.
Large retailers run enterprise bot management that scores IP reputation before any page or challenge renders, and pricing and stock endpoints are among the most heavily defended surfaces on the web - datacenter ranges are blocked wholesale and overused residential pools increasingly arrive pre-flagged. Legitimate price and availability monitoring typically needs carrier-grade mobile or provenance-verified residential exits, stable per-region sessions, and collection scoped to what robots.txt and each site's terms permit.
Marketplaces layer rate limiting, TLS and browser fingerprinting, and login walls around listing and seller data, and many of the largest players are regional, so traffic from the wrong country stands out immediately. Collection here usually calls for geo-matched mobile or residential IPs, session persistence, and respect for each platform's terms and applicable data-protection law.
Travel sites have fought automated fare and rate collection longer than almost any other vertical, because every search triggers costly backend lookups - expect strict search metering, IP-reputation gating before challenges, and prices that vary by geography and session. Legitimate fare and rate monitoring generally requires geo-accurate carrier IPs, conservative request pacing, and adherence to each provider's terms.
Property portals treat listing data as licensed core IP, back it with enterprise bot management, and the vertical has a notable history of legal enforcement against scrapers. Beyond clean mobile or residential IP provenance, teams need legal review of listing-data licensing and terms before collecting at scale.
Ticketing platforms face legal mandates against purchase bots in some jurisdictions and defend inventory with queues, challenges, purchase limits, and highly sensitive IP-reputation scoring, so even benign monitoring traffic is treated adversarially. The legitimate use case is market and price research, done with clean carrier-grade IPs, low volumes, and strict adherence to platform terms and applicable law.
Vehicle listing sites defend inventory and pricing data with bot management and rate limits, and much of the underlying data carries licensing restrictions from dealers and data providers. Teams typically need geo-matched residential or mobile exits for regional inventory views plus a clear legal basis for how listing data is used.
Job boards mix public listings with account-gated data, apply heavy fingerprinting and rate limiting, and the vertical has produced some of the most cited scraping litigation, so the legal boundaries are actively contested. Legitimate labor-market research needs clean IP provenance, restraint around logged-in surfaces, and careful terms and privacy review.
Review platforms treat their review corpus as the core asset and defend it with strong bot detection, markup obfuscation, and restrictive reuse terms, while the reviews themselves are user-generated personal data. Collection typically requires provenance-verified IPs and a compliance posture that covers both platform terms and data-protection rules.
Limited-release retail attracts the densest bot pressure on the web, so defenses run at maximum sensitivity - queues, raffles, aggressive device and browser fingerprinting, and IP ranges ever observed near a drop tend to stay pre-flagged. The legitimate use case is resale and market price monitoring, which still demands clean carrier-grade mobile IPs and full respect for platform terms.
Classifieds sites block unfamiliar IP ranges aggressively, the category has a long litigation history against automated access, and per-country sites expect local-looking traffic. Legitimate listing research calls for geo-local mobile or residential exits, modest volumes, and strict attention to terms and applicable law.
Delivery platforms are mobile-app-first, pairing device attestation on apps with bot management on the web, and menus, prices, and availability change with the exact delivery location. Meaningful collection needs city-accurate carrier IPs and location-consistent sessions, alongside compliance with platform terms.
Learning platforms gate much of their catalog and pricing behind accounts and regional storefronts, applying standard rate limiting and bot scoring rather than the extreme defenses of transactional verticals. Course and pricing research still benefits from geo-matched residential or mobile IPs and terms-compliant, logged-out collection wherever possible.
Business-intelligence platforms sell the very data being requested, so unlicensed scraping competes directly with their core business - expect strict terms, login walls, tight rate limits, and strong bot detection. Teams should evaluate licensed API access first and keep any permitted collection on clean-provenance IPs at low volume.
Reservation platforms defend table availability because automated hoarding and resale of bookings is a recognized abuse pattern, so they apply account gating, device checks, and sensitive rate limiting. Legitimate availability and market research needs city-accurate mobile IPs, low request rates, and strict adherence to platform terms.
How this is measured. Each entry is derived from public HTTP response signatures (cookies, headers and challenge assets) observed on the site's landing response, reviewed July 2026. We list a technology only when a canonical signal was present. Providers change their stack over time.
Responsible use. This registry describes defenses that exist, not how to defeat them. Respect each site's robots.txt, Terms of Service and applicable law, and collect only what you are permitted to.
Hand it to Data Works and get clean, QA-checked data delivered, or run it yourself on real 4G/5G carrier IPs from $4/GB.
Social Platforms
4 sitesSocial platforms combine login walls, device and browser fingerprinting, behavioral modeling, and per-account plus per-IP throttling, and their detection baselines are tuned to real mobile-carrier CGNAT traffic because that is where most legitimate users come from. Since much of the data is personal data, compliance with platform terms and privacy law matters as much as IP quality.