Read-only technical reference - reviewed July 2026

Anti-Bot Protection Registry

A technical reference of the anti-bot protection observed on 95 popular websites across 15 verticals, and what it takes to collect their public data legitimately.

This registry documents the anti-bot and anti-scraping technologies observed on 95 popular websites across 15 verticals - from CDN-level challenges and TLS fingerprinting to behavioral analysis and IP reputation scoring. Each entry describes the defense at the mechanism level and, at a high level, the class of IP infrastructure that legitimate large-scale data collection tends to require. It is a read-only technical reference, not a bypass guide: always respect robots.txt, a site's Terms of Service, and applicable law, and collect only data you are permitted to collect.

95 sites15 verticals10 protection technologies58 hard targets

Browse the registry

Search by site or domain, filter by category, technology or difficulty, and sort. Every entry links to a full breakdown.

Difficulty:
Showing 1-24 of 95 sites

Complete directory by category

Every site in the registry, grouped by vertical.

Retail & E-commerce

36 sites

Large retailers run enterprise bot management that scores IP reputation before any page or challenge renders, and pricing and stock endpoints are among the most heavily defended surfaces on the web - datacenter ranges are blocked wholesale and overused residential pools increasingly arrive pre-flagged. Legitimate price and availability monitoring typically needs carrier-grade mobile or provenance-verified residential exits, stable per-region sessions, and collection scoped to what robots.txt and each site's terms permit.

Online Marketplaces

12 sites

Marketplaces layer rate limiting, TLS and browser fingerprinting, and login walls around listing and seller data, and many of the largest players are regional, so traffic from the wrong country stands out immediately. Collection here usually calls for geo-matched mobile or residential IPs, session persistence, and respect for each platform's terms and applicable data-protection law.

Travel & Hospitality

11 sites

Travel sites have fought automated fare and rate collection longer than almost any other vertical, because every search triggers costly backend lookups - expect strict search metering, IP-reputation gating before challenges, and prices that vary by geography and session. Legitimate fare and rate monitoring generally requires geo-accurate carrier IPs, conservative request pacing, and adherence to each provider's terms.

Real Estate & Property Listings

8 sites

Property portals treat listing data as licensed core IP, back it with enterprise bot management, and the vertical has a notable history of legal enforcement against scrapers. Beyond clean mobile or residential IP provenance, teams need legal review of listing-data licensing and terms before collecting at scale.

Social Platforms

4 sites

Social platforms combine login walls, device and browser fingerprinting, behavioral modeling, and per-account plus per-IP throttling, and their detection baselines are tuned to real mobile-carrier CGNAT traffic because that is where most legitimate users come from. Since much of the data is personal data, compliance with platform terms and privacy law matters as much as IP quality.

Tickets & Live Events

4 sites

Ticketing platforms face legal mandates against purchase bots in some jurisdictions and defend inventory with queues, challenges, purchase limits, and highly sensitive IP-reputation scoring, so even benign monitoring traffic is treated adversarially. The legitimate use case is market and price research, done with clean carrier-grade IPs, low volumes, and strict adherence to platform terms and applicable law.

Automotive

3 sites

Vehicle listing sites defend inventory and pricing data with bot management and rate limits, and much of the underlying data carries licensing restrictions from dealers and data providers. Teams typically need geo-matched residential or mobile exits for regional inventory views plus a clear legal basis for how listing data is used.

Jobs & Recruitment

3 sites

Job boards mix public listings with account-gated data, apply heavy fingerprinting and rate limiting, and the vertical has produced some of the most cited scraping litigation, so the legal boundaries are actively contested. Legitimate labor-market research needs clean IP provenance, restraint around logged-in surfaces, and careful terms and privacy review.

Reviews & Ratings

3 sites

Review platforms treat their review corpus as the core asset and defend it with strong bot detection, markup obfuscation, and restrictive reuse terms, while the reviews themselves are user-generated personal data. Collection typically requires provenance-verified IPs and a compliance posture that covers both platform terms and data-protection rules.

Sneakers & Limited Drops

3 sites

Limited-release retail attracts the densest bot pressure on the web, so defenses run at maximum sensitivity - queues, raffles, aggressive device and browser fingerprinting, and IP ranges ever observed near a drop tend to stay pre-flagged. The legitimate use case is resale and market price monitoring, which still demands clean carrier-grade mobile IPs and full respect for platform terms.

Classifieds

2 sites

Classifieds sites block unfamiliar IP ranges aggressively, the category has a long litigation history against automated access, and per-country sites expect local-looking traffic. Legitimate listing research calls for geo-local mobile or residential exits, modest volumes, and strict attention to terms and applicable law.

Food Delivery

2 sites

Delivery platforms are mobile-app-first, pairing device attestation on apps with bot management on the web, and menus, prices, and availability change with the exact delivery location. Meaningful collection needs city-accurate carrier IPs and location-consistent sessions, alongside compliance with platform terms.

Online Learning & Other

2 sites

Learning platforms gate much of their catalog and pricing behind accounts and regional storefronts, applying standard rate limiting and bot scoring rather than the extreme defenses of transactional verticals. Course and pricing research still benefits from geo-matched residential or mobile IPs and terms-compliant, logged-out collection wherever possible.

Data & Business Intelligence

1 sites

Business-intelligence platforms sell the very data being requested, so unlicensed scraping competes directly with their core business - expect strict terms, login walls, tight rate limits, and strong bot detection. Teams should evaluate licensed API access first and keep any permitted collection on clean-provenance IPs at low volume.

Reservations & Dining

1 sites

Reservation platforms defend table availability because automated hoarding and resale of bookings is a recognized abuse pattern, so they apply account gating, device checks, and sensitive rate limiting. Legitimate availability and market research needs city-accurate mobile IPs, low request rates, and strict adherence to platform terms.

How this is measured. Each entry is derived from public HTTP response signatures (cookies, headers and challenge assets) observed on the site's landing response, reviewed July 2026. We list a technology only when a canonical signal was present. Providers change their stack over time.

Responsible use. This registry describes defenses that exist, not how to defeat them. Respect each site's robots.txt, Terms of Service and applicable law, and collect only what you are permitted to.

Have a hard target on this list?

Hand it to Data Works and get clean, QA-checked data delivered, or run it yourself on real 4G/5G carrier IPs from $4/GB.