Open dataset - free to download and cite

Anti-Bot Census: which vendors defend the web's most-scraped sites.

We probed 95 popular sites across 15 verticals for the anti-bot stack they actually run, from public HTTP signatures. Here is the market share, the difficulty spread, and the raw data - reviewed July 2026.

95
sites tracked
61%
are hard targets
114
evidenced detections
10
vendors observed

Vendor market share

Share of tracked sites observed running each bot-management family. Counted once per site per vendor, so the denominator is honest - a site running two Akamai products still counts once for Akamai. Akamai is the most commonly observed family, on 46 of 95 tracked sites (48%), as of July 2026.

Akamai
46 / 9548%
Cloudflare
33 / 9535%
HUMAN (PerimeterX)
11 / 9512%
DataDome
9 / 959%
Kasada
5 / 955%
Google reCAPTCHA
3 / 953%
hCaptcha
3 / 953%
Queue-it
2 / 952%
Imperva (Incapsula)
1 / 951%
AWS WAF
1 / 951%

Difficulty distribution

Hard target
58
61% of tracked sites
Moderate
37
39% of tracked sites

Difficulty by vertical

Where the hard targets cluster. The toughest tracked vertical is Jobs & Recruitment, where 100% of sites are hard targets.

VerticalSitesHard% hardMost common vendor
Retail & E-commerce362056%Akamaidone-for-you
Online Marketplaces12650%Cloudflaredone-for-you
Travel & Hospitality11873%Akamaidone-for-you
Real Estate & Property Listings8563%Akamaidone-for-you
Social Platforms4125%Cloudflaredone-for-you
Tickets & Live Events4250%DataDome
Automotive3267%Cloudflare
Jobs & Recruitment33100%Cloudflare
Reviews & Ratings33100%Cloudflare
Sneakers & Limited Drops33100%Cloudflare
Classifieds2150%DataDome
Food Delivery2150%Cloudflare
Online Learning & Other22100%Cloudflare
Data & Business Intelligence100%Cloudflare
Reservations & Dining11100%Imperva (Incapsula)

Methodology

Evidence-backed only

Each detection comes from a public HTTP response signature - a vendor cookie, header or challenge asset that actually fired. A site with no canonical signature is not listed. Nothing is guessed.

Site-level counting

Market share counts each site at most once per vendor family, so a site running two products from one vendor is not double-counted. The denominator is the full tracked set.

Dated and self-verifiable

Reviewed July 2026. Vendors change their stack over time, so every figure is a dated data point you can re-check yourself against the live site.

Read-only reference

This documents the defense that exists, in the spirit of BuiltWith or Wappalyzer. It is never a per-site bypass guide. Respect robots.txt, Terms of Service and applicable law.

Movements changelog. This is edition one. We re-run the detector on a recurring basis and publish the diffs - which sites changed vendor or moved difficulty tier - so you can watch the defended web shift over time. If your target just added a new layer, that is usually the moment a pipeline breaks.

Three ways to actually get this data

The census tells you what defends a site. Here is honestly how to collect it - from DIY proxies to a fully managed pipeline.

Do it yourself
From $4/GB

Best when: You have engineers and 1-2 easy targets

  • Dedicated 4G/5G carrier IPs, self-serve
  • You build, host and maintain the scraper
  • Down to $2.40/GB at volume, GB never expire
Buy proxies from $4/GB
Scraping API
Per-credit

Best when: Light, unprotected pages at low volume

  • You still write the parser and own the breakage
  • Per-record cost climbs fast on protected sites
  • "Success" billing can still charge for blocked 200s
Why APIs get expensive
Data Works (managed)
Custom-quoted flat monthly
For hard sites

Best when: 1-3 hard, defended targets like a hard target

  • A principal engineer builds, runs and repairs it
  • Real carrier IPs; we prove it in a paid pilot first
  • Delivered as clean CSV / JSON / API - breakage is ours
Scope a data project

Have a hard target in this dataset?

Point us at it. Data Works cracks it in a paid pilot on real 4G/5G carrier IPs and delivers the data - before you commit to anything.

Frequently asked questions

How is the dataset collected?

Each site is probed for public HTTP response signatures - cookies, headers and challenge assets that identify a bot-management vendor. Only evidence-backed detections are recorded; a site with no canonical signature is not listed. The current edition covers 95 sites across 15 verticals, reviewed July 2026.

What does "hard target" mean?

A hard target runs enterprise bot management (DataDome, Cloudflare Bot Management, Akamai Bot Manager, HUMAN/PerimeterX, Kasada or Imperva) that scores IP reputation and browser signals before the page renders. 61% of tracked sites fall in this tier.

Can I use this data?

Yes. The dataset is free to download and cite (CSV, NDJSON or JSON) with attribution to proxies.sx. It is a read-only technical reference in the spirit of BuiltWith or Wappalyzer - it documents the defense that exists, never a per-site bypass.

How often is it updated?

This is the first edition. We re-run the detector on a recurring basis and publish the differences as a dated "Movements" changelog, so you can see which sites changed vendor or difficulty tier over time.