Cloudflare now lets publishers charge AI agents for access. Sometimes paying is the right call — and sometimes it's paying for a door that was never locked. Here's how to tell.
Pay per crawl when the content is licensed, premium, or you need it legally at scale and the publisher sells access — you get reliable access and a clean compliance story. Use mobile proxies for the vast majority of the web that charges nothing and has no allowlist, where the only real obstacle is IP reputation. They are not competitors so much as two tools for two different halves of the web. The mistake is paying for access on a site that would have served you for free if you simply hadn't looked like a datacenter bot.
For years the only options a publisher had against an unwanted crawler were "allow" or "block." Pay-per-crawl adds a third: "charge." Conceptually it leans on HTTP 402 Payment Required — the crawler requests a page, is told there's a price, pays, and receives the content. Major publishers signed on because it turns expensive bot traffic into revenue.
For an agent developer this is genuinely useful in one scenario: when you want a specific publisher's content, that publisher participates, and the data is worth the per-request price. It is a licensing channel, not a general way onto the web — which matters because most sites are not in the marketplace at all.
| Pay Per Crawl | Mobile Proxies | |
|---|---|---|
| Coverage | Only participating publishers | Essentially the whole public web |
| Cost model | Per request, set by publisher | Per GB of traffic ($4 → $2.40) |
| Compliance | Explicit, licensed access | Depends on site ToS & use case |
| Reliability | High — you're a paying customer | High on real mobile IPs; depends on quality |
| Best use | Premium / licensed content, training data | Scraping, monitoring, testing, automation |
Named, premium source you want long-term? Pay per crawl if available. Cheaper than a lawsuit, more reliable than an arms race.
Broad data across many ordinary sites? Mobile proxies. There's nothing to pay; you just need to not get blocked.
This is exactly the layered model from our pillar guide on how AI agents access the web in 2026. If your agent uses crypto rails, our MCP server and x402 payments let it pay for bandwidth (and, where supported, content) autonomously in USDC. Not sure which path a given request needs? Use the decision tree.
Pay Per Crawl is a Cloudflare marketplace that lets publishers charge AI crawlers for access instead of simply blocking them. The crawler is asked to pay (conceptually via HTTP 402 Payment Required), and on payment the publisher serves the content. It is aimed at licensed access to high-value content rather than general-purpose web browsing.
It depends entirely on the site. Pay Per Crawl only works on participating publishers and you pay their per-request price, which can be worthwhile for premium licensed content. For the rest of the web — which charges nothing and has no allowlist — there is nothing to pay; you just need an IP that is not blocked, which is where mobile proxies at $4/GB are far more economical.
Yes, and most serious agents should. Pay where a publisher sells access and the content is worth it; route through mobile proxies everywhere else. They cover different parts of the web, so treating them as either/or is a mistake.
4G/5G mobile proxies across 17+ countries — $4/GB, free endpoints, free rotation, USDC via x402 for autonomous agents.