The open web is quietly splitting into three lanes: traffic that identifies itself, traffic that pays, and traffic that blends in. If you are building an AI agent that browses, scrapes, or buys, you need to know which lane you are in on every request.
In 2026 an AI agent reaches a website through one of three gates. Signed-agent identity (Web Bot Auth) lets the agent prove who it is so cooperative sites can allow it. Pay-per-crawl lets the agent pay the publisher for access. IP reputation — routing through real residential or 4G/5G mobile proxies — lets the agent look like ordinary human traffic on the ~99% of sites that have no allowlist and sell no crawl access. Production agents mix all three: identify or pay where it is welcomed, and rely on mobile-IP reputation everywhere else.
Until recently, "accessing the web programmatically" meant one thing: send an HTTP request and hope it wasn't blocked. That model broke in 2025–2026 for two reasons.
First, publishers started fighting AI crawlers directly. Cloudflare began blocking AI crawlers by default on new domains and launched a pay-per-crawl marketplace, while major publishers signed deals to charge for access. Second, the industry started building a way for "good" bots to identify themselves cryptographically — Web Bot Auth and signed agents, now moving through the IETF.
At the same time, anti-bot detection moved past simple fingerprinting. With JA4/JA4+ TLS fingerprinting and post-quantum TLS now widely deployed, hiding your client is harder than ever — which means the IP you arrive on carries more weight than it used to. The result is the three-gate model below.
| Gate | How it works | Works on | Best for |
|---|---|---|---|
| Signed identity | Agent signs requests (Ed25519 / RFC 9421); site verifies and allowlists | Sites that opted in & allow you | Cooperative APIs, partners, official agent access |
| Pay-per-crawl | Agent pays the publisher per request (e.g. via Cloudflare) | Publishers selling crawl access | Licensed content, training data, premium sources |
| IP reputation | Agent routes through residential / 4G/5G mobile proxies to blend in | Essentially the whole public web | Scraping, monitoring, testing, multi-account, anything without an allowlist |
These are not mutually exclusive. A well-built agent identifies itself where that earns access, pays where content is worth buying, and uses mobile-IP reputation everywhere else.
The site publishes a signed-agent allowlist and you're on it: sign your requests. It is the cleanest, most durable access and avoids the proxy arms race entirely.
The content is licensed, high-value, or you need it at scale legally: pay per crawl. You get reliable access and a clean compliance story.
The site has no allowlist and sells no access (the default case): an identified bot is just an easy block. Route through real 4G/5G mobile proxies so your agent inherits the IP reputation of ordinary mobile users.
If you only remember one rule: identity and payment are opt-in privileges granted by the site; IP reputation is the fallback that always works. That is why every serious agent stack still ends with a proxy layer. For a request-by-request walkthrough, see our AI agent blocked decision tree.
In 2026, AI agents pass through three gates: (1) signed-agent identity, where the agent cryptographically identifies itself using Web Bot Auth so the site can allow or rate-limit it; (2) pay-per-crawl, where the agent pays the publisher (often via Cloudflare) for access; and (3) IP reputation, where the agent routes through residential or mobile proxies to look like ordinary human traffic. Most production agents combine identity or payment for cooperative sites with proxies for everything else.
Yes, for most of the web. Signed agents (Web Bot Auth) only help on sites that have opted into the scheme and choose to allow your agent. The vast majority of sites have no signed-agent allowlist, so an identified bot is simply an easy bot to block. For those sites, IP reputation from real 4G/5G mobile proxies remains the most reliable way in.
Mobile proxies use real carrier IPs behind CGNAT, where hundreds of real subscribers share each address. Blocking the IP would block real customers, so anti-bot systems are far more cautious. Datacenter IPs belong to known hosting ASNs and are blocked aggressively. This is why mobile IPs report much higher success rates on heavily defended sites.
Real 4G/5G mobile proxies across 17+ countries — $4/GB, free endpoints, free rotation, and an MCP server plus x402 payments built for autonomous agents.