The web is building a way for "good" bots to prove who they are. Here's how Web Bot Auth works, who supports it, and the catch every scraper and agent developer needs to understand.
Web Bot Auth lets an AI agent cryptographically sign its HTTP requests (using Ed25519 keys and HTTP Message Signatures, RFC 9421) so a website can verify which agent is calling. Cloudflare and Google back it and the IETF chartered a working group in early 2026. The catch: it only grants access on sites that adopted it and chose to allow your agent. On the rest of the web — almost all of it — an identified bot is just an easy bot to block, so you still need IP reputation from mobile proxies.
The mechanism is deliberately boring, which is a good sign for a standard. Your agent holds a private key. On each request it adds a signature over selected headers (method, path, host, a timestamp, sometimes the body digest). The site reads two headers — broadly a Signature-Input describing what was signed and a Signature carrying the Ed25519 signature — and verifies them against your published public key.
Because the signature is bound to the request, it can't be replayed or stolen the way a leaked API key or user-agent string can. The site learns one thing it could never trust before: this request genuinely comes from the agent it claims to be. What the site does with that — allow, throttle, charge, or log — is entirely up to its policy.
Cloudflare ships verification for signed agents and pairs it with its bot controls; Google published a compatible Web Bot Auth approach for its crawlers and fetchers; and the IETF working group means the format should converge rather than fragment. That covers a meaningful slice of high-traffic infrastructure.
But "Cloudflare can verify a signature" is not the same as "the site lets you in." Adoption is opt-in per site, and being allowlisted usually means a relationship — you're a known search engine, an approved AI crawler, or a partner. For an anonymous scraping or automation agent hitting a random e-commerce site, there is no allowlist to join, and signing simply removes your anonymity.
This is the core trade-off of the verified agent web: identity is powerful where it is welcomed and a liability where it isn't. That's why mature agent stacks keep a mobile-proxy layer for everything outside the allowlist, and use a decision tree to pick per request.
Web Bot Auth is an emerging standard that lets an automated client (a bot or AI agent) cryptographically sign its HTTP requests so a website can verify which agent is making them. It builds on HTTP Message Signatures (RFC 9421) using Ed25519 keys, and is being standardized through an IETF working group chartered in early 2026. Cloudflare and Google both back compatible schemes.
No. Web Bot Auth only helps on sites that have adopted it and decided to allow your specific agent. The overwhelming majority of websites have no signed-agent allowlist, so signing your requests there just makes your bot trivially easy to identify and block. For those sites, IP reputation from real mobile proxies remains the reliable path in.
Roughly, yes — signing is the mechanism, verification is the outcome. A site that supports Web Bot Auth checks your signature against a known public key (or directory) and, if it trusts that agent, treats it as a verified bot: allowed, rate-limited, or routed differently from anonymous traffic.
4G/5G mobile proxies across 17+ countries — $4/GB, free endpoints, free rotation, MCP + x402 for agents.