Anti-Bot

What is Bot Detection?

Bot detection is the practice of separating automated traffic from human traffic. Systems score each request against network signals, transport-layer fingerprints, browser characteristics and behaviour over time, then allow, challenge or block it. Vendors including Cloudflare, DataDome, Akamai and HUMAN sell this as a service running in front of the origin server.

Scoring happens in layers, cheapest first. An IP arrives with an ASN, a reputation history and a classification as residential, mobile or hosting, and traffic from a known cloud range starts at a disadvantage before a byte of content is served. The TLS handshake then yields a JA3 or JA4 hash that identifies the client library, since Go, Python and curl each produce a distinctive one. Above that, HTTP/2 settings values and header ordering fingerprint the engine. Only afterwards does anything JavaScript-based run.

The browser stage is a challenge page. The edge serves a script that collects canvas and WebGL hashes, font and screen metrics, audio characteristics and timing measurements, solves a small proof-of-work, and posts the result back. Passing sets a clearance cookie with a lifetime measured in minutes or hours, and later requests ride on it. A scraper that solves one challenge and then throws away its cookie jar earns a fresh challenge on every page, which by itself looks nothing like a browser.

Behavioural scoring continues once a session is running. Request pacing, the order pages are visited in, whether images and stylesheets were fetched alongside the HTML, pointer and touch events, and how fast forms are completed all feed a running score. Vendors also correlate device and IP signals across their whole customer base, so a fingerprint burned on one site can arrive pre-scored at the next. For logged-in traffic, account age and past challenge outcomes weigh in too.

Downstream you see a 403, a 429, a challenge loop that never resolves, or Cloudflare's 1020 code. Mobile IPs clear the network-layer checks more often because carrier-grade NAT places thousands of real subscribers behind one address, which makes blocking that address expensive for the site. That is one input among many. A mobile exit paired with a headless container that reports SwiftShader and clicks within milliseconds of page load gets scored on everything else and blocked anyway.

Where you meet it

Bot detection announces itself as a sudden change in status codes, when a pipeline that returned 200s for weeks starts serving challenge pages after a vendor update or a traffic spike. Diagnose by layer instead of guessing. Retry the same request from a different IP class, then from a real browser, then at slower pacing, and see which change moves the response. Log the block page body, since the vendor name and a ray or incident ID are usually inside it.

Common questions

Why do I still get CAPTCHAs when using residential proxies?

The IP is one input into a score. Heavily resold residential ranges carry their own history, and everything above the network layer keeps being measured: TLS fingerprint, browser values, cookie handling, request pacing and navigation order. A clean address with a contradictory browser profile behind it still scores as automation.

What is the difference between bot detection and a WAF?

A web application firewall inspects request content for attack patterns such as SQL injection or path traversal, matching rules against what is being sent. Bot management scores who is sending it, using fingerprints and behaviour. Most edge vendors sell both in one product, so a single block page can come from either engine.

Can bot detection tell the difference between a scraper and a real user?

Not with certainty, which is why it produces a probability score rather than a verdict. Site owners tune the blocking threshold against how many genuine customers they are willing to inconvenience, so aggressive settings catch real people and relaxed ones let automation through. Traffic consistent across every layer scores like a user.

Related terms

CAPTCHA
A CAPTCHA is a challenge a site serves to decide whether a visitor is a person or a script. Modern versions rarely show distorted text; they score behaviour, browser fingerprint and IP reputation in the background, and only show a puzzle when confidence is low. reCAPTCHA, hCaptcha and Cloudflare Turnstile dominate deployment.
Rate Limiting
Rate limiting is a server-side control that caps how many requests one client may send in a time window. The server counts requests against a key such as an IP address, API token or account, then refuses anything above the ceiling, most often with HTTP 429 and a Retry-After header saying when to come back.
Throttling
Throttling is a deliberate slowdown applied to traffic a server treats as excessive or low priority. Rather than refusing the request, the system queues it or caps the bandwidth it gets, so the client sees rising latency instead of an error. APIs, origin servers and anti-bot systems all use it, and mobile carriers apply it past a data cap.
Cloudflare
Cloudflare is a CDN and security company whose reverse proxy sits in front of a large share of public websites. Traffic to a protected domain reaches Cloudflare's network first, where caching, DDoS filtering, WAF rules and bot scoring run before anything reaches the origin server. The edge challenges or blocks whatever it judges automated.
WAF
A web application firewall inspects HTTP requests and responses at the application layer and blocks those matching rules for attacks such as SQL injection, cross-site scripting and path traversal. It runs in front of the application as a reverse proxy, a server module or a cloud service, and most deployments add IP reputation feeds and rate-based rules alongside signature matching.

Real 4G/5G mobile and residential IPs

PROXIES.SX runs carrier IPs in 100+ countries with HTTP and SOCKS5 on every endpoint. $4/GB down to $2.40/GB at volume, free endpoints and rotation, and your GB never expire.

View pricing Browse the glossary