Anti-Bot

What is Throttling?

Throttling is a deliberate slowdown applied to traffic a server treats as excessive or low priority. Rather than refusing the request, the system queues it or caps the bandwidth it gets, so the client sees rising latency instead of an error. APIs, origin servers and anti-bot systems all use it, and mobile carriers apply it past a data cap.

Server-side this is usually a queue with a leak rate. The nginx limit_req module accepts a burst above the configured rate and holds the excess in a queue, releasing requests at the rate you set; add nodelay and it stops holding them and rejects the overflow instead. API gateways do the same work with token buckets. Content systems have a softer variant, answering from a stale cache or dropping expensive personalisation so the response is cheap to produce rather than slow to deliver.

From the client side, throttling and a congested network are hard to separate. Time to first byte climbs while the transfer itself stays fast, which points at a server-side hold rather than a bandwidth problem. Response times often cluster around suspiciously round values, because a queue with a fixed leak rate produces exactly that shape. Timeouts follow whenever your client's deadline is shorter than the delay the far end has decided to impose on you.

Anti-bot systems prefer throttling to blocking for a practical reason. A block tells the operator precisely what happened and invites them to change one variable and try again, while a slowdown burns their time and their bandwidth budget without giving clear feedback. Some systems pair it with degraded content that arrives as real HTML carrying thinned result sets or stale prices. That is worse than a 403, because the pipeline keeps returning 200 and quietly stores data you should not trust.

The remedy is pacing rather than volume. Cut concurrent connections per exit address and put a randomised gap between requests in the same session. Set client timeouts long enough to tell a throttle apart from a dead connection, and measure time to first byte separately from transfer time so you can see which half is degrading. On mobile networks the carrier itself may be shaping traffic after an allowance is spent, and no rotation policy fixes that.

Where you meet it

You run into throttling when a scraper's throughput halves overnight with nothing in the error log. Everything returns 200, nothing is blocked, and the queue drains slower than it did last week. The first thing to settle is where the delay sits, whether in your own concurrency, the proxy path, or a server that has decided to hold your requests. Splitting first-byte time from download time usually answers that in one run.

Common questions

How do I tell if I am being throttled or just have a slow proxy?

Compare time to first byte against download time, and run the same request through a second exit. A slow proxy inflates both numbers and affects every destination equally. A throttle looks like a long wait before the first byte from one specific host, while other sites through that same proxy answer at normal speed.

Does throttling mean I have been detected?

Often, though not always. Shared infrastructure, carrier shaping after a data cap and ordinary origin load produce the same symptom. Detection is the likelier explanation when the slowdown tracks your request rate, begins right after a burst, or hits one destination while every other target you crawl stays fast.

What delay between requests avoids throttling?

There is no universal figure, since the threshold is set per site and often per endpoint. Start with low concurrency per address and a randomised gap rather than a fixed one, raise the pace until latency starts climbing, then settle below that point. Perfectly regular intervals are themselves a signal that a script is driving.

Related terms

Rate Limiting
Rate limiting is a server-side control that caps how many requests one client may send in a time window. The server counts requests against a key such as an IP address, API token or account, then refuses anything above the ceiling, most often with HTTP 429 and a Retry-After header saying when to come back.
Bot Detection
Bot detection is the practice of separating automated traffic from human traffic. Systems score each request against network signals, transport-layer fingerprints, browser characteristics and behaviour over time, then allow, challenge or block it. Vendors including Cloudflare, DataDome, Akamai and HUMAN sell this as a service running in front of the origin server.
CAPTCHA
A CAPTCHA is a challenge a site serves to decide whether a visitor is a person or a script. Modern versions rarely show distorted text; they score behaviour, browser fingerprint and IP reputation in the background, and only show a puzzle when confidence is low. reCAPTCHA, hCaptcha and Cloudflare Turnstile dominate deployment.
Cloudflare
Cloudflare is a CDN and security company whose reverse proxy sits in front of a large share of public websites. Traffic to a protected domain reaches Cloudflare's network first, where caching, DDoS filtering, WAF rules and bot scoring run before anything reaches the origin server. The edge challenges or blocks whatever it judges automated.
WAF
A web application firewall inspects HTTP requests and responses at the application layer and blocks those matching rules for attacks such as SQL injection, cross-site scripting and path traversal. It runs in front of the application as a reverse proxy, a server module or a cloud service, and most deployments add IP reputation feeds and rate-based rules alongside signature matching.

Real 4G/5G mobile and residential IPs

PROXIES.SX runs carrier IPs in 100+ countries with HTTP and SOCKS5 on every endpoint. $4/GB down to $2.40/GB at volume, free endpoints and rotation, and your GB never expire.