What is Rate Limiting?
Rate limiting is a server-side control that caps how many requests one client may send in a time window. The server counts requests against a key such as an IP address, API token or account, then refuses anything above the ceiling, most often with HTTP 429 and a Retry-After header saying when to come back.
Implementations come down to a counter and a clock. Fixed windows are the simplest and allow a burst at the boundary, where a client spends its full quota at 11:59:59 and again a second later. Sliding windows smooth that edge out. Token buckets, common in API gateways, refill at a steady rate and let a client spend saved capacity all at once, which is why a limit advertised as ten requests per second may swallow fifty in a burst after an idle minute.
What the limit counts against decides how easily load can be spread. An API key is unambiguous. An address is not, so sites widen the counter to a /24 range or to a whole ASN once they see coordinated traffic from neighbouring addresses, which collapses a subnet of datacenter proxies into a single bucket. Logged-in traffic is normally counted per account, meaning a hundred exits behind one session share one allowance and rotation buys nothing.
Well-behaved services tell you where you stand. X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset are the widely used convention, and IETF work has been moving toward standardised RateLimit fields. A 429 with Retry-After is the polite refusal, the header carrying either a delay in seconds or an HTTP date. Plenty of sites are less generous and answer with 403, an empty 200, deliberately stale data, or a connection reset with no explanation at all.
Handling it well means backing off instead of retrying immediately. Exponential backoff with jitter stops a fleet of workers from synchronising their retries into a second wave against the same endpoint. Cap concurrency per exit address rather than globally, and treat a rising 429 rate as an instruction to slow down instead of a transient error to swallow. Adding addresses spreads load across more buckets, and does nothing when the limit is keyed to the account you logged into.
Where you meet it
This shows up as a crawl that starts clean and then degrades. The first thousand pages return 200, then 429s creep in and the worker pool retries into the same wall. The choice is whether to spread requests across more addresses or to slow each one down, and the response headers usually settle it. A limit keyed to your API token or your account follows you across every proxy you own.
Common questions
Does rotating proxies get around rate limits?
Only where the limit is keyed to the IP address. If the site counts per account, per API key or per device fingerprint, changing exit changes nothing and the extra addresses add cost without adding throughput. Check which identifier the 429 response or the API documentation names before you scale the pool.
What does HTTP 429 mean?
Too Many Requests. The server understood the request and is refusing it because you exceeded a limit. It often carries a Retry-After header with a delay in seconds or a date. Treat it as an instruction rather than a failure: wait the stated interval, and lower concurrency if 429s keep arriving after you resume.
What is the difference between rate limiting and throttling?
Rate limiting refuses the excess, usually with a 429 you can see in the log. Throttling accepts the request and makes it slow, by queuing it or shaping bandwidth. A refusal is obvious in your metrics. A throttle looks like a network problem, so teams often chase latency for hours before recognising it.
Related terms
Real 4G/5G mobile and residential IPs
PROXIES.SX runs carrier IPs in 100+ countries with HTTP and SOCKS5 on every endpoint. $4/GB down to $2.40/GB at volume, free endpoints and rotation, and your GB never expire.