What is Scrapy?
Scrapy is an open-source Python framework for large-scale web crawling. Its asynchronous engine keeps many requests in flight at once, while spiders, item pipelines and downloader middlewares keep parsing separate from transport. Proxies plug into the middleware chain, so a crawl can change exit IP addresses without any edit to spider code.
A Scrapy project is a set of spiders plus shared configuration. The engine, built on the Twisted event loop, runs in one process and handles concurrency without threads. A spider yields Requests and Items from its parse callbacks; the scheduler queues the Requests, a duplicate filter drops URLs already seen, and the downloader fetches them. Selectors take CSS or XPath expressions. Feed exports write JSON, JSON Lines, CSV or XML straight to a file or an S3 bucket, so a working crawler often needs no storage code at all.
HttpProxyMiddleware is enabled by default and reads request.meta['proxy'], falling back to the http_proxy and https_proxy environment variables. Set that meta value per request and each one leaves through a different address. The middleware strips credentials written into the proxy URL and moves them into a Proxy-Authorization header. HTTPS requests open a CONNECT tunnel through the proxy first, so a proxy that fails only on HTTPS is usually rejecting CONNECT rather than failing DNS. Rotation comes either from a short middleware of your own or from a single provider endpoint that rotates upstream.
Politeness settings decide whether a crawl finishes or gets you banned. CONCURRENT_REQUESTS defaults to 16 overall and 8 per domain, DOWNLOAD_DELAY inserts a randomised pause between requests to the same site, and AUTOTHROTTLE adjusts that delay from observed latency. RETRY_HTTP_CODES covers the usual server errors along with 429, and the retry middleware re-queues those attempts. ROBOTSTXT_OBEY is switched on in new projects. Raising concurrency without raising the size of the IP pool behind it is the fastest route to a blanket block.
Scrapy runs no JavaScript. A page whose content arrives from XHR calls will hand back an empty template, and the fix is either to call the underlying JSON API directly, which is faster anyway, or to bring in scrapy-playwright and pay browser costs per page. The other limitation is the handshake: Scrapy's TLS signature comes from Python's OpenSSL stack rather than a browser, so bot detection that fingerprints TLS can reject it regardless of which IP address it arrives from.
Where you meet it
You move to Scrapy when a script built out of requests and a loop turns into a nightly job over a few hundred thousand URLs and starts needing retries, deduplication, resumable queues and per-domain rate control. Two decisions come first: how much concurrency the IP pool behind it can support, and where the proxy assignment lives. The third is whether the target needs a rendering engine at all, or whether the page is reading a JSON endpoint you can call yourself.
Common questions
How do I rotate proxies in Scrapy?
Write a downloader middleware whose process_request sets request.meta['proxy'] to an address picked from your pool, and have process_exception put failed requests back in the queue with a different one. The simpler alternative is one rotating endpoint from a provider, where a new IP is issued upstream per request or per session.
Can Scrapy scrape JavaScript-rendered pages?
Not on its own; it fetches and parses HTML without executing scripts. Inspect the network tab first, because the data often comes from a JSON endpoint you can request directly. When rendering is required, scrapy-playwright runs a real browser for selected requests while the rest of the crawl stays lightweight.
Scrapy or requests with BeautifulSoup?
Use requests and BeautifulSoup for a few hundred pages or a one-off extraction. Scrapy earns its setup cost once you need concurrency, retry policy, deduplication, per-domain throttling, proxy middleware and structured exports, which is roughly the point where a hand-rolled script starts growing its own scheduler.
Related terms
Real 4G/5G mobile and residential IPs
PROXIES.SX runs carrier IPs in 100+ countries with HTTP and SOCKS5 on every endpoint. $4/GB down to $2.40/GB at volume, free endpoints and rotation, and your GB never expire.