What is Web Scraping?

A scraper is a loop of three stages: fetch, parse, store. The fetch is usually a plain HTTP client such as requests, httpx or curl, which is fast and cheap because it downloads markup and nothing else. Parsing pulls fields out with CSS selectors, XPath or a JSON path. Before writing selectors against rendered HTML, open the network tab and look for the internal JSON endpoint the page itself calls. It is frequently reachable without authentication and more stable across redesigns than the DOM, and the records come back cleaner.

Rendering is the expensive fallback. When content exists only after JavaScript runs, Playwright, Puppeteer or Selenium drive a real browser engine, which multiplies both time and bytes per page. An HTML document of a few dozen kilobytes turns into megabytes once images, fonts and third-party scripts load. Blocking resource types you never parse removes most of that. Sites also fingerprint the client, reading the TLS handshake shape, the HTTP/2 settings frame, header order and casing, so a Python client announcing Chrome in its User-Agent is a contradiction that detection systems read easily.

Blocks arrive in several forms. An outright 403, or a 429 with a Retry-After header, is the visible version. A CAPTCHA interstitial is the middle. The quiet one is altered content, such as truncated listings or a stale cached page, served with a 200 status. That last case is why a monitor counting only HTTP status codes reports a healthy crawl while the database fills with nulls. Track parsed field coverage per run alongside status codes, and keep raw response bodies so you can reparse without refetching.

Proxies enter once per-IP limits rather than parsing become the constraint. Requests spread across a rotating pool raise the ceiling, and geo-targeted exits return the regional version of a page. Sticky sessions hold one address through a login or a multi-step flow. Politeness still applies. Read robots.txt and keep concurrency proportionate to the size of the site you are hitting. A 429 asks you to slow down, and retrying it immediately makes the next one arrive sooner. Public data collection is routine work, though terms of service, personal data law and login-gated content raise separate questions with real consequences.