What is Browser Automation?
Browser automation is the scripted control of a real browser, navigating, clicking, typing and reading the rendered DOM in place of a person doing it by hand. Playwright, Puppeteer and Selenium drive Chromium, Firefox and WebKit over a debugging or WebDriver protocol, headless or with a visible window, for testing, scraping and agent work.
Underneath the API sits a wire protocol. Puppeteer and Playwright talk to Chromium over the Chrome DevTools Protocol on a WebSocket, which gives fine-grained control of network interception, cookies and page lifecycle. Selenium speaks W3C WebDriver to a driver binary that translates commands for each browser, and WebDriver BiDi is bringing bidirectional, event-driven control to that standard. Playwright also ships patched Firefox and WebKit builds so one script can run across engines. The protocol you land on decides what you can intercept and how you configure a proxy.
Driving a browser is expensive, so the first question is whether you need one at all. Each instance holds hundreds of megabytes and takes seconds to start, while the data you want frequently arrives through a JSON endpoint the page itself calls. Reproduce that request and you skip the browser entirely. When rendering is unavoidable, block the request types you do not need. Images, fonts, media and analytics requests are most of the bytes on a modern page, and dropping them cuts metered bandwidth sharply while speeding up every navigation.
Automated browsers are identifiable in several ways at once. navigator.webdriver reads true under automation, and older headless Chrome announced itself in the user agent string. Attaching over CDP leaves its own observable traces. What a browser does get right is the network layer, since its TLS handshake and HTTP/2 behaviour match a real browser exactly, which is where handwritten HTTP clients give themselves away. Proxy configuration differs by tool. Chromium takes a command-line proxy flag but ignores credentials embedded in that URL, so Playwright and Puppeteer accept username and password as separate options.
Flakiness is the recurring cost of maintaining these scripts, and fixed sleeps cause most of it. Wait on a condition instead, such as an element becoming visible or a network response arriving. Playwright's locators retry assertions automatically, and Selenium has explicit waits for the same purpose. For parallelism, isolated browser contexts inside one browser process are much cheaper than one process per worker, and they keep cookies and storage separate. Turn on tracing or video capture in CI, because a failure you cannot reproduce locally is otherwise unreadable.
Where you meet it
The classic moment is a script that runs green on your machine and fails in CI, where the browser is headless, the screen size is different and the traffic leaves from a datacenter address. Work through it in that order. Reproduce headless locally, then inside the container, then from the server's network. Each layer changes what the site sends back, and only the last one needs a proxy to diagnose.
Common questions
Is browser automation the same as using a headless browser?
They overlap but are separate ideas. Browser automation is controlling a browser with code; headless means running it without a visible window. You can automate a browser with its window on screen, which is how most people debug, and headless is the mode used on servers and in CI where there is no display attached.
Why does my automation script work locally but get blocked on a server?
Two things change at once. The browser typically runs headless with a different viewport and font set, and the traffic now leaves from a hosting provider's IP range rather than a home connection. Sites score both, so you can receive a challenge page, a stripped-down variant or different content while the script itself is unchanged.
Playwright, Puppeteer or Selenium?
Playwright covers Chromium, Firefox and WebKit with auto-waiting locators and per-context proxy settings, which makes it the usual choice for new work. Puppeteer is Chromium-focused and lighter. Selenium's advantage is breadth, with mature bindings in many languages and wide grid infrastructure, which decides it for large existing test estates.
Related terms
Real 4G/5G mobile and residential IPs
PROXIES.SX runs carrier IPs in 100+ countries with HTTP and SOCKS5 on every endpoint. $4/GB down to $2.40/GB at volume, free endpoints and rotation, and your GB never expire.