Data and accounts
Proxy types and providers, scraping APIs, headless browsers and what they cost per page.
15 guides15 in-depth140 min of reading
Collecting product and price data at volume is mostly a cost problem with a reliability problem inside it. Each request has a price: the proxy traffic it uses, the browser time it takes to render, and the retries when it fails. Those prices change by an order of magnitude with the target. A plain HTTP fetch of a public JSON endpoint through a datacenter IP costs almost nothing. The same page rendered in a headless browser through residential proxies, billed per gigabyte, can cost hundreds of times more. Most of that extra cost is images, fonts and scripts the scraper never reads. The guides in this topic start from that unit cost, measured per 1,000 successful pages rather than per request, because a cheap request that fails half the time is not cheap.
Residential, ISP, datacenter and mobile proxies matches each proxy type to a target. Datacenter suits unprotected public data, rotating residential suits bot-protected retail sites, and static ISP suits long-lived logins. Proxy rotation in Python handles failures: bench a proxy for a few minutes after a timeout, 403, 429 or 5xx, retry through another with backoff, and keep a sticky session for multi-step flows. Proxy health monitoring compares providers on success rate, p50 and p95 latency and cost per 1,000 successful requests, using a standard-library Python script and SQLite. Cutting residential bandwidth covers what to do once the bill arrives: fetch HTML or JSON instead of rendering, block heavy resources and cache pages that haven't changed.
The second decision is whether to run the browsers yourself. A self-hosted Playwright cluster uses a Redis stream as a job queue, retries with exponential backoff and moves jobs that keep failing to a dead-letter stream, starting at 2 browsers with 3 pages each on a 4 GB server. ZenRows, Zyte API, Firecrawl and Browserless move rendering, IP rotation or extraction to a paid API, and each guide prices that against the DIY version. When a site actively blocks automated access, how Cloudflare and DataDome score requests explains the signals and the legitimate options: an official API or feed, allowlisting, slower identified crawling, or licensed data.
Datacenter proxies are the cheapest and fastest choice for unprotected public data. Rotating residential proxies handle bot-protected retail sites. Static ISP proxies give a fixed, ISP-registered IP for long-lived account logins. Mobile proxies route through carrier networks for mobile-first social and ad platforms. Choose by use case, not price.
ZenRows renders the page in its own headless browser, routes it through rotating residential IPs and handles anti-bot challenges, then returns the finished HTML from a single GET request. With JavaScript rendering and premium proxies a page costs 25 credits, which works out to about $6.90 per 1,000 pages on the $69 Launch plan and about $2.75 per 1,000 on the $549 Scale plan. Running Playwright yourself makes the browser almost free, but on a protected store you still pay for residential proxy bandwidth and for the hours spent keeping stealth patches working: by our estimate about $190 to $460 a month for 10,000 protected pages, against $69 on ZenRows. Self-hosting wins on high-volume sites that don't block you.
Send the same set of real target URLs through every proxy you pay for, record each request's status, timing and bytes in SQLite, and compare providers on three numbers: success rate (a 200 response that contains the content you expect), p50 and p95 latency of successful requests, and cost per 1,000 successful requests. The script in this guide does that as a cron-friendly CLI with no dependencies beyond Python's standard library.
Datacenter proxies are cheap and fast, but their IP ranges belong to hosting companies, so heavily protected sites block them more often. Residential proxies route through consumer ISP addresses and are usually billed per GB at a much higher price. Start with datacenter IPs and switch only the targets that block them.
Best proxy providers in 2026: Webshare vs IPRoyal vs Decodo vs Bright Data, compared by use caseScraping publicly accessible pages is generally not unauthorized access under US law after hiQ v. LinkedIn, but that case still ended against hiQ on breach of contract, and copyright and privacy laws such as GDPR apply. Avoid logged-in areas that the terms forbid, avoid personal data, and keep request rates low. This is not legal advice.
Multiply pages per run by average page weight and add retries. Blocking images, fonts and media in the browser, or reading JSON endpoints instead of full pages, cuts the bytes per page sharply. Measure a sample run before you buy a per-GB plan.
Residential proxy bandwidth budget: measure a page load from a HAR file, project GB per month, and cut it by collecting lessRotating sessions give each request a new IP, which suits independent page fetches. Sticky sessions keep one IP for several minutes, which logins, carts and multi-step flows need, because a changing IP mid-session looks suspicious. Most providers support both on the same plan.
Proxy rotation in Python: choosing datacenter, ISP, residential or mobile IPs, sticky sessions, circuit breakers and per-host backoffFirecrawl takes a URL and a JSON schema, renders the page, and returns the product as structured JSON in one API call, for 5 credits a page (1 for the scrape plus 4 for the JSON format). 10,000 product pages a month is 50,000 credits, which fits the Standard plan at $83 a month billed yearly. A DIY pipeline with Playwright and Beautiful Soup costs little to run, but you still need an LLM call to turn the cleaned text into JSON, proxies for protected stores and a few hours a month of upkeep: by our estimate about $180 to $490 a month for the same 10,000 pages. DIY wins when you scrape a few sites with good schema.org data, or at very high volume.
Replace chromium.launch() with chromium.connect_over_cdp('wss://production-sfo.browserless.io?token=...') and your existing Playwright code drives a Chrome running on Browserless instead of your own server. Browserless bills units of up to 30 seconds of browser time per connection, so how you connect decides the bill: opening one session per page costs at least a unit a page, while reusing each session for many pages costs only the seconds you use. For 100,000 six-second pages a month, that's about $35 with reused sessions or $200 with a session per page, against about $124 for a 4 GB VPS plus an estimated two hours of upkeep. Keep workers at or below your plan's concurrent browsers, reconnect before the session time cap, and always close the browser.
Zyte API's automatic extraction takes a product URL and returns name, price, currency, availability and SKU as JSON, without any selectors, so a store redesign doesn't break your scraper. On pay-as-you-go it costs about $0.40 to $2.35 per 1,000 product pages fetched over HTTP, and about $1.30 to $17 per 1,000 when the page has to be rendered in a browser. Hand-written Playwright selectors cost almost nothing per page but need developer time every time a store changes its HTML. At an estimated half hour of maintenance per store per month, Zyte is cheaper until a single store passes roughly 16,500 to 35,000 HTTP pages a month on a moderate (tier 3) site.
Run Redis and several Playwright worker containers with Docker Compose on one VPS. Workers pull jobs from a Redis stream, render pages in a shared Chromium with a few pages open at once, retry temporary failures with exponential backoff, and move jobs that keep failing to a dead-letter stream. On a 4 GB, 2 vCPU server, start with 2 browsers and 3 pages each, then tune from measured memory and CPU.
Bot management scores every request on IP reputation, client signals, behaviour and the site owner's rules, then allows, challenges or blocks it. A challenge or a run of 403s means the owner doesn't want that traffic on those terms, so a well-built crawler detects it and stops the host instead of retrying or trying to solve it. Cloudflare marks challenge responses with the cf-mitigated: challenge header. Treat 429 as slow down, with additive increase and multiplicative decrease of the interval. Then get the data legitimately: an official API or feed, allow-listing by the owner, an identified and signed crawler, or licensed datasets. The tested script in this guide reads a crawl log and decides continue, slow down or stop per host.
Pick Apify for ready-made scrapers (Actors) for Amazon, Google Maps and social platforms, plus scheduling and storage. Pick ScrapingBee if you write code and want one API call that handles proxies and JavaScript rendering. Pick Browse AI for point-and-click monitoring of a few competitor pages, synced to Google Sheets, without code.
Webshare is the cheapest place to start, with 10 free datacenter proxies and low-cost static plans. IPRoyal suits irregular workloads because its residential traffic doesn't expire. Decodo is the balanced mid-market pick with simple dashboards and scraping APIs. Bright Data costs the most but has the deepest network and unblocking tools.
Residential proxies bill per GB in both directions, so the budget is bytes per page times pages times runs. Measure bytes per page from a HAR file of a real page load instead of guessing, then compare four ways of collecting the same data: a full browser load, a browser with images, media, fonts and analytics blocked, the HTML alone with a plain client, and the HTML only for pages that changed (conditional requests answered 304 for the rest). In the tested example below, the gap between the first and the last is about 60 times. The tested script in this guide reads a HAR and prints that comparison with your page count, schedule and price per GB.
A Squid proxy on a small VPS gives your team and your scripts one fixed datacenter IP with bandwidth included in the server price. It fits allow-listing (suppliers, APIs, admin panels that accept only known IPs), a stable egress for webhooks and monitors, and outbound filtering. It doesn't fit jobs that need many IPs or consumer locations; that's what commercial residential and ISP pools are for. Run it with authentication, a firewall rule limiting the port to your own IPs, CONNECT only to port 443, no forwarding headers and no caching, then read the access log weekly. The tested script in this guide reports usage per user and destination, refusals, probing from unknown IPs and egress against your plan.
Most product pages don't need a browser: the data is in the raw HTML, in a JSON-LD block, in framework state such as __NEXT_DATA__, or in a JSON endpoint the page calls. Check that first with a plain request, and render only the pages where the values appear after JavaScript runs. When you do render, block images, media and fonts, use one browser context per task, cap concurrency by memory, and capture the JSON responses the page fetches. The tested script in this guide makes that decision per URL and reports the JSON endpoint when one exists.
Pick the proxy type by what the job needs (datacenter for volume on unprotected sources, ISP for stable sessions, residential or mobile only where the answer depends on a consumer location), then build rotation around three rules. A 429 or 503 from the site means slow down that host, not switch IP: hold the host for Retry-After or an exponential backoff with jitter. A timeout, connection error or 403 counts against the proxy: after a few in a row, open its circuit breaker for a cooldown that doubles each time. A 407 is a credentials problem, so stop. Keep a session key on one proxy for multi-step tasks, and send one identified User-Agent. The tested standard-library module in this guide does all of this.
For Amazon, read prices and offers through the Selling Partner API (getPricing, getCompetitivePricing, getItemOffersBatch) and stay inside each operation's published rate and burst. For Walmart, use the Marketplace APIs for your own items and the affiliate API for catalog data. Crawl public pages only where the site's terms and robots.txt allow it, at a fixed interval per host with one identified User-Agent, and stop when a host challenges or blocks you. Before scheduling anything, count the calls each refresh needs against each source's limit: the tested script in this guide does that and flags any tier that can't be met.
Disclosure: these are affiliate links. If you sign up through them, I may earn a commission at no extra cost to you. How this site makes money.
ScraperAPIScraping API with Proxy Rotation
Webshare ProxiesBudget Datacenter & Residential IPs
Vultr Cloud ComputeGlobal Cloud VPS
DigitalOcean DropletsDeveloper-Friendly Linux VPS
Zyte APIAI Product Extraction API
Oxylabs Residential ProxiesBest for Scalable Scraping
BrowserlessHosted Chrome for Playwright & Puppeteer
IPRoyal ResidentialNon-Expiring Residential Traffic