David Lewin David Lewin
Proxies & Web Scraping

How to scrape product prices and inventory from Amazon and Walmart without Cloudflare or CAPTCHA blocks

· Updated · 3 min read

Disclosure: some links in this guide are affiliate links. If you sign up through them, I may earn a commission at no extra cost to you. How this site makes money.

Short answer

Standard HTTP requests fail against Cloudflare Turnstile and Akamai because of TLS JA3/JA4 fingerprinting. To scrape reliably at scale, use either a managed Web Unlocker API (Bright Data / Oxylabs) or rotate residential proxies per request with a TLS-spoofed client like curl_cffi.

David Lewin From the field

"Trying to scrape Amazon or Walmart with basic Python `requests` and datacenter proxies will get you 503 CAPTCHA pages on your third request. After building a real-time price monitor for 50,000 SKUs, here is the exact architecture that delivers 99.9% success."

Why Standard Python Scrapers Get Blocked

Anti-bot firewalls inspect the TLS ClientHello handshake (JA3/JA4) before your HTTP headers even arrive. Python's standard urllib3 and requests libraries have a distinct cipher suite order that Cloudflare blocks instantly.

The Two Production Solutions

  • Path 1: Managed Web Unlocker / Scraper APIs (Zero Maintenance): Point your script at Bright Data Web Unlocker or Oxylabs E-Commerce Scraper API. They automatically manage browser TLS signatures, residential IP rotation, header ordering, and CAPTCHA solving—you only pay for HTTP 200 successful responses.
  • Path 2: DIY Rotating Residential + TLS Impersonation: Combine rotating residential gateways from Decodo or SOAX with Python's curl_cffi library (impersonating Chrome 124 TLS handshakes).
Pro tip

Always test your proxy exit IP on IPQualityScore.com and BrowserLeaks.com before running production traffic or logging into sensitive accounts.

Tools mentioned in this guide · affiliate links

Bright Data Web Unlocker Managed Unblocking for Hard Targets
Pay-as-you-go See offer →
Oxylabs Scraper API Best for Scalable Scraping
Free trial See offer →
ScraperAPI Scraping API with Proxy Rotation
Free trial credits See offer →
Decodo Rotating Pool Rotating Residential IP Pool
Pay-as-you-go or monthly See offer →
David Lewin
David Lewin

Founder of StealthScale.io and e-commerce infrastructure engineer writing field-tested playbooks on proxies, cloud VPS, anti-detect browsers and SEO.

Proxies & Web Scraping3 min read

Puppeteer & Playwright Getting Blocked? How to run headless browsers through Cloud Unlocker nodes

Default headless Chromium instances set `navigator.webdriver = true` and leak distinct CDP (Chrome DevTools Protocol) serialization patterns that Cloudflare and DataDome detect immediately. Connect Playwright/Puppeteer via WebSocket (`wss://`) to a remote Scraping Browser cluster (Bright Data / Oxylabs) to bypass bot checks automatically.

Proxies & Web Scraping5 min read

Residential vs. ISP vs. Datacenter vs. Mobile Proxies: Which one do you actually need?

Datacenter proxies are the cheapest and fastest choice for unprotected public data. Rotating residential proxies handle bot-protected retail sites. Static ISP proxies give a fixed, ISP-registered IP for long-lived account logins. Mobile proxies route through carrier networks for mobile-first social and ad platforms. Choose by use case, not price.

Proxies & Web Scraping3 min read

Why your residential proxy bandwidth drains so fast (and how to cut it)

Residential proxies bill per gigabyte, and a full browser page load can transfer several megabytes of images, scripts and fonts you never use. Cut costs by fetching raw HTML or JSON instead of rendering, blocking heavy resources in headless browsers, caching unchanged pages, scraping less often, and moving unprotected requests to cheap datacenter proxies.