Copy-Paste Code: How to implement automatic proxy rotation, sticky sessions, and failover in Python

Most proxy rotation snippets online are a random.choice(proxies) inside a while True loop. That works until the first bad IP. Then the script hammers the same block page, retries instantly, triggers rate limits on every IP in the pool, and burns through bandwidth you pay for by the gigabyte.

Below is a small, dependency-light pattern (only requests) that handles the four things production scrapers need: failover, cool-down for bad IPs, backoff that respects the server, and sticky sessions when a flow spans several requests.

Two kinds of proxy setups

• A list of IPs (typical for datacenter and static ISP plans): you get host:port pairs and must rotate them yourself. That's what the pool below is for.
• A rotating gateway (typical for residential plans): one hostname, and the provider assigns a new exit IP per connection. You control stickiness through parameters in the username. You still need retries and backoff, but "rotation" is a username change.

The code: a proxy pool with failover and cool-down

import random
import time
import uuid

import requests

RETRY_STATUS = {403, 407, 429, 500, 502, 503, 504}


class ProxyPool:
    """Pick a random healthy proxy; bench failing ones for `cooldown` seconds."""

    def __init__(self, proxies, cooldown=300):
        self.proxies = list(proxies)
        self.cooldown = cooldown
        self.benched = {}

    def pick(self):
        now = time.monotonic()
        live = [p for p in self.proxies if self.benched.get(p, 0) <= now]
        if not live:
            raise RuntimeError("every proxy is cooling down - slow down or add IPs")
        return random.choice(live)

    def bench(self, proxy):
        self.benched[proxy] = time.monotonic() + self.cooldown


def fetch(url, pool, session=None, attempts=4, timeout=20):
    """GET url through the pool, failing over to another proxy on errors/blocks."""
    s = session or requests.Session()
    error = None
    for n in range(attempts):
        proxy = pool.pick()
        try:
            r = s.get(url, proxies={"http": proxy, "https": proxy}, timeout=timeout)
        except (requests.ConnectionError, requests.Timeout) as e:
            error = e
            pool.bench(proxy)
        else:
            if r.status_code not in RETRY_STATUS:
                return r
            error = requests.HTTPError(f"{r.status_code} via {proxy}", response=r)
            pool.bench(proxy)
            wait = r.headers.get("Retry-After", "")
            if wait.isdigit():
                time.sleep(min(int(wait), 60))
                continue
        time.sleep(min(2 ** n, 30) + random.random())  # exponential backoff + jitter
    raise error

Usage:

pool = ProxyPool([
    "http://USER:[email protected]:8000",
    "http://USER:[email protected]:8000",
    "http://USER:[email protected]:8000",
])
r = fetch("https://example.com/product/123", pool)
print(r.status_code, len(r.content))

Why each piece is there

Benching instead of deleting

A 403 or 429 usually means "this IP, right now". It doesn't mean the IP is dead forever. Removing it shrinks the pool permanently, and retrying it immediately just extends the block. A five-minute cool-down keeps the IP in rotation while letting the target's rate-limit window reset.

Which status codes trigger a retry

• 403 / 429: blocked or rate-limited. Switch IP and slow down.
• 407: proxy authentication failed. If it repeats for every proxy, your credentials or IP whitelist are wrong, and retrying won't help.
• 5xx: server or gateway trouble. Worth one or two retries.
• 404, 410 and other 4xx: the page really isn't there. Return immediately rather than paying to ask again.

Backoff with jitter, and honoring Retry-After

Waiting 1s, 2s, 4s, 8s (plus a random fraction) stops a fleet of workers from retrying in lockstep. When the server sends Retry-After, believe it. It's the cheapest signal you'll ever get about what the site will tolerate.

Sticky sessions for multi-step flows

Some tasks need the same IP for several requests: search, then open a result, then check stock and shipping. A changing IP halfway through looks like session hijacking, and many sites tie cart and session cookies to it. With a rotating gateway, stickiness is set in the username:

def sticky_proxy(user, password, host, port, minutes=10):
    """Gateway-style sticky session. The username format differs per provider - check their docs."""
    sid = uuid.uuid4().hex[:8]
    return f"http://{user}-session-{sid}-sesstime-{minutes}:{password}@{host}:{port}"

# one session = one identity for the whole flow
proxy = sticky_proxy("customer-abc", "secret", "gate.example-proxy.com", 7000)
with requests.Session() as s:
    s.proxies = {"http": proxy, "https": proxy}
    s.get("https://example.com/search?q=desk+lamp", timeout=20)
    s.get("https://example.com/item/987", timeout=20)

The -session-…-sesstime-… syntax above is a common pattern, but each provider names these parameters differently (session, sessid, life, ttl). Copy the exact format from your provider's dashboard.

Scaling up: concurrency without getting blocked

• Cap concurrency per target domain, not just globally. Twenty workers on twenty different sites is fine. Twenty on one site is a DDoS from its point of view.
• Use a thread pool (concurrent.futures.ThreadPoolExecutor) with the same ProxyPool shared between threads. For thousands of concurrent requests, move to httpx or aiohttp with an asyncio.Semaphore. The pool logic stays the same.
• Reuse sessions. A requests.Session keeps connections alive, which cuts TLS handshakes and latency.
• Log per-proxy outcomes. If one subnet produces most of the 403s, ask the provider to replace it.

When rotation isn't the problem

If every IP in a quality pool gets a 403 on the very first request, the IP isn't what's being blocked. Modern bot protection checks the client's TLS fingerprint and HTTP/2 behavior before it looks at the IP, and plain requests doesn't look like a browser. Your options:

• Fetch the site's public JSON endpoints or structured data instead of rendered pages, where the terms allow it.
• Use a client that mimics browser TLS, such as curl_cffi.
• Use a scraping API such as ScraperAPI that handles rendering and unblocking for you, and pay per successful request instead of per GB.

More on this in why scrapers get blocked by Cloudflare and DataDome.