Proxy health monitoring: measure success rate, p50/p95 latency and cost per provider with Python and SQLite

Proxy dashboards tell you how many gigabytes you used. They don't tell you how many of those requests returned the page you needed, how long the slowest ones took, or what each good page actually cost. Those three numbers decide which provider deserves your budget, and they change from week to week as pools rotate and targets update their bot protection. This guide gives you a small command-line tool that measures them on your own targets and keeps the history in SQLite, so you can compare providers on evidence instead of on their landing pages.

1. What to measure, and why these three numbers

Success rate

A request succeeds when it returns HTTP 200 and the body contains something only the real page has, such as "Add to cart" or a price element. Counting only status codes overstates quality, because many bot walls serve a CAPTCHA page with a 200 status. The --expect option catches those.

p50 and p95 latency

p50 (the median) is the typical request. p95 is the slow tail: 1 request in 20 is slower than this. Scrapers feel the tail, not the average. A pool with a fine median and a 6-second p95 will stall worker slots and push jobs into timeouts. Latency is measured on successful requests only, so fast failures don't make a bad pool look quick.

Cost per 1,000 successful requests

Providers bill bandwidth for every attempt, including blocked ones and retries. Dividing the total cost by successful requests turns a per-GB price into the number you actually pay per good page:

cost per 1,000 OK = (bytes of all attempts ÷ 1,000,000,000 × price per GB) ÷ successful requests × 1,000

A cheap pool with a low success rate can cost more per good page than an expensive one. That's the comparison this tool is built to make.

2. How the tool works

proxymon: from proxy list to alert

The CLI has three commands:

• run sends every URL through every proxy for a number of rounds and stores each result.
• report summarizes one run per provider and can exit with an error when a provider drops below a success threshold.
• trend shows the daily success rate and latency per provider, so you can spot a pool that's slowly degrading.

3. Set up the inputs

proxies.csv

One row per proxy endpoint you want to compare. Copy the gateway host and port from each provider's dashboard. For rotating residential pools, the gateway alone is enough because it gives a new exit IP per request.

provider,proxy_url,price_per_gb
webshare-dc,http://USER:PASS@GATEWAY_HOST:PORT,0.30
webshare-resi,http://USER:PASS@GATEWAY_HOST:PORT,3.50
iproyal,http://USER:PASS@GATEWAY_HOST:PORT,4.00
decodo,http://USER:PASS@GATEWAY_HOST:PORT,3.50
brightdata,http://USER:PASS@GATEWAY_HOST:PORT,5.00
oxylabs,http://USER:PASS@GATEWAY_HOST:PORT,4.00

• price_per_gb is what you pay on your plan, not the list price. The values above are placeholders to replace with your own.
• For plans priced per IP or per month (most datacenter and static ISP plans), enter an effective price: monthly cost ÷ the GB you expect to use. With unlimited bandwidth, enter the monthly cost ÷ your typical monthly GB from the dashboard.
• To compare countries or session types from the same provider, add one row each with a distinct provider label, such as decodo-us and decodo-de.

urls.txt

20 to 50 pages you really scrape, from a mix of your target sites. Don't use an IP-check or speed-test site: those are never blocked and tell you nothing about your targets.

# one URL per line; lines starting with # are skipped
https://shop.example.com/products/sample-1
https://shop.example.com/products/sample-2
https://marketplace.example.org/item/12345

Respect each site's terms and robots.txt, and keep the volume small. Monitoring needs a few hundred requests per run, not thousands.

4. The script: proxymon.py

A single file using only Python's standard library: urllib for requests through the proxy, sqlite3 for storage and concurrent.futures for parallel checks. It asks for gzip like a real browser and counts compressed bytes, which is close to what the provider meters. TLS and header overhead are estimated at 6 KB per request. Use your provider's usage page for the exact bill.

#!/usr/bin/env python3
"""proxymon: measure success rate, latency and cost per proxy provider.

Python 3.9+, standard library only. Results go to a local SQLite file.
  python3 proxymon.py run --proxies proxies.csv --urls urls.txt --expect "Add to cart"
  python3 proxymon.py report --fail-under 90
  python3 proxymon.py trend --days 14
"""
import argparse
import csv
import datetime
import gzip
import math
import sqlite3
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'
# ponytail: TLS handshake + headers estimated at 6 KB per request; the provider's usage page is the exact bill
OVERHEAD = 6000

SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (id INTEGER PRIMARY KEY, started TEXT NOT NULL, note TEXT);
CREATE TABLE IF NOT EXISTS checks (
  run_id INTEGER NOT NULL REFERENCES runs(id),
  provider TEXT NOT NULL, proxy TEXT NOT NULL, url TEXT NOT NULL,
  ok INTEGER NOT NULL, status INTEGER, error TEXT,
  ms REAL NOT NULL, bytes INTEGER NOT NULL, price_per_gb REAL NOT NULL, ts TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS checks_run ON checks (run_id, provider);
"""


def now():
    return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')


def connect(path):
    db = sqlite3.connect(path)
    db.executescript(SCHEMA)
    return db


def check(proxy, url, expect, timeout, delay):
    """One request through one proxy. Returns (ok, status, error, ms, bytes)."""
    opener = urllib.request.build_opener(urllib.request.ProxyHandler({'http': proxy, 'https': proxy}))
    req = urllib.request.Request(url, headers={'User-Agent': UA, 'Accept-Encoding': 'gzip'})
    status, raw, err = None, b'', None
    t0 = time.perf_counter()
    try:
        with opener.open(req, timeout=timeout) as r:
            status, raw = r.status, r.read()
            body = gzip.decompress(raw) if r.headers.get('Content-Encoding') == 'gzip' else raw
    except urllib.error.HTTPError as e:
        status, err, body = e.code, 'HTTP %d' % e.code, b''
    except Exception as e:  # timeouts, refused tunnels, TLS and auth errors
        err, body = '%s: %s' % (type(e).__name__, str(e)[:100]), b''
    ms = (time.perf_counter() - t0) * 1000
    ok = status == 200 and (not expect or expect.encode() in body)
    if status == 200 and not ok:
        err = 'expected text missing'  # usually a CAPTCHA or block page served with 200
    elif not ok and not err:
        err = 'HTTP %s' % status
    time.sleep(delay)
    return ok, status, err, ms, len(raw) + OVERHEAD


def cmd_run(a):
    db = connect(a.db)
    with open(a.proxies, newline='') as f:
        proxies = [p for p in csv.DictReader(f) if p['proxy_url'].strip()]
    with open(a.urls) as f:
        urls = [u.strip() for u in f if u.strip() and not u.startswith('#')]
    run_id = db.execute('INSERT INTO runs (started, note) VALUES (?, ?)', (now(), a.note)).lastrowid
    jobs = [(p, u) for _ in range(a.rounds) for u in urls for p in proxies]

    def work(job):
        p, u = job
        return job, check(p['proxy_url'], u, a.expect, a.timeout, a.delay)

    with ThreadPoolExecutor(a.workers) as ex:  # threads do the I/O; only this thread writes SQLite
        for i, ((p, u), (ok, status, err, ms, size)) in enumerate(ex.map(work, jobs), 1):
            db.execute('INSERT INTO checks VALUES (?,?,?,?,?,?,?,?,?,?,?)',
                       (run_id, p['provider'], p['proxy_url'].split('@')[-1], u, ok, status, err,
                        ms, size, float(p['price_per_gb']), now()))
            if i % 50 == 0 or i == len(jobs):
                db.commit()
                print('\r%d/%d checks' % (i, len(jobs)), end='', file=sys.stderr)
    print('\nrun %d saved to %s' % (run_id, a.db), file=sys.stderr)


def pct(xs, p):
    """Nearest-rank percentile; None for an empty list."""
    xs = sorted(xs)
    return xs[max(0, math.ceil(p / 100 * len(xs)) - 1)] if xs else None


def cmd_report(a):
    db = connect(a.db)
    run_id = a.run or db.execute('SELECT max(id) FROM runs').fetchone()[0]
    if run_id is None:
        sys.exit('no runs yet: start with "proxymon.py run"')
    by = {}
    for prov, ok, ms, size, price in db.execute(
            'SELECT provider, ok, ms, bytes, price_per_gb FROM checks WHERE run_id = ?', (run_id,)):
        by.setdefault(prov, []).append((ok, ms, size, price))
    print('run %d  %s' % (run_id, db.execute('SELECT started FROM runs WHERE id = ?', (run_id,)).fetchone()[0]))
    print('%-14s %6s %8s %8s %8s %8s %12s' % ('provider', 'checks', 'success', 'p50 ms', 'p95 ms', 'avg KB', '$/1k OK'))
    worst = 100.0
    for prov, rs in sorted(by.items()):
        good = [ms for ok, ms, _, _ in rs if ok]
        rate = 100 * len(good) / len(rs)
        worst = min(worst, rate)
        cost = sum(size * price for _, _, size, price in rs) / 1e9  # failed attempts use bandwidth too
        per_1k = '%.2f' % (cost / len(good) * 1000) if good else 'n/a'
        p50, p95 = (('%.0f' % pct(good, q)) if good else '-' for q in (50, 95))
        print('%-14s %6d %7.1f%% %8s %8s %8.0f %12s' % (
            prov[:14], len(rs), rate, p50, p95, sum(r[2] for r in rs) / len(rs) / 1000, per_1k))
    print('\ntop errors')
    for prov, err, n in db.execute('SELECT provider, error, count(*) FROM checks WHERE run_id = ? AND NOT ok '
                                   'GROUP BY 1, 2 ORDER BY 3 DESC LIMIT 8', (run_id,)):
        print('  %-14s %5d  %s' % (prov[:14], n, err))
    if a.fail_under and worst < a.fail_under:
        sys.exit('ALERT: a provider is below %.0f%% success' % a.fail_under)


def cmd_trend(a):
    db = connect(a.db)
    print('%-10s %-14s %6s %8s %8s' % ('day', 'provider', 'checks', 'success', 'avg ms'))
    for row in db.execute("SELECT date(ts), provider, count(*), 100.0 * avg(ok), avg(CASE WHEN ok THEN ms END) "
                          "FROM checks WHERE ts >= date('now', ?) GROUP BY 1, 2 ORDER BY 1, 2",
                          ('-%d days' % a.days,)):
        print('%-10s %-14s %6d %7.1f%% %8.0f' % (row[0], row[1][:14], row[2], row[3], row[4] or 0))


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('--db', default='proxymon.db')
    sub = ap.add_subparsers(dest='cmd', required=True)
    r = sub.add_parser('run', help='check every proxy against every URL')
    r.add_argument('--proxies', default='proxies.csv', help='CSV: provider,proxy_url,price_per_gb')
    r.add_argument('--urls', default='urls.txt', help='one target URL per line')
    r.add_argument('--expect', help='text that must appear in a good response')
    r.add_argument('--rounds', type=int, default=3)
    r.add_argument('--workers', type=int, default=8)
    r.add_argument('--timeout', type=float, default=20)
    r.add_argument('--delay', type=float, default=1.0, help='seconds each worker waits between requests')
    r.add_argument('--note', default='')
    p = sub.add_parser('report', help='success rate, latency and cost for one run')
    p.add_argument('--run', type=int, help='run id (default: latest)')
    p.add_argument('--fail-under', type=float, help='exit 1 if any provider is below this success %%')
    t = sub.add_parser('trend', help='daily success rate per provider')
    t.add_argument('--days', type=int, default=14)
    a = ap.parse_args()
    {'run': cmd_run, 'report': cmd_report, 'trend': cmd_trend}[a.cmd](a)


if __name__ == '__main__':
    main()

5. Run it as a CLI tool

First run

mkdir -p /opt/proxymon && cd /opt/proxymon
# copy proxymon.py, proxies.csv and urls.txt here, then:
chmod 600 proxies.csv            # it contains proxy credentials

python3 proxymon.py run --expect "Add to cart" --rounds 3 --note "baseline"
python3 proxymon.py report
python3 proxymon.py trend --days 14

With 6 providers, 30 URLs and 3 rounds, a run makes 540 requests. At 8 workers with a 1-second delay, expect it to finish in roughly 2–5 minutes, depending on pool latency.

Options

Schedule it and alert on drops

One small run every 30 minutes builds a useful history at low bandwidth cost. Add the alert to cron with crontab -e:

*/30 * * * * cd /opt/proxymon && python3 proxymon.py run --rounds 1 --expect "Add to cart" --note cron 2>>run.log && python3 proxymon.py report --fail-under 85 > last-report.txt || curl -fsS -X POST --data-binary @last-report.txt "$ALERT_WEBHOOK"

Set ALERT_WEBHOOK at the top of the crontab to a Slack, Discord or ntfy webhook URL. Any endpoint that accepts a POST works. The alert fires when a provider drops below 85% success, and the message carries the full report, including the top errors.

The database grows by about 150 bytes per check. At 30 URLs × 6 providers every 30 minutes, that's roughly 1.3 MB a day. To keep 90 days:

sqlite3 proxymon.db "DELETE FROM checks WHERE ts < date('now','-90 days'); VACUUM;"

6. Reading the report

Here's what a report looks like. The labels describe proxy types and the numbers are illustrative estimates for public product pages behind moderate bot protection. They aren't a measurement of any named provider.

run 42  2026-10-18T09:30:02+00:00
provider       checks  success   p50 ms   p95 ms   avg KB      $/1k OK
dc-shared         300    71.0%      420     1350      118         0.05
isp-static        300    90.3%      610     1600      170         0.19
mobile            300    97.0%     1650     4200      182         1.50
resi-a            300    94.0%     1050     2800      176         0.66
resi-b            300    96.7%     1010     2700      180         0.74

top errors
  dc-shared         61  HTTP 403
  dc-shared         26  expected text missing
  isp-static        22  HTTP 429
  resi-a            11  TimeoutError: The read operation timed out
  resi-a             7  expected text missing

• Shared datacenter is by far the cheapest per good page, even at 71% success. If your targets tolerate it, it's the right pool, and a retry through a second pool covers the rest.
• Static ISP errors are mostly 429s: the IPs are fine, but too few of them are taking too many requests. Spread the load across more IPs or slow down before you blame the provider.
• The two residential pools are close on success. The one with 2.7 points more success costs about 12% more per good page. Whether that's worth it depends on what a failed page costs you in retries and worker time.
• Mobile wins on success but costs more than twice as much per good page and has the slowest tail. Keep it for the targets where nothing else works.
• "Expected text missing" on a residential pool usually means a soft block: the site served a challenge page with a 200 status. If it grows across runs, the target has changed its protection.

7. Expected performance by proxy type

Based on typical industry performance for public e-commerce and catalog pages, you can expect ranges like these. Treat them as a starting point for your alert thresholds, then replace them with your own baseline after a week of runs.

Expected latency by proxy type (midpoint of the estimated range)

Alert thresholds that work as a starting point

• Success rate: alert when a provider drops more than 5 percentage points below its own 7-day average, or below a hard floor such as 85% for residential.
• p95 latency: investigate when it doubles against the provider's baseline. That usually means an overloaded region or a gateway problem, before success starts to fall.
• Cost per 1k OK: recheck it after any plan change, and whenever success moves by more than a few points.

8. What to expect from each provider

Every provider below sells several proxy types, so test the product you'd actually buy. The expected profile is based on each provider's product range and pricing model, not on our own measurements.

For a feature and pricing comparison of these providers, see Webshare vs IPRoyal vs Decodo vs Bright Data. To cut the bandwidth behind every number in the report, see how to cut proxy bandwidth costs.

9. A worked cost example

Take a 170 KB compressed product page (plus 6 KB overhead) and 100,000 good pages a month:

The provider choice moves the bill by about 12%. Fetching compressed HTML instead of full pages moves it five times over. Monitor providers to catch degradation, but fix page weight first.

10. Pitfalls that skew the numbers

• Testing against the wrong URLs. IP-lookup and speed-test sites are never blocked. Only your real targets give real success rates.
• Too few samples. With 50 requests, a single failure moves the success rate by 2 points. Compare runs of at least 200 requests per provider, or use trend across days.
• One time of day. Residential pools behave differently in the evening when home users are online. A cron schedule covers all hours automatically.
• Different GB definitions. The script uses 1 GB = 1,000,000,000 bytes. Some providers bill in GiB (1,073,741,824 bytes), a 7% difference. Match the script to your provider when you reconcile against the invoice.
• Testing like an attacker. Keep request volume small, honor robots.txt and terms, and never test login pages or accounts that aren't yours. Every provider's acceptable-use policy bans that, and it isn't monitoring.