Estimating Proxy Bandwidth Before You Buy

"How much bandwidth do I need?" is the question we are asked most often before a purchase, and the estimate people arrive at on their own is usually low by a factor of three to ten. The arithmetic is not hard; the inputs are just wrong in predictable ways.

What actually counts

Metered bandwidth is every byte crossing the proxy in both directions, not the size of the data you keep. That includes:

• Request headers and bodies (small, but not zero at scale)
• Response headers
• The HTML
• Every subresource your client fetches — CSS, JS, images, fonts, analytics beacons, tracking pixels
• Redirect hops, each a full request/response pair
• Retries, including the ones your library performs invisibly
• TLS handshake overhead on every new connection
• Failed requests. A 403 challenge page is often 40–90 KB and it counts

That last point is where estimates go wrong at low success rates. At 40% success, 60% of your bandwidth buys nothing.

The four causes of underestimation

1. Measuring the parsed output instead of the transfer. The extracted record is 2 KB. The page it came from was 340 KB. You are billed for the page.

2. Forgetting subresources. A raw HTML fetch of a typical e-commerce page is 150–400 KB. The same page in a headless browser with assets enabled is 2–6 MB. That is a 10–20x difference produced by a single configuration choice.

3. Ignoring failures and retries. Multiply the raw estimate by 1 / success_rate. At 70% success, add 43%. At 40%, you have more than doubled it.

4. Compression assumptions. If you do not send Accept-Encoding: gzip, deflate, br, you are billed for uncompressed HTML — typically 3–5 times larger. Most clients send it by default; most custom header dictionaries that override the defaults do not.

The formula

GB = (pages * avg_page_bytes) / (success_rate * 1_073_741_824)

Reference figures from customer traffic, HTML-only with gzip:

Worked example: 500,000 product pages, 320 KB each, 92% success.

(500_000 * 327_680) / (0.92 * 1_073_741_824) = 165.8 GB

Budget 180 GB. Round up — the variance on real crawls is wider than the arithmetic suggests.

Measure instead of guessing

Sample 200 pages and read the real number:

import httpx

PROXY = "http://user-country-us:[email protected]:8080"

def sample_bytes(urls: list[str]) -> None:
    """Report mean transfer per page, including failures, as the proxy meters it."""
    total = 0
    ok = 0
    with httpx.Client(proxy=PROXY, timeout=30.0, follow_redirects=True) as client:
        for url in urls:
            try:
                response = client.get(url)
            except httpx.HTTPError:
                continue
            total += len(response.content) + sum(
                len(k) + len(v) for k, v in response.headers.items()
            )
            if response.status_code == 200:
                ok += 1
    n = len(urls)
    print(f"mean {total / n / 1024:.1f} KB/page over {n} pages, {ok / n:.0%} success")
    print(f"projected for 1M pages: {total / n * 1_000_000 / 1024**3:.1f} GB")

Two hundred pages costs a few megabytes and turns a guess into a forecast. The dashboard's per-hour usage chart gives you the same figure from the other side once traffic is flowing.

Cutting usage by 60–90%

In rough order of leverage:

Block subresources in headless browsers. The single largest saving available. Playwright:

await page.route("**/*", (route) => {
  const type = route.request().resourceType();
  const blocked = ["image", "media", "font", "stylesheet"];
  return blocked.includes(type) ? route.abort() : route.continue();
});

Typically an 80–90% reduction. Verify your selectors still resolve — a few sites lay out content in CSS in ways that break when stylesheets are blocked, though far fewer than people expect.

Do not use a headless browser at all unless the content requires JavaScript. Fetch the HTML and check whether the data is there. Very often it is, in a __NEXT_DATA__ or application/ld+json block that is smaller and more stable than the rendered DOM.

Find the JSON endpoint. Open the network tab, filter to XHR, and look for the call that populates the page. One 8 KB JSON response replacing a 400 KB page render is a 50x saving and a more reliable parser.

Send conditional requests. If-None-Match and If-Modified-Since turn an unchanged page into a 304 of a few hundred bytes. On a daily re-crawl of slow-moving data this alone can cut usage by 70%.

Deduplicate before fetching. Canonicalise URLs, strip tracking parameters, and keep a seen-set. Most crawlers fetch the same page several times under different query strings.

Raise the success rate. Every point of success rate is a proportional bandwidth saving. Getting from 60% to 90% cuts consumption by a third for identical output.

Buying

Once you have a measured number: buy a small amount, run a real slice of the workload, read the dashboard, then scale. Bandwidth on FleetProxy plans does not expire, so over-buying costs you nothing except the timing of the spend — which makes the volume tiers worth taking as soon as your forecast is stable rather than topping up in small increments at the highest rate.