Loading page content.
Loading page content.
Data collection
A crawler that works on your laptop and dies at 50,000 pages a day is not a code problem, it is an exit-IP problem. Rotating residential egress moves the failure rate from double digits back under one percent without touching your parser.
The problem
Every large scrape converges on the same wall. The first few thousand requests from one address succeed, the next few thousand get slower, and then the target starts returning 403s, interstitials, or a soft block: a 200 response containing a page that is missing exactly the data you came for. The last case is the expensive one, because a naive pipeline records it as a success and quietly poisons the dataset.
The trigger is almost never the parser. It is the tuple of exit IP, request rate, and TLS fingerprint. A single datacenter address issuing 40 requests a second to one hostname is a pattern no human produces, and every commercial anti-bot vendor scores it in the first minute. Hosting ranges carry that reputation before your first request: the address was flagged for something a previous tenant did, and your crawler inherits it.
Scaling horizontally makes it worse rather than better. Twenty threads behind one IP is twenty times the rate from one address. Twenty threads behind twenty addresses in the same /24 is a pattern too, and it is a cheaper one to detect than the first.
Why proxies
Rotation spreads a fixed request budget across many addresses so that no single address exceeds the per-IP rate any target enforces. Ten thousand requests an hour through one IP is a siren; the same ten thousand spread over four thousand residential addresses is ordinary consumer traffic in aggregate and unremarkable per address.
Residential addresses matter because of who owns the range. A request from a consumer ISP allocation looks like a customer, and blocking it has a cost the target has to weigh — false positives on real shoppers. A request from a hosting ASN has no such cost, so the threshold for blocking it is far lower. That asymmetry, not raw pool size, is what moves the success rate.
Recommended product
A fresh residential IP on every request, or one held for 30 minutes
from $1.20/GB
Targets that do not weigh ASN reputation — open data portals, most APIs, documentation sites — cost roughly half as much per gigabyte through the datacenter pool. Start there and escalate only the hostnames that fight back.
from $3.50/IP/wk
A crawl that authenticates and must be recognised as the same client on every run is better served by a fixed ISP address than by rotation.
Worked example
The single highest-value change to a scraper is retrying a blocked request on a different exit rather than on the same one. With per-request rotation that is free: reissue the request and the pool hands you a new address. The snippet below also treats a suspiciously short body as a failure, which is what catches soft blocks.
import requests
from requests.exceptions import RequestException
PROXY = "http://fp_8s2k4d19-country-us:[email protected]:8080"
PROXIES = {"http": PROXY, "https": PROXY}
# A block page is small. A product page is not. Length is a cruder signal than
# parsing, and it catches the 200-with-a-challenge case before the parser does.
MIN_BODY_BYTES = 8_000
BLOCKED_STATUSES = {403, 407, 429, 503}
def fetch(url, attempts=4):
for attempt in range(attempts):
try:
response = requests.get(
url,
proxies=PROXIES,
timeout=(10, 30),
headers={"Accept-Language": "en-US,en;q=0.9"},
)
except RequestException:
continue # New exit IP on the next attempt — no backoff needed.
if response.status_code in BLOCKED_STATUSES:
continue
if len(response.content) < MIN_BODY_BYTES:
continue # Soft block: 200, but not the page we asked for.
response.raise_for_status()
return response.text
raise RuntimeError(f"{url} failed on {attempts} distinct exits")Replace the credential with the one in your dashboard. The gateway host, port and username format are the same across every product.
Pitfalls
Each of these is common, cheap to fix, and expensive to leave in place. They are listed in roughly the order teams hit them.
Anti-bot systems return challenge pages with a 200 status precisely because naive clients record them as wins. Assert on the presence of the field you came for — a price, a title, a row count — and treat its absence as a failure. Without that assertion your success metric measures how well you fetch block pages.
A fresh residential IP paired with a stale, unusual User-Agent and a request header order no browser emits is a stronger signal than the IP ever was. Rotate the header set with the exit, keep the header order consistent with the client you claim to be, and prefer a real browser engine on targets that fingerprint TLS.
A session token pinned for the length of an eight-hour job puts the whole job on one address, which is the situation rotation exists to avoid. Hold a session for the length of one logical flow — a checkout, a paginated list — and drop it. The 30-minute ceiling is a maximum, not a target.
Bandwidth billing makes images, fonts and video the largest line on the invoice while contributing nothing to the dataset. Block non-document resource types at the browser level, or fetch HTML directly. On a typical retail crawl this cuts transfer by 60 to 80 percent without changing a single extracted field.
Questions
Start on 50MB of free residential bandwidth, measure your own targets, and scale into volume tiers that step the rate down as the job grows. Unused bandwidth never expires.
No card required for the trial. Cancel or downgrade at any time.