How to Avoid IP Bans When Web Scraping

You had a working scraper. Three days in it started returning 403s, so you bought a bigger proxy pool, and now it returns 403s faster. That sequence is common enough to be a diagnostic in itself: if adding IPs does not help, the IP was not the problem.

A ban is a decision made by a system that scored your traffic. The IP is one input. Request headers, TLS handshake, timing, navigation order, and cookie behaviour are the others, and on any site that has spent money on bot management they matter more.

Read the block before you fix it

The response tells you which layer rejected you. Do not skip this step — the fixes are mutually exclusive and applying the wrong one costs days.

The last two are the ones people miss. A shadow ban — correct status, plausible HTML, wrong content — will quietly poison a dataset for weeks. Every crawler should assert on a known-good value in the parsed output, not just on the status code.

Rotate identity, not just address

An exit IP that changes every request while the User-Agent, header order, and TLS fingerprint stay identical is a more obvious signal than a single consistent IP. You have told the site that one client is arriving from two hundred networks.

Rotate the whole identity together, and keep the pieces coherent — a Windows Chrome User-Agent arriving with a Linux TLS fingerprint from a mobile carrier IP is not a plausible visitor.

import random
import httpx

GATEWAY = "http://user-country-us:[email protected]:8080"

PROFILES = [
    {
        "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
        "accept_language": "en-US,en;q=0.9",
    },
    {
        "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
              "(KHTML, like Gecko) Version/18.4 Safari/605.1.15",
        "accept_language": "en-GB,en;q=0.8",
    },
]

def fetch(url: str) -> httpx.Response:
    profile = random.choice(PROFILES)
    headers = {
        "User-Agent": profile["ua"],
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": profile["accept_language"],
        "Accept-Encoding": "gzip, deflate, br",
        "Upgrade-Insecure-Requests": "1",
    }
    with httpx.Client(proxy=GATEWAY, headers=headers, timeout=30.0) as client:
        return client.get(url)

Note what is absent: no X-Requested-With, no python-httpx in the User-Agent, no header set that a browser would never send. The cheapest ban you can earn is the one where you announce yourself.

Set a concurrency budget per domain, not per crawler

Nearly every self-inflicted block traces back to a single number: how many requests per second one origin sees from you. A pool of 10,000 IPs does not raise that ceiling, because rate limits on serious sites are applied per-account, per-fingerprint, and per-subnet as well as per-IP.

Pick a budget you can defend — 1 to 4 requests per second for a mid-size site is a reasonable start — and enforce it with a semaphore rather than hoping your worker count approximates it.

import asyncio
import httpx

GATEWAY = "http://user-country-us:[email protected]:8080"
CONCURRENCY = 4

async def worker(client: httpx.AsyncClient, sem: asyncio.Semaphore, url: str) -> str | None:
    async with sem:
        for attempt in range(4):
            response = await client.get(url)
            if response.status_code == 429:
                delay = float(response.headers.get("Retry-After", 2 ** attempt))
                await asyncio.sleep(delay)
                continue
            if response.status_code in (502, 503, 504):
                await asyncio.sleep(2 ** attempt)
                continue
            response.raise_for_status()
            return response.text
    return None

async def crawl(urls: list[str]) -> list[str | None]:
    sem = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient(proxy=GATEWAY, timeout=30.0) as client:
        return await asyncio.gather(*(worker(client, sem, u) for u in urls))

Exponential backoff with a respected Retry-After is not politeness theatre. A 429 that you hammer through becomes a 403 that lasts hours.

Hold a session wherever state exists

Per-request rotation is right for stateless list pages. It is wrong for anything involving a login, a cart, a multi-step form, or a search that carries filters in a cookie. Those flows expect one visitor at one address, and an IP that changes mid-journey reads as session hijacking to the origin — which is exactly what the defence is looking for.

Bind a sticky session by adding a session segment to the username. The same string returns the same exit IP for the life of the session:

curl -x http://user-country-us-session-a91f2c4b:[email protected]:8080 \
  -s https://example.com/cart

Generate one session identifier per logical journey, hold it for the whole journey, then discard it. Reusing one sticky session for a hundred thousand requests recreates the single-IP problem you were avoiding.

Request less

The highest-leverage change in most crawlers is not proxy configuration at all. It is fetching fewer pages.

• Deduplicate URLs before enqueueing them, including query-parameter permutations that render identically.
• Send If-Modified-Since or If-None-Match and treat 304 as success.
• Look for a JSON endpoint behind the HTML. One API call frequently replaces a page fetch plus twenty asset requests, and it is far less likely to be behind a challenge.
• Cache aggressively during development. Re-running a parser against saved HTML costs nothing and burns no reputation.

Halving your request volume halves your detection surface and your bandwidth bill at the same time.

When you should stop

Some blocks are a signal that you are on the wrong side of a line. If a site requires a login you do not have, if its terms prohibit the collection you are doing, if the data is personal, or if the defence is escalating specifically against you, the answer is not a better fingerprint. Check whether an official API exists, check what the law in your jurisdiction says about the data class, and get an opinion if the answer is not obvious.

Everything above assumes the collection itself is lawful and the traffic volume is something the origin can absorb. That assumption is doing real work, and it is worth confirming before you scale.