Using Proxies With Python: requests, httpx, and aiohttp

Every Python HTTP client supports proxies and every one of them exposes it differently. This is the reference I wish existed the first time a working requests script had to be ported to httpx under a deadline.

All examples point at a gateway of the form http://user-country-us:[email protected]:8080. Substitute your own credential; the targeting segments after the username control geography.

requests

requests takes a dict keyed by scheme. Both keys point at the same HTTP proxy — the https key means "use this proxy for https URLs", not "the proxy speaks TLS".

import requests

PROXY = "http://user-country-us:[email protected]:8080"
proxies = {"http": PROXY, "https": PROXY}

response = requests.get(
    "https://api.fleetproxy.com/v1/ip",
    proxies=proxies,
    timeout=30,
)
print(response.json()["ip"])

For anything beyond a single call, use a Session. It reuses the tunnel instead of renegotiating TLS on every request, which on a proxied connection is the difference between 400ms and 40ms per call.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
session.headers.update({"User-Agent": "fleet-crawler/1.0 (+https://example.com/bot)"})

retry = Retry(
    total=4,
    backoff_factor=0.8,
    status_forcelist=(429, 500, 502, 503, 504),
    allowed_methods=frozenset(["GET", "HEAD"]),
    respect_retry_after_header=True,
)
session.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=20))

response = session.get("https://api.fleetproxy.com/v1/ip", timeout=30)

Two things bite here. session.proxies is overridden by a per-call proxies= argument, so a stray argument in one helper silently sends that request direct. And requests reads HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment unless you set session.trust_env = False — which is how a script that works locally starts bypassing the proxy inside a CI container that exports its own.

httpx

httpx uses a single proxy argument and has the same API sync and async. Note the singular: proxies= was removed in 0.28.

import httpx

PROXY = "http://user-country-us:[email protected]:8080"

with httpx.Client(proxy=PROXY, timeout=30.0, follow_redirects=True) as client:
    response = client.get("https://api.fleetproxy.com/v1/ip")
    print(response.json()["ip"])

Async, with bounded concurrency — the shape most crawlers want:

import asyncio
import httpx

PROXY = "http://user-country-us:[email protected]:8080"

async def fetch_all(urls: list[str], concurrency: int = 8) -> list[str]:
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
    sem = asyncio.Semaphore(concurrency)

    async with httpx.AsyncClient(proxy=PROXY, limits=limits, timeout=30.0) as client:
        async def one(url: str) -> str:
            async with sem:
                response = await client.get(url)
                response.raise_for_status()
                return response.text

        return await asyncio.gather(*(one(u) for u in urls))

print(len(asyncio.run(fetch_all(["https://example.com"] * 20))))

Route different hosts through different exits by passing a mounts dict — useful when one target needs a UK exit and the rest do not:

mounts = {
    "all://*.co.uk": httpx.HTTPTransport(
        proxy="http://user-country-gb:[email protected]:8080"
    ),
    "all://": httpx.HTTPTransport(proxy=PROXY),
}
client = httpx.Client(mounts=mounts, timeout=30.0)

aiohttp

aiohttp takes the proxy per request, not per session, and authentication goes in a BasicAuth object rather than the URL.

import aiohttp

PROXY = "http://gate.fleetproxy.com:8080"
AUTH = aiohttp.BasicAuth("user-country-us", "pass")

async def fetch(url: str) -> str:
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url, proxy=PROXY, proxy_auth=AUTH) as response:
            response.raise_for_status()
            return await response.text()

Credentials embedded in the proxy URL are ignored by aiohttp, which is the single most common cause of a 407 in an otherwise correct aiohttp script. Use proxy_auth.

SOCKS5

SOCKS5 needs an extra package in every client. Install requests[socks], httpx[socks], or aiohttp-socks, then use the socks5h scheme so DNS resolves at the proxy rather than on your machine:

PROXY = "socks5h://user-country-us:[email protected]:1080"

The h matters. With plain socks5, your local resolver sees every hostname you visit and geo-aware CDNs resolve against your real location, which defeats the point of a country-targeted exit.

Sticky sessions

Per-request rotation is the default: each request may exit from a different address. To hold one address across a login or a multi-step flow, add a session segment to the username.

import uuid

def sticky_proxy(country: str = "us") -> str:
    """One exit IP for the life of a logical journey (login, cart, checkout)."""
    session_id = uuid.uuid4().hex[:12]
    return f"http://user-country-{country}-session-{session_id}:[email protected]:8080"

Sessions hold for up to 30 minutes of activity. After that the binding is released and a new address is assigned, so treat the session as a lease and be ready to re-authenticate if the flow runs longer.

A verification snippet worth keeping

Before blaming a target site, confirm what the target actually sees:

import httpx

with httpx.Client(proxy=PROXY, timeout=20.0) as client:
    print(client.get("https://api.fleetproxy.com/v1/ip").json())
    # {"ip": "..", "country": "US", "city": "..", "asn": "AS7922"}

If the country in that response is not what you asked for, the problem is the credential, not the crawler. If it is correct and the target still blocks you, the problem is fingerprint or rate, and no amount of proxy configuration will fix it.