Using Proxies with aiohttp: The Four Things That Fail Quietly

aiohttp handles proxies differently from requests, and the differences are the kind that fail quietly. Traffic still flows, the script still finishes, and the exit IP is not what you think it is.

This is the async counterpart to using proxies with Python requests. Everything below assumes aiohttp 3.9 or newer.

The basics: proxy is per request, not per session

requests takes a proxies dict on the session and applies it everywhere. aiohttp takes proxy as an argument to the individual request:

import aiohttp, asyncio

PROXY = "http://user-country-us:[email protected]:8080"

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://api.fleetproxy.com/v1/ip", proxy=PROXY
        ) as resp:
            print(await resp.json())

asyncio.run(main())

Forget the proxy= argument on one call out of forty and that call goes out on your own address. There is no warning, and in a scraper the symptom is a single unexplained block rather than an error.

Two ways to remove the risk. Either set trust_env=True and export HTTPS_PROXY, so an omission still routes through the proxy:

session = aiohttp.ClientSession(trust_env=True)

Or wrap the call so the argument cannot be forgotten:

class ProxiedSession:
    def __init__(self, session: aiohttp.ClientSession, proxy: str):
        self._session, self._proxy = session, proxy

    def get(self, url, **kw):
        kw.setdefault("proxy", self._proxy)
        return self._session.get(url, **kw)

The wrapper is better in a codebase with more than one author. Environment variables are invisible in a code review.

HTTPS through an HTTP proxy works; HTTPS to the proxy does not

This is the single most common aiohttp proxy question, and the answer is unintuitive.

Fetching an https:// URL through proxy="http://..." is fully supported — that is a CONNECT tunnel, and it is what everybody actually wants. What aiohttp does not support out of the box is speaking TLS to the proxy itself, meaning a proxy="https://..." URL. That raises ClientConnectionError or hangs, depending on version.

So http:// in the proxy URL is correct even when every target is HTTPS. Credentials are still protected: the CONNECT tunnel is established first and the TLS session is negotiated end-to-end with the target inside it.

SOCKS5 needs a separate package

aiohttp has no SOCKS support in the core library. The proxy= argument understands HTTP only, and passing a socks5:// URL raises ValueError: Only http proxies are supported.

Install aiohttp-socks and swap the connector:

from aiohttp_socks import ProxyConnector

connector = ProxyConnector.from_url(
    "socks5://user-country-us:[email protected]:1080"
)

async with aiohttp.ClientSession(connector=connector) as session:
    async with session.get("https://api.fleetproxy.com/v1/ip") as resp:
        print(await resp.json())

Note what moved: the proxy is now on the connector, not the request, so it applies to every call on that session. Use socks5h:// rather than socks5:// if you want hostnames resolved at the exit node instead of on your machine — on geo-targeted work, local resolution is what sends you to the wrong CDN edge.

FleetProxy speaks SOCKS5 on port 1080 with the same credentials as the HTTP gateway on 8080.

Connection pooling silently defeats rotation

This is the failure that matters most, and it is invisible.

aiohttp keeps a connection pool. Against a rotating endpoint, a pooled connection is a tunnel that has already been established through one exit address. Every request reusing that tunnel exits from the same IP, no matter that you asked for per-request rotation. A run of 10,000 requests can go out over a couple of dozen addresses while the dashboard shows healthy traffic and the block rate climbs.

# One tunnel, thousands of requests, one exit IP.
connector = aiohttp.TCPConnector(limit=100)

# Bounded reuse: connections retire, rotation resumes.
connector = aiohttp.TCPConnector(limit=100, ttl_dns_cache=300, force_close=True)

force_close=True closes each connection after its response, which restores true per-request rotation and costs you a TCP and TLS handshake every time. On a residential exit that handshake is not free.

The middle path is usually right: keep pooling, but cap connection lifetime with keepalive_timeout so tunnels retire on a schedule you chose.

connector = aiohttp.TCPConnector(limit=200, keepalive_timeout=15)

If you need genuinely distinct addresses per logical unit of work, do not fight the pool. Use a sticky session and make the session token the thing that varies:

def proxy_for(job_id: str, country: str = "us") -> str:
    return f"http://user-country-{country}-session-{job_id}:[email protected]:8080"

Same string, same exit. Different string, different exit. That is deterministic in a way connection pooling is not.

Timeouts are not the default you expect

aiohttp's default total timeout is 5 minutes. Through a proxy, against a target that is soft-blocking you by simply not responding, that means 300 seconds of an occupied slot per stuck request. At a concurrency of 200 the crawler stops making progress long before anything raises.

timeout = aiohttp.ClientTimeout(total=30, connect=10, sock_read=20)
session = aiohttp.ClientSession(timeout=timeout, connector=connector)

Set connect separately. A slow exit node fails at connect; a slow target fails at sock_read, and telling them apart in your metrics is the difference between blaming the proxy and blaming the site.

Retry on the right status codes

A 502 from the gateway means one exit misbehaved, and the next request will be routed through a different one — retry immediately. A 429 means you are the problem, and retrying immediately makes it worse. A 407 is a credential fault that no amount of retrying fixes.

RETRY = {502, 503, 504}

async def fetch(session, url, attempts=4):
    for i in range(attempts):
        async with session.get(url, proxy=PROXY) as r:
            if r.status not in RETRY:
                return r.status, await r.text()
        await asyncio.sleep(0.5 * 2**i)
    raise RuntimeError(f"gave up on {url}")

A checklist before you scale up

1. Print the exit IP on ten consecutive requests and confirm it changes. If it does not, the pool is reusing a tunnel.
2. Set an explicit ClientTimeout. The 5-minute default is not a timeout, it is a hang.
3. Use socks5h:// rather than socks5:// on anything geo-targeted.
4. Keep the proxy URL scheme http:// even for HTTPS targets.
5. Track success rate by exit country, not in aggregate.