aiohttp Proxy Guide: Authentication and Concurrent Requests

An aiohttp proxy request combines a reusable ClientSession, an explicit proxy URL, and credentials intended for the proxy. Build that single-request path first, then add a small amount of concurrent work. This guide covers HTTP proxies, HTTPS destinations through CONNECT, timeouts, bounded concurrency, and useful error categories. It also explains the authentication change documented in aiohttp 3.14 so older examples remain understandable.

For other Python clients, start with the Python proxies overview. The Requests guide covers synchronous scripts, while the HTTPX guide offers both synchronous and asynchronous clients.

Install aiohttp and choose explicit configuration

python -m pip install aiohttp
python -c "import aiohttp; print(aiohttp.__version__)"

Use BP_PROXY_HOST, BP_PROXY_PORT, BP_PROXY_USER, BP_PROXY_PASSWORD, and BP_TARGET_URL as application environment variables. Set the target to an endpoint you control. The downloadable script validates the endpoint components before opening a connection; the short examples assume those values have already been checked.

aiohttp does not read standard proxy environment variables by default. trust_env=True enables that path, including proxy bypass rules and applicable netrc credentials, as shown in the environment helpers. Here we retain trust_env=False and pass the proxy explicitly. Avoid combining several configuration sources while debugging, because a changed environment can otherwise change routing. The proxy formatter helps check the endpoint shape, and the authentication guide covers account authorization.

Keep proxy authentication separate from target headers

Put percent-encoded credentials in the proxy URL for the example below. aiohttp supports this format for HTTP proxy authentication, including HTTPS requests through CONNECT. Current documentation also introduces encode_basic_auth() in 3.14 and deprecates constructing BasicAuth or passing proxy_auth. The URL approach avoids constructing those deprecated helper objects and works with earlier 3.x APIs. The utility reference explains the version changes.

import asyncio
import os
from urllib.parse import quote
import aiohttp

user = quote(os.environ["BP_PROXY_USER"], safe="")
password = quote(os.environ["BP_PROXY_PASSWORD"], safe="")
host = os.environ["BP_PROXY_HOST"]
port = int(os.environ["BP_PROXY_PORT"])
proxy = f"http://{user}:{password}@{host}:{port}"

async def main():
    timeout = aiohttp.ClientTimeout(total=30, connect=10,
                                    sock_connect=5, sock_read=15)
    async with aiohttp.ClientSession(timeout=timeout, trust_env=False) as session:
        async with session.get(os.environ["BP_TARGET_URL"], proxy=proxy,
                               allow_redirects=False) as response:
            response.raise_for_status()
            await response.read()
            print("status", response.status)

asyncio.run(main())

Encode the username and password separately, exactly once. Characters such as @, /, and # can otherwise change the URL structure. A Basic authentication username cannot contain a colon. Never place the proxy password in the destination’s ordinary Authorization header. URL encoding is reversible, so the resulting proxy URL remains a secret; keep it out of logs. Match any non-ASCII credential encoding to the endpoint’s requirements.

HTTPS targets do not require an HTTPS proxy URL

The proxy URL’s scheme describes how your client connects to the proxy. With a plain HTTP proxy and an HTTPS target, aiohttp requests a CONNECT tunnel and performs destination TLS through it. https:// on the proxy URL instead means TLS to the proxy itself, which introduces separate compatibility requirements. Use the endpoint’s advertised scheme rather than changing it to match the target.

aiohttp documents limited HTTPS-proxy support, with behavior depending on the Python runtime’s TLS-in-TLS support. A working CONNECT path through an HTTP proxy is not evidence that an HTTPS proxy URL will work. Keep certificate verification enabled and check trust roots and hostnames when TLS fails. See the official proxy notes for the distinction.

Reuse a session and cap active work

A session owns a connection pool. Keep one session around a batch instead of creating one for each request. A connector limit bounds connections, while a semaphore bounds the tasks actively entering your request operation. Neither is a request-per-second rate limit. For a rate-limited API, add a separate scheduler with a suitable delay and a bounded retry policy.

async def fetch_batch(urls):
    gate = asyncio.Semaphore(4)
    connector = aiohttp.TCPConnector(limit=4, limit_per_host=4)
    timeout = aiohttp.ClientTimeout(total=30, connect=10,
                                    sock_connect=5, sock_read=15)
    async with aiohttp.ClientSession(connector=connector, timeout=timeout,
                                    trust_env=False) as session:
        async def fetch(url):
            async with gate:
                async with session.get(url, proxy=proxy,
                                               allow_redirects=False) as response:
                    response.raise_for_status()
                    await response.read()
                    return response.status
        return await asyncio.gather(*(fetch(url) for url in urls))

This block reuses the proxy and authentication variables from the first example. Four connections is an illustrative starting point. For thousands of URLs, use a bounded queue and worker tasks so the input does not create thousands of pending coroutines. The compact example propagates an error to its caller; the download catches failures per task and preserves the other results.

The response context closes each response. Reading a complete body can permit connection reuse, but read() stores the whole body in memory. The download instead streams chunks and stops if a response exceeds its two-megabyte example budget. Adjust that limit for your workload. aiohttp’s client quickstart covers body reading and streaming.

The timeout implementation uses these fields for different purposes: total covers the request, connect includes waiting for a pool connection, sock_connect bounds establishing a new connection, and sock_read bounds waiting for a portion of response data. Semaphore wait time occurs before the request starts. Add an outer batch deadline if your job requires one.

Separate proxy failures from HTTP responses

Common aiohttp proxy errors
Signal Check first
ClientProxyConnectionError Proxy address, port, DNS, firewall, and endpoint availability.
ClientHttpProxyError CONNECT status, credentials, and permitted destination port.
HTTP 407 Proxy authentication; a CONNECT rejection can appear as an exception.
asyncio.TimeoutError Which operation timed out, including pool contention and slow reads.
HTTP 403 or 429 Response source, target policy, and request rate before retrying.
Certificate verification failure Hostname, trust chain, and runtime CA configuration.

Calling raise_for_status() makes unsuccessful HTTP responses raise ClientResponseError. Catch the more specific ClientHttpProxyError before that base class. For a diagnostic log, record a job ID, proxy label, exception class, and status if available. Do not print the entire exception, its request information, proxy authorization header, URL query, or response body. The exception implementation shows the relationships; our proxy error guide adds a step-by-step diagnosis flow.

Download and check the concurrent example

Download the aiohttp concurrent proxy script. With the environment variables set, run python -B aiohttp-concurrent-proxy.py --check for configuration validation without networking. Remove --check to make four requests to your selected endpoint. The script builds an encoded authenticated proxy URL, streams response bodies, and prints redacted diagnostics.

Confirm routing by comparing the egress observed at your controlled target; an HTTP success code alone cannot prove that a proxy was used. Check one endpoint with the proxy tester before diagnosing a concurrent batch. Short-lived HTTPS scripts may need a brief transport cleanup interval after session closure, as described in aiohttp’s shutdown guidance.

Frequently asked questions

Does aiohttp support SOCKS5 natively?

Its documented built-in proxy path covers HTTP proxies. SOCKS requires another connector package; this guide does not assume one. HTTPX has an optional SOCKS extra.

Does auth= authenticate the proxy?

That argument concerns destination authentication. Use the proxy-specific configuration for your endpoint credentials.

Can I rotate proxies per request?

Yes, choose the explicit proxy URL and corresponding credentials for each request. Preserve one endpoint where a session needs stable egress, and track failures against endpoint labels.

Should I use aiohttp for a full crawler?

It supplies HTTP client primitives. For crawl scheduling and extraction middleware, see the Scrapy guide; browse the developer examples hub for other clients.

For an endpoint you can configure explicitly, review BuyProxies dedicated proxy plans and match the protocol, credentials, and concurrency budget to your application.

Official sources

Technical references reviewed October 2, 2026. See our editorial policy and testing methodology.

Scroll to Top