Python Requests proxy: proxies dict, auth, SOCKS5

Set a proxy on a request with the proxies dictionary

Requests selects a proxy from a dictionary whose keys are destination schemes. The value for https is used for https:// destinations, and the value for http is used for http:// destinations; both usually point at the same gateway. The key all applies to any scheme, and a key such as https://example.com pins one host to a specific gateway.

The value is the proxy URL, with its own scheme. http://host:port is an ordinary HTTP proxy; for an https:// destination Requests asks it to open a CONNECT tunnel. Only use https://host:port as the proxy URL when the provider terminates TLS on the proxy. Always pass a timeout; without one a stalled proxy blocks the call indefinitely.

import requests

PROXIES = {
    "http": "http://proxy.example.invalid:8080",
    "https": "http://proxy.example.invalid:8080",
}

response = requests.get(
    "https://example.com/",
    proxies=PROXIES,
    timeout=(10, 20),
)
print(response.status_code)

Use a Session so every request shares the proxy

A Session keeps the connection pool, cookies and proxy settings together. Set session.proxies once and every request made through that Session uses the gateway, while a proxies argument on an individual call still overrides it. This is also where you decide whether the process environment may contribute settings: trust_env=False stops Requests from reading HTTP_PROXY, HTTPS_PROXY, NO_PROXY, netrc credentials and the REQUESTS_CA_BUNDLE variable, so the code alone defines the route. For asynchronous workloads that need the same control over a connection pool, see the httpx async proxy guide.

import requests

with requests.Session() as session:
    session.trust_env = False
    session.proxies.update({
        "http": "http://proxy.example.invalid:8080",
        "https": "http://proxy.example.invalid:8080",
    })
    for url in ("https://example.com/", "https://example.com/robots.txt"):
        response = session.get(url, timeout=(10, 20))
        print(url, response.status_code)

Proxy authentication: encode credentials in the URL

Requests sends proxy credentials from the proxy URL itself, as http://username:password@host:port; it adds the Proxy-Authorization header for you. The auth argument is destination authentication and does not reach the proxy.

Encode the username and password individually so characters such as @, : or / cannot change the URL structure, and keep the original PROXY_URL free of credentials. Save the script below as proxy_check.py and run python proxy_check.py after installing Requests in a virtual environment with python -m pip install requests. https://proxy.example.invalid:8443 is only a nonworking illustration.

import os
from urllib.parse import quote, urlsplit, urlunsplit
import requests

gateway = urlsplit(os.environ["PROXY_URL"])
if gateway.scheme not in {"http", "https"} or not gateway.hostname:
    raise ValueError("Use your provider's HTTP(S) proxy URL")
if gateway.username is not None or gateway.path not in {"", "/"} or gateway.query or gateway.fragment:
    raise ValueError("Keep credentials and paths out of PROXY_URL")
username = quote(os.environ["PROXY_USERNAME"], safe="")
password = quote(os.environ["PROXY_PASSWORD"], safe="")
proxy = urlunsplit((gateway.scheme, f"{username}:{password}@{gateway.netloc}", "", "", ""))

with requests.Session() as session:
    session.trust_env = False
    with session.get(
        "https://example.com/",
        proxies={"http": proxy, "https": proxy},
        timeout=(10, 20),
        allow_redirects=False,
        stream=True,
    ) as response:
        response.raise_for_status()
        print({"status": response.status_code})

Environment variables and trust_env

When trust_env is left at its default of True, Requests fills in any scheme you did not set from the environment through urllib.request.getproxies(). It accepts HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY in upper or lower case, and on macOS it also reads the system proxy configuration. That is convenient on a workstation and a common source of surprise in containers, where an HTTPS_PROXY inherited from the image routes traffic you believed was direct.

NO_PROXY is a comma-separated list of hostnames, domain suffixes or IP addresses that bypass the proxy. Requests and urllib do not match it the same way (Requests honors CIDR ranges, urllib does not); the NO_PROXY matrix compares Requests with urllib, HTTPX and aiohttp. For a job whose route must be reproducible, set trust_env=False and pass the proxies explicitly; for a workstation script, setting the variables and calling Requests with no proxies argument is enough. The proxy environment variables guide shows how to observe which route a client actually selected before blaming the proxy.

export HTTPS_PROXY='http://proxy.example.invalid:8080'
export NO_PROXY='localhost,127.0.0.1,.internal.example'
python -c 'import requests; print(requests.get("https://example.com/", timeout=(10, 20)).status_code)'

SOCKS5 proxies: requests[socks], socks5 versus socks5h

SOCKS support comes from the PySocks extra: install it with python -m pip install "requests[socks]", then use a socks5:// or socks5h:// proxy URL in the same dictionary. socks5 resolves the destination hostname locally and sends the proxy an IP address; socks5h sends the hostname so the proxy resolves it, which is usually what a remote gateway expects. Username and password go in the URL exactly as for HTTP proxies.

If Requests raises InvalidSchema: Missing dependencies for SOCKS support, PySocks is not importable from the interpreter running your code, or a variable such as ALL_PROXY is supplying a SOCKS URL you never set. Fix Missing dependencies for SOCKS support covers each case, including pip failing with the same message.

import requests

PROXIES = {
    "http": "socks5h://proxy.example.invalid:1080",
    "https": "socks5h://proxy.example.invalid:1080",
}
response = requests.get("https://example.com/", proxies=PROXIES, timeout=(10, 20))
print(response.status_code)

Rotate between several proxies

When a job needs to spread requests across gateways, keep one Session for the connection pool and choose the proxies argument per request. The example cycles through a list; a real job would draw from configuration, record which gateway served each request, and retire a gateway after repeated failures rather than retrying it immediately. Whether consecutive requests should keep the same exit address is a session-model decision, and many providers express it through the username or a dedicated port. Follow the provider's documentation rather than inferring it from one successful request. In Scrapy the same rotation is a per-request proxy value in Request.meta, set by a downloader middleware; Scrapy proxies in 2.19 shows it and what happens to a hand-set Proxy-Authorization header when the proxy changes.

from itertools import cycle
import requests

GATEWAYS = cycle([
    "http://proxy-a.example.invalid:8080",
    "http://proxy-b.example.invalid:8080",
])

with requests.Session() as session:
    session.trust_env = False
    for url in ("https://example.com/", "https://example.com/robots.txt"):
        gateway = next(GATEWAYS)
        response = session.get(url, proxies={"http": gateway, "https": gateway}, timeout=(10, 20))
        print(gateway, url, response.status_code)

Know what the timeout measures

The tuple supplies separate connect and read timeouts. A read timeout limits waiting for socket data; it is not a total download deadline. Apply a separate wall-clock budget in your job runner when a whole task must finish by a fixed time. The diagnostic above streams and closes the response without downloading the full body. When a request still stalls, troubleshoot proxy timeouts one stage at a time to find which stage consumed the budget.

Setting trust_env to False also disables Requests' environment-derived CA-bundle settings. If your organization uses a custom trust bundle, pass its approved file explicitly with verify. Do not use verify=False to make a failing test pass.

Read the errors: ProxyError, 407 and timeouts

How a proxy failure surfaces depends on the destination scheme. For an https:// destination the failure happens during the CONNECT tunnel, and Requests raises requests.exceptions.ProxyError whose message includes the proxy's reply, for example Tunnel connection failed: 407 Proxy Authentication Required. For a plain http:// destination there is no tunnel, so a 407 arrives as an ordinary Response with status_code 407 and no exception at all; check for it explicitly.

Run the same gateway and destination in curl before adding retry logic. Record the exception class, attempt number and elapsed time, with credentials removed. A reproducible authentication error needs a configuration change; repeating it at higher concurrency adds noise.

• ProxyError with 407: the proxy rejected the credentials in the URL. Re-check the encoding and the provider's username format. Requests also lets an HTTPS_PROXY variable override Session.proxies; the 407 guide reproduces that and the other ways credentials fail to reach the proxy.
• ProxyError with Cannot connect to proxy or a name-resolution message: the gateway host or port is wrong, or this network cannot reach it.
• ProxyError saying Your proxy appears to only use HTTP and not HTTPS, or SSLError with WRONG_VERSION_NUMBER: a TLS handshake was answered in plaintext. The first points at an https:// proxy URL on a plain HTTP proxy port, the second at the destination port; which hop failed separates the two.
• ConnectTimeout: the connect part of the timeout tuple expired before the proxy or destination answered. ReadTimeout: the connection worked but no data arrived within the read limit.
• SSLError after a successful tunnel: certificate verification failed for the destination, or the proxy is intercepting TLS. Keep verification on and confirm with the provider.
• A successful HTTP request alone does not verify the exit's location.