Scrapy proxies in 2.19: auth, rotation and the errors you'll hit

Put the credentials in the proxy URL and pass it as meta["proxy"]:

import scrapy

PROXY = "http://user:pass@proxy.example.com:8080"


class ExampleSpider(scrapy.Spider):
    name = "example"

    async def start(self):
        yield scrapy.Request("https://example.com/", meta={"proxy": PROXY})

    def parse(self, response):
        self.logger.info("%s via %s", response.status, response.meta["proxy"])

HttpProxyMiddleware is on by default (HTTPPROXY_ENABLED = True). It moves the user and password from the URL into a Proxy-Authorization header (documentation), so response.meta["proxy"] holds the URL without credentials.

In our lab this authenticated https:// and http:// requests against a local proxy that requires Basic authentication. Everything below was recorded on 7 October 2026 with Scrapy 2.19.0, Twisted 26.4.0 and Python 3.14.4.

Three details before the errors:

• Percent-encode the password. p@ss:w/rd worked as p%40ss%3Aw%2Frd. Unencoded, the request failed with ValueError: Port could not be cast to integer value as 'w'.
• Requests made in a callback do not inherit the proxy. A request created with response.follow() went direct. A redirect kept the proxy. Set meta["proxy"] on every request, or in a downloader middleware as shown below.
• Use async def start(). A spider that defined only start_requests() opened, sent nothing and closed with finish_reason: finished and no warning. Scrapy 2.16.0 removed the method: it "is removed and no longer called" (release notes). Spider.start() replaces it. Scrapy crawls 0 pages after upgrading has the migration, spider middlewares included.

Rotating proxies: set meta["proxy"] per request

Rotation is a downloader middleware that runs before HttpProxyMiddleware (order 750) and sets a proxy URL with credentials:

from itertools import cycle

PROXIES = cycle([
    "http://user:pass@proxy-a.example.com:8080",
    "http://user:pass@proxy-b.example.com:8080",
])


class RotateProxyMiddleware:
    def process_request(self, request):
        request.meta["proxy"] = next(PROXIES)

DOWNLOADER_MIDDLEWARES = {"myproject.middlewares.RotateProxyMiddleware": 740}

In the lab a redirected request passed through this middleware twice, went to two proxies and authenticated at both.

The version that fails sets a URL without credentials and writes the header itself:

request.meta["proxy"] = next(PROXIES)  # "http://proxy-a.example.com:8080"
request.headers["Proxy-Authorization"] = "Basic dXNlcjpwYXNz"

The first request reached proxy A with the header. The target answered 301 and the redirected request went through the middlewares again. The rotation chose proxy B and set the header again, yet proxy B received the request with no Proxy-Authorization and answered 407. A retried https:// request lost the header the same way.

HttpProxyMiddleware records which proxy URL a hand-set header was sent with. When a later pass over the same request has a different proxy URL without credentials, it deletes the header. The 2.6.2 release notes give the reason: rotation middlewares changed the proxy and left the old header in place, so one proxy's credentials went to another. Since 2.7.1 a hand-set header is kept only "for as long as that proxy URL remains the same". Credentials in the URL always replace it.

Two more results from the same code:

• On the first pass the header is not tied to a proxy yet. A middleware that changed meta["proxy"] from A to B before HttpProxyMiddleware ran sent the hand-set header to proxy B.
• With no proxy at all, the header was removed before the request reached the target.

Environment variables: HTTP_PROXY is used by default

Scrapy reads the proxy variables when the crawl starts, and no setting is needed to turn that on. With HTTP_PROXY=http://user:pass@127.0.0.1:8888 and no meta["proxy"], the lab request went through the proxy and authenticated.

• HTTP_PROXY and lowercase http_proxy applied to http:// URLs, and HTTPS_PROXY to https:// URLs. With only HTTP_PROXY set, an https:// request went direct. ALL_PROXY was ignored.
• NO_PROXY=127.0.0.1 sent the request direct. meta["proxy"] took priority over the variables and ignored NO_PROXY, as the documentation says.
• meta={"proxy": None} sent one request direct with the variable set.
• HTTPPROXY_ENABLED = False stopped Scrapy from reading the variables. It also stopped the credential handling: meta["proxy"] = "http://user:pass@127.0.0.1:8888" then raised ValueError: invalid hostname: user:pass@127.0.0.1 for an http:// URL and the 407 TunnelError for https://.

To ignore the environment and keep meta["proxy"] working, replace the middleware with a subclass that discards what it read:

from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware


class ExplicitProxyMiddleware(HttpProxyMiddleware):
    def __init__(self, auth_encoding="latin-1"):
        super().__init__(auth_encoding)
        self.proxies = {}

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
    "myproject.middlewares.ExplicitProxyMiddleware": 750,
}

proxies is not a documented attribute, so test this again after an upgrade. aiohttp does the opposite: it ignores the variables unless you pass trust_env=True (aiohttp proxies in 3.14). Proxy environment variables compares other clients.

TunnelError: Could not open CONNECT tunnel with proxy

scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 127.0.0.1:8888 [{'status': 407, 'reason': b'Proxy Authentication Required'}]

The proxy rejected the CONNECT request for an https:// URL. The message was identical with no credentials and with a wrong password, and it does not contain the credentials, so it cannot tell you which of the two you have. Check the URL in meta["proxy"] on the failing request, then follow the order of checks in Fix proxy error 407.

• TunnelError is in RETRY_EXCEPTIONS. With default settings the lab proxy received the rejected CONNECT three times. The log showed Retrying <GET https://127.0.0.1:8443/> (failed 1 times), then (failed 2 times), then Gave up retrying. With "dont_retry": True in meta the proxy received one.
• handle_httpstatus_list changes nothing here. There is no response, only the exception, so handle it in an errback.

Ignoring response 407: HTTP status code is not handled or not allowed

Ignoring response <407 http://127.0.0.1:8080/>: HTTP status code is not handled or not allowed

For an http:// URL the proxy's 407 is an ordinary response, and HttpErrorMiddleware drops it. The callback was never called, this INFO line was the only sign, and the crawl closed with finish_reason: finished. With an errback the line is not logged and the errback receives scrapy.spidermiddlewares.httperror.HttpError: Ignoring non-200 response. Scrapy did not retry the request.

To get the response in the callback, allow the status for the request:

yield scrapy.Request(url, meta={"proxy": PROXY, "handle_httpstatus_list": [407]})

The spider attribute handle_httpstatus_list = [407] and the setting HTTPERROR_ALLOWED_CODES = [407] did the same (HttpErrorMiddleware). The callback then received status 407 with Proxy-Authenticate: Basic realm="lab". Without any of them, the crawl stats count the dropped responses under httperror/response_ignored_status_count/407.

UnsupportedURLSchemeError: Unsupported scheme: b'socks5'

scrapy.exceptions.UnsupportedURLSchemeError: Unsupported scheme: b'socks5'

Scrapy's default download handler has no SOCKS support. With meta["proxy"] = "socks5://127.0.0.1:1080", an http:// request raised this error once, with no retry. socks5h:// raised Unsupported scheme: b'socks5h', and a SOCKS URL in HTTP_PROXY raised the same error.

An https:// request did not raise it. Scrapy sent CONNECT 127.0.0.1:8443 as plain HTTP to the SOCKS port, the lab's SOCKS server closed the connection, and Scrapy waited for the download timeout three times before it reported DownloadTimeoutError: Getting https://127.0.0.1:8443/ took longer than 15.0 seconds. The lab set DOWNLOAD_TIMEOUT to 15. The default is 180.

There are two ways out:

• Use an HTTP proxy endpoint if your provider has one. Everything above then applies.
• Switch to HttpxDownloadHandler. It ships with Scrapy, and the documentation lists it as supporting SOCKS5 and marks it experimental and not yet recommended for production.

python -m pip install "scrapy[httpx]"

DOWNLOAD_HANDLERS = {
    "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}

With this handler socks5:// returned 200 for both schemes. For http://localhost:8080/ the SOCKS server logged a domain name, so the handler sends the hostname to the proxy. socks5h:// worked too. socks5 vs socks5h shows what other clients do with the two schemes. Its errors read differently: a rejected CONNECT surfaced as scrapy.exceptions.DownloadConnectionRefusedError: 407 Proxy Authentication Required.

Retries: RetryMiddleware repeats POST requests

RetryMiddleware retried a POST the way it retries a GET. In the lab a POST went through the proxy to a target that read the request and closed the connection without answering. Scrapy logged Retrying <POST http://127.0.0.1:8081/jobs> (failed 1 times) and sent it again. The target received the same body twice and the callback saw one 201.

For a request that is not safe to repeat, set "dont_retry": True in meta (RetryMiddleware). The target then received one POST and the errback got scrapy.exceptions.DownloadFailedError. Proxy retries: one lost response, two created jobs shows what an idempotency key changes.

Each retry is another request through the proxy. How many GB of proxies do you need measures what an HTML-only fetch uses. For diagnostic help inside a coding agent, Proxy Toolkit MCP accepts sanitized error descriptions. Keep credentials out of its arguments.

How this was tested

On Ubuntu 26.04.1 with Python 3.14.4, Scrapy 2.19.0, Twisted 26.4.0 and httpx2 2.13.1, everything on 127.0.0.1: python -m http.server as the HTTP and HTTPS targets, three proxies that require Basic authentication and log every request with its header names, a SOCKS5 server that logs each destination, and a target that loses the response to the first POST. Each of the 62 cases ran scrapy runspider in a new interpreter with a clean environment. No proxy account was used. On Python 3.15.0, pip install scrapy fails without a C compiler because brotli had no cp315 wheel on 14 October 2026; Python 3.15 for web scraping has the tested fixes.

The lab archive contains the runner, the cases, the proxy, the SOCKS5 server, the README and the recorded results.

Not tested: macOS and Windows, Digest and NTLM proxy authentication, authenticated SOCKS proxies, proxies reached over TLS, HTTP/2 and Scrapy's aiohttp-based download handler.

ipvolt is a proxy service for developers that isn't open yet; join the early-access list for one email when it opens.