# Scrapy proxies in 2.19: auth, rotation and the errors you'll hit

Source: https://ipvolt.com/guides/scrapy-proxy-setup
Markdown: https://ipvolt.com/guides/scrapy-proxy-setup.md

[Home](https://ipvolt.com/index.md) / [Guides](https://ipvolt.com/guides.md) / [Scrapy proxies in 2.19: auth, rotation and the errors you'll hit](https://ipvolt.com/guides/scrapy-proxy-setup.md)

Category: Integration
Published: 2026-10-07
Reading time: 8 minutes
Author: ipvolt

Scrapy 2.19 proxy setup: credentials in meta["proxy"], per-request rotation, HTTP_PROXY, and fixes for TunnelError 407, ignored 407 responses and socks5.

Put the credentials in the proxy URL and pass it as `meta["proxy"]`:

```python
import scrapy

PROXY = "http://user:pass@proxy.example.com:8080"


class ExampleSpider(scrapy.Spider):
    name = "example"

    async def start(self):
        yield scrapy.Request("https://example.com/", meta={"proxy": PROXY})

    def parse(self, response):
        self.logger.info("%s via %s", response.status, response.meta["proxy"])
```

`HttpProxyMiddleware` is on by default (`HTTPPROXY_ENABLED = True`). It moves the user and password from the URL into a `Proxy-Authorization` header ([documentation](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy)), so `response.meta["proxy"]` holds the URL without credentials.

In our lab this authenticated `https://` and `http://` requests against a local proxy that requires Basic authentication. Everything below was recorded on 7 October 2026 with Scrapy 2.19.0, Twisted 26.4.0 and Python 3.14.4.

Three details before the errors:

- **Percent-encode the password.** `p@ss:w/rd` worked as `p%40ss%3Aw%2Frd`. Unencoded, the request failed with `ValueError: Port could not be cast to integer value as 'w'`.
- **Requests made in a callback do not inherit the proxy.** A request created with `response.follow()` went direct. A redirect kept the proxy. Set `meta["proxy"]` on every request, or in a downloader middleware as shown below.
- **Use `async def start()`.** A spider that defined only `start_requests()` opened, sent nothing and closed with `finish_reason: finished` and no warning. Scrapy 2.16.0 removed the method: it "is removed and no longer called" ([release notes](https://docs.scrapy.org/en/latest/news.html#release-2-16-0)). [`Spider.start()`](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.start) replaces it. [Scrapy crawls 0 pages after upgrading](/guides/scrapy-start-requests-not-called) has the migration, spider middlewares included.

## Rotating proxies: set meta["proxy"] per request

Rotation is a downloader middleware that runs before `HttpProxyMiddleware` (order 750) and sets a proxy URL with credentials:

```python
from itertools import cycle

PROXIES = cycle([
    "http://user:pass@proxy-a.example.com:8080",
    "http://user:pass@proxy-b.example.com:8080",
])


class RotateProxyMiddleware:
    def process_request(self, request):
        request.meta["proxy"] = next(PROXIES)
```

```python
DOWNLOADER_MIDDLEWARES = {"myproject.middlewares.RotateProxyMiddleware": 740}
```

In the lab a redirected request passed through this middleware twice, went to two proxies and authenticated at both.

The version that fails sets a URL without credentials and writes the header itself:

```python
request.meta["proxy"] = next(PROXIES)  # "http://proxy-a.example.com:8080"
request.headers["Proxy-Authorization"] = "Basic dXNlcjpwYXNz"
```

The first request reached proxy A with the header. The target answered 301 and the redirected request went through the middlewares again. The rotation chose proxy B and set the header again, yet proxy B received the request with no `Proxy-Authorization` and answered 407. A retried `https://` request lost the header the same way.

`HttpProxyMiddleware` records which proxy URL a hand-set header was sent with. When a later pass over the same request has a different proxy URL without credentials, it deletes the header. The [2.6.2 release notes](https://docs.scrapy.org/en/latest/news.html#release-2-6-2) give the reason: rotation middlewares changed the proxy and left the old header in place, so one proxy's credentials went to another. Since [2.7.1](https://docs.scrapy.org/en/latest/news.html#release-2-7-1) a hand-set header is kept only "for as long as that proxy URL remains the same". Credentials in the URL always replace it.

Two more results from the same code:

- On the first pass the header is not tied to a proxy yet. A middleware that changed `meta["proxy"]` from A to B before `HttpProxyMiddleware` ran sent the hand-set header to proxy B.
- With no proxy at all, the header was removed before the request reached the target.

## Environment variables: HTTP_PROXY is used by default

Scrapy reads the proxy variables when the crawl starts, and no setting is needed to turn that on. With `HTTP_PROXY=http://user:pass@127.0.0.1:8888` and no `meta["proxy"]`, the lab request went through the proxy and authenticated.

- `HTTP_PROXY` and lowercase `http_proxy` applied to `http://` URLs, and `HTTPS_PROXY` to `https://` URLs. With only `HTTP_PROXY` set, an `https://` request went direct. `ALL_PROXY` was ignored.
- `NO_PROXY=127.0.0.1` sent the request direct. `meta["proxy"]` took priority over the variables and ignored `NO_PROXY`, as the documentation says.
- `meta={"proxy": None}` sent one request direct with the variable set.
- `HTTPPROXY_ENABLED = False` stopped Scrapy from reading the variables. It also stopped the credential handling: `meta["proxy"] = "http://user:pass@127.0.0.1:8888"` then raised `ValueError: invalid hostname: user:pass@127.0.0.1` for an `http://` URL and the 407 `TunnelError` for `https://`.

To ignore the environment and keep `meta["proxy"]` working, replace the middleware with a subclass that discards what it read:

```python
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware


class ExplicitProxyMiddleware(HttpProxyMiddleware):
    def __init__(self, auth_encoding="latin-1"):
        super().__init__(auth_encoding)
        self.proxies = {}
```

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
    "myproject.middlewares.ExplicitProxyMiddleware": 750,
}
```

`proxies` is not a documented attribute, so test this again after an upgrade. aiohttp does the opposite: it ignores the variables unless you pass `trust_env=True` ([aiohttp proxies in 3.14](/guides/aiohttp-proxy-setup)). [Proxy environment variables](/guides/proxy-environment-variables) compares other clients.

## TunnelError: Could not open CONNECT tunnel with proxy

```text
scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 127.0.0.1:8888 [{'status': 407, 'reason': b'Proxy Authentication Required'}]
```

The proxy rejected the `CONNECT` request for an `https://` URL. The message was identical with no credentials and with a wrong password, and it does not contain the credentials, so it cannot tell you which of the two you have. Check the URL in `meta["proxy"]` on the failing request, then follow the order of checks in [Fix proxy error 407](/guides/fix-proxy-error-407).

- `TunnelError` is in `RETRY_EXCEPTIONS`. With default settings the lab proxy received the rejected `CONNECT` three times. The log showed `Retrying <GET https://127.0.0.1:8443/> (failed 1 times)`, then `(failed 2 times)`, then `Gave up retrying`. With `"dont_retry": True` in `meta` the proxy received one.
- `handle_httpstatus_list` changes nothing here. There is no response, only the exception, so handle it in an `errback`.

## Ignoring response 407: HTTP status code is not handled or not allowed

```text
Ignoring response <407 http://127.0.0.1:8080/>: HTTP status code is not handled or not allowed
```

For an `http://` URL the proxy's 407 is an ordinary response, and `HttpErrorMiddleware` drops it. The callback was never called, this INFO line was the only sign, and the crawl closed with `finish_reason: finished`. With an `errback` the line is not logged and the errback receives `scrapy.spidermiddlewares.httperror.HttpError: Ignoring non-200 response`. Scrapy did not retry the request.

To get the response in the callback, allow the status for the request:

```python
yield scrapy.Request(url, meta={"proxy": PROXY, "handle_httpstatus_list": [407]})
```

The spider attribute `handle_httpstatus_list = [407]` and the setting `HTTPERROR_ALLOWED_CODES = [407]` did the same ([HttpErrorMiddleware](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#module-scrapy.spidermiddlewares.httperror)). The callback then received status 407 with `Proxy-Authenticate: Basic realm="lab"`. Without any of them, the crawl stats count the dropped responses under `httperror/response_ignored_status_count/407`.

## UnsupportedURLSchemeError: Unsupported scheme: b'socks5'

```text
scrapy.exceptions.UnsupportedURLSchemeError: Unsupported scheme: b'socks5'
```

Scrapy's default download handler has no SOCKS support. With `meta["proxy"] = "socks5://127.0.0.1:1080"`, an `http://` request raised this error once, with no retry. `socks5h://` raised `Unsupported scheme: b'socks5h'`, and a SOCKS URL in `HTTP_PROXY` raised the same error.

An `https://` request did not raise it. Scrapy sent `CONNECT 127.0.0.1:8443` as plain HTTP to the SOCKS port, the lab's SOCKS server closed the connection, and Scrapy waited for the download timeout three times before it reported `DownloadTimeoutError: Getting https://127.0.0.1:8443/ took longer than 15.0 seconds.` The lab set `DOWNLOAD_TIMEOUT` to 15. The default is 180.

There are two ways out:

- Use an HTTP proxy endpoint if your provider has one. Everything above then applies.
- Switch to `HttpxDownloadHandler`. It ships with Scrapy, and the [documentation](https://docs.scrapy.org/en/latest/topics/download-handlers.html#httpx-handler) lists it as supporting SOCKS5 and marks it experimental and not yet recommended for production.

```sh
python -m pip install "scrapy[httpx]"
```

```python
DOWNLOAD_HANDLERS = {
    "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}
```

With this handler `socks5://` returned 200 for both schemes. For `http://localhost:8080/` the SOCKS server logged a domain name, so the handler sends the hostname to the proxy. `socks5h://` worked too. [socks5 vs socks5h](/blog/socks5-vs-socks5h) shows what other clients do with the two schemes. Its errors read differently: a rejected `CONNECT` surfaced as `scrapy.exceptions.DownloadConnectionRefusedError: 407 Proxy Authentication Required`.

## Retries: RetryMiddleware repeats POST requests

`RetryMiddleware` retried a `POST` the way it retries a `GET`. In the lab a `POST` went through the proxy to a target that read the request and closed the connection without answering. Scrapy logged `Retrying <POST http://127.0.0.1:8081/jobs> (failed 1 times)` and sent it again. The target received the same body twice and the callback saw one 201.

For a request that is not safe to repeat, set `"dont_retry": True` in `meta` ([RetryMiddleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.retry)). The target then received one `POST` and the errback got `scrapy.exceptions.DownloadFailedError`. [Proxy retries: one lost response, two created jobs](/blog/proxy-retries-duplicate-jobs) shows what an idempotency key changes.

Each retry is another request through the proxy. [How many GB of proxies do you need](/blog/how-many-gb-of-proxies-do-i-need) measures what an HTML-only fetch uses. For diagnostic help inside a coding agent, [Proxy Toolkit MCP](/mcp) accepts sanitized error descriptions. Keep credentials out of its arguments.

## How this was tested

On Ubuntu 26.04.1 with Python 3.14.4, Scrapy 2.19.0, Twisted 26.4.0 and httpx2 2.13.1, everything on `127.0.0.1`: `python -m http.server` as the HTTP and HTTPS targets, three proxies that require Basic authentication and log every request with its header names, a SOCKS5 server that logs each destination, and a target that loses the response to the first `POST`. Each of the 62 cases ran `scrapy runspider` in a new interpreter with a clean environment. No proxy account was used. On Python 3.15.0, `pip install scrapy` fails without a C compiler because brotli had no cp315 wheel on 14 October 2026; [Python 3.15 for web scraping](/blog/python-3-15-web-scraping) has the tested fixes.

The [lab archive](https://ipvolt.com/downloads/scrapy-proxy-setup/scrapy-proxy-lab.zip) contains the [runner](https://ipvolt.com/downloads/scrapy-proxy-setup/run_lab.py), the [cases](https://ipvolt.com/downloads/scrapy-proxy-setup/cases.py), the [proxy](https://ipvolt.com/downloads/scrapy-proxy-setup/auth_proxy.py), the [SOCKS5 server](https://ipvolt.com/downloads/scrapy-proxy-setup/socks5_server.py), the [README](https://ipvolt.com/downloads/scrapy-proxy-setup/README.md) and the [recorded results](https://ipvolt.com/downloads/scrapy-proxy-setup/results.json).

Not tested: macOS and Windows, Digest and NTLM proxy authentication, authenticated SOCKS proxies, proxies reached over TLS, HTTP/2 and Scrapy's aiohttp-based download handler.

ipvolt is a proxy service for developers that isn't open yet; [join the early-access list](https://ipvolt.com/#waitlist-closing) for one email when it opens.

## Sources & further reading

- [Scrapy documentation: HttpProxyMiddleware and HTTPPROXY_ENABLED](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy)
- [Scrapy documentation: Request.meta special keys (proxy, dont_retry, handle_httpstatus_list)](https://docs.scrapy.org/en/latest/topics/request-response.html#topics-request-meta)
- [Scrapy documentation: HttpErrorMiddleware and HTTPERROR_ALLOWED_CODES](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#module-scrapy.spidermiddlewares.httperror)
- [Scrapy documentation: RetryMiddleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.retry)
- [Scrapy documentation: Spider.start()](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.start)
- [Scrapy documentation: HttpxDownloadHandler](https://docs.scrapy.org/en/latest/topics/download-handlers.html#httpx-handler)
- [Scrapy release notes: 2.16.0, start_requests() removed](https://docs.scrapy.org/en/latest/news.html#release-2-16-0)
- [Scrapy release notes: 2.6.2, Proxy-Authorization security fix](https://docs.scrapy.org/en/latest/news.html#release-2-6-2)
- [Scrapy release notes: 2.7.1, explicit Proxy-Authorization header](https://docs.scrapy.org/en/latest/news.html#release-2-7-1)

## Related guides

- [Fix proxy error 407 without guessing](https://ipvolt.com/guides/fix-proxy-error-407.md)
- [Scrapy crawls 0 pages after upgrading: start_requests() is no longer called](https://ipvolt.com/guides/scrapy-start-requests-not-called.md)
- [Proxy environment variables: HTTP_PROXY and NO_PROXY](https://ipvolt.com/guides/proxy-environment-variables.md)
- [aiohttp proxies in 3.14: auth without BasicAuth, env vars and the errors you'll hit](https://ipvolt.com/guides/aiohttp-proxy-setup.md)
- [Python Requests proxy: proxies dict, auth, SOCKS5](https://ipvolt.com/guides/python-requests-proxy.md)
- [Rotating vs sticky proxies: plan for session continuity](https://ipvolt.com/guides/rotating-vs-sticky-proxies.md)

## About ipvolt

Examples use generic proxy settings, with links to the original technical documentation. Product-specific behavior must be checked with your provider. ipvolt is still in development.

## Know when access opens.

ipvolt · In development

We’re building proxy infrastructure for developers and data teams. Join the interest list for a heads-up when ipvolt is ready.

Consent: One email when access opens. Nothing else.

[Get early access](https://ipvolt.com/guides/scrapy-proxy-setup#waitlist-closing). Use the email form on this page to join the interest list.

[Privacy](https://ipvolt.com/privacy)