# Scrapy proxy lab

Loopback lab behind the ipvolt guide [Scrapy proxies in 2.19](https://ipvolt.com/guides/scrapy-proxy-setup).
It records what Scrapy does with proxy credentials, a hand-set `Proxy-Authorization` header, proxy
environment variables, a 407 from the proxy, a `socks5://` proxy URL, retried `POST` requests and a spider
that only defines `start_requests()`. Everything runs on `127.0.0.1`. No proxy account is needed.

## Files

| File | What it is |
| --- | --- |
| `run_lab.py` | Starts the fixtures, runs every case with `python -m scrapy runspider` in a new interpreter with a clean environment, writes `results.json` |
| `cases.py` | The cases: name, environment variables and the spider file |
| `lab_common.py` | The fixture URLs and the spider base class every case imports |
| `auth_proxy.py` | HTTP proxy that requires Basic authentication, supports `CONNECT` and absolute-form requests, and answers 407 with `Proxy-Authenticate` otherwise |
| `socks5_server.py` | SOCKS5 server without authentication that logs the address type and destination of each request, and logs and disconnects a client that does not speak SOCKS5 |
| `jobs_target.py` | HTTP target that logs the header names it receives and loses the response to the first `POST` of each body |
| `requirements.txt` | `scrapy[httpx]==2.19.0` |
| `results.json` | The recorded run |

The other two targets are `python -m http.server` on port 8080 and the same server with `--tls-cert` on port
8443, which needs Python 3.14. The runner creates a two-day self-signed certificate with the `openssl` command.
Scrapy does not verify certificates by default (`DOWNLOAD_VERIFY_CERTIFICATES = False`), so the cases need no
extra TLS settings.

Three proxies run: ports 8888 and 8889 accept `user` / `pass`, port 8890 accepts `user` / `p@ss:w/rd`.

## Run it

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python run_lab.py
```

Ports 8080, 8081, 8443, 8888, 8889, 8890 and 1080 on `127.0.0.1` must be free. The runner stops if one is in
use. A full run takes a few minutes; the `socks5://` case with an `https://` target waits for three download
timeouts.

To run one case by hand, start the fixtures yourself, save the case's `code` from `results.json` as a file
next to `lab_common.py` and run `scrapy runspider that_file.py`.

## Reading results.json

The top level has the versions, the fixture ports, the `settings` passed to every case with `-s`, and the
text of `lab_common.py`. Each case has:

- `command` and `code`: what ran.
- `env`: the only variables the case received besides `PATH`, `HOME` and `PYTHONPATH`.
- `stdout`: one `CALLBACK ...` line per response that reached a callback, one `ERRBACK <class>: <message>`
  line per failure that reached the errback.
- `log`: the Scrapy log at DEBUG level. Start-up records that are the same in every case (settings, enabled
  middlewares and extensions, versions) and the per-byte trace of the httpcore2 library are left out.
- `stats`: the request, response, exception, retry, `httperror` and log counters from the crawl stats.
- `proxy_log`: the requests the proxies received during the case, with the proxy port, the method, the
  target, whether the `Proxy-Authorization` header was `ok`, `wrong` or `missing`, and the names of all
  headers received.
- `socks_log`: what the SOCKS5 server logged during the case.
- `target_log`: what the jobs target logged during the case.

An empty `proxy_log` means the request did not go through an HTTP proxy.

Four settings differ from Scrapy's defaults in every case: `LOG_FORMAT` (no timestamps, so two runs can be
compared), `TELNETCONSOLE_ENABLED` and `REMOTE_CONTROL_ENABLED` (off, so a case opens no extra port) and
`DOWNLOAD_TIMEOUT` (15 seconds instead of 180, so the SOCKS case finishes in under a minute).

## Recorded run

Recorded on 7 October 2026 on Ubuntu 26.04.1 with Python 3.14.4, Scrapy 2.19.0, Twisted 26.4.0,
pyOpenSSL 26.4.0, cryptography 50.0.2, httpx2 2.13.1 and socksio 1.0.0. 62 cases.

What the groups show:

- **Credentials.** A proxy URL with credentials in `meta["proxy"]` authenticates `https://` and `http://`
  requests. Without credentials or with a wrong password, an `https://` request ends in the same
  `TunnelError` after three `CONNECT` attempts, and an `http://` request ends in a 407 response that
  `HttpErrorMiddleware` drops.
- **Handling the 407 response.** `handle_httpstatus_list` (as a `meta` key or a spider attribute) and
  `HTTPERROR_ALLOWED_CODES` deliver the `http://` 407 to the callback. They change nothing for `https://`.
- **Environment variables.** `HTTP_PROXY`, `http_proxy` and `HTTPS_PROXY` are used without any setting,
  `NO_PROXY` bypasses them, `ALL_PROXY` is ignored, `meta["proxy"]` wins over all of them, and
  `HTTPPROXY_ENABLED = False` also breaks credentials in `meta["proxy"]`.
- **A hand-set header.** A `Proxy-Authorization` header set by hand works while the proxy URL stays the same
  and is deleted on a later pass (a redirect or a retry) when the proxy URL has changed.
- **SOCKS.** The default handler raises `UnsupportedURLSchemeError` for an `http://` request and times out
  for an `https://` request. `HttpxDownloadHandler` connects through the SOCKS5 server.
- **start() and start_requests().** A spider with only `start_requests()` sends nothing.
- **Retries.** A `POST` whose response is lost is sent again unless `dont_retry` is set.

## Not covered

macOS and Windows, proxies that use Digest or NTLM authentication, authenticated SOCKS proxies, proxies that
are themselves reached over TLS (apart from one case that records the `NotImplementedError` for an
`https://` proxy URL with an `https://` target), HTTP/2, and Scrapy's `AiohttpDownloadHandler` and
`H2DownloadHandler`.