# Scrapy start_requests() lab

Loopback lab behind the ipvolt guide
[Scrapy crawls 0 pages after upgrading: start_requests() is no longer called](https://ipvolt.com/guides/scrapy-start-requests-not-called).
It records what four Scrapy releases do with a spider that defines only `start_requests()`, with
`async def start()`, with both, and with spider middlewares that still use `process_start_requests()` or a
synchronous `process_spider_output()`. Everything runs on `127.0.0.1`. No proxy account is needed.

## Files

| File | What it is |
| --- | --- |
| `run_lab.py` | Starts the fixtures, runs every spider with `python -m scrapy runspider` in a new interpreter with a clean environment, runs the pytest check against both projects, writes `results.json` |
| `spiders/a_start_requests_only.py` | Case A: only `start_requests()` |
| `spiders/a2_start_urls_only.py` | Control: neither method, only `start_urls` |
| `spiders/b_async_start.py` | Case B: `async def start()` |
| `spiders/b2_start_urls_headers_meta.py` | `start()` over `start_urls` with headers and meta |
| `spiders/b3_proxy_meta.py` | `start()` with a proxy URL in `meta` |
| `spiders/c_both.py`, `spiders/c2_both_delegating.py` | Case C: both methods, written twice or with `start()` delegating |
| `spiders/d1_*.py`, `d1b`, `d1c`, `d1d` | Case D: middleware with `process_start_requests()` only, an unmigrated spider and middleware, `process_start()`, and both |
| `spiders/d2_*.py`, `d2b`, `d2c`, `d2d` | Case D: synchronous `process_spider_output()`, the asynchronous version without and with the `spider` argument, and the sync plus `process_spider_output_async()` pair |
| `spiders/e1_start_not_async.py`, `spiders/e2_start_returns_list.py` | `start()` without `async`, and `async def start()` that returns a list |
| `ci/test_crawl.py` | The pytest check that fails when a crawl sends no request |
| `ci/old/`, `ci/new/` | The same `myproject.spiders.quotes` spider before and after the migration |
| `forward_proxy.py` | HTTP forward proxy that requires Basic authentication (`user` / `pass`) and logs each request |
| `site/index.html` | The page served by `python -m http.server` |
| `requirements.txt` | The Scrapy installs used for the recorded run |
| `results.json` | The recorded run: command, exit code, selected stats and the full log of every case |

## Run it

```sh
python3 -m venv venv-2.19 && venv-2.19/bin/pip install "scrapy==2.19.0" pytest
python3 -m venv venv-2.16 && venv-2.16/bin/pip install "scrapy==2.16.0" pytest
python3 -m venv venv-2.15 && venv-2.15/bin/pip install "scrapy==2.15.2" pytest
python3 -m venv venv-2.12 && venv-2.12/bin/pip install "scrapy==2.12.0" "w3lib<2.3" "twisted<25" pytest
python3 run_lab.py 2.19=venv-2.19 2.16=venv-2.16 2.15=venv-2.15 2.12=venv-2.12
```

Scrapy 2.12.0 does not import with w3lib 2.5.0, which is why the last environment pins older w3lib and
Twisted releases. The runner uses ports 8047 (target) and 8048 (proxy).

## Recorded run

9 October 2026, Ubuntu 26.04.1, Python 3.14.4. 68 `scrapy runspider` runs and 8 pytest runs.

| Environment | Scrapy | Twisted |
| --- | --- | --- |
| 2.19 | 2.19.0 | 26.4.0 |
| 2.16 | 2.16.0 | 26.4.0 |
| 2.15 | 2.15.2 | 25.5.0 |
| 2.12 | 2.12.0 | 24.11.0 |

Scrapy 2.16.0 and 2.19.0 gave the same result in every row except the last two, where only the
`finish_reason` differs.

| Case | 2.16.0 and 2.19.0 | 2.15.2 | 2.12.0 |
| --- | --- | --- | --- |
| A: only `start_requests()` | 0 requests, `finish_reason: finished`, exit 0, no warning | 1 page, `ScrapyDeprecationWarning` | 1 page |
| Control: only `start_urls` | 1 page | 1 page | 1 page |
| B: `async def start()` | 1 page | 1 page | 0 requests, no warning |
| C: both methods (either file) | 1 page, no warning | 1 page, no warning | 1 page |
| D1: middleware with only `process_start_requests()`, spider with `start()` | 1 page, middleware method never called, no warning | `ValueError`, crawl aborted, exit 1 | 0 requests |
| D1b: spider and middleware both unmigrated | 0 requests, no warning | 1 page, two deprecation warnings | 1 page, request tagged |
| D1c: middleware with `process_start()` | 1 page, request tagged | 1 page, request tagged | 0 requests |
| D1d: spider and middleware define both | 1 page, request tagged | 1 page, request tagged | 1 page, request tagged |
| D2: synchronous `process_spider_output()` | `TypeError` before the spider opens, exit 1 | 1 page, two warnings | 0 requests (the spider uses `start()`) |
| D2b: asynchronous `process_spider_output(self, response, result)` | 1 page, item changed | 1 page, item changed | 0 requests |
| D2c: asynchronous with a `spider` argument | 1 page, `ScrapyDeprecationWarning` about the argument | 1 page, same warning | 0 requests |
| D2d: sync method plus `process_spider_output_async()` | 1 page, the async method ran | 1 page, the sync method ran | 0 requests |
| E1: `def start()` without `async` | `TypeError` logged, 0 requests, exit 0 (`finish_reason` `start_error` on 2.19.0, `finished` on 2.16.0) | `TypeError` logged, 0 requests | 0 requests |
| E2: `async def start()` returning a list | same as E1, `got <class 'coroutine'>` | same as E1 | 0 requests |
| pytest check, `ci/old` | fails: `assert 0 > 0` | passes | passes |
| pytest check, `ci/new` | passes | passes | fails: `assert 0 > 0` |

Not tested: macOS and Windows, Scrapy 2.13, 2.14, 2.17 and 2.18, `CrawlSpider` rules, and third-party
middlewares.