# Scrapy crawls 0 pages after upgrading: start_requests() is no longer called

Source: https://ipvolt.com/guides/scrapy-start-requests-not-called
Markdown: https://ipvolt.com/guides/scrapy-start-requests-not-called.md

[Home](https://ipvolt.com/index.md) / [Guides](https://ipvolt.com/guides.md) / [Scrapy crawls 0 pages after upgrading: start_requests() is no longer called](https://ipvolt.com/guides/scrapy-start-requests-not-called.md)

Category: Troubleshooting
Published: 2026-10-10
Reading time: 6 minutes
Author: ipvolt

Scrapy 2.16 removed Spider.start_requests(): the spider logs Crawled 0 pages and closes as finished with no warning. Fix: async def start(). Tested on 2.19.

After an upgrade to Scrapy 2.16 or later, a spider that defines only `start_requests()` opens, sends no request and closes as if it had finished:

```text
2026-10-09 12:33:33 [scrapy.core.engine] INFO: Spider opened
2026-10-09 12:33:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-10-09 12:33:33 [scrapy.core.engine] INFO: Closing spider (finished)
2026-10-09 12:33:33 [scrapy.core.engine] INFO: Spider closed (finished)
```

These are the engine and logstats lines of a Scrapy 2.19.0 run. The log had no WARNING or ERROR line, the exit code was 0 and `parse()` never ran. The stats dump had `'finish_reason': 'finished'` and no `downloader/request_count` key.

The cause: Scrapy 2.16.0 (19 May 2026) removed `Spider.start_requests()`, deprecated since 2.13.0. The [release notes](https://docs.scrapy.org/en/latest/news.html#release-2-16-0) say it "is removed and no longer called" ([pull request #7490](https://github.com/scrapy/scrapy/pull/7490)). The fix is to rename the method to `async def start()`:

```python
# Before: Scrapy 2.16 and later never call this
def start_requests(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)
```

```python
# After
async def start(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)
```

With that one change the same spider crawled its page on Scrapy 2.16.0 and 2.19.0.

| What you see on 2.16+ | Cause | Fix |
| --- | --- | --- |
| `Crawled 0 pages`, then `Spider closed (finished)`, no warning | The spider defines only `start_requests()` | Rename it to `async def start()` |
| The crawl runs, but a middleware no longer changes the first requests | A spider middleware defines only `process_start_requests()` | Add `async def process_start()` |
| `TypeError: Middleware ... doesn't support asynchronous spider output`, exit code 1 | A spider middleware has a synchronous `process_spider_output()` | Make it an async generator |

Everything below was recorded on 9 October 2026 with Scrapy 2.19.0, Twisted 26.4.0 and Python 3.14.4, and repeated on Scrapy 2.16.0, 2.15.2 and 2.12.0. A spider that only sets `start_urls` and defines neither method is not affected: it crawled on all four.

## Migrate start_requests() to async def start()

[`Spider.start()`](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.start) is an asynchronous generator: `async def` with `yield`. The requests you yield stay the same. Two slips give a different symptom, an ERROR line and still no request:

```text
TypeError: e1_start_not_async.LabSpider.start() must be an asynchronous generator, i.e. an async def method with yield statements, got <class 'generator'>
```

That is `def start()` without `async`. An `async def start()` that returns a list ends with `got <class 'coroutine'>`. In both cases Scrapy 2.19.0 closed the spider with `'finish_reason': 'start_error'` and the exit code was still 0.

If the spider overrode `start_requests()` to add headers or meta to `start_urls`, keep the loop:

```python
async def start(self):
    for url in self.start_urls:
        yield scrapy.Request(
            url,
            headers={"Accept-Language": "en-AU"},
            meta={"page": 1},
            dont_filter=True,
        )
```

The default `start()` is the same loop with `dont_filter=True` and nothing else, so keep that argument. In the lab the callback read the header from `response.request.headers` and `page` from `response.meta`.

A proxy goes in `meta` as before:

```python
async def start(self):
    yield scrapy.Request(
        "https://example.com/",
        meta={"proxy": "http://user:pass@proxy.example.com:8080"},
    )
```

The lab's `http://` request arrived at a local proxy that requires Basic authentication, with valid credentials. [Scrapy proxies in 2.19](/guides/scrapy-proxy-setup) covers rotation and the proxy environment variables, and [Fix proxy error 407](/guides/fix-proxy-error-407) covers a proxy that rejects the credentials.

## Support Scrapy below 2.13 and 2.16+ with both methods

Code that has to run on both sides of the change defines both methods. The release notes put it this way: "Use start() instead, or both to maintain support for lower Scrapy versions."

```python
async def start(self):
    for request in self.start_requests():
        yield request

def start_requests(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)
```

Scrapy 2.12.0 called `start_requests()`, and 2.15.2, 2.16.0 and 2.19.0 called `start()`. Each crawled the page, and 2.15.2 logged no deprecation warning. The opposite mistake is just as quiet: a spider with only `async def start()` sent no request on 2.12.0 and closed as finished.

## Spider middlewares: process_start_requests() is skipped without a warning

The same release removed support for `process_start_requests()` in spider middlewares. The lab middleware set `meta["tagged"] = True` on each start request. On 2.19.0, under a migrated spider, the page was crawled, the callback logged `tagged=None` and the log had no warning: the method was never called. Scrapy 2.15.2 stopped the same pair with a `ValueError`, so the upgrade turned an error into silence.

Replace it with [`process_start()`](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#scrapy.spidermiddlewares.SpiderMiddleware.process_start):

```python
class TagStartMiddleware:
    async def process_start(self, start):
        async for item_or_request in start:
            if isinstance(item_or_request, scrapy.Request):
                item_or_request.meta["tagged"] = True
            yield item_or_request
```

With this the callback logged `tagged=True` on 2.15.2, 2.16.0 and 2.19.0. To support Scrapy below 2.13 too, keep `process_start_requests()` next to it. With both methods in the middleware and both in the spider, the request was tagged on all four versions.

## TypeError: Middleware doesn't support asynchronous spider output

A synchronous `process_spider_output()` fails loudly. The crawl stops before `Spider opened`, with exit code 1:

```text
TypeError: Middleware d2_mw_sync_output.StampMiddleware doesn't support asynchronous spider output. Its process_spider_output() method should be an async generator function or it should additionally define a process_spider_output_async() method.
```

`d2_mw_sync_output.StampMiddleware` is the lab's middleware, and yours is named in its place. Scrapy 2.15.2 ran the same middleware with a warning that it "will stop working in a future version of Scrapy". Make the method an async generator:

```python
class StampMiddleware:
    async def process_spider_output(self, response, result):
        async for item_or_request in result:
            if isinstance(item_or_request, dict):
                item_or_request["stamped"] = True
            yield item_or_request
```

The signature has no `spider` parameter. With one, 2.19.0 still ran the middleware and logged `ScrapyDeprecationWarning: StampMiddleware.process_spider_output() requires a spider argument`. A middleware that already has a synchronous `process_spider_output()` and a `process_spider_output_async()` loaded without the error, and 2.19.0 ran the async one. If the class in the message comes from an installed package, look for a release of that package that supports your Scrapy version.

## Check which Scrapy version you are running

```sh
scrapy version
python -m pip show scrapy
```

In the lab the first printed `Scrapy 2.19.0` and the second `Version: 2.19.0`. The first line of the crawl log names it too: `Scrapy 2.19.0 started (bot: scrapybot)`. Run the commands in the environment that runs the crawl, such as the virtual environment, the container image or the CI job.

A `requirements.txt` line that says only `scrapy` installs the newest release on the next build. If you cannot migrate today, hold the last release line that still calls `start_requests()`:

```text
scrapy>=2.15,<2.16
```

In the lab that resolved to 2.15.2, which crawled the unchanged spider and logged a `ScrapyDeprecationWarning` that names the spider class and says the method "will stop being called in a future version of Scrapy". Treat the pin as temporary: the release notes list security bug fixes in [2.17.0](https://docs.scrapy.org/en/latest/news.html#release-2-17-0). After migrating, pin the version you tested. On Python 3.15 without a C compiler, `pip install --only-binary brotli scrapy` quietly installed Scrapy 2.17.0 instead of 2.19.0; [Python 3.15 for web scraping](/blog/python-3-15-web-scraping) shows how to catch that.

## Catch a zero-request crawl in CI

With exit code 0 and `finish_reason` `finished`, a job that only checks whether the crawl succeeded passes. Assert on the request count:

```python
from scrapy.crawler import CrawlerProcess

from myproject.spiders.quotes import QuotesSpider


def test_spider_sends_requests():
    process = CrawlerProcess(settings={"LOG_LEVEL": "WARNING"})
    crawler = process.create_crawler(QuotesSpider)
    process.crawl(crawler)
    process.start()
    assert crawler.stats.get_value("downloader/request_count", 0) > 0
```

On 2.16.0 and 2.19.0 this failed for the `start_requests()` spider with `AssertionError: assert 0 > 0` and passed after the rename. On 2.15.2 it passed for both. The lab ran one crawl per pytest process against a local `python -m http.server`, so point yours at a fixture or a page you are allowed to fetch.

For a scheduled job, read the same number from the stats dump at the end of the log. A run that sent requests has a `'downloader/request_count'` entry, and the failed run had none.

## How this was tested

On Ubuntu 26.04.1 with Python 3.14.4, everything on `127.0.0.1`: `python -m http.server` as the target and a forward proxy that requires Basic authentication. Four virtual environments held Scrapy 2.19.0 and 2.16.0 with Twisted 26.4.0, 2.15.2 with Twisted 25.5.0, and 2.12.0 with Twisted 24.11.0. Each of the 17 spiders ran on each version with `scrapy runspider` in a new interpreter with a clean environment: 68 runs, plus 8 pytest runs. No proxy account was used.

The [lab archive](https://ipvolt.com/downloads/scrapy-start-requests-not-called/scrapy-start-requests-lab.zip) contains the [runner](https://ipvolt.com/downloads/scrapy-start-requests-not-called/run_lab.py), the spiders, the [pytest check](https://ipvolt.com/downloads/scrapy-start-requests-not-called/ci/test_crawl.py), the [README](https://ipvolt.com/downloads/scrapy-start-requests-not-called/README.md) with a table of every result, and the [recorded results](https://ipvolt.com/downloads/scrapy-start-requests-not-called/results.json).

Not tested: macOS and Windows, Scrapy 2.13, 2.14, 2.17 and 2.18, `CrawlSpider` rules and third-party middlewares.

Once the spider sends requests again, each one counts against a proxy plan. [How many GB of proxies do you need](/blog/how-many-gb-of-proxies-do-i-need) measures what an HTML-only fetch uses.

ipvolt is a proxy service for developers that isn't open yet; [join the early-access list](https://ipvolt.com/#waitlist-closing) for one email when it opens.

## Sources & further reading

- [Scrapy release notes: 2.16.0, deprecation removals](https://docs.scrapy.org/en/latest/news.html#release-2-16-0)
- [Scrapy documentation: Spider.start()](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.start)
- [Scrapy documentation: SpiderMiddleware.process_start()](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#scrapy.spidermiddlewares.SpiderMiddleware.process_start)
- [Scrapy documentation: SpiderMiddleware.process_spider_output()](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output)
- [scrapy/scrapy pull request #7490: Drop start_requests() support](https://github.com/scrapy/scrapy/pull/7490)
- [scrapy/scrapy pull request #7504: Remove support for sync process_spider_output()](https://github.com/scrapy/scrapy/pull/7504)
- [Scrapy release notes: 2.17.0, security bug fixes](https://docs.scrapy.org/en/latest/news.html#release-2-17-0)

## Related guides

- [Scrapy proxies in 2.19: auth, rotation and the errors you'll hit](https://ipvolt.com/guides/scrapy-proxy-setup.md)
- [Fix proxy error 407 without guessing](https://ipvolt.com/guides/fix-proxy-error-407.md)
- [Proxy environment variables: HTTP_PROXY and NO_PROXY](https://ipvolt.com/guides/proxy-environment-variables.md)
- [aiohttp proxies in 3.14: auth without BasicAuth, env vars and the errors you'll hit](https://ipvolt.com/guides/aiohttp-proxy-setup.md)
- [Troubleshoot proxy timeouts one stage at a time](https://ipvolt.com/guides/proxy-timeout-troubleshooting.md)

## About ipvolt

Examples use generic proxy settings, with links to the original technical documentation. Product-specific behavior must be checked with your provider. ipvolt is still in development.

## Know when access opens.

ipvolt · In development

We’re building proxy infrastructure for developers and data teams. Join the interest list for a heads-up when ipvolt is ready.

Consent: One email when access opens. Nothing else.

[Get early access](https://ipvolt.com/guides/scrapy-start-requests-not-called#waitlist-closing). Use the email form on this page to join the interest list.

[Privacy](https://ipvolt.com/privacy)