Scrapy crawls 0 pages after upgrading: start_requests() is no longer called

After an upgrade to Scrapy 2.16 or later, a spider that defines only start_requests() opens, sends no request and closes as if it had finished:

2026-10-09 12:33:33 [scrapy.core.engine] INFO: Spider opened
2026-10-09 12:33:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-10-09 12:33:33 [scrapy.core.engine] INFO: Closing spider (finished)
2026-10-09 12:33:33 [scrapy.core.engine] INFO: Spider closed (finished)

These are the engine and logstats lines of a Scrapy 2.19.0 run. The log had no WARNING or ERROR line, the exit code was 0 and parse() never ran. The stats dump had 'finish_reason': 'finished' and no downloader/request_count key.

The cause: Scrapy 2.16.0 (19 May 2026) removed Spider.start_requests(), deprecated since 2.13.0. The release notes say it "is removed and no longer called" (pull request #7490). The fix is to rename the method to async def start():

# Before: Scrapy 2.16 and later never call this
def start_requests(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)

# After
async def start(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)

With that one change the same spider crawled its page on Scrapy 2.16.0 and 2.19.0.

Everything below was recorded on 9 October 2026 with Scrapy 2.19.0, Twisted 26.4.0 and Python 3.14.4, and repeated on Scrapy 2.16.0, 2.15.2 and 2.12.0. A spider that only sets start_urls and defines neither method is not affected: it crawled on all four.

Migrate start_requests() to async def start()

Spider.start() is an asynchronous generator: async def with yield. The requests you yield stay the same. Two slips give a different symptom, an ERROR line and still no request:

TypeError: e1_start_not_async.LabSpider.start() must be an asynchronous generator, i.e. an async def method with yield statements, got <class 'generator'>

That is def start() without async. An async def start() that returns a list ends with got <class 'coroutine'>. In both cases Scrapy 2.19.0 closed the spider with 'finish_reason': 'start_error' and the exit code was still 0.

If the spider overrode start_requests() to add headers or meta to start_urls, keep the loop:

async def start(self):
    for url in self.start_urls:
        yield scrapy.Request(
            url,
            headers={"Accept-Language": "en-AU"},
            meta={"page": 1},
            dont_filter=True,
        )

The default start() is the same loop with dont_filter=True and nothing else, so keep that argument. In the lab the callback read the header from response.request.headers and page from response.meta.

A proxy goes in meta as before:

async def start(self):
    yield scrapy.Request(
        "https://example.com/",
        meta={"proxy": "http://user:pass@proxy.example.com:8080"},
    )

The lab's http:// request arrived at a local proxy that requires Basic authentication, with valid credentials. Scrapy proxies in 2.19 covers rotation and the proxy environment variables, and Fix proxy error 407 covers a proxy that rejects the credentials.

Support Scrapy below 2.13 and 2.16+ with both methods

Code that has to run on both sides of the change defines both methods. The release notes put it this way: "Use start() instead, or both to maintain support for lower Scrapy versions."

async def start(self):
    for request in self.start_requests():
        yield request

def start_requests(self):
    yield scrapy.Request("https://example.com/", callback=self.parse)

Scrapy 2.12.0 called start_requests(), and 2.15.2, 2.16.0 and 2.19.0 called start(). Each crawled the page, and 2.15.2 logged no deprecation warning. The opposite mistake is just as quiet: a spider with only async def start() sent no request on 2.12.0 and closed as finished.

Spider middlewares: process_start_requests() is skipped without a warning

The same release removed support for process_start_requests() in spider middlewares. The lab middleware set meta["tagged"] = True on each start request. On 2.19.0, under a migrated spider, the page was crawled, the callback logged tagged=None and the log had no warning: the method was never called. Scrapy 2.15.2 stopped the same pair with a ValueError, so the upgrade turned an error into silence.

Replace it with process_start():

class TagStartMiddleware:
    async def process_start(self, start):
        async for item_or_request in start:
            if isinstance(item_or_request, scrapy.Request):
                item_or_request.meta["tagged"] = True
            yield item_or_request

With this the callback logged tagged=True on 2.15.2, 2.16.0 and 2.19.0. To support Scrapy below 2.13 too, keep process_start_requests() next to it. With both methods in the middleware and both in the spider, the request was tagged on all four versions.

TypeError: Middleware doesn't support asynchronous spider output

A synchronous process_spider_output() fails loudly. The crawl stops before Spider opened, with exit code 1:

TypeError: Middleware d2_mw_sync_output.StampMiddleware doesn't support asynchronous spider output. Its process_spider_output() method should be an async generator function or it should additionally define a process_spider_output_async() method.

d2_mw_sync_output.StampMiddleware is the lab's middleware, and yours is named in its place. Scrapy 2.15.2 ran the same middleware with a warning that it "will stop working in a future version of Scrapy". Make the method an async generator:

class StampMiddleware:
    async def process_spider_output(self, response, result):
        async for item_or_request in result:
            if isinstance(item_or_request, dict):
                item_or_request["stamped"] = True
            yield item_or_request

The signature has no spider parameter. With one, 2.19.0 still ran the middleware and logged ScrapyDeprecationWarning: StampMiddleware.process_spider_output() requires a spider argument. A middleware that already has a synchronous process_spider_output() and a process_spider_output_async() loaded without the error, and 2.19.0 ran the async one. If the class in the message comes from an installed package, look for a release of that package that supports your Scrapy version.

Check which Scrapy version you are running

scrapy version
python -m pip show scrapy

In the lab the first printed Scrapy 2.19.0 and the second Version: 2.19.0. The first line of the crawl log names it too: Scrapy 2.19.0 started (bot: scrapybot). Run the commands in the environment that runs the crawl, such as the virtual environment, the container image or the CI job.

A requirements.txt line that says only scrapy installs the newest release on the next build. If you cannot migrate today, hold the last release line that still calls start_requests():

scrapy>=2.15,<2.16

In the lab that resolved to 2.15.2, which crawled the unchanged spider and logged a ScrapyDeprecationWarning that names the spider class and says the method "will stop being called in a future version of Scrapy". Treat the pin as temporary: the release notes list security bug fixes in 2.17.0. After migrating, pin the version you tested. On Python 3.15 without a C compiler, pip install --only-binary brotli scrapy quietly installed Scrapy 2.17.0 instead of 2.19.0; Python 3.15 for web scraping shows how to catch that.

Catch a zero-request crawl in CI

With exit code 0 and finish_reason finished, a job that only checks whether the crawl succeeded passes. Assert on the request count:

from scrapy.crawler import CrawlerProcess

from myproject.spiders.quotes import QuotesSpider


def test_spider_sends_requests():
    process = CrawlerProcess(settings={"LOG_LEVEL": "WARNING"})
    crawler = process.create_crawler(QuotesSpider)
    process.crawl(crawler)
    process.start()
    assert crawler.stats.get_value("downloader/request_count", 0) > 0

On 2.16.0 and 2.19.0 this failed for the start_requests() spider with AssertionError: assert 0 > 0 and passed after the rename. On 2.15.2 it passed for both. The lab ran one crawl per pytest process against a local python -m http.server, so point yours at a fixture or a page you are allowed to fetch.

For a scheduled job, read the same number from the stats dump at the end of the log. A run that sent requests has a 'downloader/request_count' entry, and the failed run had none.

How this was tested

On Ubuntu 26.04.1 with Python 3.14.4, everything on 127.0.0.1: python -m http.server as the target and a forward proxy that requires Basic authentication. Four virtual environments held Scrapy 2.19.0 and 2.16.0 with Twisted 26.4.0, 2.15.2 with Twisted 25.5.0, and 2.12.0 with Twisted 24.11.0. Each of the 17 spiders ran on each version with scrapy runspider in a new interpreter with a clean environment: 68 runs, plus 8 pytest runs. No proxy account was used.

The lab archive contains the runner, the spiders, the pytest check, the README with a table of every result, and the recorded results.

Not tested: macOS and Windows, Scrapy 2.13, 2.14, 2.17 and 2.18, CrawlSpider rules and third-party middlewares.

Once the spider sends requests again, each one counts against a proxy plan. How many GB of proxies do you need measures what an HTML-only fetch uses.

ipvolt is a proxy service for developers that isn't open yet; join the early-access list for one email when it opens.