Python 3.15 for web scraping: what installs, what breaks

On Python 3.15.0, most of a scraping stack installs from ready-made wheels and passed a smoke test. Anything that needs brotli or zstandard fails on a machine without a C compiler, because neither has a wheel for 3.15, and that includes a plain pip install scrapy. The quick fix is to install gcc before pip install (Fix 1) or to stay on Python 3.14 until brotli publishes cp315 wheels (Fix 2).

Last checked: 16 October 2026, 10:09 UTC. At that time PyPI still had no cp315 wheel for brotli 1.2.0 or zstandard 0.25.0. zope.interface 8.7, released on 15 October, added cp315 wheels.

"With compiler" and "without compiler" are the official python:3.15-slim image with and without gcc, and every package was installed with both pip and uv, which agreed in every case. A pure-Python package has no compiled code, so it needs no cp315 wheel. Each install passed its smoke test.

pip install scrapy fails on Python 3.15: Failed building wheel for brotli

In the slim image, pip 26.2.1 stops at brotli:

  × Building wheel for brotli (pyproject.toml) did not run successfully.
  │ exit code: 1
      error: [Errno 2] No such file or directory: 'gcc'
  ERROR: Failed building wheel for brotli
error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> brotli

uv 0.13.1 fails at the same package and names the chain:

error: Failed to build `brotli==1.2.0`
  cause: The build backend returned an error
hint: `brotli` (v1.2.0) was included because `scrapy` (v2.19.0) depends on `brotli`

The chain is short. Scrapy 2.19.0 is pure Python, but it requires brotli>=1.2.0; implementation_name != "pypy", so every CPython install pulls in brotli. That requirement arrived in Scrapy 2.18.0 (20 August 2026); 2.17.0 did not have it. brotli 1.2.0, released 5 November 2025, has wheels for CPython 2.7 and 3.6 to 3.14 and nothing newer. Without a matching wheel, pip and uv download the source archive and compile the _brotli extension, which needs gcc. The same thing happens with httpx[brotli], aiohttp[speedups] and urllib3[brotli], which also want the brotli package on CPython, and with httpx[zstd], which wants zstandard 0.25.0.

Fix 1: add a compiler to the image

This one line, run in the slim image, was enough for every failing package:

apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev

It installed gcc 14.2.0. Afterwards pip install scrapy took 28 seconds instead of failing, uv took 19, and the spider crawled its test pages, including a brotli-compressed one. zstandard is the slow build: 63 seconds for httpx[zstd] with pip. In a Dockerfile, put the line before the pip install.

Only this Debian line was tested. On Windows, the compiler these builds need comes with Microsoft's C++ Build Tools; we did not test Windows.

Fix 2: stay on Python 3.14 until the wheels exist

Python 3.14 has brotli and zstandard wheels, so the problem does not exist there. To see whether brotli has caught up, run this with Python 3.15:

python3 -m pip download --only-binary :all: --no-deps -d /tmp/wheel-check brotli

On 16 October it still ended with ERROR: No matching distribution found for brotli. Once it downloads a .whl file, the cp315 wheels are out and pip install scrapy works without a compiler.

Upstream, google/brotli-wheels issue #25, "Python 3.15 wheels missing", was opened on 9 October and had no replies on 16 October. google/brotli pull request #1583 adds a CI job that builds cp315 wheels. On 16 October it was still open with no review, and the only comment was the CLA bot's. Its description notes that a maintainer would still need to publish the wheels to PyPI.

Not Scrapy? brotlicffi has a wheel

brotlicffi 1.2.0.2 ships an abi3 wheel that pip installed on 3.15. HTTPX, urllib3 and aiohttp accept it in place of brotli, so without a compiler these worked and decoded a brotli response:

pip install httpx brotlicffi
pip install urllib3 brotlicffi
pip install aiohttp aiodns brotlicffi

The last one is aiohttp[speedups] minus the Brotli requirement. Scrapy cannot use this route: on CPython it requires the brotli package itself.

zstd is easier than it looks. On Python 3.14 and newer, urllib3 2.8.0 and aiohttp 3.14.5 decode zstd with the standard library's compression.zstd, so both offered zstd and decoded a zstd response with no extra package installed. Only httpx[zstd] needs zstandard.

Don't use --only-binary blindly

--only-binary :all: tells pip to never build from source. It looks like the obvious workaround, and it does not fail:

pip install --only-binary :all: "httpx[brotli]" "aiohttp[speedups]"

pip printed INFO: This is taking longer than usual while it searched, then installed httpx 0.18.2 (June 2021) and aiohttp 0.13.1 (December 2014). The only other hint was one line:

WARNING: aiohttp 0.13.1 does not provide the extra 'speedups'

The resolver walked back through old releases until it found ones whose requirements could all be met with wheels. uv pip install --no-build picked the same two versions and printed warning: The package `aiohttp==0.13.1` does not have an extra named `speedups` .

Narrowing the flag to one package does not make it safe. pip install --only-binary brotli scrapy also exits 0, and it installs Scrapy 2.17.0, the last release without the brotli requirement. uv did the same. That Scrapy crawled the test pages, but it offered servers only gzip, deflate where 2.19.0 with brotli offered gzip, deflate, br, zstd. If your spiders use async def start(), 2.17.0 still runs them.

To spot a silent downgrade, read the installed versions after every install:

pip list
uv pip list

To make it fail loudly instead, pin the version you want next to the flag:

pip install --only-binary brotli "scrapy==2.19.0"

ERROR: Could not find a version that satisfies the requirement brotli>=1.2.0; implementation_name != "pypy" (from scrapy) (from versions: none)
ERROR: No matching distribution found for brotli>=1.2.0; implementation_name != "pypy"

uv with the same pin stopped with error: No solution found when resolving dependencies. A requirements.txt with exact versions for your top-level packages gives you the same protection on every build.

Fixed on 15 October: zope.interface without C speedups

Twisted, and with it Scrapy, depends on zope.interface. zope.interface 8.6 has wheels up to CPython 3.14 only. On 14 October, without a compiler, pip and uv still installed it on 3.15: its build falls back to pure Python when the C extension cannot be compiled. No error and no warning appeared in the install output.

The cost showed up in the basic interface check. In the lab, IThing.providedBy(obj) took 271 ns per call without the C extension and 101 ns with it. The lab did not measure what that means for a whole crawl.

zope.interface 8.7, released on 15 October, ships cp315 wheels. On 16 October a fresh install without a compiler got the 8.7 binary wheel with pip and uv, the C extension imported, and the same check took 102 ns. A virtual environment created before 15 October keeps the pure-Python 8.6 until you upgrade zope.interface.

To see which one you have:

python -c "import zope.interface._zope_interface_coptimizations"

A ModuleNotFoundError means the pure-Python fallback. With the 8.7 wheel, or with gcc installed, the import succeeded.

Other 3.15 changes that matter to scrapers

The one most scrapers will notice is UTF-8 mode, now on by default (PEP 686). What's New in Python 3.15 says I/O operations without an explicit encoding, such as open('flying-circus.txt'), now use UTF-8 whatever the system's environment. In the lab, with LANG=C, open() without encoding wrote UTF-8 on 3.15; with -X utf8=0 it fell back to ANSI_X3.4-1968, which is ASCII.

For scrapers this mostly shows up in exports. A CSV or JSON file written with open(path, "w") is now UTF-8 on every machine, including Windows systems where 3.14 used the locale's code page. That is usually what you want. If a downstream tool expects the old encoding, pass encoding="locale" or set PYTHONUTF8=0; better, pass the encoding explicitly in both writer and reader.

If you're upgrading Scrapy at the same time

A move to Python 3.15 often comes with a new Scrapy. Since 2.16, Scrapy no longer calls start_requests(): a spider that defines only that method opens, sends nothing and closes as finished, with no warning. Scrapy crawls 0 pages after upgrading has the one-line fix, async def start(), which the spider in this lab used.

Method and limits

Recorded on 14 October 2026 on an x86_64 Ubuntu host, in the file system of the official python:3.15-slim image (Debian 13 trixie, index digest sha256:ea4f1f29…, built 12 October 2026), run in a private Linux namespace rather than Docker. The interpreter reported Python 3.15.0 (main, Oct 12 2026, 22:33:31) [GCC 14.2.0], with pip 26.2.1 as bundled and uv 0.13.1. The compiler copy added gcc 14.2.0 with the apt line above. The table comes from those runs. On 16 October the Scrapy, --only-binary brotli and zope.interface cases were run again without a compiler: everything matched 14 October except zope.interface, which now came from the 8.7 wheel.

Every case got a new virtual environment and no package cache. The installed versions and whether each came from a binary wheel, a pure-Python wheel or a local build were read from the wheel tags. The smoke tests ran against a local http.server that compresses only with encodings the client offers. Playwright was installed but its browsers were not. No proxy was used.

The lab says nothing about macOS, Windows, ARM, musl-based images or free-threaded builds. PyPI changes daily, so rerun the wheel check before acting on any row. The script, the smoke tests and the raw output are in the download folder: lab315.py, results.csv, results.json and the 16 October rerun, results-2026-10-16.json.

Related

• Scrapy proxies in 2.19, for auth, rotation and the proxy errors Scrapy logs.
• aiohttp proxies and curl_cffi proxies, both of which installed from wheels on 3.15.
• Fix "Missing dependencies for SOCKS support", the other install-time error in this stack.
• How many GB of proxies do I need?, for sizing traffic once the stack runs. Compression matters there: a client that offers fewer encodings can download more bytes.

ipvolt is building proxy infrastructure for developers and agents. Join the waitlist for one email when access opens.

Tagged:ProxiesTroubleshooting