# Python 3.15 for web scraping: what installs, what breaks Source: https://ipvolt.com/blog/python-3-15-web-scraping Markdown: https://ipvolt.com/blog/python-3-15-web-scraping.md [Home](https://ipvolt.com/index.md) / [Blog](https://ipvolt.com/blog.md) / Python 3.15 for Web Scraping: What Installs, What Breaks Category: Analysis Published: 2026-10-16 Updated: 2026-10-16 Author: ipvolt Reading time: 8 minutes Tags: proxies, troubleshooting pip install scrapy fails on Python 3.15 without a C compiler: brotli has no cp315 wheel. We tested 19 scraping packages with pip and uv to see what else breaks. On Python 3.15.0, most of a scraping stack installs from ready-made wheels and passed a smoke test. Anything that needs brotli or zstandard fails on a machine without a C compiler, because neither has a wheel for 3.15, and that includes a plain `pip install scrapy`. The quick fix is to install `gcc` before `pip install` ([Fix 1](#fix-1-add-a-compiler-to-the-image)) or to stay on Python 3.14 until brotli publishes cp315 wheels ([Fix 2](#fix-2-stay-on-python-314-until-the-wheels-exist)). **Last checked: 16 October 2026, 10:09 UTC.** At that time PyPI still had no cp315 wheel for brotli 1.2.0 or zstandard 0.25.0. zope.interface 8.7, released on 15 October, added cp315 wheels. | Package | Version | cp315 wheel | With compiler | Without compiler | Notes | | --- | --- | --- | --- | --- | --- | | `scrapy` | 2.19.0 | pure Python; no (brotli 1.2.0) | installs | fails: brotli build | requires brotli on CPython since 2.18.0 | | `requests[socks]` | 2.34.2 + PySocks 1.7.1 | pure Python | installs | installs | | | `httpx[socks]` | 0.28.1 + socksio 1.0.0 | pure Python | installs | installs | | | `httpx[brotli]` | 0.28.1 | no (brotli 1.2.0) | installs | fails: brotli build | `httpx brotlicffi` installs without a compiler | | `httpx[zstd]` | 0.28.1 | no (zstandard 0.25.0) | installs | fails: zstandard build | slowest build | | `aiohttp` | 3.14.5 | yes | installs | installs | | | `aiohttp[speedups]` | 3.14.5 + aiodns 4.0.4 | no (Brotli 1.2.0) | installs | fails: Brotli build | `aiohttp aiodns brotlicffi` installs | | `aiohttp-socks` | 0.12.0 | pure Python | installs | installs | | | `urllib3[brotli]` | 2.8.0 | no (brotli 1.2.0) | installs | fails: brotli build | `urllib3 brotlicffi` installs | | `curl_cffi` | 0.16.3 | abi3 | installs | installs | | | `playwright` | 1.64.0 | not version-specific | installs | installs | browsers not tested | | `lxml` | 6.1.3 | yes | installs | installs | | | `beautifulsoup4` | 4.15.0 | pure Python | installs | installs | | | `parsel` | 1.12.1 | pure Python | installs | installs | | | `selectolax` | 1.0.0 | yes | installs | installs | 1.0: `selectolax.parser` raised ImportError, use `selectolax.lexbor` | | `pandas` | 3.0.6 + numpy 2.5.4 | yes | installs | installs | | | `pydantic` | 2.14.1 + pydantic-core 2.50.1 | pydantic-core: yes | installs | installs | | | `orjson` | 3.13.0 | yes | installs | installs | | | `tenacity` | 9.2.1 | pure Python | installs | installs | | "With compiler" and "without compiler" are the official `python:3.15-slim` image with and without `gcc`, and every package was installed with both pip and uv, which agreed in every case. A pure-Python package has no compiled code, so it needs no cp315 wheel. Each install passed its smoke test. ## pip install scrapy fails on Python 3.15: Failed building wheel for brotli In the slim image, pip 26.2.1 stops at brotli: ```text × Building wheel for brotli (pyproject.toml) did not run successfully. │ exit code: 1 error: [Errno 2] No such file or directory: 'gcc' ERROR: Failed building wheel for brotli error: failed-wheel-build-for-install × Failed to build installable wheels for some pyproject.toml based projects ╰─> brotli ``` uv 0.13.1 fails at the same package and names the chain: ```text error: Failed to build `brotli==1.2.0` cause: The build backend returned an error hint: `brotli` (v1.2.0) was included because `scrapy` (v2.19.0) depends on `brotli` ``` The chain is short. Scrapy 2.19.0 is pure Python, but it requires `brotli>=1.2.0; implementation_name != "pypy"`, so every CPython install pulls in brotli. That requirement arrived in Scrapy 2.18.0 (20 August 2026); 2.17.0 did not have it. brotli 1.2.0, released 5 November 2025, has wheels for CPython 2.7 and 3.6 to 3.14 and nothing newer. Without a matching wheel, pip and uv download the source archive and compile the `_brotli` extension, which needs `gcc`. The same thing happens with `httpx[brotli]`, `aiohttp[speedups]` and `urllib3[brotli]`, which also want the `brotli` package on CPython, and with `httpx[zstd]`, which wants zstandard 0.25.0. ### Fix 1: add a compiler to the image This one line, run in the slim image, was enough for every failing package: ```sh apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev ``` It installed gcc 14.2.0. Afterwards `pip install scrapy` took 28 seconds instead of failing, uv took 19, and the spider crawled its test pages, including a brotli-compressed one. zstandard is the slow build: 63 seconds for `httpx[zstd]` with pip. In a Dockerfile, put the line before the `pip install`. Only this Debian line was tested. On Windows, the compiler these builds need comes with Microsoft's [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/); we did not test Windows. ### Fix 2: stay on Python 3.14 until the wheels exist Python 3.14 has brotli and zstandard wheels, so the problem does not exist there. To see whether brotli has caught up, run this with Python 3.15: ```sh python3 -m pip download --only-binary :all: --no-deps -d /tmp/wheel-check brotli ``` On 16 October it still ended with `ERROR: No matching distribution found for brotli`. Once it downloads a `.whl` file, the cp315 wheels are out and `pip install scrapy` works without a compiler. Upstream, [google/brotli-wheels issue #25](https://github.com/google/brotli-wheels/issues/25), "Python 3.15 wheels missing", was opened on 9 October and had no replies on 16 October. [google/brotli pull request #1583](https://github.com/google/brotli/pull/1583) adds a CI job that builds cp315 wheels. On 16 October it was still open with no review, and the only comment was the CLA bot's. Its description notes that a maintainer would still need to publish the wheels to PyPI. ### Not Scrapy? brotlicffi has a wheel brotlicffi 1.2.0.2 ships an `abi3` wheel that pip installed on 3.15. HTTPX, urllib3 and aiohttp accept it in place of `brotli`, so without a compiler these worked and decoded a brotli response: ```sh pip install httpx brotlicffi pip install urllib3 brotlicffi pip install aiohttp aiodns brotlicffi ``` The last one is `aiohttp[speedups]` minus the `Brotli` requirement. Scrapy cannot use this route: on CPython it requires the `brotli` package itself. zstd is easier than it looks. On Python 3.14 and newer, urllib3 2.8.0 and aiohttp 3.14.5 decode zstd with the standard library's `compression.zstd`, so both offered `zstd` and decoded a zstd response with no extra package installed. Only `httpx[zstd]` needs zstandard. ## Don't use --only-binary blindly `--only-binary :all:` tells pip to never build from source. It looks like the obvious workaround, and it does not fail: ```sh pip install --only-binary :all: "httpx[brotli]" "aiohttp[speedups]" ``` pip printed `INFO: This is taking longer than usual` while it searched, then installed httpx 0.18.2 (June 2021) and aiohttp 0.13.1 (December 2014). The only other hint was one line: ```text WARNING: aiohttp 0.13.1 does not provide the extra 'speedups' ``` The resolver walked back through old releases until it found ones whose requirements could all be met with wheels. `uv pip install --no-build` picked the same two versions and printed ``warning: The package `aiohttp==0.13.1` does not have an extra named `speedups` ``. Narrowing the flag to one package does not make it safe. `pip install --only-binary brotli scrapy` also exits 0, and it installs Scrapy 2.17.0, the last release without the brotli requirement. uv did the same. That Scrapy crawled the test pages, but it offered servers only `gzip, deflate` where 2.19.0 with brotli offered `gzip, deflate, br, zstd`. If your spiders use `async def start()`, 2.17.0 still runs them. To spot a silent downgrade, read the installed versions after every install: ```sh pip list uv pip list ``` To make it fail loudly instead, pin the version you want next to the flag: ```sh pip install --only-binary brotli "scrapy==2.19.0" ``` ```text ERROR: Could not find a version that satisfies the requirement brotli>=1.2.0; implementation_name != "pypy" (from scrapy) (from versions: none) ERROR: No matching distribution found for brotli>=1.2.0; implementation_name != "pypy" ``` uv with the same pin stopped with `error: No solution found when resolving dependencies`. A `requirements.txt` with exact versions for your top-level packages gives you the same protection on every build. ## Fixed on 15 October: zope.interface without C speedups Twisted, and with it Scrapy, depends on zope.interface. zope.interface 8.6 has wheels up to CPython 3.14 only. On 14 October, without a compiler, pip and uv still installed it on 3.15: its build falls back to pure Python when the C extension cannot be compiled. No error and no warning appeared in the install output. The cost showed up in the basic interface check. In the lab, `IThing.providedBy(obj)` took 271 ns per call without the C extension and 101 ns with it. The lab did not measure what that means for a whole crawl. zope.interface 8.7, released on 15 October, ships cp315 wheels. On 16 October a fresh install without a compiler got the 8.7 binary wheel with pip and uv, the C extension imported, and the same check took 102 ns. A virtual environment created before 15 October keeps the pure-Python 8.6 until you upgrade zope.interface. To see which one you have: ```sh python -c "import zope.interface._zope_interface_coptimizations" ``` A `ModuleNotFoundError` means the pure-Python fallback. With the 8.7 wheel, or with gcc installed, the import succeeded. ## Other 3.15 changes that matter to scrapers The one most scrapers will notice is UTF-8 mode, now on by default ([PEP 686](https://peps.python.org/pep-0686/)). [What's New in Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html#whatsnew315-utf8-default) says I/O operations without an explicit encoding, such as `open('flying-circus.txt')`, now use UTF-8 whatever the system's environment. In the lab, with `LANG=C`, `open()` without `encoding` wrote UTF-8 on 3.15; with `-X utf8=0` it fell back to `ANSI_X3.4-1968`, which is ASCII. For scrapers this mostly shows up in exports. A CSV or JSON file written with `open(path, "w")` is now UTF-8 on every machine, including Windows systems where 3.14 used the locale's code page. That is usually what you want. If a downstream tool expects the old encoding, pass `encoding="locale"` or set `PYTHONUTF8=0`; better, pass the encoding explicitly in both writer and reader. ## If you're upgrading Scrapy at the same time A move to Python 3.15 often comes with a new Scrapy. Since 2.16, Scrapy no longer calls `start_requests()`: a spider that defines only that method opens, sends nothing and closes as finished, with no warning. [Scrapy crawls 0 pages after upgrading](/guides/scrapy-start-requests-not-called) has the one-line fix, `async def start()`, which the spider in this lab used. ## Method and limits Recorded on 14 October 2026 on an x86_64 Ubuntu host, in the file system of the official `python:3.15-slim` image (Debian 13 trixie, index digest `sha256:ea4f1f29…`, built 12 October 2026), run in a private Linux namespace rather than Docker. The interpreter reported `Python 3.15.0 (main, Oct 12 2026, 22:33:31) [GCC 14.2.0]`, with pip 26.2.1 as bundled and uv 0.13.1. The compiler copy added gcc 14.2.0 with the apt line above. The table comes from those runs. On 16 October the Scrapy, `--only-binary brotli` and zope.interface cases were run again without a compiler: everything matched 14 October except zope.interface, which now came from the 8.7 wheel. Every case got a new virtual environment and no package cache. The installed versions and whether each came from a binary wheel, a pure-Python wheel or a local build were read from the wheel tags. The smoke tests ran against a local `http.server` that compresses only with encodings the client offers. Playwright was installed but its browsers were not. No proxy was used. The lab says nothing about macOS, Windows, ARM, musl-based images or free-threaded builds. PyPI changes daily, so rerun the wheel check before acting on any row. The script, the smoke tests and the raw output are in the [download folder](/downloads/python-3-15-web-scraping/README.md): [lab315.py](/downloads/python-3-15-web-scraping/lab315.py), [results.csv](/downloads/python-3-15-web-scraping/results.csv), [results.json](/downloads/python-3-15-web-scraping/results.json) and the 16 October rerun, [results-2026-10-16.json](/downloads/python-3-15-web-scraping/results-2026-10-16.json). ## Related - [Scrapy proxies in 2.19](/guides/scrapy-proxy-setup), for auth, rotation and the proxy errors Scrapy logs. - [aiohttp proxies](/guides/aiohttp-proxy-setup) and [curl_cffi proxies](/guides/curl-cffi-proxy-setup), both of which installed from wheels on 3.15. - [Fix "Missing dependencies for SOCKS support"](/guides/fix-missing-dependencies-for-socks-support), the other install-time error in this stack. - [How many GB of proxies do I need?](/blog/how-many-gb-of-proxies-do-i-need), for sizing traffic once the stack runs. Compression matters there: a client that offers fewer encodings can download more bytes. ipvolt is building proxy infrastructure for developers and agents. [Join the waitlist](https://ipvolt.com/#waitlist-hero) for one email when access opens. ## Sources - [Python.org: Python 3.15.0 release page](https://www.python.org/downloads/release/python-3150/) - [What's New in Python 3.15: UTF-8 is the default encoding](https://docs.python.org/3.15/whatsnew/3.15.html#whatsnew315-utf8-default) - [PEP 686: Make UTF-8 mode default](https://peps.python.org/pep-0686/) - [PyPI: brotli 1.2.0 files](https://pypi.org/project/brotli/1.2.0/#files) - [PyPI: zstandard 0.25.0 files](https://pypi.org/project/zstandard/0.25.0/#files) - [PyPI: zope.interface 8.6 files](https://pypi.org/project/zope.interface/8.6/#files) - [PyPI: zope.interface 8.7 files](https://pypi.org/project/zope.interface/8.7/#files) - [PyPI: Scrapy 2.19.0](https://pypi.org/project/Scrapy/2.19.0/) - [google/brotli-wheels issue #25: Python 3.15 wheels missing](https://github.com/google/brotli-wheels/issues/25) - [google/brotli pull request #1583: Build and test CPython 3.15 wheels in CI](https://github.com/google/brotli/pull/1583) - [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) ## Know when ipvolt access opens. ipvolt is in development. Leave your email and we’ll notify you once when access opens. Consent: One email when access opens. Nothing else. [Notify me](https://ipvolt.com/blog/python-3-15-web-scraping#waitlist-blog-end). Use the email form on this page to join the interest list. [Privacy](https://ipvolt.com/privacy) ## Related posts - [Check a Proxy for IP, DNS and WebRTC Leaks with ToDetect](https://ipvolt.com/blog/proxy-leak-check-todetect.md) (Analysis, Oct 23, 2026, 9 min read): Check a browser proxy for IP, DNS, WebRTC and time zone leaks with ToDetect's free tools, using measured results from a 4G mobile line as the worked example. - [Japanese price parsing: yen, width and tax labels](https://ipvolt.com/blog/japanese-price-parsing.md) (Analysis, Oct 6, 2026, 6 min read): Parse Japanese price fields without losing currency or tax basis. Run a tested Python fixture for fullwidth yen, mixed prices and deliberate review cases. - [SOCKS5 vs SOCKS5h: What 6 Clients Actually Send (Tested)](https://ipvolt.com/blog/socks5-vs-socks5h.md) (Comparison, Oct 4, 2026, 7 min read): socks5:// does not mean local DNS in every client. We logged what curl, Requests, HTTPX, aiohttp, Playwright and Node send a SOCKS5 proxy for each scheme. ## Related guides - [Scrapy proxies in 2.19: auth, rotation and the errors you'll hit](https://ipvolt.com/guides/scrapy-proxy-setup.md): Scrapy 2.19 proxy setup: credentials in meta["proxy"], per-request rotation, HTTP_PROXY, and fixes for TunnelError 407, ignored 407 responses and socks5. - [Scrapy crawls 0 pages after upgrading: start_requests() is no longer called](https://ipvolt.com/guides/scrapy-start-requests-not-called.md): Scrapy 2.16 removed Spider.start_requests(): the spider logs Crawled 0 pages and closes as finished with no warning. Fix: async def start(). Tested on 2.19. - [aiohttp proxies in 3.14: auth without BasicAuth, env vars and the errors you'll hit](https://ipvolt.com/guides/aiohttp-proxy-setup.md): Set up an HTTP or SOCKS proxy in aiohttp 3.14: replace the deprecated BasicAuth and proxy_auth, turn on trust_env, and read the 407 and Bad status line errors. ## About ipvolt Technical analysis from the ipvolt team. ipvolt access is not open yet.