# Accept-Encoding defaults lab

Companion lab for the ipvolt guide [curl --compressed and Accept-Encoding defaults](https://ipvolt.com/guides/curl-compressed-accept-encoding).

For each HTTP client setup, the lab records which `Accept-Encoding` header the client sends, what a local
server sends back, how many body bytes cross the wire, how many bytes a local forward proxy relays, and
whether your code receives the decoded body. It covers curl, Wget, Python (urllib, http.client, Requests,
httpx, aiohttp, Scrapy), Node.js (built-in fetch, node:http, axios, got, node-fetch), Go, PHP (ext-curl,
Guzzle, Laravel Http) and Java (java.net.http.HttpClient) at pinned versions.

**These are synthetic, loopback observations.** The origin server and the proxy run on 127.0.0.1 and
serve local files. Nothing here measures a public site. No byte count here is what a proxy provider
bills: wire body bytes exclude response headers, TLS records, the CONNECT exchange and retries.

## Results at a glance

Recorded on 2026-09-27 (UTC) on macOS 15.7.4 (Apple M4 Pro) in one run of the whole matrix: 567 records,
all `run`. The full tables are in `tables.md`; every record is in `results.json`
and `results.csv`.

### Sends no Accept-Encoding, or identity

A server that compresses only when asked sends these clients uncompressed text. No header: curl, node:http, node:https, PHP ext-curl, Guzzle, Laravel Http, Java HttpClient. `identity`: Wget, urllib, http.client.

| Client | Fix |
|---|---|
| curl | Add `--compressed` |
| Wget | Add `--compression=auto` |
| urllib, http.client | Use Requests or httpx |
| node:http, node:https | Use `fetch`, got or axios |
| PHP ext-curl | Set `CURLOPT_ACCEPT_ENCODING` to `''` |
| Guzzle, Laravel Http | Set `decode_content` to `'gzip'` |
| Java HttpClient | Send `Accept-Encoding: gzip`, unzip with `GZIPInputStream` |

Versions run: curl 8.7.1, 8.22.0; Wget 1.25.0; urllib.request, http.client (Python 3.13.15, 3.14.7); node:http, node:https (Node.js 22.23.3, 24.21.0, 26.10.0); PHP 8.5.8 ext-curl (libcurl 8.20.0); Guzzle 8.2.0, Laravel Http 13.33.0; Java 25.0.4.1 HttpClient.

- curl: a hand-set `-H 'Accept-Encoding: gzip'` is sent, but the body stays compressed.
- Wget: with `--compression=auto` it sent `gzip` and decoded the reply.
- urllib.request and http.client return compressed bodies as they arrived.
- node:http and node:https return compressed bodies as they arrived.
- PHP ext-curl: with `CURLOPT_ACCEPT_ENCODING` set to `''`, this libcurl sent `deflate, gzip, br, zstd`. A hand-set header is sent but not decoded.
- Laravel Http: `withOptions(['decode_content' => 'gzip'])`.
- Java HttpClient never decodes: wrap the body in `GZIPInputStream` when the reply is gzip.

### Already asks for compression

Nothing to change: these clients ask for compression and decode the reply.

| Client | Header sent |
|---|---|
| Requests | Python 3.13.15: `gzip, deflate`; Python 3.14.7: `gzip, deflate, zstd` |
| httpx | `gzip, deflate` |
| aiohttp | Python 3.13.15: `gzip, deflate`; Python 3.14.7: `gzip, deflate, zstd` |
| Scrapy | `gzip, deflate, br, zstd` |
| Node.js built-in fetch | Node.js 22.23.3/24.21.0: `br, gzip, deflate`; Node.js 26.10.0: `br, gzip, deflate, zstd` |
| Go net/http | `gzip` |

Versions run: Requests 2.34.2; httpx 0.28.1; aiohttp 3.14.3; Scrapy 2.19.0; built-in fetch (Node.js 22.23.3, 24.21.0, 26.10.0); Go 1.27.1 net/http.

- Requests: `brotli` adds br; on Python 3.13, `backports.zstd` adds zstd.
- httpx: `brotli` adds br and `zstandard` adds zstd.
- aiohttp: `Brotli` adds br; on Python 3.13, `backports.zstd` adds zstd.
- Built-in fetch over `http://`: `gzip, deflate`.
- Go: setting the header yourself turns decoding off.

## What it measures

- **Cells.** A cell is one client setup × one path × one server mode × one body. Paths: direct; `http://`
  through the proxy (absolute-form, so the proxy can read the request headers); `https://` through the
  proxy with `CONNECT` (an opaque TLS tunnel). Core cells: 193; pitfall cells: 196;
  corpus cells: 138; Node.js resource-timing cells: 18;
  certificate negative controls: 21; proxy-framing quirk: 1.
- **Server policy (`negotiate` mode).** This is the harness's own policy, not a claim about any real server. A
  missing or empty `Accept-Encoding` gets an uncompressed body, which is common practice (RFC 9110
  §12.5.3 actually says a missing header means any coding is acceptable). Otherwise the highest `q` wins,
  with ties going to `zstd, br, gzip, deflate`. Encoders: gzip level 6, brotli quality 5, zstd level 3.
- **Forced modes.** For the pitfall cells the server sends gzip, br, zstd or four concatenated zstd frames
  regardless of the request, the way some servers compress without being asked (`forced-identity`
  sends the body uncompressed).
- **Bodies.** `harness/fixture/fixture.html` is synthetic HTML, 100,129 B, SHA-256 `3183884a6113544e297fb83c165a2dc4d39806e338c33f45e6e12af8767fb1e9`
  (gzip 17,750 B, br 18,451 B, zstd 19,298 B). Its ratios are a **fixture ratio, not
  typical**. The corpus is 23 pages of the official CPython 3.14.7 HTML documentation
  (PSF License), downloaded from python.org by `./setup.sh`, checked against a pinned SHA-256 and not
  redistributed here; see `corpus-manifest.txt`. Its ratios describe **this corpus**, not the web.
- **Per cell** (see "Reading a record"): header received, `Content-Encoding` served, wire body bytes,
  proxy bytes in both directions, decoded size, whether the client decoded, the exact error text.
- **What Node.js fetch reports.** The 18 resource-timing cells read
  `performance.getEntriesByType('resource')` after each fetch and keep `encodedBodySize`,
  `decodedBodySize` and `transferSize` next to the origin's and the proxy's own byte counts
  (`summaries.resource_timing` in `results.json`). `harness/snippets/node-fetch-bytes.mjs` prints
  `encodedBodySize` for your own URL.
- **Certificate checks.** Every https cell trusts a throwaway local CA explicitly, per client. The
  21 negative controls repeat https requests without that CA; every client refused the certificate.
- **Verify snippets.** `harness/snippets/` holds the reader-facing snippets. `./run.sh snippets` runs them
  unchanged except that `https://example.com/` becomes a loopback URL, and checks each printed number
  against the origin log (16 runs recorded, all consistent).

## What it cannot tell you

- How any real site, CDN or provider behaves. Real servers choose their own codings, minimum sizes and
  content types; some never compress, some compress without being asked.
- What a provider bills. Providers count headers, TLS, CONNECT, retries and failed requests in their own
  ways. The proxy counters here are this proxy's client leg only.
- A typical compression ratio. The fixture and the corpus ratios are properties of those files.
- Anything about HTTP/2 or HTTP/3 (the origin offers HTTP/1.1 only), other operating systems, or client
  versions other than the pinned ones. Defaults change between releases; rerun before relying on a row.
- Browsers. They negotiate compression themselves and are not in this lab.

## Requirements

- **Everything:** bash, `curl` and `shasum` (or `sha256sum`), `openssl`, Python 3.14 (runs the origin,
  the proxy and the 3.14 cells) and Python 3.13 (the 3.13 cells). [uv](https://docs.astral.sh/uv/) is
  optional (without it setup falls back to `python -m venv` and pip).
- **curl and Wget cells:** `/usr/bin/curl` (macOS system curl), `/opt/homebrew/opt/curl/bin/curl`
  (Homebrew curl) and `wget` on `PATH`.
- **Node.js and Go cells:** `go` on `PATH`. Setup downloads Node.js 22.23.3, 24.21.0 and 26.10.0 from
  nodejs.org and checks each tarball against `SHASUMS256.txt` (and its signature when `gpgv` is present);
  npm packages come from the npm registry; Go fetches the go1.27.1 toolchain if yours is another version.
- **PHP and Java cells:** macOS on arm64 only (pinned builds): PHP 8.5.8 (static-php-cli), Temurin JDK
  25.0.4.1+1 and Composer 2.10.3, each checked against a pinned SHA-256, then `composer install` from the
  committed lock files.
- **Downloads during setup:** pinned packages from PyPI, one archive from python.org, the Node.js,
  PHP, JDK and Composer archives above, and the npm and Composer packages in the lock files.

The matrix itself talks only to 127.0.0.1. A part of setup that cannot be completed prints one line and
setup moves on; the cells that need it are recorded `not-run` with the reason, and the run exits 1 unless
you pass `--allow-missing`. `./run.sh` before `./setup.sh` stops with `run ./setup.sh first`.

## Run it

```sh
unzip accept-encoding-lab.zip
cd accept-encoding-lab/harness
./setup.sh             # CA, 21 venvs, corpus, Node.js/Go and PHP/Java runtimes, all under harness/.lab-state
./run.sh               # the whole matrix, every verify snippet and environment.txt into harness/out/
python3 ../compare.py ../results.json out/results.json
```

The published results came from exactly these two commands, run once from an empty state directory.
Useful variations: `./setup.sh --core` (only what the curl, Wget and Python cells need),
`./run.sh matrix --only 'curl-*' 'wget-*'` (a subset by cell id glob), `./run.sh matrix --cells
cells/php-java/*.json` (one cell family), `./run.sh list` (every cell id), `LAB_STATE_DIR=/some/dir` and
`LAB_OUT_DIR=/some/dir` (move state or output). Add `--ignore-not-run` to `compare.py` when your machine
lacks some runtimes.

Every file is also served on its own at `https://ipvolt.com/downloads/accept-encoding-defaults/` plus the path listed in `SHA256SUMS`. The zip
keeps the executable bits the scripts need; with loose files, run `chmod +x harness/*.sh` before setup:

```sh
curl -fsSO "https://ipvolt.com/downloads/accept-encoding-defaults/SHA256SUMS"
awk '{print $2}' SHA256SUMS | while read -r f; do
  mkdir -p "$(dirname "$f")" && curl -fsS -o "$f" "https://ipvolt.com/downloads/accept-encoding-defaults/$f"
done
shasum -a 256 -c SHA256SUMS && chmod +x harness/*.sh
```

## Compare with the published run

`compare.py` matches records by id and compares only the ids both files have, so a subset run works.
Behaviour fields (header sent, coding served, body bytes, decoded bytes, decode outcome, whether an error
was raised) must match or it exits 1. Detail fields (proxy byte counts, client versions, error text with
ports and paths masked) depend on your builds and TLS stack and are reported; `--strict` makes them fail
too. `--ignore-not-run` skips cells your machine could not run.

## Regenerate the tables

```sh
python3 render_tables.py results.json --out tables.md                  # every table, English
python3 render_tables.py results.json --section answer                 # just the two tables above
python3 render_tables.py results.json --section timing                 # what Node.js fetch reports
python3 render_tables.py harness/out/results.json --out my-tables.md   # the tables of your own run
python3 render_tables.py --emit-strings > strings.xx.json               # labels to translate
python3 render_tables.py results.json --strings strings.xx.json --parity strings.xx.json
```

Every number, header and version comes from `results.json`. Each row of the answer tables also carries
checks against the records (for example "curl --compressed decodes"); if a rerun breaks one, the script
names it and exits 1 instead of printing stale advice. `--parity` checks that a translated render carries
the same code spans and numbers as the English one.

Your own run's `harness/out/results.json` is the runner's output, not a published file like
`results.json`: the script computes the CONNECT pairs from its records, reads the snippet results from
`harness/out/snippets/`, lists that run's not-run cells as its coverage gaps and marks its anomalies
unreviewed. A not-run cell never backs a row; when a section needs cells your run did not execute, the
script names them and exits 1.

## Reading a record

| Field | Meaning |
|---|---|
| `id`, `group`, `client`, `variant` | cell identity (`group`: core, pitfall, corpus, resource-timing, tls-control, proxy-quirk) |
| `client_version`, `runtime`, `runtime_version`, `executable_path` | what ran |
| `scheme`, `via`, `mode`, `body` | `http`/`https`; `direct`, `absolute-form` or `CONNECT`; `negotiate` or `forced-*`; `fixture` or `corpus/...` |
| `request_header_set` | an `Accept-Encoding` the cell set by hand (null: the client's default) |
| `header_present`, `header_received` | whether the origin received the header, and its exact value |
| `content_encoding_served` | the coding the origin used (null: not compressed) |
| `identity_body_bytes`, `origin_wire_body_bytes` | body size before encoding and as sent |
| `proxy_bytes_up`, `proxy_bytes_down`, `proxy_connections` | client-side bytes at the proxy, including headers, TLS and CONNECT |
| `proxy_saw_header` | the `Accept-Encoding` recorded from absolute-form requests; inner CONNECT headers were not parsed |
| `decoded_bytes`, `auto_decoded`, `decode_check` | what your code received, and whether it equals the original body |
| `exception_text`, `exit_code` | the client's own error, verbatim |
| `timestamp_utc`, `elapsed_s`, `status`, `locator`, `anomalies`, `notes`, `extra` | when, how long, `run`/`source-read`/`not-run`, the cell definition, harness sanity flags, source notes and client-reported details (`extra.resource_timing` on the resource-timing cells) |

Paths in `results.json` use `$LAB_STATE_DIR` for the state directory of the recorded run. `results.csv`
has the same records without `notes`, `extra` and `elapsed_s`. The runner flagged
3 records; each is kept and explained under `reviewed_anomalies`.
Coverage gaps are listed under `not_run`.

## How the published results were produced

One run on 2026-09-27: `./setup.sh && ./run.sh` from `harness/`, with an empty state directory. The matrix ran
from 2026-09-27T02:44:37Z to 2026-09-27T02:45:53Z (UTC): 567 records from 10 cell files.
Clients: curl (macOS system 8.7.1, Homebrew 8.22.0), Wget 1.25.0, CPython 3.13.15 and 3.14.7 (urllib.request, http.client, Requests, httpx, aiohttp, Scrapy), Node.js 22.23.3, 24.21.0 and 26.10.0 (built-in fetch, node:http/https, axios, got, node-fetch), Go 1.27.1 net/http, PHP 8.5.8 ext-curl (libcurl 8.20.0), Guzzle 8.2.0 (cURL and stream handlers), Laravel Http (illuminate/http v13.33.0), Java 25 java.net.http.HttpClient (Temurin 25.0.4.1+1).

Harness tree SHA-256 `c586261f08f49693b7f32cd9a5efea1dc805d363d0308110b6238a45b60d2a77` (per-file hashes in `results.json`,
`harness_source`); the shipped `harness/` directory is that tree. Before publication the zip was unpacked
into an empty directory, set up and run again with the two commands above, and `compare.py --strict`
matched every record. `environment.txt` holds the OS, hardware and every version output (the host name
is written as `<hostname>`).

## Safety

- No credentials are used anywhere. The proxy needs none, binds 127.0.0.1 and refuses non-loopback targets.
- Certificate verification is never switched off. Each client trusts the lab CA explicitly
  (curl `--cacert`, Wget `--ca-certificate`, Python `ssl.create_default_context(cafile=...)` or
  `verify=`, Scrapy through `SSL_CERT_FILE` with `DOWNLOAD_VERIFY_CERTIFICATES` switched on (Scrapy 2.19
  does not verify certificates by default), Node.js `NODE_EXTRA_CA_CERTS`, Go `RootCAs`, PHP
  `CURLOPT_CAINFO` and Guzzle `'verify'`, Java an `SSLContext` built from the CA certificate).
- `make-ca.sh` creates the CA and a 127.0.0.1 server certificate (RSA 2048, valid 30 days) on your machine.
  Keys stay in `harness/.lab-state`. Delete `harness/.lab-state` and `harness/out` to undo everything;
  nothing is installed system-wide.

## Files

`SHA256SUMS` lists every file below. The zip's own checksum is published next to it as
`accept-encoding-lab.zip.sha256`.

```text
README.md
SHA256SUMS
compare.py
corpus-manifest.txt
environment.txt
render_tables.py
results.csv
results.json
tables.md
harness/fetch-corpus.sh
harness/make-ca.sh
harness/run.sh
harness/setup-node-go.sh
harness/setup-php-java.sh
harness/setup.sh
harness/venvs.json
harness/cells/core.json
harness/cells/corpus.json
harness/cells/pitfalls.json
harness/cells/tls-controls.json
harness/clients/node_fetch_example.mjs
harness/clients/pyclient.py
harness/fixture/fixture.html
harness/labcore/__init__.py
harness/labcore/bodies.py
harness/labcore/encodings.py
harness/labcore/envcapture.py
harness/labcore/fetch_corpus.py
harness/labcore/origin.py
harness/labcore/proxy.py
harness/labcore/redact.py
harness/labcore/runner.py
harness/labcore/setup_venvs.py
harness/labcore/snippets.py
harness/nodego/__init__.py
harness/nodego/envcapture.py
harness/nodego/snippets.py
harness/phpjava/__init__.py
harness/phpjava/envcapture.py
harness/snippets/curl-verify.sh
harness/snippets/go-verify.go
harness/snippets/httpx-verify.py
harness/snippets/node-fetch-bytes.mjs
harness/snippets/node-fetch-verify.mjs
harness/snippets/node-http-verify.mjs
harness/snippets/requests-verify.py
harness/tools/gen_fixture.go
harness/tools/gen_node_go_cells.py
harness/cells/examples/node-fetch.example.json
harness/cells/node-go/go.json
harness/cells/node-go/node.json
harness/cells/php-java/make_cells.py
harness/cells/php-java/php-java-core.json
harness/cells/php-java/php-java-pitfalls.json
harness/cells/php-java/php-java-proxy-quirk.json
harness/cells/php-java/php-java-tls-controls.json
harness/clients/go/goclient.go
harness/clients/node/load.mjs
harness/clients/node/nodeclient.mjs
harness/clients/node/package-lock.json
harness/clients/node/package.json
harness/clients/php-java/JavaHttpClientLab.java
harness/clients/php-java/guzzle_client.php
harness/clients/php-java/labcommon.php
harness/clients/php-java/laravel_http_client.php
harness/clients/php-java/php_curl_client.php
harness/clients/php-java/apps/guzzle/composer.json
harness/clients/php-java/apps/guzzle/composer.lock
harness/clients/php-java/apps/laravel/composer.json
harness/clients/php-java/apps/laravel/composer.lock
```