# ETag monitoring: a no-cache header changed our results

Source: https://ipvolt.com/blog/etag-monitoring-cache-control
Markdown: https://ipvolt.com/blog/etag-monitoring-cache-control.md

[Home](https://ipvolt.com/index.md) / [Blog](https://ipvolt.com/blog.md) / ETag monitoring: a no-cache header changed our results

Category: Analysis
Published: 2026-09-11
Updated: 2026-09-11
Author: ipvolt team
Reading time: 7 minutes
Tags: proxies, troubleshooting

A live proxy test found that one request header changed 304 responses into full downloads. Inspect the measured bytes, request policies and reproducible data.

Before buying more proxy traffic for a content monitor, inspect the headers your client actually sends. On our live website, adding `Cache-Control: no-cache` changed unchanged-page checks from body-free `304` responses to full `200` downloads, even though the requests carried the matching ETag.

In a controlled comparison, **12 conditional requests without that override transferred zero response-body bytes; 12 with it transferred 271,968 bytes of unchanged HTML.** The six initial downloads needed to obtain the ETags cost another 135,984 bytes and are included in the published records.

That is a finding about one deployed Next.js 16.3.4 site and a particular request policy. It is not advice to remove freshness requirements from a monitor. The useful decision is whether to change the client policy, choose a smaller representation, or investigate the server's revalidation behavior.

## The task: monitor known content without downloading it unnecessarily

We tested two existing ipvolt articles: the [proxy benchmark guide](https://ipvolt.com/blog/what-a-proxy-benchmark-should-measure) and [proxy error explanation](https://ipvolt.com/blog/proxy-status-codes-407-429-502), each as HTML and Markdown. Our monitor watched the article title and two specified section headings. It also checked media types and, for HTML, the page title and canonical URL.

Those checks establish that the watched information is present. They do not make Markdown a substitute for layout, JavaScript behavior or a working signup journey. The exact assertions are in the [study configuration](https://ipvolt.com/downloads/content-monitoring/main-study.json).

Collection used authenticated third-party proxy connections targeting the US, UK and Netherlands. The initial experiment also included a direct control. Country-check responses reported the requested countries at the checkpoints; an exit address changed during the initial US run, so these are route observations rather than fixed-IP or country-wide reliability results. The connection provider is omitted; this was not a test of an available ipvolt proxy service.

All three HTTP experiments ran on 11 September 2026. They made **234 requests and transferred 2,975,066 response-body bytes**, including initial downloads, verification downloads and unsuccessful hypotheses. Every attempted request completed without a curl transport error. Additional geographic probes and browser QA are recorded separately in the [method note](https://ipvolt.com/downloads/content-monitoring/method.json); they are outside that HTTP-experiment total.

## The smaller first download did not answer the caching question

The initial experiment requested gzip consistently and measured received body bytes. Across twelve initial observations per representation, the sizes were:

| Article representation | Received body bytes per initial request |
| --- | ---: |
| Benchmark HTML | 26,521 |
| Benchmark Markdown | 7,056–7,061 |
| Error guide HTML | 18,807 |
| Error guide Markdown | 5,081 |

Both HTML endpoints supplied ETags. Neither Markdown endpoint supplied an ETag or Last-Modified validator in this experiment. HTML conditional checks without a Cache-Control override returned `304`; the runner reused the matching, previously accepted body instead of interpreting the empty response as a new document.

The exploratory run made 144 requests: 120 full `200` responses and 24 accepted `304` responses. It skipped 36 conditional slots because the Markdown had no validator or the homepage was marked `no-store`. Skipped requests are not counted as successes or zero-byte network observations.

We then ran five successive checks per representation through each of the three proxy routes. Each check explicitly sent `Cache-Control: no-cache`, and the client retained eligible validators between checks. All 60 requests returned `200`, including the 24 HTML rechecks carrying a matching ETag. The expectation that repeated HTML validation would avoid downloading the body did not hold under this policy.

## Isolating the header that changed the response

To investigate, we held the URL, route configuration, Accept, language, gzip preference and exact seed ETag constant. For each of the six route/article combinations, we made one seed request and four conditional requests. The order was A/B/B/A or B/A/A/B, where A omitted a Cache-Control override and B sent `no-cache`.

| Conditional request policy | Attempts | Responses | Received body bytes |
| --- | ---: | --- | ---: |
| Matching If-None-Match; no Cache-Control override | 12 | 12 × 304 | 0 |
| Same If-None-Match; Cache-Control: no-cache | 12 | 12 × 200 | 271,968 |

All 24 conditional observations matched the seed's watched values and decoded content. Including the six seeds, this controlled experiment transferred 407,952 body bytes across 30 requests. It used balanced ordering, not random sampling, and establishes the observed behavior of these endpoints during this short window.

The implementation explains a plausible mechanism. We checked the actual deployed dependency files against the local Next.js 16.3.4 installation: the relevant source hashes matched. Its ETag response helper calls `fresh` before returning `304`; the bundled freshness check returns false when the request contains `no-cache`, before evaluating the matching ETag. A separate local function test reproduced the same branch. [Next.js response helper](https://github.com/vercel/next.js/blob/v16.3.4/packages/next/src/server/send-payload.ts), [fresh implementation](https://github.com/jshttp/fresh/blob/v0.5.2/index.js).

This is stack-specific behavior consistent with the live results. HTTP does not define `no-cache` as a universal instruction to download a complete body. Nor does receiving `304` prove that the request reached the origin rather than a responding cache. [HTTP caching](https://www.rfc-editor.org/rfc/rfc9111.html).

## The useful optimization depends on the freshness you need

For the five-check experiment that explicitly used `no-cache`, the actual totals were:

| Representation | Requests, including initial downloads | Received body bytes |
| --- | ---: | ---: |
| HTML | 30 | 679,920 |
| Markdown | 30 | 182,070 |
| Difference | Same request count | 497,850 |

**Markdown transferred 73.2% fewer response-body bytes for the watched content under that policy.** It did not reduce the request count. These are measured payload differences, not a measured invoice saving or a forecast of monthly traffic. curl's download-size metric excludes headers and other network overhead. [curl payload accounting](https://curl.se/libcurl/c/CURLINFO_SIZE_DOWNLOAD_T.html).

There is another option when freshness requirements allow it: the Markdown responses advertised `public, max-age=300, must-revalidate`. A cache may reuse a still-fresh stored representation without making a request. Missing validators do not make it uncacheable. Our runners deliberately checked over the network; they did not implement that freshness-based scheduling policy. [Freshness and reuse](https://www.rfc-editor.org/rfc/rfc9111.html#section-4).

Use the result to choose the next action:

| Your monitoring requirement | Action supported by this case |
| --- | --- |
| Reuse within the server's freshness window is acceptable | Evaluate a local freshness-aware cache before scheduling another download. |
| Ask the responding server or cache to validate every check | Test the final request headers against the actual endpoint; do not assume an ETag guarantees a body-free response. |
| Need the watched text, and each check returns a full body | Compare compressed HTML with a text or Markdown representation that contains those same facts. |
| Need rendered layout or an interactive journey | Keep browser checks; the smaller text representation does not cover that job. |

Do not strip `no-cache` solely to reproduce the zero-byte row. The two header policies can have different freshness consequences. Establish what your monitor must detect, then measure the permitted implementation.

## Reproduce the result and adapt the check

Download [the offline analysis script](https://ipvolt.com/downloads/content-monitoring/reproduce.py) and [the sanitized recordings](https://ipvolt.com/downloads/content-monitoring/recordings.zip) into the same directory. Python 3.10 or newer is sufficient; this command makes no network requests:

```sh
python3 reproduce.py recordings.zip
```

It recomputes the attempt counts, body totals, header comparison and Markdown percentage. Compare its output with [the recorded result](https://ipvolt.com/downloads/content-monitoring/results.json). The dataset includes timestamps, request policies, selected response headers, content checks and hashes. Credentials, exit IPs, unfiltered headers and raw body captures remain private; replay verifies the published arithmetic rather than independently authenticating historical responses.

The [download instructions](https://ipvolt.com/downloads/content-monitoring/README.md) link the exact collection programs, configurations and 27 passing offline tests. The live programs require curl 8.4 or newer, preserve TLS verification, bound requests and body transfer, and pass proxy authentication privately through curl's input. Their allowlist is restricted to this case's public site; adapt both the allowlist and content assertions for a property you operate.

We did not observe a real content update or measure detection delay. Decoded content stayed stable for every tested endpoint; a few gzip byte streams differed despite identical decoded text. That is a reason to compare the content your monitor cares about, not evidence that this study eliminated operational false alerts. This is one site, two articles and a short observation window—not a provider ranking or a universal caching benchmark.

*Method: agent-assisted collection and analysis, with separate evidence, editorial and search reviews. All reported network observations were actually executed; the offline tests exercise controlled fixtures and are labeled separately.*

[Notify me when access opens](https://ipvolt.com/blog/etag-monitoring-cache-control#waitlist-blog-end). One email when access opens. Nothing else.

## Sources

- [RFC 9110: If-None-Match and conditional requests](https://www.rfc-editor.org/rfc/rfc9110.html#section-13.1.2)
- [RFC 9111: request and response cache directives](https://www.rfc-editor.org/rfc/rfc9111.html)
- [curl: downloaded response-body size](https://curl.se/libcurl/c/CURLINFO_SIZE_DOWNLOAD_T.html)
- [Next.js 16.3.4: ETag response handling](https://github.com/vercel/next.js/blob/v16.3.4/packages/next/src/server/send-payload.ts)
- [fresh 0.5.2: request freshness evaluation](https://github.com/jshttp/fresh/blob/v0.5.2/index.js)

## Know when ipvolt access opens.

ipvolt is in development. Leave your email and we’ll notify you once when access opens.

Consent: One email when access opens. Nothing else.

[Notify me](https://ipvolt.com/blog/etag-monitoring-cache-control#waitlist-blog-end). Use the email form on this page to join the interest list.

[Privacy](https://ipvolt.com/privacy)

## Related posts

- [Amazon listing updates: accepted is not live](https://ipvolt.com/blog/amazon-listing-update-reconciliation.md) (Analysis, Sep 14, 2026, 8 min read): Diagnose accepted Amazon listing updates by separating submitted attributes, live offers, inventory and buyability, with a practical reconciliation matrix.
- [Bookmaker odds feeds: validate before comparing](https://ipvolt.com/blog/bookmaker-odds-feed-validation.md) (Analysis, Sep 14, 2026, 7 min read): Compare bookmaker odds only after checking market identity, settlement rules, timestamps and status. Use a local validator to expose false comparisons.
- [Cloudflare AI Agent Blocks: What to Fix First](https://ipvolt.com/blog/cloudflare-ai-agent-proxy-setup.md) (Analysis, Sep 14, 2026, 9 min read): Diagnose Cloudflare failures in AI-agent jobs, configure a stable browser proxy, and reject challenges or invalid content before RAG ingestion.

## Related guides

- [Use a proxy with curl](https://ipvolt.com/guides/curl-proxy-setup.md): Test an HTTP proxy with curl, separate proxy authentication from destination authentication, and read connection failures without exposing credentials.
- [Configure a proxy in Python Requests](https://ipvolt.com/guides/python-requests-proxy.md): Make a Python Requests proxy configuration explicit, encode credentials correctly, and distinguish connect and read timeouts from a total job deadline.
- [Troubleshoot proxy timeouts one stage at a time](https://ipvolt.com/guides/proxy-timeout-troubleshooting.md): Separate proxy DNS, TCP, CONNECT, TLS and response delays with curl timings, then set request deadlines and decide whether a retry is safe.

## About ipvolt

Technical analysis from the ipvolt team.

ipvolt access is not open yet.