# Lost POST responses: count requests and application effects separately

This executable lab creates synthetic job records at a temporary local
origin. A local HTTP forward proxy receives the complete first POST response,
then drops it before sending any response headers to the client. Every
subsequent POST is an explicit action in the example. The proxy never retries.
No real job is executed, and no provider account or external target is used.

Requires Python 3.11+ (`asyncio.timeout` is used). The recorded run used
CPython 3.14.7 on macOS arm64, HTTPX 0.28.1 and HTTPcore 1.0.9. The full
tested dependency set is pinned in `requirements.txt`.

## Reproduce the download

On macOS or Linux, after downloading `proxy-retry-jobs.zip`:

```sh
unzip proxy-retry-jobs.zip
cd proxy-retry-jobs
python3 -m venv .venv
.venv/bin/python -m pip install --requirement requirements.txt
.venv/bin/python retry_jobs_demo.py > result.json
.venv/bin/python retry_jobs_demo.py --csv result.json
.venv/bin/python -W error::ResourceWarning -m unittest -v test_retry_jobs.py
```

Package installation accesses the package index. The demo and tests connect
only to temporary `127.0.0.1` sockets. CSV conversion reads the already
generated JSON and makes no requests. The demo exits 0 only after verifying
its scenario matrix, response-loss ordering and cleanup; a nonzero exit
means the experiment failed. `example-output.json` is safe raw output from
the recorded run; `example-matrix.csv` was derived directly from that JSON.

## Read the result

Each row is an independent fixture with fresh state. Counts come from
origin request receipts and its job-record list, independently of client
success responses. The proxy also records its own POST/GET receipts.

| Case | Explicit action after the lost response | Origin POSTs | Origin GETs | Job records | Next HTTP status |
|---|---|---:|---:|---:|---:|
| `blind_retry` | Repeat the POST without a key contract | 2 | 0 | 2 | 201 |
| `ignored_stable_key` | Repeat with a key the endpoint ignores | 2 | 0 | 2 | 201 |
| `supported_stable_key` | Repeat the same key and input under the fixture contract | 2 | 0 | 1 | 201 |
| `new_key_on_retry` | Change the key for the same intended record | 2 | 0 | 2 | 201 |
| `changed_payload` | Reuse the supported key with changed input | 2 | 0 | 1 | 409 |
| `reconcile_confirmed_effect` | Confirm the record through a lookup; do not send another POST | 1 | 1 | 1 | 200 |

In all six recorded cases, the first client error was `RemoteProtocolError`.
The ordered events show the origin creating a record and sending its response,
the proxy receiving that entire response, the proxy dropping it, and only
then the client reporting failure. This is one observed failure phase under
this pinned stack; other forms of response loss can produce different errors.

The supported-key replay returns status 201 and exactly the saved result
body, including the original record ID. The 409 conflict response is this
fixture's chosen contract. Real APIs define their own status and replay rules.
The ignored-key case logs the same key arriving at the origin twice and
still creates two records. A client-supplied header alone is insufficient.

The final case uses a separate stable `X-Operation-Ref` to look up the
original record after the dropped response. It does not need the ID from
that lost response. It demonstrates an authoritative **positive** lookup
only. A negative lookup does not prove that an earlier request cannot apply
later. When a real operation remains uncertain, these local results do not
authorize a blind repeat.

## What the supported-key fixture promises

The teaching contract has one caller and one origin process. Its key scope
is `(POST, path, Idempotency-Key)`. It compares the parsed JSON payload and
the operation reference; JSON object key order does not change the match.
Matching input returns the saved result, while different input returns 409.
Different keys can intentionally create distinct records with identical
payloads. Content equality alone is not an operation identity.

An `asyncio.Lock` encloses the saved-result check, the synthetic record
creation and result storage. The concurrency tests hold the first request
inside that critical section until a second POST has reached the origin.
They verify one record for the same key/input, two records for distinct
operation keys, and a conflict without a second effect for changed input.
They also check that a changed operation reference conflicts and that the
original winning input can still replay its saved result.

This is an atomic in-process example under normal task execution. It is not
a durable or distributed exactly-once implementation. State is memory-only;
there is no crash recovery, persistence, retention expiry, account scoping,
external side effect, transactional database or multi-worker coordination.
Do not copy this in-memory dictionary into production as a complete
deduplication design. A real contract must define those boundaries.

## Checks and transport limits

The five tests cover the entire six-case matrix and event ordering, receipt
counts, body/result equality, same-key concurrency, independent operation
keys, concurrent payload conflicts, reference mismatch, external-destination
refusal and cleanup. Normal cases require no forced handler cancellation,
no open fixture writers/handlers/listeners, a closed HTTPX client and no
new pending asyncio tasks. Readiness uses events and socket completion,
not timing sleeps; client, fixture and total-demo deadlines bound failures.

This small proxy handles HTTP/1.1 GET/POST with Content-Length framing and
one request per connection. It can reach only its own fixed loopback origin.
The client uses explicit proxy routing with `trust_env=False`, no redirects
and no configured automatic retries. HTTPS CONNECT, TLS, proxy authentication,
HTTP/2, SOCKS, remote DNS, production failures and provider behavior are not
tested. No performance or service reliability result can be inferred.

For the underlying protocol distinction, see
[RFC 9110, idempotent methods and automatic retry conditions](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2).
The lab's key and lookup behavior is explicitly defined above, rather than
attributed to HTTP itself or to an arbitrary API.