Before retrying a POST after a proxy failure, establish what the destination may already have done. Losing the response leaves the operation's outcome uncertain. A second request can create another job even though the first one already exists.
In our local demonstration, the client received an error after the origin created a job. Repeating the POST created a second job. Repeating it with the same key also created a second job when the receiving endpoint ignored the key. Only the endpoint's implemented deduplication contract made a same-key replay return the original job.
These are synthetic job records in memory, not completed production tasks. The useful result is the difference between requests attempted and application effects observed. You can reproduce both in the download.
Lose the response after creating the job
Download and extract the proxy retry demonstration. From the extracted directory, run:
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python retry_jobs_demo.py > result.json
python retry_jobs_demo.py --csv result.jsonThe example requires Python 3.11 or later and pins HTTPX 0.28.1 with HTTPcore 1.0.9. It uses asyncio.timeout, added in Python 3.11. Installing dependencies uses the package index. The demonstration itself starts a temporary HTTP origin and forward proxy on loopback; no provider account or external destination is involved.
In each of six separate cases, the origin applies the first POST and sends its response. The proxy reads that complete response, then deliberately closes the downstream connection without sending response headers. The client records RemoteProtocolError in this pinned run. Any repeat request or lookup is a subsequent, explicit client action.
The captured ordering is:
origin: job applied
origin: response sent
proxy: complete origin response received
proxy: response dropped before client headers
client: request failedThe origin's own records produced this matrix. POST counts include the original attempt; a lookup is a separate GET.
| Action after the lost response | POSTs received | GETs received | Jobs created | Next client result |
|---|---|---|---|---|
| Repeat the POST without a replay contract | 2 | 0 | 2 | 201, second job |
| Reuse a key that the endpoint ignores | 2 | 0 | 2 | 201, second job |
| Reuse the supported key and unchanged input | 2 | 0 | 1 | 201, original saved result |
| Generate a new key for the retry | 2 | 0 | 2 | 201, second job |
| Reuse the supported key with changed input | 2 | 0 | 1 | 409, input conflict |
| Look up the confirmed operation; send no second POST | 1 | 1 | 1 | 200, existing job found |
A 201 on the second attempt therefore meant two different things in this fixture: a newly created second job or the saved result for the first job. We checked job identities and origin effect counts to distinguish them. Idempotency concerns the intended effect of repeating an operation; the status code alone cannot demonstrate it. HTTP idempotency semantics.
The important boundary is between applying the operation and delivering its response. This is different from a connection attempt that failed before the application request was sent. For an HTTPS destination, a failure while establishing the CONNECT tunnel also happens at a different stage from losing the response to a POST sent through an established tunnel. The timeout guide explains those stages.
The proxy in this demonstration does not retry requests. A second POST is an explicit action by the client. That distinction matters: HTTP semantics prohibit a proxy from automatically retrying a non-idempotent request. A client also needs a basis for treating a non-idempotent operation as repeatable, or evidence that the original was not applied. RFC 9110, section 9.2.2.
Choose the next action from the operation's state
An exception class describes a client observation. It is not an application receipt. Use evidence about the logical operation, meaning the job the caller intended to create, to decide what happens next.
| Evidence available | What you know | Next action |
|---|---|---|
| An authoritative result identifies the created job | The creation took effect | Use or reconcile that result; do not create the job again |
| Outcome is unknown, but the endpoint supports replay with your retained operation key and unchanged input | Its documented contract may let you repeat the attempt without repeating the effect | Reuse that identity within the endpoint's scope and retention rules |
| Reliable evidence establishes that no attempt applied or can still apply | The logical operation has not taken effect | A new attempt can be considered within the normal request budget |
| Outcome is unknown and no applicable replay contract exists | Repeating the POST could create a second effect | Use the service's status/reconciliation process or escalate the unresolved operation |
For the lookup case, the caller retained X-Operation-Ref: operation-1; the fixture's GET /operations/operation-1 returns the associated job. That reference was known before the POST, so the lookup does not need a job ID from the lost response.
Treat an empty or failed lookup carefully. “Not found” at one instant does not establish that an earlier request cannot arrive or complete later. The positive lookup in the demonstration works because it returns the job already created. It does not test negative lookup safety. This distinction follows from the late-arrival problem discussed in the AWS Builders Library.
A backoff delay changes when you try again. It does not establish whether the first operation happened. A 502, 504, timeout or disconnected response still needs its request phase and application context. Use the proxy status-code article to identify the responding layer; avoid turning its error categories into a universal list of safe POST retries.
A header needs a receiving application that understands it
The supported-key fixture records the job and its response under an in-process lock. Within its single-caller scope, the key is bound to the POST endpoint and compared against the operation reference and parsed payload. A matching repeat receives the saved job result. A changed input receives the fixture's explicit 409 response without creating another job.
The ignored-key case changes the receiving application's behavior, not the spelling of the header. The new-key case changes the operation identity presented to the application. Both created two jobs. Reuse the identity of the intended operation when its contract permits a replay; generating a new key on every attempt defeats that connection.
This fixture retains records only in memory for its lifetime. Its lock demonstrates coordination inside one process. It does not make job creation durable, cover a restart, expire keys safely, or coordinate external effects such as sending a message. Those properties need an application design beyond this example.
Real APIs define their own contracts. For example, Stripe documents storing a request's result for reuse with the same key, checking subsequent parameters, and treating a reused key as a new request after its stored record is removed. Those are Stripe's rules; sending a similarly named header to another endpoint does not import them. Stripe idempotent requests.
Before relying on a key in production, verify four things with the receiving service:
- Identity and scope: which caller, account, endpoint and operation does the key identify?
- Input and concurrency: how are changed inputs and overlapping attempts handled?
- Retention and recovery: how long does protection last, and what survives a restart or partial failure?
- Result semantics: what is returned for an existing, pending, failed or completed operation?
AWS's design discussion connects request identity with atomic recording of the mutation and describes why late arrivals and changed intent need explicit handling. It is useful background for these questions, not evidence that every API implements the same guarantees. Making retries safe with idempotent APIs.
Keep client retries and business outcomes separate
Do not use the client's successful-response count as the job count. Record the logical operation identity separately from each transport attempt. When the response is lost, preserve an unresolved outcome until the service's contract or authoritative state supports a stronger conclusion. That makes a recovered response, a replayed result and a newly created job distinguishable in your investigation.
Be precise about library settings too. HTTPX documents its built-in transport retries for connection errors and connection timeouts, not arbitrary read/write failures or status codes. In the pinned HTTPX 0.28.1 implementation, the HTTP-proxy branch does not pass the transport's retries setting into its proxy pool. This experiment uses explicit client actions and zero default retries; it does not depend on that option replaying a proxied POST. HTTPX transport documentation, tagged transport source.
For monitoring, count attempts, observed response failures, confirmed jobs and unresolved logical operations separately. The benchmark methodology article explains why an attempt denominator matters. For a state-changing integration, add the application outcome before calling recovery successful.
Downloads and test boundaries
The archive contains the complete demonstration, tests, pins, README, recorded JSON and its derived CSV. You can also inspect the code, tests, requirements, README, event output and result matrix separately.
ipvolt ran the local checks on 13 September 2026 with CPython 3.14.7 on macOS arm64 and the pinned packages above. The checks cover the six-case matrix, overlapping same-key requests, changed-input conflicts, distinct identities, refusal to forward to an external destination and fixture cleanup. Recorded cases finished with the client closed and no remaining fixture handlers or writers.
The test uses HTTP/1.1 forwarding. It does not test CONNECT, TLS, authentication, SOCKS, external providers, durable storage, key expiry or a remote service's late-arrival behavior. It makes no provider reliability or exactly-once processing claim.
If you want to hear when ipvolt access opens, join the early-access list. One email when access opens. Nothing else. This demonstration does not describe an available ipvolt API or an ipvolt idempotency feature.