# Proxy retries: one lost response, two created jobs

Source: https://ipvolt.com/blog/proxy-retries-duplicate-jobs
Markdown: https://ipvolt.com/blog/proxy-retries-duplicate-jobs.md

[Home](https://ipvolt.com/index.md) / [Blog](https://ipvolt.com/blog.md) / Proxy retries: one lost response, two created jobs

Category: Analysis
Published: 2026-09-13
Updated: 2026-09-13
Author: ipvolt
Reading time: 8 minutes
Tags: proxies, troubleshooting

A local proxy drops a response after a POST creates a job. See when retrying duplicates work, when an idempotency key helps, and when to reconcile.

Before retrying a POST after a proxy failure, establish what the destination may already have done. Losing the response leaves the operation's outcome uncertain. A second request can create another job even though the first one already exists.

In our local demonstration, the client received an error after the origin created a job. Repeating the POST created a second job. Repeating it with the same key also created a second job when the receiving endpoint ignored the key. Only the endpoint's implemented deduplication contract made a same-key replay return the original job.

These are synthetic job records in memory, not completed production tasks. The useful result is the difference between requests attempted and application effects observed. You can reproduce both in the download.

## Lose the response after creating the job

Download and extract the [proxy retry demonstration](https://ipvolt.com/downloads/proxy-retry-jobs/proxy-retry-jobs.zip). From the extracted directory, run:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python retry_jobs_demo.py > result.json
python retry_jobs_demo.py --csv result.json
```

The example requires Python 3.11 or later and pins HTTPX 0.28.1 with HTTPcore 1.0.9. It uses [`asyncio.timeout`, added in Python 3.11](https://docs.python.org/3/library/asyncio-task.html#asyncio.timeout). Installing dependencies uses the package index. The demonstration itself starts a temporary HTTP origin and forward proxy on loopback; no provider account or external destination is involved.

In each of six separate cases, the origin applies the first POST and sends its response. The proxy reads that complete response, then deliberately closes the downstream connection without sending response headers. The client records `RemoteProtocolError` in this pinned run. Any repeat request or lookup is a subsequent, explicit client action.

The captured ordering is:

```text
origin: job applied
origin: response sent
proxy: complete origin response received
proxy: response dropped before client headers
client: request failed
```

The origin's own records produced this matrix. POST counts include the original attempt; a lookup is a separate GET.

| Action after the lost response | POSTs received | GETs received | Jobs created | Next client result |
|---|---:|---:|---:|---|
| Repeat the POST without a replay contract | 2 | 0 | 2 | 201, second job |
| Reuse a key that the endpoint ignores | 2 | 0 | 2 | 201, second job |
| Reuse the supported key and unchanged input | 2 | 0 | 1 | 201, original saved result |
| Generate a new key for the retry | 2 | 0 | 2 | 201, second job |
| Reuse the supported key with changed input | 2 | 0 | 1 | 409, input conflict |
| Look up the confirmed operation; send no second POST | 1 | 1 | 1 | 200, existing job found |

A 201 on the second attempt therefore meant two different things in this fixture: a newly created second job or the saved result for the first job. We checked job identities and origin effect counts to distinguish them. Idempotency concerns the intended effect of repeating an operation; the status code alone cannot demonstrate it. [HTTP idempotency semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2).

The important boundary is between applying the operation and delivering its response. This is different from a connection attempt that failed before the application request was sent. For an HTTPS destination, a failure while establishing the CONNECT tunnel also happens at a different stage from losing the response to a POST sent through an established tunnel. The [timeout guide](https://ipvolt.com/guides/proxy-timeout-troubleshooting) explains those stages.

The proxy in this demonstration does not retry requests. A second POST is an explicit action by the client. That distinction matters: HTTP semantics prohibit a proxy from automatically retrying a non-idempotent request. A client also needs a basis for treating a non-idempotent operation as repeatable, or evidence that the original was not applied. [RFC 9110, section 9.2.2](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2).

## Choose the next action from the operation's state

An exception class describes a client observation. It is not an application receipt. Use evidence about the logical operation, meaning the job the caller intended to create, to decide what happens next.

| Evidence available | What you know | Next action |
|---|---|---|
| An authoritative result identifies the created job | The creation took effect | Use or reconcile that result; do not create the job again |
| Outcome is unknown, but the endpoint supports replay with your retained operation key and unchanged input | Its documented contract may let you repeat the attempt without repeating the effect | Reuse that identity within the endpoint's scope and retention rules |
| Reliable evidence establishes that no attempt applied or can still apply | The logical operation has not taken effect | A new attempt can be considered within the normal request budget |
| Outcome is unknown and no applicable replay contract exists | Repeating the POST could create a second effect | Use the service's status/reconciliation process or escalate the unresolved operation |

For the lookup case, the caller retained `X-Operation-Ref: operation-1`; the fixture's `GET /operations/operation-1` returns the associated job. That reference was known before the POST, so the lookup does not need a job ID from the lost response.

Treat an empty or failed lookup carefully. “Not found” at one instant does not establish that an earlier request cannot arrive or complete later. The positive lookup in the demonstration works because it returns the job already created. It does not test negative lookup safety. This distinction follows from the late-arrival problem discussed in the [AWS Builders Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/).

A backoff delay changes when you try again. It does not establish whether the first operation happened. A 502, 504, timeout or disconnected response still needs its request phase and application context. Use the [proxy status-code article](https://ipvolt.com/blog/proxy-status-codes-407-429-502) to identify the responding layer; avoid turning its error categories into a universal list of safe POST retries.

## A header needs a receiving application that understands it

The supported-key fixture records the job and its response under an in-process lock. Within its single-caller scope, the key is bound to the POST endpoint and compared against the operation reference and parsed payload. A matching repeat receives the saved job result. A changed input receives the fixture's explicit 409 response without creating another job.

The ignored-key case changes the receiving application's behavior, not the spelling of the header. The new-key case changes the operation identity presented to the application. Both created two jobs. Reuse the identity of the intended operation when its contract permits a replay; generating a new key on every attempt defeats that connection.

This fixture retains records only in memory for its lifetime. Its lock demonstrates coordination inside one process. It does not make job creation durable, cover a restart, expire keys safely, or coordinate external effects such as sending a message. Those properties need an application design beyond this example.

Real APIs define their own contracts. For example, Stripe documents storing a request's result for reuse with the same key, checking subsequent parameters, and treating a reused key as a new request after its stored record is removed. Those are Stripe's rules; sending a similarly named header to another endpoint does not import them. [Stripe idempotent requests](https://docs.stripe.com/api/idempotent_requests).

Before relying on a key in production, verify four things with the receiving service:

- **Identity and scope:** which caller, account, endpoint and operation does the key identify?
- **Input and concurrency:** how are changed inputs and overlapping attempts handled?
- **Retention and recovery:** how long does protection last, and what survives a restart or partial failure?
- **Result semantics:** what is returned for an existing, pending, failed or completed operation?

AWS's design discussion connects request identity with atomic recording of the mutation and describes why late arrivals and changed intent need explicit handling. It is useful background for these questions, not evidence that every API implements the same guarantees. [Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/).

## Keep client retries and business outcomes separate

Do not use the client's successful-response count as the job count. Record the logical operation identity separately from each transport attempt. When the response is lost, preserve an unresolved outcome until the service's contract or authoritative state supports a stronger conclusion. That makes a recovered response, a replayed result and a newly created job distinguishable in your investigation.

Be precise about library settings too. HTTPX documents its built-in transport retries for connection errors and connection timeouts, not arbitrary read/write failures or status codes. In the pinned HTTPX 0.28.1 implementation, the HTTP-proxy branch does not pass the transport's `retries` setting into its proxy pool. This experiment uses explicit client actions and zero default retries; it does not depend on that option replaying a proxied POST. [HTTPX transport documentation](https://www.python-httpx.org/advanced/transports/#http-transport), [tagged transport source](https://raw.githubusercontent.com/encode/httpx/0.28.1/httpx/_transports/default.py).

For monitoring, count attempts, observed response failures, confirmed jobs and unresolved logical operations separately. The [benchmark methodology article](https://ipvolt.com/blog/what-a-proxy-benchmark-should-measure) explains why an attempt denominator matters. For a state-changing integration, add the application outcome before calling recovery successful.

## Downloads and test boundaries

The [archive](https://ipvolt.com/downloads/proxy-retry-jobs/proxy-retry-jobs.zip) contains the complete demonstration, tests, pins, README, recorded JSON and its derived CSV. You can also inspect the [code](https://ipvolt.com/downloads/proxy-retry-jobs/retry_jobs_demo.py), [tests](https://ipvolt.com/downloads/proxy-retry-jobs/test_retry_jobs.py), [requirements](https://ipvolt.com/downloads/proxy-retry-jobs/requirements.txt), [README](https://ipvolt.com/downloads/proxy-retry-jobs/README.md), [event output](https://ipvolt.com/downloads/proxy-retry-jobs/example-output.json) and [result matrix](https://ipvolt.com/downloads/proxy-retry-jobs/example-matrix.csv) separately.

ipvolt ran the local checks on 13 September 2026 with CPython 3.14.7 on macOS arm64 and the pinned packages above. The checks cover the six-case matrix, overlapping same-key requests, changed-input conflicts, distinct identities, refusal to forward to an external destination and fixture cleanup. Recorded cases finished with the client closed and no remaining fixture handlers or writers.

The test uses HTTP/1.1 forwarding. It does not test CONNECT, TLS, authentication, SOCKS, external providers, durable storage, key expiry or a remote service's late-arrival behavior. It makes no provider reliability or exactly-once processing claim.

If you want to hear when ipvolt access opens, [join the early-access list](https://ipvolt.com/#waitlist-closing). One email when access opens. Nothing else. This demonstration does not describe an available ipvolt API or an ipvolt idempotency feature.

## Sources

- [RFC 9110: idempotency and retry semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2)
- [AWS Builders Library: making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
- [Stripe: idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [HTTPX: transport retries](https://www.python-httpx.org/advanced/transports/#http-transport)
- [HTTPX 0.28.1: transport implementation](https://raw.githubusercontent.com/encode/httpx/0.28.1/httpx/_transports/default.py)
- [Python: asyncio.timeout](https://docs.python.org/3/library/asyncio-task.html#asyncio.timeout)

## Know when ipvolt access opens.

ipvolt is in development. Leave your email and we’ll notify you once when access opens.

Consent: One email when access opens. Nothing else.

[Notify me](https://ipvolt.com/blog/proxy-retries-duplicate-jobs#waitlist-blog-end). Use the email form on this page to join the interest list.

[Privacy](https://ipvolt.com/privacy)

## Related posts

- [Amazon listing updates: accepted is not live](https://ipvolt.com/blog/amazon-listing-update-reconciliation.md) (Analysis, Sep 14, 2026, 8 min read): Diagnose accepted Amazon listing updates by separating submitted attributes, live offers, inventory and buyability, with a practical reconciliation matrix.
- [Bookmaker odds feeds: validate before comparing](https://ipvolt.com/blog/bookmaker-odds-feed-validation.md) (Analysis, Sep 14, 2026, 7 min read): Compare bookmaker odds only after checking market identity, settlement rules, timestamps and status. Use a local validator to expose false comparisons.
- [Cloudflare AI Agent Blocks: What to Fix First](https://ipvolt.com/blog/cloudflare-ai-agent-proxy-setup.md) (Analysis, Sep 14, 2026, 9 min read): Diagnose Cloudflare failures in AI-agent jobs, configure a stable browser proxy, and reject challenges or invalid content before RAG ingestion.

## Related guides

- [Troubleshoot proxy timeouts one stage at a time](https://ipvolt.com/guides/proxy-timeout-troubleshooting.md): Separate proxy DNS, TCP, CONNECT, TLS and response delays with curl timings, then set request deadlines and decide whether a retry is safe.
- [Configure a proxy in Python Requests](https://ipvolt.com/guides/python-requests-proxy.md): Make a Python Requests proxy configuration explicit, encode credentials correctly, and distinguish connect and read timeouts from a total job deadline.
- [Use a proxy with curl](https://ipvolt.com/guides/curl-proxy-setup.md): Test an HTTP proxy with curl, separate proxy authentication from destination authentication, and read connection failures without exposing credentials.

## About ipvolt

Technical analysis from the ipvolt team.

ipvolt access is not open yet.