# Durable listing-notification lab

Run a local SQLite inbox and single-worker reconciler using synthetic inputs.
The demo performs no network requests and no queue acknowledgements.
Requires Python 3.11+ and its standard library; no package installation.

From this directory:

```sh
python3 -B demo.py
python3 -B -m unittest -v test_consumer.py
```

Both commands use disposable temporary databases. The demo's JSON is deterministic;
compare it with `evidence/demo.stdout.json`. The recorded runtime and exact file
hashes are in `evidence/metadata.json`. Python 3.11 compatibility is a code target;
the runtime versions actually exercised are listed in that evidence.

The trace shows a hard process exit immediately after a committed notification,
redelivery after restart, an older event, a notification arriving during a read,
a failed read, a wrong-key result, and a conflicting duplicate identity.
It ends with five inbox records and three independent clean listing keys. The
primary key's generations progress from 1 to 3; its generation-2 completion is
rejected after generation 3 arrives. All listing states and reads are invented
fixtures, not observations of Amazon.

## Exact normalized input

`Store.ingest()` takes **one JSON object as a string**, using exactly these fields.
The complete example is `initial_event` in `fixtures/scenario.json`.

| Field | Lab contract |
| --- | --- |
| `schema_version` | Integer `1` (not a Boolean). |
| `source` | Stable, validated source namespace. |
| `subscription` | Stable subscription namespace within that source. |
| `notification_id` | Stable notification identity across repeated delivery. |
| `seller_id`, `sku`, `marketplace_id` | Explicit validated target scope; all three required. |
| `notification_type` | `listing_status_changed` or `listing_issues_changed`. |
| `event_time` | Valid original event time, `YYYY-MM-DDTHH:MM:SSZ`. Stored, never used to discard older events. |
| `payload` | A JSON object preserved for identity comparison; never applied as listing state. |

The six identity/scope strings must have 1–256 characters, no leading/trailing
whitespace and no ASCII control characters below U+0020. Unknown top-level fields,
duplicate JSON members, invalid Unicode, non-finite numbers, nesting beyond 20 and
raw or canonical JSON exceeding 65,536 UTF-8 bytes are rejected. These are **lab
bounds**, not Amazon limits. Object member order and formatting do not affect
identity comparison; arrays and values do. Numeric `1` versus `1.0` conservatively
counts as different content. Read documents are also bounded JSON objects.

This contract is **not a raw Amazon, EventBridge or SQS parser**. A production
adapter must validate the real envelope/version, trusted source and subscription,
stable business identity, authorized seller and explicit marketplace routing
before creating it. Do not guess a missing marketplace. A transport delivery ID
must not be substituted for a stable notification ID without a verified mapping.
Do not put receipt handles, receive times or delivery-attempt counters into the
normalized event: those would change a repeated notification's content.

## Persistence and refresh contract

`consumer.py` supplies the executable contract; `test_consumer.py` probes failures.

1. `ingest` starts `BEGIN IMMEDIATE`, compares the identity tuple
   `(source, subscription, notification_id)`, inserts a new inbox row, and bumps
   `(seller_id, sku, marketplace_id)`'s generation in **one transaction**.
   New events advance it regardless of event-time order.
2. An identical existing notification returns `duplicate` without changing work.
   Different normalized content for the same identity raises `IdentityConflict`;
   the original record and work remain intact. Validation, storage and conflict
   errors do not return an acknowledgement decision.
3. Only after successful commit does `ingest` return `may_ack=True`. This is a
   logical decision for a future queue adapter. No queue deletion is implemented.
   The hard-exit test leaves committed work recoverable before any external ack.
4. On startup and between runs, the worker scans `dirty_keys()`. `begin_refresh`
   captures the target key and generation. `refresh_once` calls the supplied reader
   outside the database transaction; a thrown read error leaves work dirty.
5. `complete` requires a `ReadResult` for the ticket's exact key. It saves that
   document and advances `applied_generation` only while the stored generation
   still equals the ticket and has not already been applied. A new event during
   the read makes completion return `False` / `refresh_once` return `superseded`.
   The next scan retries dirty work. The last accepted local snapshot stays intact.

`dirty` means `generation > applied_generation`. A `clean` row means a local read
was accepted against the currently persisted generation; it is **not proof of
Amazon freshness or atomicity**. New events after completion make the row dirty
again. Missing or not-yet-ingested events cannot be detected by this local fence.
Failed reads, superseded reads and restart recovery can all repeat reads. This is
not an exactly-once-effects system.

The supplied reader is a trusted adapter: before returning `ReadResult`, it must
validate the real response status/body, requested datasets and seller/SKU/marketplace
scope, and reject failed, partial or unexpected responses. Missing required data
must raise an error, not return `{}` to clear work. This core checks only JSON
shape and the adapter-declared key; it does not validate a provider response or
prove that the contents belong to that key. An empty JSON object passes its shape
check, so the reader's semantic validation is required.

## Scope and test coverage

Use one reconciliation worker. Tickets are not leases, work claims or security
tokens; concurrent reconcilers, ownership recovery and external writes are outside
this example. Separate ingester connections can commit during a read, as the race
test demonstrates. Do not share one SQLite connection across threads. SQLite uses
WAL and `synchronous=FULL` on a local filesystem; tests cover a process exit, not
power loss, disk corruption or remote-filesystem durability.

The 12 tests cover deduplication even after work is clean, identity namespaces,
conflicting payload/key/type/time, rollback of both work insertion and update,
hard-exit recovery, out-of-order events, the in-flight generation race, failed
reads, wrong keys and independent seller/marketplace/SKU scope, repeated completion,
strict input rejection and invalid read documents. The failure injection aborts
SQLite work statements after the inbox insert to test transaction rollback.

The database has no retention or pruning; removing inbox identities weakens the
deduplication horizon. No schema migrations, fairness, polling schedule, backoff,
rate limiting, reconciliation audits or health monitoring are included. Repeated
events can keep work dirty. A production adapter also needs a durable policy for
invalid/conflicting inputs (for example, reviewed dead-letter handling), so they
do not loop indefinitely, plus transport recovery and a way to reconcile missed
events. None of those provider integrations or guarantees is tested here.