AI Agent Observability: Prove No Accepted Work Vanished

An agent queue can report the expected depth, workers can keep emitting spans, and completed work counters can keep rising while one accepted job has disappeared. The practical answer is to stop comparing totals and reconcile identities. At one bounded cutoff, record every accepted work id and require it to occupy exactly one current bucket: The ⊎ matters: this is a disjoint union, not ordinary addition. An ID found in two buckets is a contradiction. An accepted ID found in none is missing. A terminal or active ID that was never accepted is a phantom. This work conservation check does not promise exactly once execution, but it can prove a narrower and useful fact: the control plane evidence accounts for every accepted unit of work exactly once. Equal totals can conceal a missing job Suppose the acceptance ledger contains 1,000 IDs. The queue, lease table, and terminal ledger also contain 1,000 records in total. A count based dashboard turns green. That arithmetic permits a bad substitution: accepted work 417 is absent while terminal record ghost 92 has appeared. The counts still match. Identity sets do not. Define the evidence at cutoff t0 : A : IDs durably acknowledged as accepted at or before t0 ; Q : accepted IDs visible as queued in the cutoff snapshot; L : accepted IDs with a lease whose expiry is later than t0 ; T : accepted IDs with one recognized terminal record. Then check both directions: The terminal set needs an explicit policy. completed is terminal. cancelled can be terminal when the cancellation is authorized and durable. dead lettered can also be terminal for conservation purposes, even though it is a failed outcome that still needs attention. A dead letter queue is useful precisely because it isolates work that was not processed successfully for diagnosis and possible redrive, as the Amazon SQS dead letter queue documentation explains. Do not treat leased as a timeless state. In SQS, receiving a message does not delete it; the message becomes temporarily invisible and should become visible again if it is not deleted before the visibility timeout expires. Standard queues also use at least once delivery, so a visibility window is not an exactly once lock. Those documented visibility timeout semantics are why the invariant uses valid lease , not every lease row ever written. Run the six scenario reconciliation The companion fixture makes the decision rule inspectable without requiring a broker. It contains six snapshots at 2026 07 27T00:00:00Z : Scenario Evidence Expected verdict Mixed current states one queued, one unexpired lease, one completed CONSERVED Vanished after acceptance accepted ID in no current bucket MISSING Queued and completed same ID in two buckets DUPLICATE STATE Lease expired lease ended before the cutoff, no return evidence EXPIRED LEASE Phantom terminal unknown completed ID balances the count PHANTOM Explicit cancellation accepted ID has a durable cancelled record CONSERVED Download work conservation fixture.json and audit work conservation.mjs from the article evidence bundle, or reproduce their fields in a local directory, then run: The exact observed counts were: PASS means the auditor classified all six fixtures as expected; it does not mean every fixture was healthy. Four deliberately unhealthy cases were detected. The phantom case is the important trap. It has one accepted ID and two observed records: the real ID remains queued while an unknown ID claims completion. A naive equality check can be made to look balanced by choosing one aggregate from each subsystem. Set reconciliation reports the unaccepted terminal ID instead. The duplicate state case exposes the inverse problem: every ID is known, yet the same work appears ready for another attempt and already completed. This is also why message traces alone are insufficient. The OpenTelemetry messaging conventions distinguish receiving, processing, and settling, and describe creation context for correlating producers with consumers. That context is valuable evidence. A process span, however, does not by itself establish the current queue state or a durable terminal outcome. Take one coherent cut of the lifecycle The invariant becomes misleading when its inputs describe different moments. Imagine reading the acceptance ledger at 12:00:00, the queue at 12:00:03, and terminal outcomes at 12:00:08. A job can legitimately move between those reads and appear missing or duplicated. Use the strongest consistency mechanism your stack supports: 1. assign an immutable, namespace qualified work id before acknowledging acceptance; 2. write the acceptance record durably in the same transaction as enqueueing, or retain a recoverable outbox relation; 3. capture queue and lease state at a named watermark, offset, database snapshot, or brief observation barrier; 4. include lease expires at , not merely leased=true ; 5. append terminal outcomes with the same work id , an outcome kind, and a durable timestamp; 6. reconcile only records whose visibility rules place them on the same side of t0 . If the systems cannot provide a coherent cut, return UNVERIFIABLE rather than inventing a healthy result. A short quiet period can reduce churn but is not a substitute for a consistency contract. Document the maximum snapshot skew and delay an alert until a work item has remained anomalous beyond that bound. Stable IDs are equally important. A retry should normally retain the logical work id and receive a separate attempt id . If every attempt gets a new work identity, the audit cannot distinguish a retry from new work. If IDs are reused across tenants or queues, a legitimate terminal record can look like a phantom or falsely satisfy another job. An expired lease deserves its own verdict. It might already be visible in the broker again, but an old lease row cannot prove that transition. Require fresh queue evidence or a new lease generation before calling it active. This keeps “waiting” separate from “stuck”: a valid lease can represent working; queued work can represent legitimate waiting; an expired lease with no observed return is unresolved. Map each verdict to a bounded response The audit should prepare investigation, not launch a broad replay. For MISSING , first verify snapshot skew, then inspect the acceptance to enqueue boundary and any transactional outbox. Preserve evidence before retrying. Recreating work from an incomplete record can duplicate an external effect. For DUPLICATE STATE , stop automatic pickup for that ID if the queue permits a reversible hold. Compare terminal evidence with any effect receipt before deciding whether the queued copy is stale. A terminal label alone may be wrong; a still visible queue item may also be a delayed replica. For EXPIRED LEASE , ask the broker for fresh visibility and check the worker generation. A bounded re lease may be safe only after the previous holder is fenced or proven dead. The conservation audit detects the unresolved state; it does not authorize recovery. For PHANTOM , verify ID namespace, ingestion bugs, and terminal ledger provenance. Do not delete the unknown record merely to make the equation balance. It could be valid work from another tenant, queue, or observation window. Finally, conservation is not outcome quality. A completed record can still point to a missing or incorrect deliverable. Follow this lifecycle test with deterministic outcome verification where possible. The two checks answer different questions: “Is every accepted job accounted for?” and “Did the job produce the intended result?” Sidewisp is currently in private preview. Production agent health collection, queue adapters, and runtime recovery are not generally shipped. Work conservation is an example of the evidence a future health layer could evaluate; it is not a claim that Sidewisp currently collects these records or acts on live agents. The useful near term move is to instrument stable work IDs and test the invariant locally before adding any automated response.