Agent Observability Across Restarts: The Handoff Receipt Pattern

Agent observability usually answers what happened inside one run. That is useful, but it is not enough when an agent delegates work, exits, restarts, or waits for another agent. The practical fix is a durable handoff receipt : a small record written outside either process that says who accepted the work, what outcome is expected, and what evidence will close it. Keep traces for debugging. Add receipts for continuity. A trace can show that a handoff tool returned successfully; the receipt tells an operator whether the recipient accepted the task and whether the promised artifact was later verified. The short answer: observe the boundary, not only the run A handoff is healthy only when four different events can be distinguished: 1. the sender delegated a bounded task; 2. the recipient acknowledged the same task; 3. useful progress or a legitimate wait was recorded; 4. the expected outcome was verified. Those events may occur in different processes and different traces. They may be separated by a queue delay, a host restart, or a human approval. Treating them as one in memory span creates a fragile dependency: the context that explains the work can disappear with the process. OpenTelemetry describes context propagation as the mechanism that lets spans from different processes be assembled into a trace. It also provides span links for causally related asynchronous operations where the later work cannot be a simple child span. That solves correlation. It does not define your application’s promise, acceptance, or outcome verification rules. The receipt fills that gap. It is deliberately smaller than a transcript and more explicit than a log line. Where an ordinary trace stops helping Consider a research agent that hands a source checking task to a second worker. The sender records a successful handoff span and exits. Ten minutes later the worker starts under a new process, finds one inaccessible source, and waits for approval to use an alternative. Three states can now look deceptively similar: the task is still in a queue and has never been accepted; the worker accepted it and is legitimately waiting; the worker completed a command but never produced the requested evidence file. The span that wrapped the handoff cannot decide among them. Its successful end means the handoff operation returned without an error. OpenTelemetry is explicit that span status describes the operation tracked by that span. It is not proof that a later business outcome exists. The OpenAI Agents SDK illustrates the same boundary from another direction. Its built in tracing records runs, tool calls, handoffs, guardrails, and custom events. A group id can associate multiple traces, and a handoff span can show delegation. The SDK also notes that trace export is batched and may require an explicit flush when immediate delivery matters. Rich tracing improves the evidence available for debugging; it still needs an external rule for “the deliverable passed inspection.” This is why agent observability should not collapse command completion into outcome completion. A minimal handoff receipt contract Store one append only record per state transition. The storage can be a database table, a durable queue log, or an NDJSON file on a single host. The important property is that neither participating process owns the only copy. Here is a compact event shape: Six fields carry most of the value: operation id is the durable identity of the user visible job. It survives retries and restarts. handoff id identifies one delegation attempt. A retry gets a new handoff ID instead of overwriting history. event is one of delegated , accepted , progress , waiting , completed , or outcome verified . expected artifact names a deterministic verification target. It can also name a test, API condition, or review decision. trace id points back to detailed telemetry without making the receipt depend on that telemetry. reason explains a wait, rejection, or verification failure in bounded operational terms. Do not put prompts, credentials, model output, or raw tool payloads in this record. A receipt is an index and a state machine, not a second tracing backend. The reasonable default is append only transitions plus a derived current status. Updating one mutable row is tempting, but it destroys the evidence needed to tell a delayed acknowledgement from a missing one. Reproduce the failure cases before choosing alerts The companion fixture for this article contains four operations: a verified handoff, an unacknowledged delegation, a legitimate approval wait, and a false success completion with no verified artifact. The classifier is intentionally deterministic. Run it with: The expected result is: Two observations fall out of this small test. First, acknowledgement latency and outcome verification are independent. op 101 can be accepted quickly and still fail later; op 102 is already unhealthy before any model call or tool execution begins. A trace centric dashboard that starts at recipient execution will not see the orphan. Second, waiting needs a declared dependency. op 103 has no recent progress, but treating it as stuck would be wrong because the receipt names the approval it needs. The absence of activity becomes actionable only when combined with state and expectation. The 120 second acknowledgement limit in the fixture is an example, not a universal threshold. Set it from the queue’s observed delivery latency and the task’s urgency. Batch work might tolerate minutes; an interactive handoff might tolerate seconds. The invariant is the transition, not the number. Preserve causality without turning metadata into a leak Use the trace ID as a pointer and propagate only identifiers that downstream workers actually need. OpenTelemetry’s Baggage guidance warns that baggage is commonly sent in HTTP headers, can reach unintended third parties, and has no built in integrity checks. That makes raw goals, customer text, filesystem paths, and credentials especially poor propagation values. A safer boundary looks like this: propagate an opaque operation id and handoff id ; create a span link from the receiving trace to the delegating trace when the runtime supports it; keep the expected artifact and approval state in trusted durable storage; resolve identifiers to sensitive context only inside the authorized host boundary; authenticate receipt writers, because correlation metadata is not proof of identity. There is a trade off. A minimal receipt cannot explain why a model chose a tool or reconstruct an entire conversation. That is intentional. Use traces and logs for detailed investigation, subject to your retention and privacy rules. Use receipts to answer a smaller operational question reliably: did responsibility move, and was the promised outcome observed? Turn receipts into operator states Avoid a single red or green status. The receipt history supports five states with different responses: Working : accepted with recent useful progress. Do not interrupt it. Waiting : a named external dependency or human decision is outstanding. Route the request instead of retrying. Stuck : accepted, not waiting, and no useful progress within the task’s evidence window. Prepare one bounded recovery. Uncertain : records disagree, the writer is untrusted, or required evidence is unavailable. Ask before acting. Failed outcome : execution completed, but the artifact check failed or never occurred. Reopen the outcome, not the entire trace. The recovery boundary matters. An orphaned handoff may justify re delivery if the action is idempotent and the retry limit is known. A waiting handoff should not be retried merely because a timer expired. A false success state should run the missing verifier or request the missing artifact; replaying the whole agent can duplicate side effects. For every automated response, record the authority, maximum attempts, cost or time limit, and the evidence that will mark recovery as successful. “Retry command exited zero” is not enough when the original promise was a published report, a merged change, or a delivered message. What this means for Sidewisp This receipt pattern matches the operational questions Sidewisp is being designed to clarify: whether an agent is working, waiting, stuck, uncertain, or missing its promised outcome. It also respects a necessary product boundary: diagnosis comes before any recovery, and consequential actions require explicit authority. Sidewisp is currently in private preview. Its public site and article system are live, while production agent health collection, runtime adapters, and automated recovery are not generally shipped. The pattern above is therefore a runtime neutral design you can implement and test today, not a claim that Sidewisp already collects these receipts. If cross run handoffs are where your agents become opaque, join the private preview and describe the runtime, receipt storage, and approval boundary you need. That evidence is more useful than a generic request for “more traces.”