Agent Observability: Catch False Success with an Outcome Contract

Agent observability should answer a harder question than “did the run end?”: did the expected result exist, belong to this run, and pass its acceptance check? The practical default is to define that result before execution, observe it outside the agent’s own completion message, and record a compact outcome receipt. A terminal event can trigger verification; it cannot substitute for verification. This distinction catches false success without requiring a second model to reread the whole transcript. It also avoids the opposite mistake: treating a legitimate approval wait as a broken run. The receipt described below records an opaque artifact identifier, freshness, a content digest when appropriate, and a deterministic validator result. Missing evidence remains unverified or a specific failure state instead of being rounded up to healthy. A terminal event is evidence of execution, not delivery Traces are the right place to understand how work ran. They are not automatically a proof that the requested external state now exists. The current OpenTelemetry semantic conventions for GenAI agent spans describe operations such as invoke agent , plan , and execute tool , plus agent, provider, model, timing, and error attributes. The document is explicitly marked Development. Those signals can show that an operation happened and whether it reported an error. They cannot know that your particular invoice was stored, your pull request contains the requested change, or your report matches an approved schema. That acceptance rule belongs to the application. The OpenAI Agents SDK tracing reference makes the same boundary concrete. Its default trace can include model generations, function calls, guardrails, handoffs, and custom spans. This is rich execution evidence. The SDK also warns that generation and function spans may contain sensitive inputs and outputs, and lets operators disable that capture. An outcome receipt can therefore be both narrower and more decisive: retain the proof needed to judge the deliverable, not a second copy of every prompt and tool payload. A good operational model uses both: the trace explains the path, retries, tools, and failure location; the outcome receipt proves the intended result or names the missing proof; a waiting signal records a known dependency or approval, rather than pretending the task finished; a progress signal shows useful movement while work is still active. Conflating these signals creates bad alerts. Activity is not useful progress. A clean terminal event is not a verified result. A declared wait is not a stall. Write the outcome contract before the run An outcome contract is small enough to review at task creation and strict enough to evaluate without asking the agent what it meant. Start with the cheapest deterministic check that matches the real result. Field Purpose Example artifact id Names the expected result without exposing a secret or absolute path monthly report run started at Establishes the freshness boundary 2026 07 25T14:00:00Z observed at Shows when the evidence was collected 2026 07 25T14:08:12Z modified at Rejects an artifact left by an earlier run 2026 07 25T14:07:55Z expected sha256 Pins exact bytes when byte identity matters a 64 character digest validator Names the acceptance check report schema v3 validator exit code Records the deterministic verdict 0 evidence source Says where the observation came from local file stat Do not require every field for every job. A database migration may need a schema query rather than a file digest. A deployed page may need HTTP status, canonical content, and a browser assertion. A human approval task should remain waiting until the authority event arrives. The contract should represent the outcome, not force every workload into a file shaped model. The default classification order matters. Check for the absence of evidence first, then identity, freshness, digest, and validator result. This produces actionable states: 1. missing — no observed artifact exists; 2. wrong artifact — the observation belongs to a different target; 3. stale — the artifact predates the run; 4. hash mismatch — exact bytes were required and differ; 5. validator failed — the artifact exists but does not meet acceptance criteria; 6. unverified — the required check did not run or its evidence is unavailable; 7. verified — every required condition passed. Keep the receipt privacy minimal. Opaque identifiers are safer than customer names or filesystem paths. A digest can prove byte identity, but a plain hash does not hide a predictable secret from enumeration. Use a keyed HMAC when the value is sensitive and low entropy, or avoid retaining the value entirely. Evidence collection should happen near the artifact so raw content does not need to leave the host. Run the six case false success test I tested the rule against a synthetic six run fixture. Every run carries the same runtime terminal state: completed . Two observations are fresh and valid. Four represent a different false success mode: no artifact, an artifact older than the run, a content mismatch, and a validator failure. The classifier is deliberately boring. It evaluates facts in a fixed order: Running the included fixture produces: The falsifiable claim is narrow: for this supplied fixture, a terminal status rule accepts six runs, while the outcome contract verifies two and rejects four with specific evidence states. This is not a measured production failure rate. It is a boundary test showing that identical terminal states can conceal materially different outcomes. The useful metric is not “percentage of runs that said complete.” It is verified outcomes / runs expected to deliver an outcome , reported beside the coverage of the checks. If only half of your task types have deterministic validators, show that limitation. Do not silently classify the uninstrumented half as healthy. Attach verification at the completion boundary The receipt works best when the runtime exposes a completion boundary but the check itself remains independent. At that boundary, collect evidence, run the validator, persist the receipt, and only then update the operational state. Claude Code provides one concrete implementation point. Its current hooks reference says TaskCompleted runs when a task is being marked complete. A command hook can exit with code 2 to prevent completion and return feedback when tests or another acceptance check fail. This makes a deterministic gate possible without trusting a prose claim. It is a Claude Code specific mechanism, not a universal agent standard, and a hook that ran successfully still needs to test the right artifact. For runtimes without a blocking completion hook, use a two stage state transition: Do not automatically retry every non verified state. missing after a known upload delay may need a short bounded observation window. validator failed may justify one reversible repair attempt if the user already authorized it. unverified means the evidence channel failed; it does not prove the deliverable is bad. A task awaiting an irreversible decision belongs in waiting or needs human , not in a recovery loop. Also separate command success from outcome success. A validator process that exits 0 proves only what that validator actually checks. Version the validator name, record its evidence source and observation time, and review the contract when the deliverable changes. A stale acceptance rule can produce a perfectly documented false positive. Escalate uncertainty; do not manufacture success An outcome contract is only as complete as its declared expectations. It can miss an unlisted artifact, accept a weak validator, or read from a stale evidence source. Those are reasons to expose coverage and confidence, not reasons to add a model judge by default. Use an LLM evaluation only for criteria that cannot be checked deterministically, keep its rubric and version visible, and avoid letting the same agent both produce and conclusively grade its own work. When evidence conflicts, prefer uncertain and ask for authority before changing external state. Sidewisp is currently in private preview. The public site and article library are live; production agent health collection, runtime adapters, and recovery are not generally shipped. Sidewisp is intended as a health layer alongside existing runtimes, not a replacement runtime or autonomous fixer. The operating rule is simple: let the runtime terminal event start the check, let external evidence decide the outcome, and let missing evidence remain unknown. If that health model matches how you run agents, the private preview signup is the appropriate next step.