AI Observability: Build a Four-Layer Signal Contract

AI observability is not a dashboard category. It is the ability to answer four different questions with evidence: Was the work expected? What actually ran? Is it waiting on a legitimate dependency? Did the intended outcome exist and pass verification? A stack that answers only the second question can produce beautiful traces while a scheduled agent never starts, a human approval sits unnoticed, or a “successful” run leaves no deliverable. The practical default is therefore a four layer signal contract: schedule, execution, dependency, and outcome . Keep model latency, tokens, errors, tool calls, and spans, but do not mistake them for the whole contract. This article tests that rule on a small NDJSON fixture and gives you an audit you can adapt before buying or instrumenting another platform. Treat AI observability as a coverage problem Search results for AI observability mix several legitimate concerns: model quality, data drift, GPU and application performance, agent traces, security, governance, and cost. That breadth is why “we have observability” is hard to evaluate. Two teams can use the same phrase while collecting disjoint evidence. For an agent that performs scheduled or delegated work, use the intended job as the unit of analysis. Then require one layer for each question that can change the operational verdict. Layer Minimum evidence Failure it can expose Schedule expected time, deadline, start time, schedule identity the run never started Execution run ID, step or span, tool result, terminal status, error class the run stalled, looped, retried, or failed Dependency explicit waiting state, dependency type, approval or external system reference legitimate waiting was mislabeled as stuck Outcome artifact or side effect identity, deterministic check, verification time execution said “success” but the work was absent or wrong These layers are not four vendor products. They are four joins around one stable run ID. A trace backend may hold most execution events. A scheduler may own expected times. An approval system may own waiting evidence. The destination itself—object storage, a repository, a ticketing API, a database—usually owns the strongest outcome check. This framing also keeps monitoring and evaluation separate without forcing them apart. A rubric based quality score can be an outcome verifier when no deterministic check exists. It should not silently replace a file hash, test result, row count, or API receipt when one of those is available. Why a complete trace can still miss the incident Tracing standards are improving quickly. At commit 74fd2e0 , the OpenTelemetry generative AI semantic conventions cover model and agent spans, metrics, events, exceptions, provider specific conventions, and MCP. The document marks the GenAI conventions as Development , an important version boundary when you design long lived schemas. The agent span specification defines operations such as create agent , invoke agent , invoke workflow , plan , and execute tool . It also carries useful attributes including gen ai.operation.name , gen ai.agent.name , and conditionally required error.type ; see the pinned agent spans source. That is strong execution evidence. It tells an investigator which operation happened, how spans relate, how long they took, and whether a reported error ended the operation. Framework tracing can be richer still. The OpenAI Agents SDK tracing documentation says its default trace covers runner invocations, task and turn spans, agents, generations, function tools, guardrails, and handoffs. It also supports custom spans and processors. That makes it possible to attach missing business evidence. But neither a finished span nor a terminal ok establishes that a scheduled run was expected in the first place. Nor does it prove that weekly report.pdf exists, has the new reporting period, and passed a parser. The absence is not a defect in tracing. It is a boundary between execution telemetry and operational outcome evidence. That boundary is falsifiable: take two runs with identical successful execution events, add an outcome verified event to only one, and the operational verdict must differ. If your current alerting gives both runs the same green state, it cannot detect false success. Run a four layer audit on a fixed fixture The accompanying fixture contains four runs observed at 2026 07 25T02:42:00Z : run alpha starts, calls its report tool, finishes, and records a verified artifact; run beta has the same successful execution shape but no verified outcome; run gamma explicitly waits for approval approve 42 ; run delta passes its expected deadline without a start event. Run the audit with Node.js 20 or newer: The classifier returns one run in each state: The code uses a deliberately boring decision order. A verified outcome wins. An explicit wait with both a reason and an approval reference is waiting, not stuck. A run that never started before its deadline is missed. A finished run without outcome evidence is false success. A started run past its deadline is stuck. Everything else remains working rather than being promoted to healthy. This is an experiment, not a benchmark. Four handcrafted cases cannot estimate production error rates, and a real classifier needs duplicate event handling, clock skew tolerances, late arriving outcomes, and per job deadlines. The fixture is useful because every verdict is inspectable and changing one event changes one result. Preserve waiting as its own state A binary healthy/unhealthy field destroys information at the moment an operator needs it. Consider run gamma : the process is not making progress, yet restarting it would be the wrong default. It has an explicit approval dependency. The correct action is to surface the request to the right person while preserving its scope, age, and authority boundary. Store at least: Use the same approach for rate limit reset times, external job IDs, maintenance windows, and upstream data arrivals. A free text message such as “still waiting” is weak evidence: it is hard to route, expire, or correlate. A typed dependency plus an opaque reference supports a bounded response without copying the secret, prompt, or approval content into telemetry. Activity is equally easy to overvalue. Repeated tool calls show that a process is busy; only deltas in state or outcome show useful progress. A retry counter therefore belongs beside the last meaningful change time, not beside a generic “last event” timestamp that a loop can refresh forever. Make the outcome check destination native The strongest verifier lives where the work should have landed. For a file, record a stable object key, size, digest, and parser result. For a pull request, record the repository, PR number, target branch, and required check conclusion. For a CRM update, record the non secret entity ID, expected field transition, and read after write result. Do not put the raw deliverable into every trace. Store the smallest evidence needed to repeat the check. The OpenAI Agents SDK documentation warns that generation and function spans may contain sensitive inputs and outputs, and describes controls for disabling that capture. Apply the same principle to your custom events: identifiers and digests are usually safer than prompts, responses, credentials, absolute local paths, or customer content. Outcome verification also needs freshness. A file left by yesterday’s run is not proof that today’s run succeeded. Join the artifact to the current run through a run ID, an expected reporting period, a creation window, or a digest calculated after the current start time. There is a trade off. Destination native checks add integration work and can fail independently. Treat an unavailable verifier as unknown , not healthy and not automatically failed. Surface the missing evidence, its last successful check, and the confidence of the resulting verdict. Audit tools against the contract before you compare features A useful product evaluation starts with four rows, not a logo grid. For each candidate stack, ask where each layer originates, how it joins to the run, how long it is retained, and which query proves coverage. 1. Can it import or derive expected runs, including timezone and deadline? 2. Can it trace model, tool, handoff, retry, and error events without requiring sensitive payload capture? 3. Can it represent waiting with a typed dependency and an escalation target? 4. Can it ingest or link deterministic outcome receipts from the destination? 5. Can it distinguish unavailable evidence from a healthy result? 6. Can you export the data through an open format or API if the tool changes? Do not reject a focused trace tool because it lacks scheduler or outcome semantics. Pair it with the missing sources if the joins are reliable. Reject the architecture when it cannot represent the distinction you need, hides missing data behind green status, or requires raw sensitive content for routine health checks. The four layer contract resolves the original question: AI observability for operational agents is complete only when it can explain expectation, execution, dependency, and verified outcome separately. Traces are essential evidence, but they are one layer. Sidewisp is currently in private preview. Its intended role is a health layer alongside existing runtimes, with evidence and human authority boundaries; production monitoring adapters and recovery are not generally shipped today. If this signal contract matches the failures you need to catch, you can join the early access list without replacing your runtime or model gateway.