AI Agent Observability: Measure Useful Progress, Not Just Activity

AI agent observability is the ability to explain what an agent did from external evidence: traces, logs, metrics, tool events, model calls, latency, tokens, and cost. That evidence is necessary, but it is not a health verdict. A trace can be complete while the requested file is missing. A tool call can succeed while the agent repeats the same step. A quiet run can be correctly waiting for approval. The practical answer is to keep the telemetry and add a small decision layer above it. For each task, observe five things together: reachability, useful progress delta, declared dependencies, deterministic outcome checks, and cost over the same window. Evaluate them in that order before alerting or recovering anything. This guide turns that rule into an executable five case fixture. It is intentionally runtime neutral: the same reasoning can sit above OpenTelemetry spans, Claude Code events, an OpenClaw run log, or a custom agent. A trace proves what ran, not what changed The current OpenTelemetry semantic conventions for GenAI agent spans define operations for creating and invoking agents, invoking workflows, planning, and executing tools. The document is marked Development , which matters when you design a long lived schema: use the conventions where they fit, but isolate version sensitive attributes behind your own normalization layer. Runtime telemetry is already becoming detailed. Claude Code's monitoring documentation describes metrics, events, and beta distributed traces. Its trace tree can include an interaction, model requests, tool calls, time blocked on a user decision, and tool execution. Documented fields include duration, tokens, estimated cost, tool result size, and success . Those fields answer valuable questions: Did the runtime respond? Which model and tools ran? Did a specific tool body return an error? How long did permission waiting and execution take? How many tokens and dollars did the run consume? They do not define your task's success. A shell command returning exit code 0 proves that command completed under its own contract. It does not prove that an article is public, a sitemap contains its URL, a pull request passes CI, or a customer record reached the intended system. Applications can add those checks, but a generic tool success field cannot invent them. Activity and progress can therefore move in opposite directions. Nineteen successful tool calls with no output delta may be a loop. Two tool calls followed by silence may be a healthy wait for a reviewer. One final message saying “done” may be a false success if the promised artifact is absent. Build one health window from five signals Choose a task specific observation window before looking at the result. Five minutes may suit a small code edit; an hour may be reasonable for a scheduled research job. Avoid a global threshold that labels every long operation as stuck. Within that window, collect five signals: Signal Minimal evidence What it prevents Reachability heartbeat age, process/session response, or scheduler run receipt treating an unreachable runtime as a reasoning failure Useful progress a monotonic, task specific delta mistaking repeated activity for movement Dependency typed wait reason, owner, and due time retrying work that legitimately needs a person or external system Outcome deterministic predicate for the promised result accepting a completion message without a deliverable Cost tokens, calls, time, or money in the same window ignoring expensive retries that create no progress “Useful progress” must be concrete. For a coding agent it might be a changed test result, a new commit, or a reduced failing test count—not lines emitted to a terminal. For a publishing agent it might be a CMS draft ID, then a public API match, then a live URL in the sitemap. For a support agent it might be a validated ticket transition rather than another model response. Prefer a deterministic outcome predicate when one exists: Use an evaluator only when the outcome cannot be checked mechanically, and store its rubric, version, and uncertainty. Do not collect hidden chain of thought as a shortcut. Tool selections, explicit plans, outputs, timestamps, and state changes provide operational evidence without requiring private reasoning. Cost belongs in the window, but cost alone is not health. Ten dollars that produces a verified migration may be expected. Fifty cents spent repeating an unchanged search can be the anomaly. A useful derived measure is: The max avoids division by zero; it does not make zero progress healthy. Alert separately when progress delta == 0 and cost continues to rise. Classify working, waiting, stuck, unreachable, and false success Decision order matters. Check reachability first. Then honor an explicit dependency that is still within its due time. Check a claimed completion against the outcome predicate before accepting it. Only then interpret progress and elapsed time. Here is a complete NDJSON fixture. Save it as health window fixture.ndjson : Run this classifier with Node.js: Expected output: The fixture makes the thesis falsifiable. A naive rule such as tool calls 0 marks both run stuck and run false success as active. A rule based only on silence marks run waiting as unhealthy. The five signal rule separates them because it preserves dependency and outcome evidence. The example is a decision skeleton, not a universal scoring model. Production code also needs freshness, confidence, source identity, and an uncertain path when signals disagree. Map each state to a bounded response Classification exists to prevent the wrong action, not to decorate a dashboard. State Evidence required Default response Working recent reachability and positive progress delta leave it alone; sample again later Waiting typed dependency, owner, and unexpired due time notify the responsible person once; do not retry the blocked step Stuck reachable, past its window, no dependency, no progress delta inspect the repeating step; prepare one reversible retry or nudge within limits False success completion declared, deterministic outcome failed reopen the task and report the missing predicate; do not call it complete Unreachable stale heartbeat or failed runtime contact check host/runtime availability before changing prompts Uncertain missing or contradictory evidence collect the absent signal or ask; do not automate recovery This separation has a useful precedent outside AI systems. Kubernetes distinguishes startup, liveness, and readiness probes because “the process exists” and “the service should receive traffic” are different decisions. Its documentation also warns that badly designed liveness probes can cause cascading failures through unnecessary restarts. The analogy has a boundary: an AI agent is not a Pod, and useful progress is task dependent. The transferable lesson is narrower—do not let one ambiguous green or red signal authorize every intervention. For active recovery, attach limits to the action: one retry, not an unbounded retry loop; a maximum elapsed time and cost; a reversible step; an explicit approval boundary for destructive or external actions; a post action check of progress or the outcome. Command completion is not recovery. Clear the incident only after the expected state changes. Instrument the contract, not every thought A compact normalized health record can sit beside your existing trace data: Keep evidence references access controlled and redact secrets, prompts, raw tool payloads, and absolute local paths unless they are strictly required. The health record should say what was checked and where authorized operators can inspect it; it should not become a second copy of sensitive telemetry. Version four things explicitly: 1. the runtime adapter that normalized the evidence; 2. the progress definition; 3. the outcome predicate; 4. the classifier rules and thresholds. Without those versions, a changed test command or renamed deliverable can look like a sudden agent regression. Know where the method stops The hard part is not collecting another span. It is defining useful progress and the promised outcome honestly. Some tasks have no monotonic progress measure. A research agent may discard a weak hypothesis and return to an earlier stage; that can be valuable work even though a counter falls. Some dependencies do not expose a reliable due time. Some outcomes require judgment rather than a checksum. In those cases, preserve the evidence and report uncertain . Do not manufacture precision with a universal health score. Also separate online health checks from offline evaluation. Offline datasets can reveal whether a new agent version is more accurate across known cases. The live health window answers a different question: is this particular run reachable, moving, legitimately waiting, or missing its result now? You usually need both. Start with one consequential workflow. Define one progress delta and one deterministic outcome predicate. Replay known working, waiting, stuck, unreachable, and false success cases. Only after the classifications match reality should an alert—or a bounded recovery action—depend on them. Sidewisp is currently in private preview. Its public site and article system are live, but production agent health collection, runtime adapters, and recovery are not generally shipped. The product direction is a health layer that makes evidence, freshness, confidence, and approval boundaries easier to act on without replacing the agent runtime. Primary references OpenTelemetry: Semantic Conventions for GenAI agent and framework spans — fetched July 24, 2026; document status: Development. Claude Code: Monitoring — official telemetry fields and configuration, fetched July 24, 2026. Kubernetes: Liveness, Readiness, and Startup Probes — official probe purposes and recovery cautions, fetched July 24, 2026.