AI Agent Dashboard: Six Signals That Reveal Operational Health

An AI agent dashboard should answer one operational question before it draws a token chart: does this run need attention now? The smallest useful row combines six signals—collector freshness, schedule timeliness, heartbeat activity, useful progress, an explicit waiting dependency, and outcome verification. Apply them in a fixed order so that a quiet approval wait is not mistaken for a stall and a busy retry loop is not mistaken for healthy work. Keep latency, model calls, tool calls, tokens, cost, and traces. They are valuable diagnostic evidence. They do not, by themselves, prove that a scheduled run started, a report changed, an approval is pending, or the promised deliverable exists. Start with a state row, not a wall of charts The reasonable default for an LLM application dashboard is service telemetry. Sentry's official AI Agents dashboard, for example, includes agent runs, LLM calls, duration, model use, tokens, tool calls, errors, and trace detail. Those panels help answer “what happened inside this execution?” and “which dependency became slow or expensive?” An operator facing twenty autonomous or scheduled runs has an earlier decision: which one should I open? A compact state row can answer it: Field Example Decision it supports Agent and workflow researcher / weekly brief Which work is affected? State waiting:human approval Does it need intervention, routing, or patience? Evidence age collector 14s ago Is the verdict based on fresh data? Last useful progress 3 sources added, 4m ago Is the outcome moving, not merely the process? Next expected event approval by owner What should happen next? Outcome check brief.md schema: pending What must be true before “complete” is credible? The state is a verdict derived from evidence, not a copy of the runtime's last status string. Put the decisive evidence beside it. “Stuck” without a progress window is an opinion; “no source set change for 13 minutes while heartbeats remained fresh” is inspectable. This separation has a useful precedent outside agent systems. Kubernetes does not collapse startup, liveness, and readiness into one probe because the correct response differs: wait for startup, restart after a genuine liveness failure, or stop routing traffic when a workload is not ready. Its documentation also warns that a badly designed liveness probe can create cascading failures. An agent dashboard has the same control problem: a label is only safe when it implies the right next action. Collect six signals with explicit freshness The six signals below form a compact health contract. They can be stored as fields on a run record or calculated from runtime events. Each needs a timestamp, source, and unavailable state. 1. Collector freshness Record when the dashboard last reached the runtime or received a trusted event. If the collector is stale, classify the run as unreachable or uncertain before interpreting older heartbeats and progress. Otherwise a disconnected host can look peacefully idle. Use a threshold tied to the collection cadence. A 120 second freshness limit is reasonable for a one minute poller in an example; it is nonsense for a job that syncs every hour. Show both the observed age and the configured limit. 2. Schedule timeliness Store the expected start, actual start, timezone, grace window, and scheduler policy. “No run” means little without those fields. A scheduler may be paused, an overlap policy may intentionally skip a start, or a catch up window may defer missed work. Temporal's Schedule documentation makes these distinctions concrete: schedules can be paused; overlap policies can skip, buffer, cancel, terminate, or allow concurrent executions; catch up windows decide which missed actions run after an outage. Other runtimes use different names, but the dashboard still needs to preserve the policy that explains the gap. 3. Heartbeat activity A heartbeat proves recent contact or execution activity. It is useful for separating an unreachable runtime from a live process. It should not advance the progress clock. Make the event narrow: heartbeat at , source , and perhaps a monotonically increasing sequence. Do not call the run healthy merely because that sequence keeps changing. 4. Useful progress Define one task specific delta. A coding agent might change the patch digest or increase the passing test count. A research agent might add a reachable primary source or move a brief from schema invalid to schema valid. A support agent might create the promised ticket. The progress record needs last progress at , a small description of the delta, and a verifier version. Avoid vague counters such as “steps completed” unless each step maps to the outcome. 5. Waiting dependency Represent legitimate waiting explicitly: human approval , credential , rate limit , external job , or another named dependency. Add an owner, requested action, and deadline when known. This field changes the action. A run waiting for approval needs routing to the authorized person, not a restart. A rate limit wait may need patience. A missing credential needs a human who can supply it without exposing the secret to the dashboard. 6. Outcome verification Write the completion predicate before the run starts. Examples include: report exists, parses, and contains two reachable primary sources ; pull request exists and named checks pass ; ticket ID was returned and can be fetched ; scheduled export contains the expected date partition . Store declared complete separately from outcome verified . The second should be true , false , or unavailable , with the verifier and check time. A successful command is activity. The intended artifact is the outcome. Apply one precedence order The dashboard should not stack every possible warning on the same row. Evaluate the safest, most explanatory state first: 1. stale collector → unreachable ; 2. expected start beyond its grace window → missed schedule ; 3. named dependency → waiting:<dependency ; 4. declared complete without a verified outcome → false success ; 5. fresh heartbeat plus stale, zero progress → stuck ; 6. verified outcome → complete ; 7. positive progress delta → working ; 8. insufficient or conflicting evidence → uncertain . That order is a product decision, not a law of nature. It is still better than allowing three independent alerts to claim that the same disconnected run is simultaneously stuck, late, and missing its outcome. Preserve the underlying signals for investigation, but give the operator one primary state and one next action. The accompanying fixture tests six cases against explicit example limits: collector freshness of 120 seconds, schedule grace of 300 seconds, heartbeat freshness of 60 seconds, and a progress timeout of 600 seconds. Run it with Node.js 20 or newer: The exact output is: The most important pair is approval wait versus retry loop . Both have fresh heartbeats, no progress delta, and old progress. The named dependency makes the first legitimate waiting; the absence of a dependency makes the second a stuck candidate after its timeout. The false success case has recent progress and a completion claim, but the outcome verifier is false, so the completion claim wins no trust. Put diagnostics behind the verdict Once the state row identifies a run worth opening, the detail view can explain why. Organize it around the transition that failed rather than around whichever telemetry source is easiest to chart. For missed schedule , show the schedule expression, timezone, enabled state, expected and actual starts, overlap policy, and recent run history. For waiting , show the dependency, owner, requested authority, age, and a bounded reminder action. For stuck , show heartbeat cadence beside progress evidence and repeated tool signatures. For false success , show the completion claim beside the failed predicate. Then add trace, latency, token, cost, model, and tool panels. A token spike attached to a stuck run is actionable; the same spike attached to a verified result may be a cost review item rather than an incident. Tool call repetition can explain a stall, but identical calls are not proof of a loop until the task's progress evidence also stops changing. Use a timeline for causality: This view makes quiet periods understandable. It also gives every alert an evidence age. If an adapter stops reporting after 12:03, the dashboard should move to unreachable or uncertain instead of keeping a green state forever. Treat thresholds and data collection as product boundaries The fixture is a counterexample set, not a production benchmark. Ten minutes without a file change may be normal for deep analysis and disastrous for a one minute queue worker. Calibrate windows per workflow, then record the configuration next to the verdict. Progress is the hardest signal. Prefer deterministic evidence such as a digest, row count, status code, test result, or schema check. When the outcome is qualitative, a versioned evaluator can contribute evidence, but its score has uncertainty. Keep unavailable as a real state; do not convert missing evidence into healthy. Collect the minimum needed to establish health. A state row usually does not need prompts, responses, secrets, raw tool payloads, or absolute local paths. An opaque run identifier, timestamps, small counters, verifier results, and redacted dependency classes can drive the first decision. Richer traces can have separate retention and access rules. Finally, do not wire the primary state directly to irreversible automation. Kubernetes' liveness warning is relevant here: an overconfident health test can make recovery worse. A dashboard may recommend a bounded retry, pause, or reminder, but the action should respect authority, retry, time, and cost limits—and the issue should clear only after useful progress or the expected outcome is observed. Where Sidewisp fits Sidewisp's product direction is a health view around agents people already run: reachability, useful progress, memory and tool access, outcome evidence, cost signals, and controlled recovery with explicit approval boundaries. It is not intended to replace the runtime, scheduler, model gateway, or raw tracing system. That description is product direction, not a claim of generally available monitoring. Sidewisp is currently in private preview. The public site, interactive demonstration, and article system are live; production agent health collection, runtime adapters, automated recovery, cron management, and token cost analytics are not generally shipped. Join the private preview if this six signal dashboard contract matches the operational problem you need to solve. Primary sources Sentry AI Agents Dashboards — official fields for runs, LLM calls, duration, models, tokens, tools, errors, and traces. Kubernetes liveness, readiness, and startup probes — official separation of health checks, reactions, thresholds, and failure cautions. Temporal Schedules — official schedule, pause, overlap, catch up, and failure policy semantics.