AI Agent Observability: Stop Stale Green with a Freshness Proof

An AI agent health verdict must expire. If a dashboard says “healthy” without showing when its decisive evidence was observed, it can keep a disconnected runtime green long after the facts have changed. The practical default is a freshness certificate evaluated at one explicit time. For every required signal, keep the source event time, the collector observation time, a monotonically increasing sequence, and a task specific maximum age. Mark the run healthy only while every required signal is present, fresh, non replayed, and consistent with the claimed outcome. If required evidence expires, the verdict becomes unknown —not healthy and not automatically failed. This article makes that rule testable. The accompanying five case fixture changes no model, trace vendor, or agent prompt. It changes only evidence age, arrival time, sequence, or outcome value and shows why a cached green status cannot be trusted by itself. A health verdict needs an expiry time Observability systems are good at retaining the latest value. Agent health requires knowing whether that value is still admissible. Imagine a scheduled research agent with three green fields: heartbeat: ok ; useful progress: 3 sources accepted ; outcome check: brief schema valid . Those values describe different promises. A heartbeat might expire after two collection intervals. Progress can reasonably remain unchanged during a bounded reading phase. An outcome receipt may remain valid for the specific run forever, but it must still be joined to that run rather than inherited from yesterday. A single global “last updated” timestamp erases those distinctions. The safer certificate stores an expiry for each required fact: The minimum matters. A fresh outcome receipt cannot prove that the runtime is currently reachable; a fresh heartbeat cannot prove that the promised artifact exists. When the earliest required signal expires, the combined healthy verdict expires with it. This is not a novel requirement invented for LLMs. Kubernetes uses the Lease API for node heartbeats: each kubelet updates a Lease’s spec.renewTime , and the control plane uses that timestamp to determine node availability. The official Kubernetes Lease documentation at source revision 9a8df52 is useful here because it treats availability as a time bounded claim, not a permanent property of the last successful update. Prometheus makes a related boundary explicit. Its querying documentation at revision ab225f6 describes a default five minute lookback and stale series behavior. That five minute default is a Prometheus query rule, not an appropriate universal timeout for agents. The transferable principle is that an old sample must eventually stop answering a current question. Use two clocks and one evaluation time An event can have at least two relevant times: when the source says it occurred and when the collection system observed it. Keep both. The stable OpenTelemetry Logs Data Model at revision f62b146 defines Timestamp as the time measured by the origin clock and ObservedTimestamp as the time the collection system observed the event. It also says the source timestamp may be absent. That separation prevents one overloaded field from pretending to answer event ordering, transport delay, and monitor freshness at once. For a health certificate, evaluate age in one clock domain: Use source time to order source events only when you know the source clock’s synchronization bound. Subtracting a host clock from a collector clock and calling the result network latency is unjustified without that bound. If a laptop clock is four minutes fast, a newly collected heartbeat can appear to come from the future; if it is slow, a live agent can appear stale. collector time has a narrower but dependable meaning: the monitor had this evidence by that moment. It cannot prove when the underlying action really happened, but it can prove whether the monitor’s own view is recent. Record collection delay separately when the source provides trustworthy clocks or a transport receipt. One evaluation time is equally important. Without evaluated at , a certificate cannot be replayed in a test or incident review. “Fresh now” is not an inspectable statement. “Fresh at 2026 07 26T00:42:00Z under policy freshness v1 ” is. Build a certificate, not a latest value cache The smallest useful record is deliberately plain: Each field closes a specific loophole: Field What it prevents run id accepting an outcome from another run evaluated at a moving, irreproducible “now” collector time retaining evidence after the monitor’s freshness window source time losing source event ordering when that clock is trustworthy sequence accepting a replayed or out of order heartbeat as new max age s applying one arbitrary timeout to unlike signals required silently treating missing decisive evidence as optional policy changing thresholds without an audit trail Do not infer progress from heartbeat sequence. A loop can emit perfect heartbeats while producing no useful change. Give heartbeat, progress, waiting dependency, and outcome their own evidence records. A named approval can be fresh and valid while progress is intentionally paused; the certificate should classify that run as waiting , not stuck. The verdict precedence should preserve uncertainty: 1. a fresh deterministic failure yields attention ; 2. a missing, stale, future arriving, or replayed required signal yields unknown ; 3. a fresh named dependency yields waiting ; 4. only complete, fresh, current run evidence can yield healthy . This order does not hide failure behind missing telemetry. A fresh failed outcome is stronger evidence than a stale heartbeat. Conversely, stale evidence alone does not establish that the agent failed; it establishes that the monitor cannot currently support a healthy claim. Run five counterexamples The fixture accompanying this article contains five runs evaluated at the same instant: fresh run has three fresh, advancing, successful signals; stale green still carries heartbeat: ok , but the collector last saw it 240 seconds ago against a 120 second limit; delayed batch contains a progress record whose collector time is after the evaluation time, so that evidence was not yet available; replayed sequence repeats an outcome sequence rather than advancing it; failed outcome has fresh evidence that the required artifact check failed. Run the Node.js 20+ classifier: Its exact output is: The falsifiable thesis is narrow: changing only evidence admissibility must be able to revoke a green verdict. fresh run and stale green both contain heartbeat: ok ; only the collector age differs. If a dashboard still renders both as healthy at the evaluation time, it is displaying a cached value rather than a current health conclusion. The delayed batch case handles a subtler error. Backfilled telemetry can improve historical diagnosis, but it must not rewrite what the monitor knew at an earlier decision time. Comparing collector time with evaluated at keeps incident replay honest. The sequence check blocks a different false freshness. A queue reconnect can redeliver the last heartbeat with a new transport arrival time. If the monitor refreshes age using arrival alone, the replay makes a dead source look current. Require a source sequence, boot identifier, or another monotonic token where the runtime can provide one. Across restarts, pair the sequence with an incarnation ID so a legitimate reset is not mistaken for replay. The fixture is an inspectable counterexample set, not a production benchmark. It does not estimate false positive rates, account for every queue, or prove the example limits suit your workload. Its value is that each verdict has a one field explanation and can be changed by editing one timestamp or sequence. Calibrate freshness around promises Choose limits from the workflow’s expected behavior, not from a universal dashboard default. For a poller expected every 60 seconds, a heartbeat limit might be two missed intervals plus measured jitter. For a compiler phase expected to run silently for eight minutes, progress may use a phase deadline rather than a 60 second change rule. For an outcome receipt tied immutably to one run and artifact hash, freshness may mean “belongs to this run and was verified after its start,” not “was created in the last five minutes.” Test these boundaries before alerting: normal scheduler jitter; collector restart and queue backlog; host clock skew; duplicate and out of order delivery; runtime restart with a reset sequence; a legitimate human approval wait; a completed command whose destination check fails; a monitor outage while the agent keeps working. Record the first breach separately from the latest observation. Require persistence when the evidence is noisy, but do not let a new heartbeat reset a stale progress clock. Clear an incident only with fresh evidence that addresses the condition that opened it. A restarted process is not proof that the missing report now exists. There is a cost to this rigor: more fields, per workflow policy, and an explicit unknown state. The benefit is avoiding a more expensive fiction—green status backed by evidence the monitor no longer has the right to use. Start with the three signals that can change the action: reachability, useful progress or dependency, and outcome verification. Keep diagnosis and product claims bounded A freshness certificate sits above logs, traces, metrics, scheduler records, and destination checks. It does not replace them. It records which evidence was admissible for one health decision and when that decision expires. It also should not trigger irreversible recovery by itself. unknown calls for restored evidence or human inspection. attention can justify a bounded recommendation, but secrets, destructive changes, and uncertain diagnoses still require explicit authority. Recovery is complete only after fresh progress or the intended outcome is observed. That resolves the original question: AI agent observability should never preserve healthy status indefinitely. Give each decisive signal a collector timestamp, policy bound maximum age, current run identity, and replay guard; evaluate them at one named instant; and expire the combined verdict at the earliest required boundary. Sidewisp is currently in private preview. Its intended role is a health layer alongside existing agent runtimes, with visible evidence and human approval boundaries. Production agent health collection, runtime adapters, automated recovery, cron management, and token cost analytics are not generally shipped today. If freshness proofs match the failures you need to catch, you can join the private preview.