Agent Monitoring After a Fix: Prove Recovery in Five Steps

An AI agent is not recovered merely because a restart, retry, or nudge returned successfully. The practical default for agent monitoring is to verify recovery in five steps: record the authorized intervention, confirm reachability and readiness, observe task specific progress, verify the promised outcome independently, and watch one stability window for relapse. Until the strongest applicable check passes, keep the state as recovering , uncertain , or needs human —not healthy. This distinction matters because the repair command and the user’s work live at different layers. A process can restart while its credential remains expired. A heartbeat can resume while the agent repeats the same tool call. An agent can declare completion while the file, ticket, message, or deployment is still absent. Recovery is an evidence claim, not an activity event. Clear an incident with a verification ladder Use a ladder rather than one green status. Each step answers a different question and should preserve its own timestamp, source, and confidence. Step Question Minimum evidence State if it fails Intervention Was the exact bounded action authorized and executed? approval reference, action type, attempt ID, exit or API result needs human or intervention failed Reachability Is the runtime contactable and ready for its task? fresh heartbeat plus a task relevant readiness check unreachable or alive only Progress Has useful work changed since the intervention? monotonic artifact, completed unit, cursor, test delta, or destination change alive only or recovering Outcome Does the promised result exist and satisfy its deterministic check? destination native lookup, digest, test, revision, or receipt false recovery , recovering , or uncertain Stability Did the diagnosed failure stay absent long enough to recur? one workload specific observation window with no repeated symptom relapsed or recovered The reasonable default is conservative: an agent that is reachable but has not moved useful work is alive only ; one that has resumed measurable work but has not reached its outcome is recovering ; only fresh outcome evidence that survives the stability window earns recovered . Kubernetes uses a related separation for containers. Its probe documentation gives startup, liveness, and readiness different jobs: liveness can trigger a restart, while readiness controls whether a container should receive traffic. It also warns that incorrect liveness probes can cause cascading failures. The analogy has a boundary—an agent task is not a Pod—but the operating lesson transfers: “the process should restart” and “the work is ready” must not be the same test. A stability window is not an arbitrary five minute sleep. Choose the shortest interval in which the original failure had a fair chance to return. For a loop that repeated every two tool calls, observe at least two clean tool call opportunities. For a scheduled publisher, wait through its next outcome deadline. For a credential failure, exercise the affected permission once with a non destructive check. The window should be long enough to falsify the repair, but not so long that the incident stays ambiguous after decisive evidence exists. Record one recovery attempt, not a loose sequence of commands Tie diagnosis, authority, intervention, and verification to one immutable attempt ID. Otherwise, the monitor can join a heartbeat from a later manual restart to an earlier automated nudge and report a recovery nobody can explain. A privacy bounded event can look like this: This event does not need prompts, responses, secrets, raw tool payloads, or absolute paths. It does need the action boundary and the evidence boundary. Store action completed as a receipt, not as the recovery verdict. That rule is especially important for asynchronous APIs. RFC 9110 section 15.3.3 says an HTTP 202 Accepted response means processing has not completed and might never occur; the response ought to describe current status and point to a status monitor. If an agent’s recovery adapter receives 202 , follow that monitor or query the destination. Do not translate “accepted” into “fixed.” Traces have the same scope boundary. OpenTelemetry defines a span as a unit of work and its status as the status of the operation it tracks. A clean restart agent span proves that operation did not report an error. It does not define whether a report was produced, a ticket arrived, or a deployment serves the intended revision. Link the intervention span to later progress and outcome evidence; do not overload its status. Authority belongs in the record too. If a restart, credential refresh, message send, or rollback requires approval and no valid approval exists, the monitor should emit needs human . It must not attempt the action and then ask for retrospective permission. Recovery also needs a retry and time budget. A second intervention after the first one fails is a new decision, not an invisible extension of the original command. Reproduce the heartbeat false positive The accompanying fixture contains eight synthetic post intervention cases: missing authority, an action error, heartbeat only activity, resumed progress, verified stable recovery, relapse, stale verifier evidence, and agent declared completion with a missing destination outcome. Run the classifier from the artifact directory: The decisive output is: The naïve rule—action completed and heartbeat present—reports six recoveries. The ladder reports one. That is not because the ladder is pessimistic. One case is genuinely recovering : useful progress resumed and the outcome is still pending. Another has fresh outcome evidence but then repeats the diagnosed failure, so it is relapsed . A third has a verified looking result that is ten minutes old under a two minute freshness contract, so it is uncertain , not failed or healthy. The fixture’s 120 second freshness threshold is illustrative, not a production default. Evidence freshness belongs to the verifier. A file digest on local storage may be decisive immediately. An eventually consistent search index may need a documented delay. If the verifier itself is unavailable, preserve uncertain and expose the missing signal. Do not restart the agent again merely to make the dashboard turn green. Google’s Monitoring Distributed Systems chapter separates symptoms from causes and black box from white box evidence. It also treats a successful protocol response with wrong content as an error that may require end to end testing. In this recovery ladder, the intervention receipt and runtime telemetry are white box cause evidence; the destination native outcome check is the black box symptom test. Both are useful, but only the latter resolves what the user actually lost. Turn recovery verification into an operating contract For each monitored task class, define the ladder before an incident: the failure states that permit a bounded intervention; the person or policy allowed to approve each action; the reversible action and its hard attempt, time, and cost limits; the runtime readiness check after the action; one useful progress field with an expected direction; the deterministic outcome verifier, its deadline, and its freshness limit; the recurrence opportunity that closes the stability window; the rollback or escalation path when the attempt fails. Keep the verdict vocabulary small. Needs human means authority, a secret, or an irreversible decision is missing. Intervention failed means the approved action did not complete. Alive only means the runtime is ready but useful progress is absent. Recovering means progress has resumed while the outcome or stability check remains open. Uncertain means decisive evidence is missing or stale. False recovery means the outcome deadline passed without the promised result. Relapsed means the original symptom returned. Recovered means the task specific outcome is verified and the recurrence window stayed clean. There are honest limits. Some outcomes cannot be checked deterministically. “The customer accepted the analysis” may require a human decision; “the summary is good” may need a rubric whose reliability is itself measured. A destination can also commit a side effect before its response times out. Reconcile with an idempotency key or an independent lookup before retrying. When evidence cannot resolve the state, keep uncertainty visible. Sidewisp is currently in private preview. Its production monitoring adapters, token cost analytics, and recovery execution are not generally shipped. The intended direction is a health layer alongside existing runtimes that makes diagnosis, authority, evidence freshness, useful progress, and outcome verification explicit. It should not become a mandatory model gateway or an autonomous fixer. If that operating model fits your agents, Join the private preview. Sources Kubernetes: Liveness, Readiness, and Startup Probes Google SRE Book: Monitoring Distributed Systems RFC 9110: HTTP Semantics, 202 Accepted OpenTelemetry: Traces