AI Agent Monitoring: A Quiet Alert Policy for Real Failures

AI agent monitoring should interrupt a person only when it can name an ongoing failure, show the evidence, and point to a bounded next action. A tool call, a token spike, or a long trace may help explain a problem; none of them proves that the agent has stopped delivering useful work. For a practical first policy, monitor three things separately: 1. Runtime freshness: did the scheduled run start, and is its heartbeat still current? 2. Useful progress: did task specific evidence change within the expected window? 3. Outcome verification: does the promised deliverable exist and pass its acceptance check? Then route the result. Page for a persistent, user relevant failure. Create a ticket or owner notification for a legitimate wait or a slow investigation. Suppress a single bad sample and healthy work. This article turns that rule into a small event contract and an executable eight case fixture. The page should name the broken promise An agent can be online while its work is wrong. It can also be quiet because it is correctly waiting for an approval. That is why “process running” is too weak for agent monitoring and “no recent tool call” is too noisy for paging. Google’s Monitoring Distributed Systems chapter draws a useful line between white box evidence and black box symptoms. Internal telemetry is essential for diagnosis, but a page should represent a clear failure affecting the service. The chapter also notes that a successful protocol response can still be an error when the returned content is wrong. For an agent, the corresponding failure is a run that says completed while the required artifact is absent or invalid. Start by writing one monitoring contract per workflow: Contract field Example for a repository agent Why it exists Expected start Weekdays at 09:00 UTC, five minute grace Detect a missed schedule Heartbeat Runtime observation no older than ten minutes Detect an unreachable or dead run Progress evidence New commit, changed test result, or a recorded blocker Separate motion from repeated activity Legitimate wait Approval ID plus responsible owner Keep waiting work out of the stall page Completion claim Runtime status is completed Record what the agent declared Outcome predicate Target branch contains the commit and required checks pass Verify the promised result independently The last row should be deliberately specific. “Generated an answer” may be enough for a chat task. “Created a file” is not enough for a release task if the file is invalid, unpublished, or attached to the wrong destination. The monitor cannot infer this contract from a span; the workflow owner has to define it. Instrument the run without treating spans as completion OpenTelemetry’s Generative AI semantic conventions now define agent and workflow operations such as invoke agent , invoke workflow , plan , and execute tool . The current agent span document also provides fields such as gen ai.agent.id , gen ai.agent.name , gen ai.agent.version , and error.type . These are useful correlation and diagnostic fields. They are not an outcome schema. The document is marked Development , so pinning a version matters. It also warns that captured input and output messages are likely to contain sensitive information. You can implement the alert policy below without storing prompts, responses, secrets, or full tool payloads. A compact event can look like this: Keep runId stable across the scheduler, runtime telemetry, and outcome checker. Store a low cardinality agent or workflow name for aggregation. Put diagnostic trace identifiers behind the alert rather than inside its identity. Otherwise, every retry can create a new incident for the same broken promise. The upper track in the illustration is busy but circular. The lower track changes state and produces an inspectable result. That distinction is the center of the policy: activity is evidence for debugging; progress and outcomes decide health. Test the policy with eight inconvenient cases The artifact accompanying this article uses one NDJSON record per observed run. It covers verified completion, false success, a missed schedule, an unreachable runtime, a legitimate approval wait, a persistent no progress run, one transient bad sample, and healthy active work. Run it from the artifact directory: Expected output: The evaluator uses a fixed precedence. A false success result wins over stale telemetry because the broken outcome is already known. A missed schedule wins when the run never started. An unreachable runtime wins over a no progress diagnosis because the monitor lacks fresh execution evidence. An explicit wait wins over the stall rule. Only then does a stale progress timestamp become stuck . This prevents one record from opening three incidents. It also makes each decision explainable: the output can name the condition, the evidence timestamp, and the threshold that was crossed. The included thresholds are examples, not universal defaults: five minutes after an expected start; ten minutes without a heartbeat; fifteen minutes without useful progress; two consecutive bad samples for page conditions; three consecutive bad samples for a no progress ticket. A coding agent running a two minute fix and a research agent reading papers for an hour should not share those numbers. The important part is the sequence and the requirement for persistence, not the particular duration. Add persistence before escalation Prometheus alerting rules provide two relevant mechanics. The documented for clause keeps a newly active condition pending until it has remained active for a duration. keep firing for can hold an alert open after the last matching sample to reduce flapping or false resolution caused by missing data. The same ideas apply even if you do not use Prometheus: require repeated observations before paging on silence; record the first breach time separately from the latest sample; group alerts by workflow and broken promise, not by retry or trace; hold the incident open until fresh evidence confirms recovery; re page only when severity or affected outcome changes. Do not put every condition behind the same delay. A completion claim whose required artifact fails a deterministic check is stronger evidence than one missed heartbeat. Conversely, an LLM quality score near a threshold is weaker evidence and may belong in a review queue rather than a pager. A quiet routing table is more useful than a long metric inventory: Observed condition Default route Clear condition Completion claimed; required outcome check fails after its verification grace Page when user relevant, otherwise ticket Outcome predicate passes or claim is corrected Expected run has not started after grace and two checks Page when the run has a current obligation Run starts or scheduler expectation is explicitly changed Runtime heartbeat is stale for two checks Page when active work is affected Fresh heartbeat plus a new health sample Named approval, secret, or irreversible decision is outstanding Notify the responsible owner or create a ticket Dependency is supplied or work is cancelled Activity continues but task evidence has not changed for three checks Ticket for investigation Progress evidence changes or a legitimate wait is recorded One stale or missing sample No human notification Re evaluate on the next sample Work the edge cases before choosing a tool Alert policy errors usually appear at the boundaries, not in the happy path. Verification lag: a publisher can report completion seconds before a CDN or search index updates. Give the outcome predicate a documented grace period, then verify again. Do not treat an arbitrary sleep as proof; the second check must inspect the real destination. Human waits: store both the dependency and its owner. waitingOn: "approval" without a responsible person merely hides the stall. A wait can remain healthy for the agent while still creating an overdue human task. Long silent work: a research or compilation step may be healthy without frequent tool events. Choose progress evidence that the runtime can emit safely: a completed shard, a changed content hash, a new test phase, or an explicit bounded phase deadline. Retries: retries can mask provider failures while inflating activity and cost. Group them under the same run and record attempt count as diagnostic context. A retry should not reset the first breach time unless it produces useful progress. Unknown signals: missing telemetry is not green. Report it as unavailable and avoid automated recovery when the monitor cannot distinguish stuck from disconnected. An uncertain diagnosis should ask for inspection, not execute a destructive fix. Recovery: closing an incident because a restart command returned zero repeats the false success problem. Use the same outcome or progress predicate that opened the incident. Recovery is complete only when fresh evidence shows the work is moving or the promised result exists. What this experiment proves—and what it does not The fixture makes one narrow thesis falsifiable: with the documented precedence and thresholds, the eight supplied cases produce exactly three pages, two tickets, and three suppressed notifications. You can edit one timestamp or breach count and see the route change. It does not prove that the thresholds fit your workload. The cases are synthetic, and the evaluator reads already normalized records. Real integrations must handle clock skew, duplicate delivery, late samples, scheduler policy, time zones, and collector outages. They also need a privacy boundary for anything derived from prompts or tool calls. The policy does not replace traces, evaluations, or runtime logs. Those signals explain why an outcome failed. Nor does it guarantee that a task specific predicate captures every quality problem. Some outcomes are deterministic, such as a file hash or test result; others need sampling, review, or an evaluation process with an explicit uncertainty level. Most importantly, the policy should not authorize autonomous recovery. A monitor can recommend a bounded retry or prepare a repair step, but irreversible actions, secret access, and uncertain diagnoses still require human authority. Turn the fixture into an acceptance test Before connecting a real alert destination, replace the synthetic cases with recent examples from one workflow: 1. Define the expected start and acceptable lateness. 2. Choose one heartbeat produced outside the model response. 3. Name the smallest evidence of useful progress. 4. Record legitimate wait reasons and owners. 5. Implement the outcome predicate at the actual destination. 6. Replay known healthy, waiting, stuck, missed, unreachable, and false success cases. 7. Run the policy silently long enough to review false pages and missed incidents. Only after that review should a page route become active. Keep the raw evidence, decision, threshold version, and resolution check inspectable so an operator can understand why the monitor spoke. Sidewisp is designed around this health first boundary: detect, explain, ask for authority where needed, and verify the result. Sidewisp is currently in private preview. The production monitoring adapters and recovery engine are not generally shipped today. If this approach matches how you operate agents, you can join the private preview and describe the runtime and failure cases you need covered.