Agent Watchdog: One Owner, Fresh Evidence, Verified Work

An agent watchdog is a separate observer that follows another agent’s run, decides whether it is working, legitimately waiting, stuck, uncertain, failed, or finished, and audits the claimed result. The useful default is one read mostly watchdog with a lease, a slow evidence driven cadence, and no automatic mutation authority. “The process is alive” and “the requested work is correct” are separate verdicts. That definition also resolves an ambiguity in the current search results. “Watchdog” can mean an old infrastructure restart daemon, an AI security product, or an agent that supervises another agent. This guide addresses the third meaning. Builder.io’s current agent watchdog skill describes the same concrete handoff: wait for another agent, reconstruct the request, then check claims against diffs, files, tests, CI, screenshots, and review state. Its repair mode is separate and requires authorization. That separation is the right starting point. Give exactly one watchdog ownership A watchdog needs its own identity and lease. Without them, two scheduled checks can both conclude that they own the same run. Even if both diagnoses are correct, two nudges, retries, or restarts can produce duplicate effects. Use a record like this: runId binds the observer to one piece of work. watchdogId identifies the owner. The expiry forces re election after a crashed observer instead of leaving permanent ownership behind. observedAt says how fresh the verdict is; it is not interchangeable with lastProgressAt . An observer can hold a recent progress record while its own connection has gone stale. Before every verdict, enforce three ownership rules: 1. There must be exactly one unexpired watchdog lease for the run. 2. The observer must refresh evidence before classifying or recommending an intervention. 3. A replacement watchdog may take over only after the previous lease expires or is explicitly released. If two live IDs exist, return conflict . Do not let “both are only helping” become an implicit concurrency policy. Separate observation, intervention, and outcome gates Kubernetes documents startup, readiness, and liveness probes separately because they answer different questions and trigger different actions. Its documentation also warns that a bad liveness rule can turn restarts under load into cascading failure. An AI agent watchdog needs an equivalent separation, with an additional outcome gate. Observation gate: Is the evidence fresh enough to classify the run? Check the observer timestamp, agent reachability, progress receipt, waiting metadata, and current terminal state. A timeout or missing sample yields uncertain ; it does not prove stuck . Intervention gate: Is an action justified and authorized? A read only watchdog may report a stale credential, an unowned wait, or ten minutes without useful progress. It may not infer permission to restart, cancel, edit files, send messages, or spend more budget. Give each permitted action its own retry, time, and cost limit. Outcome gate: Did the requested work pass its verifier? A terminal process status is only activity evidence. For a code task, the receipt might combine an expected commit, a clean targeted test, and a required screenshot. For a publishing task, it might require API parity, an HTTP 200 page, sitemap inclusion, and rendered assets. For an external side effect, it may need a destination read back or idempotency record. Google’s SRE guidance makes the same practical distinction from another direction: white box signals explain internals, while black box checks expose wrong content that a successful protocol status cannot detect. The watchdog should preserve both. Logs can explain why the run stopped; the outcome receipt determines whether the user’s request was satisfied. Use a state rule that preserves uncertainty The following order matters. Ownership and freshness come before progress. A terminal self report comes before a generic stall timer, but it still does not bypass verification. The two minute observation freshness and ten minute progress window are fixture values, not universal defaults. Derive them from the workflow. A deployment that normally produces a milestone every forty minutes needs a different progress window from an interactive coding run that changes files every minute. Run the rule against cases that vary one condition at a time: Case Changed evidence Verdict Bounded next action Fresh progress New test milestone working Observe later Owned wait Owner and future deadline waiting Notify near deadline Unowned wait No owner or deadline needs human Assign both Stalled run No progress for 18 minutes stuck Prepare one diagnostic Stale observer Last observation is four minutes old uncertain Refresh evidence Duplicate watchdogs Two live watchdog IDs conflict Elect one owner Reported done No outcome receipt audit required Verify artifact Verified done Receipt passes complete Release lease In the executable fixture used for this article, all eight expected classifications passed. Two comparisons are especially useful. Fresh progress with one live observer is working ; the same evidence with two live observer IDs is conflict . A completed self report with a missing receipt is audit required ; adding a verified receipt is the only change needed to reach complete . Choose cadence from expected evidence changes Polling faster does not necessarily detect failure sooner. It can create cost, noise, rate limit pressure, and repeated judgments over unchanged data. Set cadence from the evidence’s expected change rate and the consequence of delay. For a long research run, a five minute observation may be reasonable if milestones normally arrive every fifteen minutes. A scheduled deliverable needs checks around its expected start and deadline, not constant polling all day. A human approval wait needs a named owner and escalation deadline; repeated “still waiting” calls add no information. A useful schedule has four numbers: observation interval — when to refresh reachability and state; freshness limit — when the observer’s own evidence becomes unusable; progress window — the longest normal gap between meaningful milestones; action cooldown — the minimum delay before another authorized intervention. Record the last evidence hash as well as the timestamp. New log lines are not necessarily new progress. A repeated tool call, unchanged test failure, or regenerated identical draft should not reset the progress clock merely because the process is active. The reasonable recovery default is still report first. If the run is stuck , prepare one bounded diagnostic or nudge. If it is waiting , route the decision to the named owner. If it is uncertain , collect better evidence. If it is conflict , remove extra supervisors. Only a separately approved policy should permit a reversible restart, and the watchdog must verify useful progress afterward. Keep the watchdog smaller than the work An agent watchdog should not become a second runtime, an unrestricted reviewer, and an autonomous fixer at once. Its minimum useful inputs are the original request, later scope changes, a stable run identity, fresh progress evidence, waiting ownership, terminal status, and a task specific outcome verifier. Everything else should earn its collection cost. This design has a hard limit: generic telemetry cannot prove an arbitrary deliverable. Someone must define what “done” means for the task. When no deterministic check exists, the watchdog can route the result to a human or a narrowly scoped judge, preserve the evidence, and label confidence. It should not manufacture a green state. Sidewisp’s intended product territory is the health layer around existing agents: reachability, useful progress, waiting, tools, outcomes, and safe recovery boundaries. Sidewisp is currently in private preview. Its production monitoring engine and runtime adapters are not generally shipped, so the contract in this article is an operating pattern you can implement in your current runtime—not a claim that Sidewisp already watches or repairs live agents. Start with one run and one observer. Require a lease, keep observation read mostly, preserve uncertain , and define the outcome receipt before the work starts. That is enough to make an agent watchdog useful without letting supervision become another source of failure. Sources Builder.io agent watchdog README at commit 51bb048 Kubernetes: Liveness, Readiness, and Startup Probes Google SRE Book: Monitoring Distributed Systems