Agentic AI Observability: Diagnose Deadlocks with a Wait-for Graph

Agentic AI observability needs to answer a coordination question that ordinary request telemetry leaves open: who is waiting for whom, and is that wait still legitimate? The practical default is two layers. Keep traces for causal execution detail, then maintain a small application level wait for graph for active dependencies. Each directed edge says that one agent cannot continue until another agent, a person, or an external system supplies something specific. An acyclic edge to a working agent is usually a normal wait. An edge to a person is a needs human state. A closed agent to agent cycle is a warning, but it becomes a deadlock candidate only when every participant has also stopped making useful progress for a calibrated interval. That final condition matters. Restarting a healthy agent because its span is quiet destroys context. Restarting an entire group because a graph contains a fresh cycle can duplicate tool calls. The operator needs topology, freshness, and an expected outcome—not a single “busy” light. Traces show execution; dependency edges explain the pause Microsoft’s current agentic AI observability guidance recommends end to end traces that link agent execution steps, with run identifiers, tool and agent invocations, and enough context to reconstruct incidents. It also says that latency, errors, and throughput are too narrow to establish AI system quality and reliability. That is a useful base, but it does not define your application’s promise between two collaborating agents. The emerging OpenTelemetry GenAI agent span conventions provide vocabulary for operations such as invoke agent , invoke workflow , plan , and execute tool , plus stable agent identifiers where available. The document is explicitly marked Development . Treat it as an evolving interoperability layer, not permission to pretend that a proposed attribute proves your workflow completed. Imagine a research agent invokes a reviewer. The invocation span can finish successfully even though the reviewer has not accepted the assignment, the reviewer is waiting for a destination contract, or the requested source bundle never arrived. The span describes the invocation. The dependency record describes why the larger operation cannot advance. Use both. Do not overload span status with application outcomes it cannot support. Record the minimum wait contract A useful edge is deliberately small. It needs stable operational identifiers and freshness, not prompts or chain of thought: operation id contains the graph. waiter → blocker supplies its direction. blocker type distinguishes an agent from a human or external service. since tells you how long the dependency has existed, while last progress at refers to useful movement in this operation—not a heartbeat, token stream, or repeated retry. reason should name the missing condition. authority prevents an approval gate from being “recovered” as though it were a failed process. Keep content out of the edge. A task identifier, artifact hash, redacted reason code, and trace correlation ID are usually enough. Microsoft’s guidance places data minimization, retention, access control, and privacy alongside forensic usefulness. A wait graph should help locate evidence; it should not become a second store for prompts, tool arguments, secrets, or deliverable contents. The illustration above is a normal pause. The chain is acyclic, upstream work is fresh, and the final blocker is explicit human authority. No agent restart can supply that authority. The correct action is to route the decision to the owner with the relevant evidence. A graph cycle is evidence, not a verdict Cycle detection is cheap. Diagnosis is not. For each active operation, build a directed graph from agent to agent edges and run a depth first search or strongly connected components algorithm. A component with more than one node—or a self edge—contains a cycle. Then evaluate three independent conditions: 1. Coverage: are all active participants and dependencies represented, or is telemetry missing? 2. Freshness: when did each participant last change the expected artifact, decision, or verified task state? 3. Authority: does any edge intentionally wait for a person, credential, irreversible choice, or external service? Only call the cycle stale when the youngest last progress age in that cycle exceeds the threshold. Using the oldest age is a subtle bug: one long idle participant could make a cycle look stuck even while another participant changed the output seconds ago. The threshold is workload specific. Five minutes is plausible for a fast local code review loop and absurd for a research agent waiting on a long build. Start from an expected progress interval for that operation, add transport and scheduling slack, and preserve an UNCERTAIN CYCLE state when the evidence is too fresh or incomplete. This distinction also protects against clock skew. If timestamps come from several hosts, either normalize them through a trusted collector or carry a source clock and uncertainty bound. A negative or implausibly large age should reduce confidence, not trigger recovery. Reproduce five coordination states I tested the rule with eight NDJSON edges grouped into five operations. The fixture time is 2026 07 24T18:30:00Z , and the illustrative stale window is 300 seconds. The classifier applies this order: That order is part of the safety contract. If you test for cycles before authority, a human approval edge represented by an agent proxy can be mishandled. If you test only freshness, an unreachable artifact store and a mutually waiting agent group collapse into the same generic timeout even though their repairs differ. Running the deterministic classifier produced: The two cycles have the same shape. Their operational conclusions differ because the two agent cycle changed useful state 15–20 seconds ago, while every member of the three agent cycle has been quiet for at least 1,860 seconds. Topology alone cannot make that decision. The test also exposes a failure mode in “all agents online” dashboards. Every participant in op cycle can answer a heartbeat and still be unable to advance. Conversely, op approval can remain quiet for twenty minutes without being broken. Reachability is one signal; dependency resolution is another. Break one edge, then verify the outcome A stale cycle calls for investigation before intervention. Open the correlated traces, inspect the dependency reasons, and identify the smallest reversible edge that can be resolved without inventing missing authority. For the three agent fixture, the publisher waits for a source bundle, the researcher waits for review notes, and the reviewer waits for a destination contract. If the destination contract already exists in durable storage, attaching that evidence to the reviewer’s task may break one edge safely. Restarting all three agents would be a much larger action and could repeat external work. Use a bounded recovery record: the edge selected and evidence supporting the diagnosis; the exact reversible action proposed; who approved it, when approval is required; a time, retry, and cost limit; the progress signal expected afterward; the artifact or outcome that must be independently verified. Do not clear the incident because a nudge or retry command returned zero. First look for an edge transition, then for useful progress, and finally for the intended output. If the expected report is still missing, the operation is unresolved even if every trace is green. The OpenTelemetry agent observability overview explains why standardized telemetry matters across frameworks and why agent telemetry also feeds evaluation. The wait graph complements that telemetry; it does not replace traces, logs, evaluations, or outcome checks. Know what the graph cannot prove A wait for graph is only as complete as its writers. A crashed agent may fail before emitting its edge. A faulty agent may report meaningless “progress.” An expired credential might appear as a generic external wait. A queue can delay an event long enough to create a temporary false cycle. Treat these as explicit limitations: Missing edges produce false negatives. Stale or skewed clocks distort freshness. Progress counters without artifact deltas reward activity. A cycle can disappear while the requested deliverable remains missing. A malicious or compromised writer can falsify its own state. Mitigate them with collector freshness, authenticated writers, monotonic sequence numbers, artifact hashes, trace correlation, and outcome verification. When signals disagree, return UNCERTAIN and ask for evidence. Uncertainty is an operational state, not a reason to make the recovery more aggressive. The resolved rule is compact: trace the work, record active dependencies, detect cycles, require stale useful progress evidence, preserve human authority, change one bounded edge, and verify the intended outcome. That is enough to turn “the agents are busy but nothing is happening” into a diagnosis an operator can inspect. Sidewisp is currently in private preview. Its production monitoring adapters and recovery engine are not generally shipped. If a health view that separates working, waiting, stuck, uncertain, and needs human agents would help your operation, you can join the private preview without replacing your existing runtime.