AI Agent Observability for Tool Failures: A Five-Layer Test

An AI agent can fail to use a tool even when its model is responsive, its process is alive, and the tool call appears in a trace. The practical default for AI agent observability is therefore to test a tool call in five ordered layers: transport, protocol, authorization, execution, and outcome. Stop at the first failed layer. That rule turns an ambiguous “tool unavailable” alert into a bounded repair, and it prevents a green response envelope from being mistaken for delivered work. This guide applies the rule to MCP style tools over HTTP and JSON RPC, but the diagnostic shape also works for custom REST tools and local command adapters. The goal is not to collect every prompt or payload. It is to retain the smallest evidence needed to answer: where did the call stop, what authority is required, and did the promised result reach its destination? Start with the first failed layer A single tool call failed counter collapses failures that require incompatible responses. A DNS error may justify a bounded connectivity retry. An expired token may justify one refresh. Missing scope requires a person or administrator; retrying the same token is waste. A valid tool response with no file, ticket, message, or database change needs outcome investigation, not transport repair. Use this precedence: Layer Minimum evidence Failure examples Reasonable next move Transport connection result, HTTP status, elapsed time DNS failure, refused connection, HTTP 503 check reachability; retry only within a fixed budget Protocol request ID, method, JSON RPC error code 32601 method missing, 32602 invalid params refresh discovery or fix the request contract Authorization HTTP status, sanitized auth error, required scope 401 invalid token , 403 insufficient scope refresh once or request the missing authority Execution tool result status, timeout, output schema verdict MCP isError: true , timeout, malformed structured output inspect the tool implementation or input Outcome destination verifier and freshness response says “created,” but the artifact is absent verify the destination; do not declare completion The order matters. If DNS resolution failed, authorization and outcome are unobserved , not failed. Emitting five failures for one early break inflates incident counts and sends responders toward evidence that never existed. Keep protocol failure separate from tool failure MCP tool invocation uses tools/call , while the tool definition carries an inputSchema and may carry an outputSchema . The current MCP Tools specification also shows a tool result with isError: false . Those are distinct checkpoints: the client can reach the server, exchange a valid JSON RPC response, and still receive a tool level failure. JSON RPC makes the outer distinction explicit. Its 2.0 specification reserves 32601 for “Method not found” and 32602 for “Invalid params”; an error response contains error , while a successful response contains result . A JSON RPC result only proves that the protocol exchange completed. It does not prove that the tool accepted the operation, that its structured output matches the advertised schema, or that the external side effect exists. Record the boundary without storing sensitive arguments: This event deliberately omits the bearer token, tool arguments, response body, ticket text, and absolute paths. Hash or map identifiers when cross run joins are needed. A trace ID is useful only if the health record can still explain the first failed layer and the outcome verdict when the raw trace is unavailable. Do not retry an authority problem as if it were packet loss Authorization deserves its own layer because 401 and 403 imply different actions. The MCP authorization specification requires clients to handle 401 Unauthorized and describes protected resource discovery through WWW Authenticate . It also recommends scope guidance so a client can learn the authority required for the current request. RFC 6750 defines invalid token for an expired, revoked, malformed, or otherwise invalid bearer token and associates it with HTTP 401. It defines insufficient scope for a token that lacks required privileges and associates it with HTTP 403. That gives an operator a safe decision rule: 1. For invalid token , attempt the configured refresh path once. If the refreshed credential fails, stop and surface the credential owner. 2. For insufficient scope , do not loop. Show the required scope if the server provides it and request explicit authority. 3. Never put the token, refresh token, authorization header, or raw challenge into general purpose telemetry. This distinction also prevents a damaging automation pattern: broadening permissions whenever a tool call fails. A connectivity incident must not become a privilege escalation, and a scope denial must not be “fixed” by silently switching to a more powerful credential. Reproduce the false success gap The accompanying fixture contains eight synthetic calls: two transport failures, one JSON RPC method error, two authorization failures, one tool execution error, one false success, and one verified delivery. Run the classifier from the artifact directory: The decisive output is: Three calls returned a result envelope, but only one produced a destination verified outcome. One envelope carried an execution error; another claimed success while its promised artifact was absent. Counting protocol results would report a 37.5% success rate. Counting verified outcomes reports 12.5%. The difference is not a detector score or an LLM judgment: it comes from changing the completion criterion. The fixture is intentionally deterministic. Real systems add ambiguity: a ticket API can commit a record and time out before returning its ID; a destination lookup can be stale; an idempotency key may allow a safe reconciliation query. Mark those cases uncertain . Do not retry a side effecting call until you know whether the first attempt committed. Turn the evidence into an operating rule Instrument one health event per attempted tool operation, linked to the owning run. Preserve the first failed layer, the sanitized code, evidence freshness, retry owner, and outcome verifier. Then apply four controls: Alert on grouped incidents, not every attempt. Five calls failing on the same expired credential are one authority incident. Cap retries by layer. Transport failures may receive bounded backoff; protocol and scope failures usually need a contract or human change. Distinguish waiting from stuck. A call waiting for an approved OAuth flow is not making progress, but it is not an execution loop. Clear the incident only after the failed layer passes and the intended outcome is observed. A successful retry command is activity, not recovery. The useful dashboard row is consequently small: affected agent, first failed layer, impact, evidence time, confidence, retry count, required authority, and verifier result. Raw traces can remain a drill down. This is operational agent health, not a requirement to replace the runtime or route every model request through a new gateway. There is also a hard limitation. Not every outcome has a deterministic verifier. A file can be checked by path and digest; a ticket by stable ID; a deployment by health endpoint and revision. “The research is good” may require a rubric or human review. Label the method and confidence beside the verdict instead of converting missing evidence into healthy. Sidewisp is currently in private preview. Its production monitoring adapters and recovery execution are not generally shipped. The planned direction is a health layer alongside existing runtimes that separates reachability, progress, tool access, and verified outcomes while keeping humans in control. If that operating model matches your agents, join the private preview. Sources Model Context Protocol: Tools, specification version 2025 11 25 Model Context Protocol: Authorization, specification version 2025 11 25 JSON RPC 2.0 Specification RFC 6750, OAuth 2.0 Bearer Token Usage