MCP Observability: Build a Five-Layer Health Contract

MCP observability should answer a harder question than “did the request return?” For each Model Context Protocol session, preserve one correlated evidence chain from authorization and initialization through tool discovery, invocation, external effect, and the user’s expected outcome. A green tools/call response is only one link in that chain. The reasonable default is a five layer health contract: 1. Session: did the client and server negotiate a supported protocol version and the capabilities this run will use? 2. Catalog: did the client read the complete current tool list and react to a later list change signal? 3. Request: can you join the JSON RPC request, progress notifications, cancellation, response, and deadline? 4. Effect: if the tool changes an external system, is there destination evidence for what actually happened? 5. Outcome: did the artifact, state change, or decision the agent was supposed to produce pass its verifier? This contract deliberately separates protocol activity from useful progress. It also gives an operator a concrete state to route: needs auth , incompatible session , stale catalog , working , protocol error , tool failed , effect unknown , false success , or healthy . Start with one evidence chain, not one dashboard number The fresh US results for “mcp observability” emphasize sessions, connections, tool analytics, transport latency, throughput, and errors. Those are useful. Grafana’s current MCP Observability documentation lists session establishment, connection stability, protocol compliance, tool performance, and transport reliability. The gap is not that these signals are wrong. The gap is that none of them, alone, proves that the agent’s intended work reached its destination. A health record can stay compact: The hashes are identifiers, not permission to upload tool schemas, arguments, prompts, tokens, or returned content. Keep sensitive values on the host. Record the smallest fields needed to correlate state and prove the decision. The contract should also have a precedence rule. Authorization failure comes before session health; an incompatible session comes before catalog freshness; an unresolved catalog comes before request interpretation; a transport timeout at a side effect boundary becomes effect unknown , not an automatic retry; and a successful call with a missing deliverable becomes false success , not healthy. Make protocol state observable before measuring tool latency The MCP lifecycle specification makes initialization the first client server interaction. The two sides agree on a protocol version, exchange capabilities, and then enter normal operation. The same document says subsequent communication must respect the negotiated version and capabilities. That gives observability a clean first checkpoint: store the requested version, accepted version, capability set, and the moment the client sent notifications/initialized . Do not collapse every failure before that point into “server down.” For HTTP transports, authorization is a separate gate. The current MCP authorization specification defines protected resource discovery and requires clients to handle a 401 Unauthorized challenge. A server can be reachable and correct while the client lacks a token, has the wrong scope, or cannot discover the authorization server. Route that as needs auth ; do not page it as a transport outage. Tool discovery needs its own completeness proof. The tools specification says tools/list is paginated and that servers declaring listChanged can emit notifications/tools/list changed . Therefore, “we received one page” is not a fresh catalog. Record: the initialization capability that advertised tools; every pagination cursor until no nextCursor remains; a canonical hash of names and input/output schemas; the last successful refresh time; any tools/list changed notification and the refresh that followed it. This is narrower than logging every schema. A client can calculate the hash locally and retain only the hash, tool count, page count, and refresh evidence. If a change notification arrives and the refresh fails, classify the session stale catalog . The server may still answer pings, but the model could be choosing from an obsolete tool contract. Long running calls introduce another trap. MCP progress notifications use a request provided token that must be unique among active requests; progress values must increase, and notifications stop after completion. That evidence can justify working while the call is inside its maximum deadline. It does not prove the work is useful, and it must never extend the deadline forever. A repeated message with no increasing value, a token attached to the wrong request, or progress after a terminal response is inconsistent evidence. Use two clocks: an inactivity window, which may reset on valid monotonic progress; an absolute maximum deadline, which does not reset. That distinction prevents two opposite errors: killing legitimate long work because it has not returned yet, and accepting an endless stream of progress notifications as health. A successful tool call is not the outcome MCP distinguishes protocol errors from tool execution errors. According to the tools specification, malformed requests and unknown tools use JSON RPC errors, while business or input failures can return a tool result with isError: true . Keep those states separate because the next action differs: fix the client contract for protocol error ; adjust inputs, permissions, or the downstream dependency for tool failed . The more dangerous case is no response after a side effect may have happened. Suppose publish report times out. Retrying immediately can create a duplicate report because transport uncertainty says nothing about the destination. Mark the call effect unknown , reconcile using a stable operation key or destination query, and retry only after evidence proves the first attempt did not commit. Even a normal result is not enough. The server can return isError: false while the promised file is absent, the remote record is still a draft, or the URL is private. Add an outcome verifier chosen before the call: file hash, database row plus version, public HTTP status, test result, or another deterministic receipt. Use an LLM judge only when the outcome cannot be checked directly, and label that weaker evidence. This creates three distinct terminal looking states: Protocol result Destination effect Expected outcome Health state timeout unknown unknown effect unknown success verified failed or missing false success success verified verified healthy The middle row matters most. It stops a successful protocol exchange from becoming a false claim that the agent completed the user’s task. Replay the contract against inconvenient cases The accompanying artifact is an illustrative nine case fixture and a deterministic classifier. It contains no production telemetry. Run it with: The fixture covers one healthy export and eight inconvenient boundaries: an authorization challenge, unsupported protocol version, unreconciled tool list change, long running call with valid progress, malformed call, tool execution error, timeout after a possible side effect, and success with a missing deliverable. The classifier returned one case in each expected state: This is the falsifiable part of the method: change the evidence and the state must change predictably. If tools/list changed is followed by a complete refresh, the stale catalog case should advance. If the timed out call gains a destination receipt but its deliverable verifier fails, it should move from effect unknown to false success . It reaches healthy only when session, catalog, request, effect, and outcome evidence all agree. The artifact does not validate whether a tool’s business semantics are correct. It does not discover an undocumented side effect, prove that an OAuth issuer is trustworthy, or decide how long your deadline should be. Those are deployment specific reviews. Its job is smaller: prevent missing evidence from being silently rendered green. Alert on the state that needs action Do not send every non healthy state to the same channel. needs auth goes to the credential or authorization owner with the challenged resource and required scope, never the token value. incompatible session goes to the integration owner with client version, server version, and negotiated capabilities. stale catalog triggers one bounded re discovery attempt; repeated failure becomes an integration incident. working stays quiet while progress is valid and the absolute deadline remains. protocol error goes to the client implementer with the method, request identifier, and sanitized error class. tool failed follows the tool’s retry policy only when the failure is known reversible. effect unknown blocks automatic retry at the side effect boundary and starts reconciliation. false success opens an outcome incident even though MCP itself completed. This routing keeps “waiting” distinct from “stuck.” An authorization prompt assigned to a person can be a legitimate wait; a progress token advancing within a bounded request can be work; an unchanged catalog after a list change signal is neither. Start with a single high value tool and one real outcome verifier. Capture the chain for a week, inspect every unknown state, then add coverage. A perfect fleet wide dashboard built on uncorrelated events is less useful than one tool call whose session, effect, and outcome can be explained end to end. Where this contract stops MCP observability can establish protocol and operational evidence, but it cannot infer all user intent from the wire. A valid output schema proves shape, not truth. A destination receipt proves an effect, not that the effect was wise. A monotonic progress signal proves movement reported by the server, not useful progress toward the user’s goal. Those boundaries require application checks and sometimes human judgment. Sidewisp is currently in private preview. Its production monitoring adapters and recovery systems are not generally shipped. The contract in this article is an operating method and an inspectable local artifact, not a claim that Sidewisp already collects MCP sessions or repairs them. The intended product direction is to make health evidence, uncertainty, and approval boundaries easier to see alongside existing runtimes. If you are designing an MCP integration now, keep the five layer record local, redact content, and refuse to call a run healthy until the expected outcome has its own receipt. That single rule turns protocol telemetry into an operational decision instead of another green chart.