Grafana LLM Observability: Prove the Outcome Layer

Grafana LLM observability can give you a strong account of model calls, agent generations, traces, tool activity, latency, tokens, cost, and evaluations. It cannot, by itself, prove that an agent was reachable when expected, waited on the right person, applied an external effect exactly once, or produced the requested deliverable. The practical default is therefore two part: use Grafana for the telemetry it documents, then add a small operational evidence contract for the questions that telemetry does not answer. Test those parts separately. A conversation appearing in Grafana is not proof that traces and metrics arrived, and a clean trace is not proof that the destination changed. This guide implements that boundary against Grafana's current documentation and the @grafana/agento11y JavaScript package. It also includes a seven case audit that makes the acceptance rule executable rather than leaving it as dashboard advice. Start with the documented path, then test it in pieces Grafana currently exposes two related routes. Its general AI Observability setup sends OpenTelemetry traces, metrics, and logs through the Grafana Cloud OTLP gateway by default; an OpenTelemetry Collector or Grafana Alloy is the documented alternative when you need routing, transformation, or better control at higher volume. Its newer Agent Observability surface adds agent oriented generations, conversations, tool calls, workflow steps, token and cost data, evaluations, and framework integrations. For JavaScript, the current package is @grafana/agento11y . The npm latest tag returned version 0.9.0 when checked on July 27, 2026. Treat that as a dated implementation snapshot, not a permanent version recommendation. The shortest documented start looks like this: Then configure the SDK from environment variables rather than putting an access token in source: The relevant environment variable names are documented as AGENTO11Y ENDPOINT , AGENTO11Y PROTOCOL , AGENTO11Y AUTH MODE , AGENTO11Y AUTH TENANT ID , and AGENTO11Y AUTH TOKEN . Do not print their values in logs or attach them to traces. There is an easy to miss boundary in Grafana's JavaScript instrumentation guide: generation export and OpenTelemetry export are not the same path. The SDK can send generation data, but it needs an application configured TracerProvider and MeterProvider to export the spans and metrics it emits. Without those providers, the guide says the traces and metrics are silently lost. That makes one visible conversation a weak setup test. Use three independent probes instead: 1. Create a generation with a unique test ID and find it in Conversations . 2. Find a span carrying the same correlation ID in the traces data source. 3. Query one SDK emitted metric over the test window and confirm its resource attributes identify the expected service and environment. Fail setup if any probe is missing. Do not average the three into a comforting score: generation=yes, trace=no, metric=no is partial telemetry, not a mostly healthy installation. A complete Grafana trace is still activity evidence Grafana documents useful coverage. Agent Observability can capture generations, tool calls, workflow steps, latency, errors, tokens, costs, quality scores, and version comparisons. Those signals can answer questions such as: Did the LLM call fail or slow down? Which model and agent version handled the run? Which tools were invoked, and in what trace? How many tokens and how much attributed cost did the run consume? Did a configured evaluation or guard produce a score? The OpenTelemetry GenAI agent conventions give that activity a portable vocabulary. At the time of research, the agent conventions were explicitly marked development status and included operations for invoking an agent or workflow, planning, and executing a tool. Development status matters: pin the instrumentation version and expect attributes or span shapes to evolve. None of those span names defines your business outcome. An execute tool span can report a successful HTTP response while the provider applies the request asynchronously, rejects it after validation, or writes to the wrong object. A completed workflow span can coexist with a missing file. A quality evaluation can score generated text while a scheduled run never started. Use an evidence map that states both the proof and its boundary: Evidence plane It can establish It does not establish Release probe Generation export A recorded model generation reached Agent Observability Traces and metrics were exported Find a unique conversation ID Traces Instrumented operations and their causal path The external destination now has the intended state Query the correlated span Metrics Aggregated rate, duration, error, token, and cost signals One required run or deliverable succeeded Query the exact test window Evaluation A named scorer ran against supplied material A deterministic destination assertion passed Retain scorer version and input scope Runtime receipt The runtime was reachable and fresh at an expected time The task completed Compare heartbeat age with its SLA Wait receipt A pause has a reason, owner, deadline, and resume handle The owner will decide in time Verify routing and escalation Effect receipt An external operation can be reconciled The user's full outcome is present Read back by stable operation ID Outcome receipt A destination specific assertion passed Future stability Recheck during a stability window This is not an argument for uploading more content. Prefer IDs, timestamps, versions, low cardinality states, hashes, counts, and destination assertions over raw prompts, completions, tool payloads, secrets, or file paths. Rich telemetry and data minimization are compatible when the contract is designed before instrumentation. Run the seven case coverage audit I converted the map into a small deterministic fixture. Each case records whether generation, trace, and metric export succeeded; whether telemetry and the runtime heartbeat are fresh; whether the run has a legitimate wait; whether an external effect was reconciled; and whether the expected outcome has an authoritative receipt. The classifier applies precedence rather than arithmetic: The seven fixtures cover: no generation arrived; generation arrived but spans and metrics did not; all telemetry exists but is stale; the agent is legitimately waiting with an owner, deadline, and resume token; a tool attempt has no reconciled effect receipt; telemetry and tool evidence are green but the deliverable is missing; the same completed case has an authoritative deliverable receipt. The local replay passed all seven expected classifications: The most useful comparison is between the last two cases. Both have generation, trace, and metric export. Both are fresh. Both report the run completed and the tool effect verified. The first has no destination assertion and is classified false success ; the second adds object:report 17 plus a content hash and becomes verified . That change is deliberately narrow. No dashboard panel, token count, model score, or trace status changes. Only the evidence required by the user's request changes. The audit has a hard limitation: it trusts the facts supplied to it. A malicious or broken collector can lie, and a receipt from the wrong destination is not proof. In production, generate outcome receipts at the authoritative boundary—a storage read after write, a database constraint query, a deployment health probe, a sent message provider lookup, or another deterministic assertion tied to the original work ID. Turn coverage into a production acceptance gate Run one instrumented canary before enabling a new agent version, framework integration, collector route, or sampling change. Give the canary a unique work ID and a harmless expected outcome. Then require every applicable assertion: Generation: the generation exists under the canary ID. Trace: the root span and required child operations are queryable. Metric: the canary window contributes to the expected series. Freshness: collector delay stays below a declared limit. Runtime: a heartbeat or scheduler receipt proves the runtime was reachable when expected. Waiting: any pause names its reason, authorized owner, decision deadline, escalation route, and resume handle. Effect: side effecting operations reconcile against a stable idempotency or provider operation ID. Outcome: an authoritative destination check proves the requested artifact or state exists. Privacy: the test query confirms that prohibited prompt, response, secret, and path fields are absent. Keep the verdicts inconvenient. telemetry partial should block an instrumentation rollout. health unknown should prevent a green status when evidence is stale. waiting should notify the owner without restarting legitimate work. uncertain effect should stop blind retries. false success should reopen the task or incident even when the trace ended normally. Sampling needs a separate decision. Heavy traces can be sampled when volume requires it, but the compact health receipts needed to classify a required run should not disappear with them. Retain enough identifiers to correlate sampled traces with unsampled run, wait, effect, and outcome receipts. Otherwise a cheaper telemetry policy quietly becomes a weaker correctness policy. There is also an operating cost. Destination read backs add latency and provider calls; runtime heartbeats add storage and freshness checks; wait receipts need routing ownership. Apply the smallest deterministic check that resolves the decision. A file existence and hash check is better than asking another model whether the file probably exists. An LLM judge remains useful when the outcome is semantic, but record its version, rubric, input scope, and uncertainty beside deterministic checks. Where Sidewisp fits Grafana is a capable place to inspect and correlate telemetry. The missing layer described here is operational judgment across reachability, progress, waiting, effects, and outcomes—not another trace viewer. Sidewisp is intended to become a health layer around existing agent runtimes, with explicit evidence, freshness, uncertainty, approval boundaries, and verification. It is not a replacement runtime or mandatory model gateway. Production monitoring adapters and recovery are not shipped today. Sidewisp is currently in private preview. If this evidence boundary matches how you operate agents, the appropriate next step is to join the private preview waitlist and describe the runtime and outcome checks you need. Until then, keep Grafana's telemetry verdicts precise and attach completion to the destination evidence your work actually requires.