LLM Monitoring: What to Measure Beyond Latency, Errors, and Tokens

LLM monitoring should start with model call health: latency, errors, request volume, tokens, and output quality. That is the reasonable default for a chat feature, retrieval pipeline, or API wrapper. Once the same application can plan, call tools, wait for approval, resume later, or declare a task complete, add a second ledger for operational health. The two ledgers answer different questions. The first asks, “Did the model service behave normally?” The second asks, “Did the agent make useful progress and produce the expected result?” Join them with a run id ; do not turn one green latency chart into a claim that the work is healthy. Start with the default LLM monitoring layer A useful first dashboard does not need dozens of panels. It needs enough evidence to separate provider failure, application failure, cost drift, and output quality drift. Signal Question it answers Practical first alert What it cannot prove Request error rate Are model calls failing? Rate over a window, split by provider and model Whether successful calls advanced the task p50/p95 latency Did response time regress? Compare like for like operations and models Whether a slow run eventually delivered Input/output tokens Did context or generation grow? Change from a task specific baseline Whether the extra tokens were useful Throughput Is load changing? Requests per minute plus concurrency Whether a scheduled run was missed Quality score Did sampled output meet a rubric? Versioned evaluator plus human calibration Whether a file, ticket, or deployment exists Trace coverage Can an operator reconstruct execution? Missing or incomplete traces by runtime Whether the intended outcome is present This baseline matches the current search intent. Langfuse describes monitoring around latency, throughput, and error rates, with traces for execution paths and evaluation for output quality. Splunk and Dynatrace expand the set to resource, safety, cost, feedback, and application signals. Those are useful views of an LLM application; none should be discarded merely because an agent layer exists. OpenTelemetry's GenAI semantic conventions make the separation visible in the instrumentation itself. The development specification defines gen ai.client.token.usage and gen ai.client.operation.duration , then separate workflow and agent instruments such as gen ai.invoke agent.duration , inference call count, and tool call count. “Development” matters here: pin the version you implement and expect names to change. The clean implementation is a model call ledger keyed by run id , operation , provider , model , and timestamp. Aggregate it for service level alerts, but retain a path back to the individual run. A token spike without a run identifier is a bill; a token spike attached to a stalled run is an incident clue. Add a task health ledger when the application becomes an agent An LLM backed feature crosses the operational boundary when it owns work over time. It might call a database, write a report, open a pull request, wait for a person, or wake on a schedule. At that point, successful model responses are only intermediate events. The OpenAI Agents SDK illustrates how rich those events can become. Its built in tracing records generations, function tool calls, handoffs, guardrails, and agent runs. That is valuable debugging evidence. The same documentation also notes that generation and function spans may contain sensitive inputs and outputs, which is a reason to make content capture an explicit choice rather than a monitoring prerequisite. A task health ledger can stay smaller than the trace. For each run, record: expected outcome : a predicate such as report exists and parses , not the phrase “finish the task”; outcome verified : true , false , or unavailable , with verifier version; progress delta : a task specific count or digest change over a declared window; waiting on : a named dependency such as human approval , or null ; last heartbeat at and last progress at , because activity and progress are different clocks; declared complete : what the runtime reported; collector freshness : when these facts were last observed. This ledger deliberately does not repeat every prompt, completion, or span. It stores the minimum evidence needed to decide whether the run is working, waiting, stuck, unreachable, or complete. Raw traces remain available for investigation when policy permits. The important modeling choice is tri state evidence. If a collector cannot check the deliverable, record outcome verified: "unavailable" . Do not convert missing evidence into true , and do not call an unknown run failed merely because its signal is absent. Reproduce the gap with six runs The accompanying fixture contains six synthetic runs. The LLM ledger alerts when request errors are at least 20%, p95 latency exceeds 5,000 ms, or token use exceeds 20,000. The task health ledger checks explicit waiting, declared completion against a deterministic outcome, and progress freshness. Run the audit with Node.js 20 or newer: The exact result is: The disagreement is the result, not a defect in either ledger. The provider error run needs model service investigation even though the agent is still making progress. The slow run completed with a verified outcome, so it is a performance problem rather than a missing work incident. The approval wait and false success run look normal to the LLM monitor because their calls were fast, cheap, and successful. Only the task contract exposes what needs attention. The numbers are a counterexample, not a benchmark. Six synthetic records cannot establish universal alert thresholds. Replace the cutoffs with baselines from your runtime, and replace progress delta with evidence tied to the actual job. Turn the task contract into alerts Begin with one high value workflow. Write its completion predicate before adding another dashboard. A research job might require a Markdown file, at least two reachable sources, and a schema valid evidence ledger. A coding job might require a clean patch plus a named test command. A support agent might require a created ticket or a recorded escalation. Then evaluate signals in an order that preserves meaning: 1. If the runtime or collector is stale, mark the run unreachable or uncertain. 2. If waiting on is explicit, route the dependency instead of restarting the run. 3. If the runtime declares completion, evaluate the outcome predicate. 4. If the run is active but progress delta stays zero beyond its window, mark it stuck. 5. If none applies and useful progress is fresh, leave it working. This order prevents three noisy interventions. A legitimate approval wait is not a stall. A command that exited zero is not automatically a completed task. A busy trace with repeated tools is not progress if the relevant artifact never changes. Alert on the next safe action, not just the symptom. A provider error alert goes to the application owner with model, operation, error class, and trace link. An approval wait goes to the person who can decide, with the exact scope requested. A missing outcome alert points to the failed predicate. A retry loop recommends a bounded pause or investigation; it should not authorize an irreversible fix. Keep the join useful without collecting everything Use one opaque run id across both ledgers. Do not put customer text, secrets, absolute paths, or tool payloads in that identifier. A useful correlation record can contain: Keep retention and access rules different if the data risks differ. Aggregated latency and token histograms may need longer retention than prompt bearing spans. Outcome evidence can often be a digest, count, status code, or schema result rather than the artifact itself. When an operator drills into a trace, show its freshness and sampling boundary so absence is not mistaken for proof. There is also a boundary to automated evaluation. Deterministic checks are preferable for files, HTTP status, database rows, tests, and structured fields. If the intended outcome is qualitative, a versioned evaluator can help, but its score is evidence with uncertainty—not ground truth. Calibrate it against human review and preserve an unavailable state. Where Sidewisp fits Sidewisp's product direction is the second ledger: a health view around existing agent runtimes, with evidence, freshness, issue priority, and explicit approval boundaries. It is not meant to replace the runtime, model gateway, or raw tracing system. That is direction, not a shipped monitoring claim. Sidewisp is currently in private preview. The public site and article system are live, while production agent health collection, runtime adapters, and recovery execution are not generally shipped. Join the private preview if this model call versus outcome boundary matches the operational problem you need to solve. Primary sources OpenTelemetry GenAI metrics semantic conventions — development status metric names for client, workflow, agent, and tool operations. OpenAI Agents SDK tracing guide — traced event types, export behavior, and sensitive data controls. Langfuse: What is LLM observability and monitoring? — a same intent benchmark for latency, throughput, errors, tracing, and evaluation.