Splunk LLM Observability: Keep Evaluator Calls Out of Agent Health

Splunk LLM observability can show useful performance, quality, token, estimated cost, and trace evidence for an instrumented agent. The safe operational default, however, is not “the AI Agents page is populated, therefore the agent is healthy.” First prove that the measurement pipeline is complete, that evaluator calls are not being counted as application work, that evaluation coverage has a known denominator, and that the requested result exists outside the trace. This guide builds that proof without collecting prompts or responses. The included eight case fixture produces a content free receipt and routes each run to one of these states: ingestion incomplete, metric contract mismatch, evaluator self observed, high cardinality risk, content policy review, evaluation coverage dropped, observable but unverified, or healthy with covered and verified evidence. Audit the measurement path before reading the score Splunk's current setup documentation makes two telemetry details operationally important. First, histogram metrics are required for the AI Agent Monitoring pages. When the SignalFx exporter is used, the documented Collector setting is send otlp histograms: true . The setup also specifies delta temporality through: A visible trace does not prove that this metric path is correct: spans and histograms can fail independently. Second, the Python AI instrumentation can run evaluations in the same process as the application. Splunk documents this switch, whose default is false: In that default mode, LLM calls made by evaluators such as DeepEval can be instrumented alongside application calls. Setting it to true runs evaluations in a child process with the OpenTelemetry SDK deactivated, preventing evaluator calls from polluting application telemetry. Splunk marks this isolation as required for OpenAI instrumentation when evaluations are enabled and optional for other documented frameworks. That distinction changes the meaning of a chart. Suppose one agent invocation calls a model twice, then one evaluator makes three more model calls. If the evaluator shares the instrumented process, a naive aggregate can report five calls, their combined tokens, and their combined latency. The extra activity is real compute, but it is not evidence that the agent made more progress. It is measurement work observing measurement work. Record a content free receipt per deployment: None of those fields needs a prompt, response, tool argument, secret, or customer identifier. They describe the health of the evidence pipeline. The first three checks answer different questions: histogramsExported : did the Collector export the histogram telemetry required by the AI monitoring pages? deltaTemporality : does the metrics contract match the documented configuration? aiSpanVisible : did at least one fresh GenAI span reach the expected Splunk view? Do not collapse these into one telemetry ok boolean. If spans arrive but histograms do not, trace investigation may work while aggregate panels remain incomplete. If the data is old, a populated page can still be stale. Keep the evidence source and observation time beside the receipt in a real implementation. Give every quality score a coverage denominator Splunk describes an AI agent quality score as the percentage of evaluations that passed for a metric. Its AI Agents documentation says spans are sampled to calculate those scores, and a score below 80% flags a quality issue. That can be a useful rule for the evaluated sample. It is not, by itself, proof that every eligible invocation was evaluated or that the sample represents every task type. Track four numbers together: 1. eligible application spans; 2. configured evaluation sample rate; 3. completed evaluation results; 4. evaluation queue drops. For a deterministic audit window, calculate: With 400 eligible spans, a 0.25 sample rate, 100 evaluations, and zero drops, the observed coverage is 25% and matches the configured expectation. That does not mean the other 300 spans passed. It means their evaluation state is outside the sample. Splunk's Python configuration also exposes an evaluation queue size. A positive bound applies backpressure; when the queue is full, new items are dropped with a warning. The documentation recommends a bound in the 100–1000 range depending on throughput and memory, while zero or unset leaves the queue unbounded. Either choice has a trade off: an unbounded queue can turn evaluation lag into memory pressure; a bounded queue can preserve the process but reduce evaluation coverage; a queue drop counter of six means 94 completed evaluations cannot honestly stand in for 100 eligible evaluations at a 100% sample rate. The audit therefore returns EVALUATION COVERAGE DROPPED , not healthy and not failed agent. The agent may have completed useful work; the evidence needed for the quality verdict is incomplete. Metric dimensions need a similar boundary. Splunk allows GenAI context attributes to be copied into metric dimensions, but explicitly warns that gen ai.conversation.id may cause high cardinality issues. Keep per conversation identity on spans when it is needed for diagnosis. Do not automatically turn it into a metric dimension. A low cardinality deployment environment or tenant tier is usually safer for aggregation; the correct set still depends on traffic and tenant limits. Keep content capture an explicit exception Splunk's LLM service documentation says prompt and response collection is off by default and warns that the content can contain sensitive or personally identifiable information. Its setup path also notes that large captured inputs and outputs can exceed backend limits and cause performance problems. This audit does not need content. It can verify histogram export, temporality, process isolation, sampling, drops, dimensions, trace visibility, and a destination receipt using metadata alone. If a separate quality investigation genuinely requires captured content, route it through a content policy review: identify the exact evaluator that needs content; state whether capture occurs on spans, events, or both; document retention, access, masking, and deletion; test payload size behavior; verify that tool definitions are not captured merely because message capture was enabled; disable capture after the bounded investigation if continuous collection is not justified. The fixture returns CONTENT POLICY REVIEW when capture is enabled without an approval receipt. That verdict is deliberately neither healthy nor broken. It says the instrumentation has crossed a data boundary that the operational health check cannot authorize. The same restraint applies to estimated cost. Splunk states that its agent cost estimate multiplies published provider cost by available token counts and does not represent actual billing. Label it estimated, retain the pricing date, and do not reconcile it silently with an invoice or a provider specific cache discount. Run the eight case evidence audit The reproducible artifact uses one precedence rule. Earlier findings block later green states: Run the saved fixture: It replays eight cases and returns eight distinct results: covered and verified — HEALTHY COVERED VERIFIED : Required telemetry, declared sample, zero drops, and outcome agree. evaluator self observed — EVALUATOR SELF OBSERVED : Judge calls can enter application telemetry. histograms missing — INGESTION INCOMPLETE : A trace does not prove the required metrics arrived. wrong temporality — METRIC CONTRACT MISMATCH : The exported metric contract differs from the documented setup. conversation id as metric — HIGH CARDINALITY RISK : Per conversation identity was promoted into a metric dimension. content capture unreviewed — CONTENT POLICY REVIEW : A sensitive data boundary was crossed without a receipt. evaluation queue dropped — EVALUATION COVERAGE DROPPED : 94 results cannot represent an expected 100. trace without outcome — OBSERVABLE NOT VERIFIED : Execution evidence exists, but the promised result does not. This is a synthetic configuration audit, not a live Splunk conformance test. It cannot prove a Collector is reachable, a role includes the required capability, retention covers an incident window, or a destination contains the expected deliverable. Replace the fixture values with observations from your environment and preserve unknown when a value cannot be measured. Stop the trace before the health verdict Splunk's trace and AI interaction views answer important questions about model operations, errors, token use, latency, and evaluated response quality. An agent health verdict has one more boundary: did the requested work happen? Choose the strongest deterministic destination check available: a file exists at the expected path and matches a schema or hash; a pull request exists at the expected repository and commit; a message exists at the intended destination with the expected idempotency key; a database mutation is visible under the intended tenant and operation id; a test suite passed against the produced artifact; a human approval is still pending, so the run is waiting , not failed. Join that receipt to the trace with a privacy minimal run identifier. Keep the deliverable content at its source. A successful span plus a missing receipt is OBSERVABLE NOT VERIFIED ; it is not automatic permission to retry, because the external effect might exist but be temporarily unreadable. The practical rule is simple: trust the Splunk view after its own measurement path passes, interpret quality scores within their known coverage, and clear agent health only when the intended outcome is verified separately. Sidewisp follows the same evidence first product direction: distinguish activity from useful progress, expose missing evidence, and keep outcome verification separate from trace completion. Sidewisp is currently in private preview. Its production monitoring adapters and recovery engine are not presented here as generally available; the public site is an early access experience and product demonstration. Sources Set up AI Agent Monitoring, Splunk Observability Cloud documentation. Configure the Python agent for AI applications 0.1.14 and higher, Splunk Observability Cloud documentation. Monitor AI agents, last updated June 16, 2026. Monitor LLM services, last updated May 12, 2026.