Elastic LLM Observability: Audit Support Before Green

Elastic LLM observability can tell you a great deal about model calls, but a green Kibana view is not yet an agent health verdict. Before trusting it, verify five layers in order: the collection path, support for the exact language/provider pair, telemetry freshness, required GenAI fields, and the destination outcome. That order matters. Elastic documents two collection methods: provider integrations for metrics and logs, and application tracing through the Elastic Distributions of OpenTelemetry (EDOT). Those methods have different coverage. Even a supported, error free LLM span does not prove that application specific business logic ran or that the promised deliverable exists. This guide turns those boundaries into a small audit you can run before clearing an incident. Identify the collection path before reading the dashboard Elastic's LLM and agentic AI observability overview describes a broad set of provider integrations, APM traces, metrics, logs, and dashboards. The first operational mistake is compressing all of that into one capability called “Elastic monitoring.” Keep two collection planes separate: Plane What produces the evidence Useful for What it does not establish Provider integration A provider or cloud service sends metrics and logs Provider errors, latency, usage, guardrail events, platform health That your application emitted an LLM span or completed its own tool effect EDOT application tracing An instrumented Java, Node.js, or Python process exports OTLP spans Request flow, model calls, duration, errors, token fields, correlation That unsupported libraries were instrumented or the external deliverable exists This is not a product weakness. It is an evidence boundary. A Bedrock integration can be delivering fresh service metrics while a Java application has no documented EDOT Bedrock LLM instrumentation. Conversely, an OpenAI client span can be complete while a later custom database write is uninstrumented. Elastic's current EDOT LLM support page makes the language/provider boundary concrete: Provider path EDOT Java EDOT Node.js EDOT Python : : : OpenAI client Supported Supported Supported AWS Bedrock Not listed Not listed Supported Google Vertex AI Not listed Not listed Supported The page labels LLM observability in the three EDOT distributions as tech preview and directs operators to SDK specific pages for exact versions. Treat that table as a dated support snapshot, not an eternal capability promise. The audit therefore starts with two questions that a dashboard cannot answer for you: 1. Which plane is expected to contain the evidence for this incident? 2. Does the deployed language, provider, client package, and version have documented instrumentation on that plane? If the answer to the second question is no, do not wait for a missing span to appear. Classify the path as unsupported, choose documented native or manual OpenTelemetry instrumentation, or change the evidence requirement. “No errors in Elastic” is not meaningful when the relevant event was never expected to be captured. Test support and schema as separate gates Supported does not mean observed, and observed does not mean complete. The EDOT Python technology table documents Python versions, client package ranges, tracer names, and semantic convention status. At the time of this review, its OpenAI instrumentation row labels the semantic conventions as development . The same page explicitly says automatic instrumentation cannot cover custom or proprietary frameworks, closed source unsupported components, or application specific business logic. That gives us four distinct checks: 1. Support: the documented matrix includes the language/provider pair. 2. Reachability: the collector and ingest path accept current telemetry. 3. Presence: the run produces the expected LLM span. 4. Schema: the span contains the fields required for the operator's decision. A minimal field contract might require: Do not mistake this example for a universal schema. Pin the semantic convention and instrumentation versions used by your deployment. OpenTelemetry's former GenAI convention pages now point to a dedicated GenAI semantic conventions repository, which is another reason to record provenance rather than assuming that an attribute set is timeless. The classifier below preserves the important failure states: Run the audit against a fixture rather than testing only the happy path: The accompanying nine case fixture produced nine expected verdicts: This precedence prevents a common monitoring error: letting a later green signal hide an earlier evidence gap. A successful span cannot override an unreachable collector check, and a completed run cannot override a missing destination receipt. Preserve waiting, missing evidence, and failure as different states An approval pause is not an instrumentation failure. A missing span is not automatically a provider failure. An unsupported path is not stale telemetry. Those distinctions change the operator's next move: Verdict Meaning Bounded next action UNSUPPORTED PATH The expected automatic LLM instrumentation is outside the documented matrix Add documented native/manual spans or change the evidence contract TELEMETRY STALE Relevant evidence exists, but not within the run's freshness window Check export, collector, ingest, clock, and query window INSTRUMENTATION GAP The path is supported and fresh telemetry arrives, but the LLM span is absent Verify package range, bootstrap, disabled instrumentation, and tracer identity SCHEMA GAP The span exists but cannot answer the required question Check convention version and field mapping; report the field as unavailable meanwhile WAITING A named dependency or approval is outstanding Notify the recorded owner; do not retry the tool blindly FALSE COMPLETE Elastic shows clean completion but the promised result is unverified Run a deterministic destination check before closing the incident Notice what the table does not recommend: treating every gap as a reason to restart the agent. Recovery without diagnosis can duplicate external effects, spend more tokens, or erase useful evidence. For legitimate waiting, retain an owner, reason, start time, deadline, and resume condition. That transforms an ambiguous pause into an inspectable operational state. If the deadline passes, the state may become stuck or need human attention, but the original pause was not a failure merely because no new spans arrived. Require an outcome receipt outside the trace An LLM span answers a model call question. The deliverable belongs to the application. Suppose an agent asks a model to prepare an invoice, calls an internal API, and reports completion. Elastic may show: a fresh trace; the expected provider and model; no exception; plausible latency and token counts; a completed root transaction. The invoice can still be absent. The custom API may have accepted the request without committing it, an asynchronous worker may have failed, or the agent may have skipped the tool and produced only a textual claim. Define the smallest deterministic receipt that proves the promised effect. Examples include: the expected object exists at the destination and matches a content hash; a database row has the intended business key and committed state; a pull request exists at the expected repository and head SHA; a report endpoint returns the new version and passes schema validation; a message provider returns a delivery identifier that can be reconciled later. Store only the minimum safe evidence: The trace ID provides correlation. It is not the proof itself. In the fixture, the only difference between FALSE COMPLETE and HEALTHY is outcomeVerified: true ; none of the Elastic span fields change. That is the central operating rule: clear an incident only when telemetry coverage and application outcome evidence agree. Treat privacy and version drift as part of health Elastic's overview says LLM tracing can capture prompts and responses. That can be useful for diagnosis, but it also changes the data boundary. Decide explicitly whether content is permitted before enabling it. Prefer identifiers, lengths, hashes, classifications, token counts, and redacted error categories when full content is not required. Record these values with every coverage audit: Elastic deployment and EDOT distribution version; language runtime and instrumented client package version; active instrumentation package and tracer name; semantic convention source and revision; collector and ingest path; required fields and freshness window; content capture policy; destination verifier version. Re run the fixture when any of those values change. A package upgrade can add support, rename or migrate fields, or alter default instrumentation. A dashboard saved object can remain green while its assumptions quietly become stale. The practical default is modest: use Elastic for the model and application evidence it actually collects, keep unsupported or missing signals explicit, and add one deterministic receipt for the result the user cares about. That produces a defensible health decision without pretending one dashboard owns every layer. Sidewisp is currently in private preview. Its product direction is to turn evidence like reachability, progress, tool access, context, cost, and verified outcomes into a clear health view; live Elastic monitoring adapters are not currently shipped. If this distinction between trace completion and real work matters in your agent stack, the private preview waitlist is the appropriate place to share the failure case you need covered.