OpenTelemetry LLM Observability: Pin the Schema Before Green

OpenTelemetry LLM observability is useful only if the telemetry has a declared meaning. Before treating a GenAI trace as operational evidence, pin the semantic convention revision, validate required fields for each operation, reject unapproved content capture, prove expected operations are present, and then evaluate freshness, work state, and the external outcome separately. That order matters in July 2026. The OpenTelemetry website now points GenAI conventions to a dedicated repository. At the source revision inspected for this article, 434c91dcc34ed038e3048c07720ddfed2c6bddfc , the repository's README still lists its schema URL as TODO, and the generative client span document is marked Development . This is not a reason to avoid OpenTelemetry. It is a reason to make compatibility explicit. A dashboard containing GenAI spans can still combine an old producer, a current collector, and a query written for a third attribute shape. The safe default is a small, versioned acceptance profile beside your collector configuration. Pin the contract you actually operate OpenTelemetry's GenAI repository covers LLM clients, agents, tool execution, evaluation, memory, retrieval, and MCP. Its docs are partly generated from YAML models, which is valuable because the source is inspectable and testable. It also means that "we use OTel" is too vague to be a compatibility claim. Record four identities with every instrumentation deployment: Identity Example Why it matters Convention source Repository plus commit SHA Defines the attribute and operation contract you reviewed Instrumentation package Package name and version Identifies what emitted the span Collector pipeline Config digest and deployment ID Identifies transforms, filters, and exporters Query contract Dashboard or alert version Identifies which fields the verdict expects Do not infer the convention revision from whatever fields happen to arrive. That turns silent drift into apparent compatibility. If a producer cannot declare its revision, classify the batch as schema drift until you test and register that producer shape. This is especially important when framework native and external instrumentation coexist. OpenTelemetry's agent observability guidance describes the maintenance trade offs of baked in instrumentation and warns external packages can collide with it. The same model call can otherwise be observed twice, or one route can remain on an older convention after a partial upgrade. The practical rule is simple: one expected telemetry producer per operation path, one declared convention revision per rollout cohort, and a canary trace that must pass before the cohort is promoted. Validate a small operational profile before the full schema A complete semantic convention validator can be generated from the repository models. An operational gate should start smaller. Enforce only fields that affect your current decisions, then expand the profile as you use more operations. At the pinned revision, the inference span table marks: gen ai.operation.name as required; gen ai.provider.name as required; error.type as conditionally required when the operation ends in an error; gen ai.request.model as conditionally required when available; input messages, output messages, system instructions, and tool definitions as opt in. Those requirement levels should not be flattened into "field present or absent." An error span without error.type has lost a class of evidence the convention expects. A missing request model may be legitimate when it was unavailable. Prompt and response content should remain absent unless an explicit policy permits collection. The companion audit implements this narrow profile: The content check looks only at attribute keys . It does not read or store prompt text, response text, system instructions, or tool arguments. That is enough to catch accidental opt in capture without turning the validator into another sensitive data sink. There is a deliberate limitation here: this profile is not the whole OpenTelemetry specification. It tests one reviewable contract used for an operational verdict. When the upstream source changes, update the pinned revision, compare the generated definitions, adjust the fixture, and rerun it before upgrading producers. Audit coverage before interpreting a clean trace A conformant span can still be incomplete evidence. If the application expected invoke agent , chat , and execute tool , but the trace contains only the first two, the right verdict is coverage gap , not healthy. Build expected operations from the workflow topology rather than from observed spans: This avoids a circular test in which the telemetry defines its own completeness. The expected set can come from a release manifest, registered tool route, or workflow definition. It should be small enough to maintain and specific enough to expose a missing instrumentation path. Automatic retries need care. The current client span prose says one logical span should cover the operation duration including automatic retries. Your application may also retain attempt level transport spans. Do not count those two layers as duplicate agent work. Decide whether coverage is expressed at the logical operation, attempt, or both, then make the relationship explicit. Likewise, a provider name is not necessarily the ultimate model owner. The convention notes that instrumentation may know a proxy or hosting platform rather than the transparent upstream provider. Treat gen ai.provider.name as a format discriminator and routing fact within its documented scope, not as a billing or model provenance oracle. Keep schema, health, and outcome as separate verdicts Once schema and coverage pass, the trace is eligible to inform a health decision. It is not the decision by itself. Use an explicit precedence: 1. Schema identity — does the producer match the pinned revision? 2. Schema validity — are required and conditional fields valid? 3. Content policy — are opt in fields allowed for this route? 4. Coverage — are all expected operations represented? 5. Freshness — is the evidence recent enough for the workflow? 6. Work state — is the agent working, waiting, stuck, uncertain, or complete? 7. Outcome — does the promised result exist at its destination? The ten case fixture for this article keeps those layers inconvenient on purpose. It includes: an unpinned producer and a legacy producer; an inference span missing gen ai.provider.name ; an error span missing error.type ; an opt in content key without permission; a workflow missing its expected tool span; a complete trace without an outcome receipt; a verified but stale trace; a legitimate human approval wait; one fresh, covered, outcome verified case. Run the artifact with: The executed result classified exactly ten cases: The complete no receipt case is the key boundary. Its producer revision matches. Its spans contain the fields required by the profile. The expected agent and model operations exist. The telemetry is fresh. It still returns unverified because no destination receipt proves the ticket, file, deployment, or other promised result exists. The waiting for approval case preserves a different boundary. Recent, conformant spans stop at a named human approval with an owner and deadline. That is not a stall. Paging an operator as if the agent had failed would destroy useful state information. Turn the audit into a release and incident control Run this test at three moments. Before an instrumentation upgrade , capture the current revision, package versions, collector digest, and expected operations. Replay the fixed fixture against the proposed stack. A changed verdict must be explained before promotion. During rollout , emit a content free canary through every registered model, agent, and tool route. Check that each canary arrives once, carries the expected producer identity, passes the operational profile, and remains searchable within the freshness window. During an incident , preserve the four layers instead of collapsing them into "observability is broken." Schema drift calls for producer and query reconciliation. A coverage gap calls for instrumentation repair. Stale telemetry calls for collector or exporter diagnosis. waiting calls for the named owner. unverified calls for a destination check, not another model retry. Do not automatically repair an agent because a telemetry field changed. Convention drift can make evidence uncertain without making the underlying work unhealthy. Freeze active recovery, identify the failed evidence layer, and use the smallest reversible test that restores confidence. The schema profile also needs an owner. Pinning a commit forever is not safety; it is eventual stagnation. Assign a review cadence, watch the GenAI repository, and require a fixture diff when moving the pin. If OpenTelemetry publishes a stable GenAI schema URL later, adopt it when your producers and queries support it, but keep coverage, freshness, waiting, and outcome checks independent. Sidewisp is currently in private preview. Its public experience is an early access website and interactive demonstration; production agent health collection, host adapters, and recovery are not shipped in the current website repository. The product direction is a health layer around existing runtimes, not an OpenTelemetry collector, tracing backend, or autonomous fixer. If revision pinned evidence and separate outcome verification match the failures you need to catch, you can join the Sidewisp private preview and describe the agent runtime and telemetry path you operate.