Datadog LLM Observability Documentation: Audit Instrumentation Coverage

The useful way to read the Datadog LLM observability documentation is as a versioned instrumentation contract, not as proof that every important agent step is visible. Before trusting a trace, verify four things: your framework and tracer versions are supported together, exactly one setup path is active, sampling preserves the evidence required for health decisions, and every external effect has a deterministic outcome receipt. That distinction matters because an apparently complete trace can still hide a custom database write, an unsupported integration, a legitimate approval wait, or a missing deliverable. The audit below turns those gaps into eight explicit verdicts. It uses no prompts, responses, API keys, or customer data. Start with the support contract, not the dashboard Datadog's Agent Observability overview says that an application request appears as a trace and that spans represent choices or workflow steps. That is the right model for investigating latency, errors, token usage, and the path through an agent. It is not a promise that an arbitrary application is completely instrumented. The automatic instrumentation reference limits automatic tracing to supported frameworks and libraries. It explicitly directs operators to manual instrumentation for other API calls, database queries, and internal functions. Treat the documentation as four linked contracts: Contract Evidence to record Failure that a green trace can hide Compatibility runtime, framework version, tracer version, module mode unsupported code path emits partial or no spans Startup one enabled setup mode, site, application name, transport duplicate initialization or data sent to the wrong destination Coverage sampling policy and a manifest of operations that must be visible a required health event is sampled away or never instrumented Outcome wait receipt and destination specific completion receipt the agent reports completion but the requested effect is absent These contracts should be checked against a dated documentation snapshot. On July 30, 2026, Datadog's Python table listed LangGraph =0.2.23 with ddtrace =3.10.1 . The same page listed different minimums for other frameworks and languages. “We installed the latest package” is therefore weaker evidence than “this exact application/tracer pair meets the checked support row.” The SDK reference adds another boundary: Python command line setup with ddtrace run and in code setup with LLMObs.enable() are alternatives. Its in code section warns not to combine them. The reference also exposes DD LLMOBS SAMPLE RATE , which means trace retention is an operator decision rather than an intrinsic health guarantee. Run an eight case instrumentation audit The audit fixture fixes one support row—Python, LangGraph 0.2.23 , and ddtrace 3.10.1 —then changes one operational fact per case. Save the structure below as instrumentation cases.json and extend its cases array with the eight conditions described by the recorded output: Use this decision rule in audit datadog instrumentation.mjs : Run the audit: The recorded run evaluated eight fixtures and matched all eight expected verdicts: The order of the checks is deliberate. Compatibility comes first because a missing span from an unsupported pair should not be diagnosed as an application failure. Startup comes next because two setup paths create an ambiguous collection state. Coverage follows because a correctly initialized tracer can still omit required evidence. Waiting and outcome checks come last because they describe the work, not the telemetry transport. This is a configuration and evidence contract audit. It does not connect to a Datadog tenant or prove ingestion. After it passes, send a canary run and confirm that the expected root and child spans appear under the intended application, site, environment, and time window. Make sampling a coverage decision Datadog documents a configurable Agent Observability sample rate. Sampling is useful when full diagnostic traces are expensive, but it creates a strict operational consequence: absence of a sampled trace cannot prove absence of a run, a tool call, or a failure. Keep two evidence paths when health decisions must cover every run: 1. Diagnostic traces may be sampled. They retain rich span detail for investigation. 2. Mandatory health receipts remain compact and unsampled. They record run identity, state, freshness, wait ownership, expected destination, and outcome verification status. The fixture returns coverage gap when sampleRate is below 1 and no mandatory health ledger exists. It does not call the run failed. The correct conclusion is narrower: the available evidence cannot support an every run health claim. This also prevents a common alerting mistake. A missing sampled trace should not page an operator as an outage. First compare the unsampled run receipt, runtime heartbeat, and collection freshness. Escalate only when the evidence establishes a user relevant failure or remains unavailable past an explicit deadline. Add manual spans at effect boundaries Automatic instrumentation is a starting point, not a map of your business outcome. Suppose an agent calls a supported model and tool, then runs an internal function named write release manifest . The model and tool spans can finish normally while that last function fails silently. Create a small operation manifest before rollout: The audit returns instrumentation gap when a required operation lacks automatic or manual coverage. Do not repair this by adding spans to every helper function. Instrument boundaries that change the operator's decision: calls across services, durable writes, permission checks, approval transitions, retries with external effects, and destination verification. The strongest receipt should come from the destination. For a file, verify the expected path and digest. For a pull request, query the hosting service for the PR and required checks. For a message, retain the provider's accepted identifier and reconcile delivery when the workflow requires it. “Span ended without error” is activity evidence; it is not the same as “the requested result exists.” Preserve waiting instead of mislabeling it as stuck Agent traces often include long pauses. Some are failures; others are correct waits for a person, a provider, or a scheduled window. Duration alone cannot tell them apart. A legitimate wait needs a small receipt: Without an owner, deadline, and resumable identity, the fixture returns ambiguous wait . It does not immediately return stuck , because the evidence is insufficient. With a valid receipt, monitoring can remain quiet until the deadline, route the request to the right person, and later verify that work resumed. This distinction keeps operators from “recovering” healthy work by retrying it. An unsafe retry at an external effect boundary can create duplicate tickets, messages, payments, or deployments even when the original run was merely waiting for confirmation. Read the Datadog trace and the outcome receipt together Use the Datadog trace to answer: Did the expected framework and model calls appear? Which span failed, slowed down, or consumed unusual tokens? Did the trace tree preserve the expected parent child structure? Is the evidence fresh and associated with the intended application? Use the separate health receipt to answer: Was the run expected at this time? Is it working, waiting, stuck, uncertain, or complete? Did every required effect occur once? Does the destination contain the promised result? The final fixture is intentionally inconvenient: the trace path is supported, setup is clean, sampling is complete, and no operation is missing—yet the run claims complete without an outcome receipt. Its verdict is false green . That is the boundary to keep. Datadog can provide detailed trace, evaluation, latency, error, and token evidence. Your application must still define what success means and verify it where the result lives. Sidewisp applies the same health first distinction in its product direction: activity is not useful progress, waiting is not automatically stuck, and command completion is not a verified outcome. Sidewisp is currently in private preview. Its production monitoring adapters are not generally shipped today; the public site is an early access experience and product demonstration. The practical rollout sequence is short: pin the documentation date, record the support row, choose one startup path, declare the sampling boundary, list required effect operations, replay the eight fixtures, then send a live canary and verify its destination. Only after all of those receipts agree should a green trace become a green agent health verdict.