Datadog LLM Observability Experiments: Add a Promotion Gate

Datadog LLM Observability Experiments can tell you that a candidate prompt, model, provider, or agent architecture scored better and ran faster. That is useful evidence, but it is not yet a release decision. The reasonable default is to compare the candidate and baseline inside the same Datadog project, pin the dataset version, keep evaluator definitions stable, and inspect the distribution and traces rather than trusting one average. Then add a small promotion gate outside the scorecard: prove that every required record ran, expected waits remained waits, mutating cases produced exactly one effect, and completed cases have a destination receipt. This article implements that last step with an eight record fixture. The candidate moves evaluator pass rate from 75% to 100% and mean duration from 980 ms to 738.75 ms . Promotion is still blocked. One refund has no verified destination receipt, and one cancellation produced two effects. First prove that the two experiments are comparable Datadog's Experiments overview describes three core operations: version datasets, run experiments, and compare results. The current setup documentation defines an experiment as a task executed sequentially across dataset records, with record evaluators and optional summary evaluators. Task internals can use the same tracing decorators as production code. Those primitives give you good material for a comparison. They do not remove the need to define what must remain identical between the two runs. Before looking at scores, record a content free manifest: The hashes identify evaluator code; they are not hashes of prompts, outputs, credentials, or customer data. Hash the canonical JSON serialization of this manifest and attach the fingerprint to both experiment records. The fixture produces: If the fingerprints differ, stop calling the result a paired comparison. A changed dataset version may add hard cases. A changed evaluator may move the threshold. A missing record may improve an average by removing the failure that matters. Those changes can be legitimate, but they require a new baseline. Datadog's product walkthrough says datasets support version control and can be pinned to specific versions. It also explains that the comparison surface exposes averages, distributions, latency, cost, token counts, outputs, and span traces. Use all of that. The manifest does not replace Datadog's evidence; it prevents accidental comparison drift before you interpret it. Run a gate that can disagree with the averages The inspectable artifact for this article is a standard library Python script. It contains eight synthetic, content free records and emits JSON. Run it with: The core rule is deliberately small: This does not ask a second model whether the first model looked successful. It checks explicit fields whose meaning you control. The complete fixture yields: Measure Baseline Candidate : : Records present 8/8 8/8 Evaluator pass rate 75% 100% Mean duration 980 ms 738.75 ms Blocking records — 2 Promotion decision — Block The Datadog evaluation developer guide supports typed evaluator results, an optional pass/fail assessment, metadata, tags, and summary evaluators. That makes an evaluator a suitable source for evaluatorPass . It does not make every evaluator a destination verifier. If your evaluator compares generated text with an expected answer, it can pass while a refund never reaches the ledger. Keep these evidence lanes separate: Evaluator lane: Did the output satisfy the chosen quality rule? Trace lane: Which LLM, retrieval, tool, and task steps ran, and with what errors or latency? State lane: Was the case working, legitimately waiting, complete, uncertain, or in need of a person? Effect lane: Did the intended external mutation occur exactly once? Outcome lane: Does the destination now contain the promised result? The first two lanes are natural inputs from Datadog Experiments. The remaining lanes come from your task contract, fixtures, and destination systems. You can submit their results as custom evaluations or experiment metadata, but define them at the boundary that owns the truth. The two blocked records reveal different failures The issue refund record has a passing evaluator, a complete state, and one attempted effect. Its outcomeReceipt is missing . This is not evidence that the refund failed; it is evidence that completion is unverified. The safe verdict is uncertain , not green and not an automatic retry. The cancel subscription record has a passing evaluator and a verified destination receipt, but effectCount is two. A successful final state does not erase the duplicate effect. The candidate is blocked because the change weakened effect safety even while improving the aggregate score. The approval required record demonstrates the opposite boundary. Its expected state is waiting , and the candidate supplies an owner, deadline, and resume token. It passes. A pause with a known dependency is not a stall, so a gate that requires every row to end in complete would punish correct behavior. These cases are intentionally inconvenient. If a promotion gate only repeats the evaluator score, it cannot change the release decision. A useful gate must be able to say: the comparison is invalid because record coverage changed; the candidate is better on quality but unsafe on effects; the candidate is faster but left an outcome unknown; the wait is valid and should not count as failure; the evidence conflicts, so a person must decide. Attach the gate to a real Datadog experiment Datadog's current setup guide requires ddtrace =4.3.0 for the documented Python Experiments path and requires both an API key and application key. Keep those secrets in environment variables; do not put them in a manifest, dataset row, article artifact, or trace attribute. For an existing experiment workflow, add the gate in five bounded steps. 1. Freeze the comparison contract. Record the Datadog project, dataset identity and version, ordered record IDs, evaluator code hashes, task revision, candidate configuration, and expected state for each record. Store only opaque IDs or hashes when content is sensitive. 2. Reconcile coverage before calculating a rate. Compare expected record IDs with observed experiment records. Missing, duplicate, or unexpected IDs fail comparability. Do not silently reduce the denominator to the rows that returned. 3. Emit deterministic fields at the task boundary. For a read only task, the receipt may be a schema valid response tied to the expected source revision. For a mutating task, use an idempotency key, effect identity, and a destination side lookup. Count committed effects, not retry attempts. 4. Preserve non terminal states. A waiting case needs a reason, owner, deadline, and resume identity. A credentials case may correctly end as needs human . Do not manufacture an answer to make an offline score easier. 5. Promote only on conjunction. The candidate can be promoted when the manifest matches, coverage is complete, required evaluators pass, operational budgets hold, every state matches its contract, every completed outcome is verified, and every mutating case has exactly one intended effect. A weighted average is not a substitute because a high score can compensate mathematically for a single catastrophic effect. One practical policy is: Tune thresholds for advisory quality metrics, latency, and cost. Keep missing receipts, duplicate effects, secret exposure, and invalid comparisons as hard failures unless your domain provides a safer explicit rule. Know what this audit does not prove The fixture proves its own eight cases and rule implementation. It does not prove that your dataset represents production traffic, expected outputs are correct, evaluator hashes correspond to reviewed code, or destination receipts cannot be forged. It also does not measure Datadog's product quality or compare models. Those questions need dataset review, source control, access controls, production sampling, and reconciliation with the real destination. Datadog Experiments remains the place to study experiment runs, distributions, traces, and evaluator results. The promotion gate adds a narrower operational decision: is this specific comparison complete and safe enough to authorize the change? Sidewisp's product direction is to make agent health evidence, uncertainty, and verification easier to interpret across existing runtimes. It does not replace Datadog, your runtime, or your release process. Sidewisp is currently in private preview. The public site and interactive demonstration are live; production agent health collection and Datadog integration are not generally shipped.