MLflow LLM Observability: Prove the Trace Became Evidence

MLflow LLM observability is useful for agent operations only when a trace becomes durable, searchable evidence. A completed handler is not that proof. With asynchronous logging, the application can finish before the trace reaches the tracking backend; a full queue can discard new traces; an expired retry window can discard failed writes; and trace level sampling can intentionally omit an entire trace. The practical default is a five stage receipt: 1. the request was eligible for tracing; 2. the trace was admitted to the asynchronous export path; 3. the configured backend stored it; 4. a backend search found a fresh trace with the required spans; 5. a separate deterministic check verified the requested outcome. Stages one through four establish observation coverage. Stage five establishes that the agent delivered what the user asked for. Do not merge them into a single green status. This article tests that boundary against MLflow 3.14.0, the current PyPI release when checked on July 30, 2026. The experiment uses a local SQLite tracking backend and content free attributes; it does not send prompts, responses, credentials, or customer data. A trace can disappear after the handler returns MLflow's production tracing guide recommends asynchronous trace logging for production workloads. It documents three operational boundaries that matter before a trace can support an incident decision. First, async logging is enabled by default for open source MLflow and Databricks non notebook workloads. Databricks notebooks use a different default. An audit therefore needs the effective runtime mode , not an assumption copied from another environment. Second, MLFLOW ASYNC TRACE LOGGING MAX QUEUE SIZE defaults to 1,000. The documentation is explicit: when that queue is full, new traces are discarded. A successful application response can coexist with missing observability evidence because request execution and trace admission are separate events. Third, failed trace writes are retried only within MLFLOW ASYNC TRACE LOGGING RETRY TIMEOUT , documented with a 500 second default. After that window, the trace is discarded. Raising the timeout can improve resilience during a short tracking backend outage, but it also extends memory pressure and recovery work. It is not a durability guarantee. Sampling is different again. MLFLOW TRACE SAMPLING RATIO selects whole traces: the spans of a selected trace stay together, while an unselected trace is intentionally absent. That is a policy result, not exporter failure. Your health logic should say deliberately unobserved , not trace lost , when the sampling decision is known. These distinctions change the alert. An intentionally unsampled request should affect coverage calculations. Queue overflow or retry exhaustion is an observability incident. A backend outage may make the verdict uncertain. Treating all three as “no trace” hides both cause and the safe next action. Prove persistence with a canary, not a process exit MLflow 3.14.0 exposes persistence controls that let a test distinguish pending background work from queryable backend evidence: mlflow.flush trace async logging() flushes pending trace writes; mlflow.get trace(trace id, flush=True) flushes and retries when the trace is not found; mlflow.search traces(..., flush=True) flushes before searching. The relevant API behavior is documented in the MLflow Python reference. The flush option is especially useful in tests, deployment probes, short lived jobs, and controlled canaries. Flushing every production request would defeat much of the latency benefit of asynchronous logging. Here is a minimal content free canary: Run this against the same tracking URI, credentials, network path, experiment location, and package combination used by the worker you want to trust. A canary against a developer's local file store says nothing about a production container pointing at a remote tracking server. In the recorded MLflow 3.14.0 experiment, the difference was visible. Immediately after ending the span, get trace(..., flush=False) returned no trace and search traces(..., flush=False) returned zero results. After flush trace async logging() , retrieval succeeded, search returned one trace, and that result contained the canary trace ID. That is one observation, not a universal latency benchmark. A fast backend may persist before the first query; a slow or failing backend may take longer. The durable rule is the assertion after a controlled flush, not the exact before flush count. For a long running service, schedule the canary at a rate that is cheap enough to retain at 100% sampling. Record: worker and deployment identity; effective tracking URI fingerprint, never the credential; trace ID and experiment or location; enqueue time, flush completion time, and search time; expected root and required child span names; a freshness deadline; the result of a separate destination check. Those fields let an operator distinguish a worker that never created the span from an exporter that could not persist it. Route eight states without a false green A useful audit needs more than found: true . The following eight case fixture gives each failure boundary a different verdict. Evidence Verdict Operator decision Sampling policy excluded the request deliberately unobserved Recalculate coverage or increase sampling for critical paths Queue rejected a new trace discarded queue full Reduce pressure, increase bounded capacity, or scale exporters Export retries exhausted their timeout discarded retry expired Investigate the backend or network; evidence is unavailable Local work ended but backend storage is unproven backend persistence unproven Flush in a probe and query the configured backend Stored trace is older than its evidence deadline stale evidence Re run the canary; do not reuse old green Trace is fresh but a required tool or destination span is absent incomplete trace Fix instrumentation before using it for diagnosis Trace is complete but the deliverable is unverified observed outcome unverified Check the destination or artifact directly Trace is fresh, complete, and outcome receipt passes verified Admit this evidence to the health decision The precedence matters. If a request was deliberately unsampled, there is no reason to diagnose queue admission for that request. If persistence is unproven, span completeness is unknowable. If the trace is complete but the external artifact is missing, the result is false success, not an instrumentation win. The executable audit accompanying this article replayed exactly one case for each verdict and allowed only verified to become green. It also checked the MLflow 3.14.0 function signatures and stored a canary in a fresh SQLite backend. This fixture is intentionally small: its value is the decision boundary, not load test realism. A complete trace and a completed task are different receipts MLflow Tracing can capture inputs, outputs, metadata, model calls, retrievals, tool calls, and other intermediate steps. Its tracing overview presents those traces as evidence for debugging, monitoring, evaluation, feedback, and dataset collection. That evidence can explain how the run behaved . It cannot generically prove that every external promise was fulfilled. A tool span with a successful status may show that an API call returned. It does not necessarily prove that the requested file exists at the agreed path, a pull request contains the intended diff, a message reached the correct recipient, or a scheduled report contains current data. Define the outcome receipt from the task contract: for a file, verify path, type, size floor, checksum or content predicate; for a deployment, verify the target revision and a live acceptance probe; for a message, verify destination identity and provider receipt; for a database mutation, verify the intended row state and idempotency key; for a scheduled run, verify its expected window and the freshness of its output. Link the receipt to the trace with a content free operation or request ID. Keep secrets and raw prompt content out of the join key. If the destination cannot be queried safely, classify the outcome as unknown and ask for the missing authority or evidence. This also limits what the canary proves. One flushed trace verifies one path at one moment. It does not measure every worker, guarantee future queue capacity, reconstruct sampled out traces, test backup retention, or prove a user outcome. Load tests, backend availability checks, retention drills, and outcome specific probes remain separate. The operational default Use asynchronous logging for production latency, but pay the durability debt explicitly: 1. pin the MLflow package and record the effective async, queue, retry, and sampling configuration; 2. retain a low rate, 100% sampled canary for every critical worker path; 3. flush and search only inside probes, tests, shutdown handling, or other bounded verification points; 4. require fresh backend search plus expected span coverage before admitting trace evidence; 5. verify the task destination separately; 6. alert differently for deliberate sampling, exporter loss, stale evidence, incomplete traces, and false success. That policy makes MLflow observability useful without pretending it is an outcome oracle. Sidewisp is currently in private preview. It is being designed as a health layer alongside agent runtimes and observability systems, with evidence freshness, uncertainty, and outcome verification kept explicit. Sidewisp does not currently ship MLflow monitoring or automated recovery.