MLflow LLM Evaluation: Add a Runtime-Health Release Gate

MLflow LLM evaluation can tell you whether an application’s outputs satisfy the criteria you chose. It cannot, by itself, prove that the agent was reachable, started on time, completed an external side effect, or left the promised deliverable at its destination. The practical default is to keep the MLflow evaluation result and the runtime health verdict as two evidence layers, then require both before release. I tested that boundary with MLflow 3.14.0. A code based scorer gave three agent runs a perfect exact deliverable/mean of 1.0 . A separate four receipt health rule allowed only one of those runs to pass. The difference was not a flaw in MLflow. It was a mismatch between the question the scorer answered and the broader operational decision. Reproduce the documented MLflow evaluation path MLflow’s current evaluation guide defines an evaluation from three components: a dataset, one or more scorers, and an optional prediction function. The dataset supplies inputs and expectations. A prediction function generates outputs when they are not already present. Scorers turn the available evidence into feedback or metrics. That division is useful because it makes the evaluation question explicit. If the question is “Did this output equal the expected deliverable name?”, a deterministic code based scorer is more appropriate than an LLM judge. MLflow’s custom scorer documentation allows scorers to read inputs , outputs , expectations , or a full trace, and to return a primitive result or richer Feedback . The experiment used an inline list, pre generated outputs, and this scorer: MLflow logged exact deliverable/mean = 1.0 . That is the correct result for the defined evidence: every output string matched its expectation. The result is reproducible, cheap, and easy to explain. It is also narrower than “all three agent runs are healthy.” MLflow’s evaluation dataset documentation describes datasets as selected examples for regression prevention, version comparison, and targeted quality testing. That is the right mental model. A dataset is a test suite for the claims encoded in its examples and scorers; it is not automatically a census of every production failure that matters. Put operational receipts beside the evaluation result The same fixture attached a small runtime receipt to each task. It recorded four facts that the exact output scorer did not inspect: heartbeatFresh : the runtime is recently reachable; scheduleOnTime : the expected run started within its allowed window; effectVerified : the external destination confirms the intended side effect; deliverableVerified : the promised artifact exists and passes its destination check. The resulting comparison was: Task Exact deliverable Runtime evidence Release decision run 101 pass all four receipts present release run 102 pass stale heartbeat; effect and deliverable unverified block as unreachable run 103 pass schedule missed its allowed window block as late All three evaluation rows passed. Only one run was releasable. A correct string can survive in a cached response after a worker disappears. A correct payload can arrive after the business deadline. A tool call can return a plausible acknowledgement while the destination remains unchanged. None of those cases invalidates the output scorer; they demonstrate why the release decision needs additional evidence. Keep the layers joined by a stable task id or run id , but do not collapse them into one vague score. A compact record can look like this: The join key matters. Without it, a team may compare a current health receipt with an evaluation from another version, environment, or retry. Include the application version, dataset version, scorer version, environment, and observation time when those dimensions can change the verdict. MLflow can retain evaluation and trace evidence; the runtime or destination must still supply facts that only it can know. Treat missing evidence as unknown , not pass . A missing heartbeat may mean collector failure rather than agent failure. A missing destination receipt may mean the write failed, the verifier failed, or the integration cannot expose the fact. Those states call for investigation; they do not justify a green release. Use a two gate decision instead of one blended score A practical release rule is deliberately boring: Each clause should retain its own evidence, freshness, and failure reason. That gives an operator a bounded next action: An evaluation failure routes back to the prompt, model, tool policy, dataset, or scorer. A stale heartbeat routes to runtime or collector diagnosis. A missed schedule routes to the scheduler, queue, or capacity boundary. An unverified effect blocks retries until the external destination is reconciled. A missing deliverable routes to the producer or destination verifier. This separation also prevents an LLM judge from becoming an authority it was not designed to be. Judges are valuable when correctness or quality requires semantic assessment. MLflow explicitly supports built in, guideline based, custom, and code based scorers. Use those tools for their stated criteria. Prefer deterministic destination checks for file existence, database state, API resources, test results, or signed receipts. Do not read the experiment as “MLflow lacks production monitoring.” MLflow documents evaluation, traces, monitoring, datasets, feedback, and several scorer types. The narrower conclusion is falsifiable: the exact evaluation run performed here did not establish four operational facts because its data and scorer did not test them. You can add health oriented scorers when the relevant evidence is present, or keep the health classifier alongside MLflow when the evidence lives in the runtime and external systems. The fixture is intentionally small. It does not benchmark LLM judges, test MLflow at scale, compare vendors, or measure monitoring coverage. Its value is the controlled mismatch: three identical evaluation passes, three different operational states, and one inspectable rule that explains the release decision. For teams operating agents, this boundary is the useful one: evaluate output quality with the strongest appropriate scorer, verify operational facts at their source, and join the evidence before declaring success. Sidewisp is currently in private preview. Its planned role is a health layer alongside existing runtimes, not a replacement for MLflow or an automatic claim that a passing evaluation means a healthy agent. The current public experience is an early access site and product demonstration; production monitoring adapters are not generally shipped.