LLM Evaluation: Route Each Decision to the Right Evidence

LLM evaluation is the practice of testing whether a model powered system meets a defined criterion on a defined target. The useful default is simple: name the decision first, then collect the narrowest evidence that can support it. A curated dataset can support a pre release quality claim. A baseline comparison can support a regression decision. Sampled production runs can reveal live quality drift. None of those, by itself, proves that an agent was reachable, completed a tool effect, or delivered the promised artifact. That boundary matters because “the eval passed” sounds broader than it is. A score always has a target, a clock, and missing evidence. Treating it as a universal health verdict creates false green: the response looks good while the process or outcome is broken. Start with the decision, not the metric Before choosing exact match, an embedding score, an LLM judge, or a platform, write one sentence in this form: At this time , decide this action from this target using this evidence . That sentence routes the work into a lane: Decision Target Clock Minimum evidence Strongest supportable claim Is a candidate good enough? Curated examples Before deployment Dataset, task specific scorer The candidate meets the named criterion on this dataset Did the release regress? Baseline and candidate At release Paired runs, threshold The candidate did not cross the named regression limit Is live quality drifting? Sampled production runs During traffic Fresh sample, production evaluator This sample meets or misses the live criterion Is the agent healthy? Runtime and expected work At the expected run time Heartbeat, schedule, progress receipt The runtime is reachable and useful work is moving Did the intended result happen? External destination After the effect or completion deadline Destination receipt, checksum, acceptance test The effect or deliverable exists and passes verification The last two rows are not “better evals.” They answer different questions. An evaluator can inspect a response or trace; operational health needs evidence about the process that should be running; outcome verification needs evidence from the destination where the result should exist. Offline evaluation supports bounded pre release claims OpenAI's evaluation guidance defines a useful workflow: state the objective, collect a dataset, define metrics, run comparisons, and continue evaluating as the system changes. It also warns against generic metrics and “vibe based” evaluation. The practical implication is that an offline score needs a release decision attached to it. Suppose a support agent must select the correct tool, pass the correct account identifier, and return a policy compliant answer. Do not collapse those into one average. Use three checks: exact or schema based checks for the selected tool and arguments; a task specific quality rubric for the answer; a deterministic check for any output field the application contract requires. Then compare the current candidate with the release baseline on the same cases. A candidate that improves answer style but reduces account ID accuracy is not “0.7% better.” It traded one failure class for another. The release gate should say whether that trade is allowed. Regression testing is therefore a particular use of offline evaluation, not a synonym for all evaluation. It requires a stable baseline, paired cases, and a threshold tied to a release action. Record the dataset version, scorer version, model and prompt configuration, sample count, and disagreements. Without that manifest, a score change cannot be attributed safely. The reasonable floor can be small. Five to ten carefully reviewed examples per critical component are more useful than a large synthetic set with unclear acceptance criteria. Expand the set from real incidents, edge cases, and reviewer disagreements. A dataset should become harder because the system taught you where it fails, not because a dashboard rewards case count. Online evaluation watches live behavior, with weaker references LangSmith's evaluation concepts make an important target distinction. Offline evaluations run on datasets and examples, often with reference outputs. Online evaluations run on production runs or threads, where a correct reference usually is not available. That changes what the verdict can mean. An online evaluator can flag a safety pattern, malformed output, topic drift, low user satisfaction, or an unusual trajectory. It can also harvest difficult live cases for the offline regression set. It cannot silently inherit the confidence of a reference backed test. Its sample may be stale, filtered, unrepresentative, or scored by a judge that has drifted. For each online rule, retain: the sampling policy and exclusions; the run or thread identifier, without leaking sensitive content; the evaluator version and rubric; the observation time and freshness window; the action triggered by a miss; a path for human review and disagreement capture. Google's Agent Development Kit documentation separates evaluation of tool use trajectory from evaluation of the final response. That is useful, but trajectory matching needs restraint. Two valid agents can solve the same task through different tool sequences. Exact trajectory matching is appropriate when order is part of the safety contract; otherwise, verify required effects and forbidden actions rather than demanding one ideal path. Production monitoring also contains non evaluation signals. Latency, error rate, token use, and trace completeness describe service behavior. A response quality evaluator describes sampled content or behavior. Neither one establishes that tomorrow's scheduled worker is reachable. Keep these claims separate even if one platform displays them together. The outcome boundary needs receipts, not another judge Three cases from the article's fixture produced the same trap: a generic LLM score passed, yet the only defensible verdict was UNKNOWN . 1. The runtime health case had an expected schedule and a progress record, but no fresh heartbeat. Old good work did not prove current reachability. 2. The tool effect case had a stable operation ID, but no receipt from the destination. A timeout could hide either no effect or a completed effect. 3. The final deliverable case had an acceptance test definition, but no artifact checksum. There was nothing concrete to test. An LLM judge cannot repair these gaps. Asking a model whether a worker is probably alive does not create a heartbeat. Asking whether an email was probably sent does not create a provider receipt. Asking whether a file sounds complete does not prove that the file exists at the required path. Prefer deterministic evidence near the boundary: a fresh heartbeat and an expected run record for availability; a progress receipt tied to a non secret run ID for execution; an idempotency key plus a destination read for an external effect; a checksum, schema validation, test result, or destination query for a deliverable; an authorized decision receipt for an irreversible action. Missing evidence should remain missing. UNKNOWN is an operationally useful state because it routes investigation without inventing success or failure. Reproduce the eight case evidence routing audit The inspectable artifact used for this article contains eight decision cases. Each case declares the decision, available evidence, expected lane, and expected verdict. The core policy is deliberately mechanical: The fixture covers candidate quality, release regression, live quality drift, runtime health, external effects, final deliverables, subjective review, and human authority. Running it produced: All eight cases reached their expected evidence lane. Five had enough evidence for their bounded claim. Three were unknown, and all three would have looked green if the policy had accepted a generic passing score. This is not a universal standard. Replace the fixture with decisions from one real workflow. Add the exact evidence names your runtime and destinations can produce. Keep the failure behavior: if a required signal is absent, return unknown and list the missing fields. Do not convert absence into a zero score, because zero suggests measurement occurred. Choose the scorer only after the evidence target Once the target is correct, scorer selection becomes easier. Use code when the property is deterministic: JSON shape, tool name, argument range, checksum, file presence, test status, or destination state. Use an LLM judge when the property is genuinely qualitative and you have a clear rubric, a calibration set, and a disagreement review path. OpenAI's guidance notes that models are often more reliable at discriminating between options than producing open ended judgments, so pairwise comparison or classification may be stronger than an unconstrained score. Use human review when the decision involves taste without a stable rubric, legal or policy authority, high impact ambiguity, secrets, or irreversible action. A judge can summarize evidence for the reviewer; it cannot become the authorized person. MLflow's evaluation documentation describes datasets, scorers, prediction functions, human feedback, systematic evaluation, and production monitoring as related capabilities. That is a useful implementation menu. The routing rule still belongs to the application owner: the tool can calculate a score, but only the owner can define what decision the score is allowed to support. Keep four verdicts in the release and operations record A compact evaluation record should answer four questions independently: Did the candidate meet its offline quality criteria? Did it avoid a prohibited regression against the baseline? Does a fresh production sample meet its online criteria? Is the expected work healthy, and is the promised outcome verified? Do not average those answers. A release can pass offline evaluation while live evidence is not yet available. A production sample can look healthy while a scheduled run is missed. A trace can look complete while the final artifact is absent. Preserve each verdict, its evidence time, and its scope. This produces a calmer operating rule: evaluate model and application behavior with datasets and sampled runs; assess runtime health with reachability, schedule, waiting, and progress evidence; verify external outcomes at their destination. Escalate only the missing or failed lane. Sidewisp is currently in private preview. It is being designed as a health layer for existing agent runtimes, but production monitoring adapters and recovery systems are not generally shipped. If the distinction between a good looking run and a verified outcome is the problem you are trying to solve, the private preview waitlist is the appropriate next step.