LLM Self Evaluation: Calibrate the Judge Before Trusting It

LLM self evaluation is useful when the criterion is genuinely qualitative, the evaluator has been tested against human labels for that exact criterion, and its answer is treated as supporting evidence. It should not decide whether a file exists, a payment landed, a test passed, or a person authorized an irreversible action. Those questions already have stronger owners: deterministic checks or accountable humans. The practical default is therefore a router, not a universal judge. Send mechanically verifiable claims to code. Send bounded qualitative criteria to a calibrated model evaluator. Send uncovered, unstable, or high impact decisions to a person. If the evidence is missing, keep the verdict unknown . An LLM judge can be useful without being authoritative Self evaluation means asking an LLM to assess its own output or another model's output. A second pass can catch an obvious contradiction, compare two summaries, or grade a response against a rubric. Learn Prompting's overview shows the basic pattern: generate an answer, then ask a model to critique or revise it. Research gives this pattern a credible but bounded role. Ren and colleagues reformulated open ended self evaluation as token level predictions and reported better selective generation accuracy and stronger correlation with output quality on TruthfulQA and TL;DR for their tested PaLM 2 and GPT 3 settings. A newer study of multi step tasks found that stepwise self evaluation outperformed holistic scoring for failure detection on two benchmark datasets, with up to a 15% relative increase in AUC ROC. Neither result turns a model opinion into a delivery receipt. They show that self evaluation can improve a decision under specific tasks, prompts, models, and datasets. The unit of trust is that tested evaluator configuration, not the phrase "LLM judge." There are also known failure modes. The MT Bench and Chatbot Arena paper reports that strong judges reached more than 80% agreement with human preferences in its setting, while documenting position, verbosity, self enhancement, and reasoning biases. Agreement is encouraging; the bias list is operationally decisive. A judge can be useful on average and still reverse the one verdict that matters. Position bias is easy to test. Evaluate candidates A and B, swap their order without changing their content, and evaluate again. Wang and colleagues showed that order changes could materially alter pairwise rankings and proposed balanced position calibration plus human escalation. If the preferred candidate changes after the swap, record unstable . Do not average the contradiction into a confident score. Freeze the criterion before calibrating the judge "Quality" is too wide to calibrate. A judge that recognizes a clear summary may still be unreliable at factual grounding, policy interpretation, or evidence sufficiency. Define one criterion with a narrow contract: The calibration set needs examples the evaluator did not write, labels from the people who own the criterion, and enough difficult negatives to expose flattering or overly permissive grading. Freeze the model identifier, judge prompt, rubric, examples, output parser, and threshold together. A change to any of them creates a new evaluator version. Human labels are not an embarrassment in this process; they define the target. EvalGen's authors describe "criteria drift": people often refine what they mean by a criterion after seeing real outputs. That is a reason to version criteria and labels, not a reason to hide judgment behind a model score. Measure at least these properties per criterion: agreement against held out human labels; false pass rate, because an evaluator that approves bad output creates false success; unknown rate, which reveals missing coverage; stability when candidate order and irrelevant formatting change; coverage by language, task family, impact class, and output length; disagreement clusters, kept as examples for the next review. Do not combine unrelated criteria into one reassuring average. Suppose clarity has 100% agreement, groundedness has 75%, and safety has only two labeled examples. An 88% blended score can conceal that no defensible safety gate exists. Keep the three rows separate. Thresholds are policy choices, not constants from a paper. A team may accept 80% agreement for a low impact style suggestion and require deterministic proof for a deployment effect. The important property is that the threshold is frozen before the release candidate is scored. Replay a calibration gate The following fixture uses four criteria and eight routing cases. Its threshold is deliberately simple: at least three human labeled examples, at least 80% agreement, and no winner change under order reversal. The decision rule is small: Run the complete fixture with: The replay produces: Two observations matter more than the compact numbers. First, the fixture contains a missing deliverable that the judge calls pass . The router returns deterministic fail . It also contains a checksum match that the judge dislikes; the router returns deterministic pass . A model can comment on the presentation, but it does not get to overrule evidence about whether the promised object exists and matches. Second, pairwise quality has perfect agreement on four labeled examples but changes its winner when the candidates swap order. The gate still fails. Historical agreement does not excuse a present contradiction. This small replay is not evidence that 80% is safe for your application. It is an inspectable pattern for proving that every verdict reached the correct evidence owner. Route the live decision to the strongest evidence owner Once calibration passes, the runtime decision should preserve three lanes. Deterministic lane. Use filesystem checks, schema validation, test results, database constraints, provider receipts, signatures, checksums, and destination reads when they can answer the question directly. Record the observed value and freshness. A command exiting zero proves only that command's contract; verify the intended external outcome separately. Judge support lane. Use a pinned evaluator for criteria such as clarity, relevance, tone, or rubric adherence when calibration coverage matches the current case. Retain the criterion, evaluator version, prompt hash, input reference, verdict, explanation, and calibration set version. The model supports the verdict; the surrounding policy decides what that evidence permits. Human review lane. Route high impact disagreements, new languages, uncovered task families, order instability, policy exceptions, and irreversible authority here. Include the conflicting evidence and a concrete question. "Please review" is weak routing; "The deterministic receipt is missing while the judge says complete—may this incident be closed?" is actionable. An unknown result is useful. It says the system cannot currently establish the claim. Turning unknown into pass merely protects a dashboard from looking incomplete. Recalibrate on drift, not only on a calendar A judge can drift when the model alias changes, the prompt is edited, examples are reordered, a new language arrives, outputs become longer, or the product begins handling a different task family. Trigger recalibration on those changes and monitor live disagreement samples between scheduled reviews. Keep two boundaries visible: 1. Self evaluation estimates a bounded qualitative property; it does not prove reachability, useful progress, tool effects, memory persistence, or deliverable existence. 2. A calibrated evaluator reduces review load; it does not receive human authority over secrets, spending, publication, deletion, or other irreversible actions. The result is a less theatrical form of AI evaluation. The judge gets a useful job, deterministic evidence keeps its stronger job, and people retain the decisions that belong to them. Sidewisp is currently in private preview. It is being developed as a health layer around existing agent runtimes, but production agent health collection and recovery are not shipped in the current website repository. If this evidence routing problem matches how you operate agents, you can join the private preview waitlist without treating the article as a claim of a live monitoring integration.