Towards a Science of AI Agent Reliability: Audit the Evidence

The short operational reading of Towards a Science of AI Agent Reliability is this: keep the paper’s four dimensional profile, but never accept a row of twelve scores without the evidence that produced them. For a deployment decision, also add current proof of reachability, expected runs, owned waits, tool effects, and destination outcomes. A benchmark profile and a live health verdict answer related but different questions. The June 2026 arXiv v3 paper by Stephan Rabanser and colleagues measures reliability independently of raw task accuracy. It evaluates 15 models on GAIA and a cleaned subset of τ bench, then decomposes reliability into consistency, robustness, predictability, and safety. The authors’ result is deliberately inconvenient: capability gains over the studied release window did not automatically produce comparable reliability gains. That is useful for an operator, but only after translating each metric into an evidence contract. The checklist below does that translation and makes the remaining production boundary explicit. Read the twelve metrics as evidence requests The paper’s Table 2 contains twelve metrics. They are not twelve interchangeable quality points. Dimension Paper metrics Minimum evidence to retain Consistency outcome; trajectory distribution; trajectory sequence; resource use Task set version, independent run count, outcome oracle, action sequences, resource units, and every per run record Robustness fault; environment; prompt Baseline trials, paired perturbed trials, a versioned perturbation specification, and proof that the environment or prompt change preserved the task’s meaning Predictability calibration; discrimination/AUROC; Brier Confidence captured before grading, binary outcome evidence, binning or ranking method, sample counts, and the source of confidence Safety compliance; harm severity Versioned constraints, every violation, severity rules, judge identity and version, and a human validation sample The first operational rule is therefore simple: a metric without its denominator, protocol, and evidence location is unavailable, not low and not healthy . Consider outcome consistency. Re running forty tasks five times gives 200 trials, but a score alone does not tell you whether the same tasks alternated between success and failure or whether a fixed subset failed every time. Those patterns can have similar average accuracy and very different operating consequences. The paper’s metric exists to expose that distinction; discarding the trial level evidence would recreate the problem it solves. The robustness metrics need even more care. A fault injection score is interpretable only relative to a baseline. A prompt robustness score is valid only if the variants remain semantically equivalent. Renaming a JSON field can test environmental brittleness; removing information required to solve the task changes the task. Store the perturbation generator, its version, and a reviewed sample beside the result. Predictability requires a similarly strict timestamp. Capture confidence before the outcome grader runs. Confidence written after an agent has seen a test result is not a forecast. Also record whether the value came from the agent, a separate model, a calibrated classifier, or a heuristic. Calibration, AUROC, and Brier scores can all be calculated correctly from the wrong confidence signal. Finally, preserve the paper’s safety boundary. Its public methodology reports safety separately from the aggregate reliability score. That is the right default for operations. A strong consistency score must not average away one unauthorized destructive action. Compliance measures how often constraints were respected; conditional harm severity asks how bad the violations were. You need both the frequency and the tail. Lint the profile before debating thresholds Threshold arguments are premature when the evidence package is incomplete. I built a small profile linter that first checks structure: all twelve named metrics exist; every metric has a score, numerator, denominator, protocol, and evidence URI; robustness results include paired baseline and perturbation counts plus a versioned specification; predictability results identify the confidence source; safety results identify constraints and judging method; safety is not folded into the overall reliability aggregate; an autonomous deployment includes a non empty human validated safety sample. It then checks six live health evidence classes: freshness, reachability, schedule coverage, waiting ownership, tool effect receipts, and destination outcome receipts. The included illustrative profile has all twelve paper metrics and a 30 case human safety review. Its result is still: The two blockers are missing tool effect and destination outcome receipts. The linter test then supplies those two evidence references and changes the result to PASS . That does not prove the fictional agent is safe; it proves the evidence manifest is complete enough to review. The distinction matters. Run the audit with: This is a deliberately modest artifact. It validates presence and shape, not scientific validity. It cannot decide whether a benchmark represents your users, whether a confidence report is honest, whether two prompts mean the same thing, or whether an LLM judge graded harm correctly. Those remain domain review tasks. Do not turn the profile into one release number The paper computes dimension aggregates to support comparison, but its own choices reveal why operators should keep the profile visible. The overall reliability aggregate combines consistency, predictability, and robustness; safety remains separate. The paper also states important limits: two benchmarks cover only a narrow task slice, one scaffold is used per benchmark, the safety evaluation depends on an LLM judge, metric and aggregation choices are subjective, and temperature zero may overestimate reliability available under settings chosen to maximize accuracy. Those limits should appear beside a deployment decision, not in an archived methodology note. A practical review can use three layers: 1. Profile completeness: Are the twelve metrics backed by inspectable evidence? 2. Application thresholds: Which dimensions and tail conditions are acceptable for this task and authority level? 3. Operational evidence: Is the current deployed system reachable, progressing, bounded, and producing the intended external result? The second layer is necessarily application specific. A draft writing assistant with mandatory human review can tolerate different inconsistency from an unattended refund agent. The paper makes the same general point: reliability requirements should scale with autonomy. Avoid copying a leaderboard score into a universal release threshold. For consequential side effects, use vetoes before averages. One sensible local policy is: This ordering keeps attractive averages from hiding unknown or severe states. Add the production evidence the paper does not claim to measure A benchmark evaluates behavior under a defined protocol. An operator must also know what is happening now. Add these six receipts without pretending they are extra paper metrics: Evidence freshness: when each health signal was last checked and when it expires. Reachability: whether the runtime and required tools can be contacted now. Schedule coverage: which expected runs started, completed, overlapped, or were missed. Waiting ownership: whether a legitimate dependency has an owner, deadline, and resumable state. Tool effect reconciliation: whether a timed out or retried action changed its destination zero, one, or multiple times. Destination outcome verification: whether the promised file, database change, message, test result, or other deliverable exists and satisfies its acceptance rule. This addendum prevents two category errors. First, repeated benchmark success does not prove that a production runtime is currently reachable. Second, a successful command or model response does not prove that the external effect or deliverable exists. The HAL harness repository linked from the project site is helpful here because it emphasizes reproducible evaluations, traces, isolation, and cost tracking. Those are strong evaluation inputs. They still need a deployment specific outcome oracle and current operational receipts when the agent acts outside the benchmark. Use the paper as a profile, not a permission slip For a real review, start with one workflow rather than an entire fleet: 1. Pin the runtime, model, scaffold commit, tool versions, task set version, and evaluation date. 2. Run repeated nominal trials and retain outcomes, trajectories, and resource records. 3. Define fault, environment, and prompt perturbations before inspecting the result. 4. Capture confidence before grading. 5. Define constraints and severity levels; human validate a safety sample. 6. Keep safety separate from the aggregate. 7. Attach the six live health evidence classes. 8. Record unknown wherever proof is missing. 9. Set thresholds and vetoes for this workflow’s authority and consequences. 10. Re run the profile after material model, prompt, scaffold, tool, or environment changes. This preserves the paper’s main contribution: reliability is a shape, not a synonym for accuracy. It also stops that shape from becoming a decorative dashboard. The observable outcome of the review is not “a reliability score was computed.” It is a versioned decision with inspectable evidence, explicit unknowns, and a clear list of what must be verified before authority expands. Sidewisp’s intended territory is the operational side of that boundary: reachability, useful progress, context, tools, outcomes, and cost around agents that already run elsewhere. Sidewisp is currently in private preview. Its production monitoring adapters and recovery systems are not generally shipped, so this article describes an operating method, not a current automated Sidewisp capability. If that evidence boundary matches the failures you need to catch, you can join the private preview from the Sidewisp site. Sources Rabanser et al., “Towards a Science of AI Agent Reliability,” arXiv v3, June 2026 Holistic Agent Leaderboard reliability methodology HAL reproducible evaluation harness ICML 2026 poster page OpenReview record