AI Agent Evaluation Metrics: Five Signals, No Composite Score

AI agent evaluation metrics should answer a release question, not decorate a dashboard. For a tool using agent, a compact default is five measurements kept separate: task quality, useful progress, tool effect integrity, cost per verified outcome, and false success rate. Record evidence coverage beside each one. Then make missing outcomes and unknown tool effects hard gates before you optimize averages. That ordering matters. A candidate can write better prose, finish more quickly, and look cheaper per run while producing an unverified deliverable. Averaging those signals into one score turns an unsafe release into a numerical improvement. Keep five denominators instead of one score Agent evaluation has more than one object of interest. Anthropic’s engineering guide to agent evals distinguishes tasks, repeated trials, graders, transcripts, and the final state of the environment. Its flight booking example is the useful boundary: the transcript may say a flight was booked, while the outcome is whether the reservation exists in the database. The current Vertex AI agent evaluation description makes another separation. Final response evaluation asks whether the agent achieved the goal; trajectory metrics inspect the path, including exact or ordered action matches, precision, recall, and single tool use. Both views are useful, but neither should silently stand in for the other. Use this five metric contract: Metric Numerator Denominator Preserve separately Release role Task quality Sum of versioned rubric or deterministic quality scores Trials for which the quality grader actually ran Grader version, coverage, disagreement Optimize after hard evidence passes Useful progress Active observation windows with a task specific evidence delta Observed active windows Explicit waiting windows and stale collectors Floor, then optimize Tool effect integrity External effects verified at the destination Verified + failed + unknown effect bearing attempts Intentionally denied calls, unknown effects Hard floor Cost per verified outcome Total covered run cost Destination verified outcomes Estimated cost, missing billing coverage Budget floor and optimization False success rate Declared completions whose outcome check failed All declared completions Completions whose verifier was unavailable Hard veto Each denominator prevents a different lie. Mean quality without grader coverage rewards the easiest examples. Tool success without destination evidence confuses a successful call with a successful effect. Cost per run rewards cheap failures. Completion rate rewards an agent for grading its own claim. Operational telemetry still belongs in the ledger. The pinned OpenTelemetry GenAI metrics snapshot defines development status instruments for token usage, operation duration, agent duration, inference call count, tool call count, and tool duration. Those measurements explain workload and efficiency. They do not assert that a file exists, an invoice was created once, or a scheduled report reached its destination. Write the evidence record before choosing thresholds Start from one row per trial. Do not begin with an aggregate chart. A minimal record needs enough information to reproduce every numerator, denominator, exclusion, and unavailable state: The important value is not 0.90 ; it is the disagreement around it. The response grader liked the output, but the outcome verifier was unavailable and the tool effect remained unknown. This trial may teach you about response quality. It cannot support a release claim. Use three evidence states wherever absence is possible: verified means the named predicate ran against the intended destination and passed. failed means the predicate ran and the expected state was absent or unacceptable. unavailable means no verdict was possible because the verifier, collector, permission, or destination evidence was missing. Do not convert unavailable to zero or success. Zero is a measurement; unavailable is a coverage defect. Waiting needs similar care. A run with an explicit approval owner, requested decision, deadline, and resumable state is not stuck. Count useful progress during the active windows before the wait, then stop the active progress clock. Keep the denied high impact tool call as evidence that the boundary worked; do not count an intentional denial as tool failure. Task quality is the one metric likely to need several grader types. Use deterministic tests for structured fields, files, database rows, HTTP status, and exact policy conditions. A model grader can score tone, relevance, or an open ended artifact, but store its prompt, model, rubric version, and calibration sample. Human review remains necessary when the judgment is consequential or the model grader’s disagreement rate is not understood. Reproduce the release reversal The artifact retained with this article contains ten synthetic trials: five for stable v1 and five for fast v2 . Run it with Node.js 20 or newer: The declared thresholds are: The exact summary is: Variant Quality Useful progress Tool effects verified Cost / run Cost / verified outcome False success Unverified complete Gate : : : : : : : stable v1 0.818 (4/4) 14/14 5/5 $0.41 $0.52 0/4 0 PASS fast v2 0.902 (5/5) 15/16 4/5 $0.38 $0.63 1/5 1 FAIL Three observations change the decision. First, fast v2 looks cheaper when the denominator is runs: $0.38 instead of $0.41. Once spend is divided by verified outcomes, the conclusion reverses: $0.63 instead of $0.52. The faster version spent less on each attempt and more on each result you could trust. Second, the higher 0.902 quality score does not repair a missing outcome. One polished completion failed its environment check; another had no usable verifier. Quality and useful progress averages remain diagnostic, but they are not authority to publish, pay, message, delete, or otherwise accept an external effect. Third, stable v1 contains a legitimate approval wait. It is absent from the completed trial quality denominator, but the progress made before waiting and the intentionally denied tool call remain visible. The metric contract neither rewards completion theater nor punishes a correct pause. This experiment is a counterexample, not a benchmark. The prices, trials, and thresholds are constructed to test the accounting rule. They do not estimate a production false success rate, and $0.60 is not a universal budget. Gate evidence before optimizing quality Apply the metrics in a fixed order: 1. Check coverage. Every declared completion needs an outcome verdict or an explicit unavailable state. Every effect bearing tool attempt needs verified, failed, or unknown destination evidence. 2. Block false success. If a declared completion failed its outcome check, stop the release. Investigate the completion predicate, not the response style. 3. Block unknown effects. An unknown side effect is an incident boundary. Query the destination or reconcile an idempotency key before retrying. 4. Check progress and tool floors. Compare like for like tasks and preserve valid waits. A progress regression or failed tool effect rate may justify a rollback even when final quality rises. 5. Optimize quality and cost. Only now compare rubric scores, latency, tokens, and cost per verified outcome. This is deliberately not a weighted composite. Weights invite negotiation between unrelated harms: enough fluency can mathematically cancel one duplicate payment; enough cheap runs can hide a missing report. A policy can still use weights inside the task quality rubric, but evidence coverage and external effects remain outside that score. Choose thresholds from the workflow’s consequences and baseline. A read only research agent may tolerate a lower tool effect floor because most calls are reversible. A publication, billing, customer message, or access control agent should require destination receipts, idempotency, and a zero false success target for the covered test suite. Report confidence intervals when the trial count is large enough, and show the raw count when it is not. Five trials are 0/5 , not proof of a zero population failure rate. Keep evaluation and production health connected but distinct. Offline trials tell you whether a candidate survives known scenarios. Production monitoring tells you whether real schedules, credentials, dependencies, costs, and outcomes remain healthy after release. A passing suite is permission to deploy under the declared scope, not proof that future runs cannot fail. Sidewisp’s product direction is to put outcome, progress, tool, availability, memory, and cost evidence into one health view around existing agent runtimes. It is not a replacement runtime, mandatory gateway, or autonomous fixer. Sidewisp is currently in private preview. The public site and article library are live; production agent health collection, runtime adapters, token cost analytics, and recovery execution are not generally shipped. Use the five metric contract on one consequential workflow first. If the version with the best looking average still fails an outcome or tool effect gate, the metrics have done their job.