AI Agent Testing Framework: Choose by Evidence, Not Feature Count

An AI agent testing framework should be chosen by the evidence it can reproduce and verify, not by the number of metrics in its catalogue. For a stateful, tool using agent, the reasonable first prototype is Inspect AI when disposable environments, external agent execution, and code scorers are central. LangWatch Scenario is the better shaped prototype when simulated users and multi turn behavior dominate. MLflow fits teams whose evaluation datasets and experiment history are already the organizing layer; DeepEval fits a pytest centered regression workflow. That is a prototype order, not a universal leaderboard. None of these frameworks knows whether your invoice was created once, your memory survived a restart, or your promised deliverable is valid. Those are application oracles. A selection process is honest only when it names that remaining work before adoption. Start with the evidence the framework must leave behind Agent tests are not enlarged prompt tests. An agent changes state over several turns, crosses permission boundaries, calls tools, waits, retries, and may finish with an external artifact. A fluent final response is one observation among several. Anthropic's agent evaluation guide separates the task, trials, transcript, outcome, evaluation harness, and graders. It also distinguishes code based, model based, and human grading. That decomposition gives us a useful selection question: where will each decisive fact come from? Use four gates: Gate Evidence required Common false substitute Replay The same fixture can restore relevant inputs, tool state, permissions, and starting data Re sending the same prompt Tool effect The destination proves the intended effect happened with the right identity and count A trace says the tool was called Memory continuity Required decisions survive the boundary you actually fear: restart, compaction, or handoff The conversation has several turns Outcome verification A deterministic check proves the promised artifact or state exists and is valid The agent says it finished A framework may expose hooks for all four without implementing your four proofs. That is acceptable. The bad outcome is hiding a custom database query, restart fixture, or artifact validator under the vague label “integration.” The distinction matters most at two boundaries. A tool call can time out after the destination committed its change, so transport success and effect success can disagree. A multi turn test can preserve state in one process while the deployed agent loses the same decision after a restart. If a framework comparison collapses either pair, its score is not useful for agent reliability. Four current frameworks, read through those gates I reviewed current official documentation on 27 July 2026 and recorded only documented surfaces. A level of “custom” below is not a criticism; it means the framework supplies an extension point while the application must supply the truth. Inspect AI documents composable datasets, agents, tools, and scorers, external agent execution, evaluation logs, and several sandbox backends. Its official overview even uses an agent acting through tools inside a Docker sandbox. That makes it a strong starting surface for executable, stateful tasks. A custom scorer can inspect the resulting environment. It still needs a connector to the real destination and a deliberately constructed restart memory oracle. LangWatch Scenario begins from multi turn simulation rather than a static input output row. Its agent simulation documentation shows intermediate tool call expectations, custom assertions such as a created ticket, error recovery tests, and reproduction of issues found in production. That shape is attractive for support, voice, and other interactive agents. The documented conversation state is not proof that a decision survived process death, and a ticket assertion is still application code. MLflow organizes evaluation around datasets, prediction functions, scorers, run results, human feedback, and monitoring. The current GenAI evaluation overview makes custom scoring a first class part of an evaluation run. This is useful when the team already treats datasets and experiment lineage as its source of truth. The selection cost is the stateful harness around the prediction function: restoring a tool world, forcing a restart, and reconciling external effects. DeepEval offers local test runs, single and multi turn cases, thresholds, tracing, and regression comparison in a pytest shaped workflow. Its quickstart is an easy on ramp for teams that want evaluations beside application tests. The quickstart leans on model based metrics, so an operational proof of fit should add deterministic effect and outcome checks rather than assume that a judge score proves destination state. Framework Documented center of gravity Prototype it first when Explicit adapters to test Inspect AI Executable agent tasks, tools, sandboxes, scorers The agent changes files or other inspectable state Real destination effect; restart memory LangWatch Scenario Multi turn simulation and step assertions User behavior and recovery paths drive failures Real destination effect; restart memory MLflow Datasets, scorers, run history, feedback Evaluation lifecycle and lineage already live in MLflow Stateful fixture; effect; restart memory DeepEval Pytest style metric regression and tracing The team needs a lightweight test suite entry point Stateful fixture; effect; restart memory; deterministic outcome This table is deliberately narrower than a product comparison. It says nothing about hosted pricing, support, maintainer responsiveness, or every integration. It answers one question: which documented execution surface is closest to the evidence needed for this agent? Run the selector, then distrust the score The accompanying framework evidence.json records four evidence levels for each candidate. 0 means no retained primary source evidence, 1 means an explicit custom adapter or scorer is required, and 2 means the documentation presents a first class workflow. The stateful tool scenario weights replay at 2 , tool effects at 4 , memory continuity at 4 , and verified outcomes at 5 . Run the artifact: The decisive part of the output is: Inspect and LangWatch Scenario both receive a weighted evidence score of 22 . Inspect wins this fixture only because the declared workload is stateful tool , which adds a three point fit bonus. Change the workload to multi turn simulation and the order should change. Change the weights and the result may change. That sensitivity is a feature: it makes the team's assumptions reviewable. The score must never erase gaps. A result that claimed zero adapter work here would be less credible, not more. The matrix cannot know the schema of the destination, the identity rule for an effect, the decisions that memory must retain, or the validity rule for the deliverable. The artifact also has a hard limitation: it is a dated documentation audit. It does not install all four frameworks or measure integration time. Use it to choose the order of experiments, then let two inconvenient cases decide. Make the proof of fit fail in two different ways The first case tests an ambiguous tool effect. Arrange a destination fixture in which the tool commits one object and the transport then returns a timeout. The harness passes only if it can: 1. keep a stable operation identity across the retry boundary; 2. inspect the destination rather than trust the call result; 3. classify the state as committed, not blindly retry it; 4. show the evidence in a run record a developer can debug. The second case tests restart continuity. Let the agent choose a bounded plan, persist only the allowed decision state, terminate its process, and resume with a fresh process. The harness passes only if it can: 1. prove the restart occurred; 2. restore the same fixture without leaking hidden answer state; 3. verify the required decision survived; 4. detect stale, missing, or contradictory memory; 5. verify the final artifact independently. Include a legitimate wait as a control if the agent requests approval. A test harness that marks every pause as failure will pressure the product to remove safe authority boundaries. The evidence should distinguish working, waiting, stuck, and complete rather than reward uninterrupted activity. Time box the prototype. A small team should not build a general adapter layer before it has reproduced these two failures. Give each candidate the same fixture, the same outcome oracle, and the same debugging budget. Prefer the framework that makes the evidence chain shortest and most inspectable, even if another candidate produces more aggregate metrics. The outcome is not “framework X is best.” It is “framework X reaches our two hard proofs with these named adapters, and framework Y does not within the same budget.” That statement can survive a code review. Testing evidence is not live agent health Pre release evaluation answers whether a build can handle known tasks and controlled failures. Live health asks whether a particular deployed agent is reachable now, making useful progress, keeping required context, reaching its tools, producing the expected result, and staying within sensible time and cost boundaries. Passing an evaluation does not prove a scheduler fired last night, a credential is valid today, or a deliverable reached its real destination. Sidewisp's intended territory is that health layer around existing runtimes: distinguish working from waiting or stuck, show evidence and freshness, and verify outcomes before clearing an issue. Sidewisp is currently in private preview. The public experience is an early access website and interactive demonstration; production agent health collection, runtime adapters, and automated recovery are not generally shipped. So keep the boundary explicit. Use the chosen test framework to make controlled regressions visible before release. Use runtime specific evidence and independent outcome checks to establish live health after release. A green test run is valuable evidence, but it is not permission to treat an unobserved deployed agent as healthy.