AI Voice Agent Testing: Prove the Call Did the Work

AI voice agent testing should not pass a release because the transcript sounds plausible. A defensible test proves five separate things: the call connected, the agent responded within a workflow specific limit, interruption actually stopped assistant audio, each side effecting tool reached a known result, and the caller's requested outcome exists outside the conversation. That distinction matters because the usual artifacts answer narrower questions. A recording proves audio existed. A transcript proves that speech recognition produced text. An LLM rubric estimates whether that text met a criterion. A terminal telephony status proves the call ended. None of those facts alone proves that an appointment was booked, a cancellation was applied, or a transfer reached its destination. The practical default is to keep the simulator and transcript evaluator, then add a small deterministic call receipt beside them. A fluent transcript is only one layer Voice simulation is useful. Vapi's current voice testing documentation describes a real phone call between the target agent and a testing agent, followed by recording, transcription, and LLM evaluation against a rubric. That exercises far more of the path than a text only prompt test. It still leaves observable gaps. A transcript can omit the exact boundary between the caller finishing and audio playback starting. It can read cleanly after the caller talked over the assistant. It may contain a tool's confident confirmation even when the remote system timed out. Telephony completion has a similarly narrow meaning. Twilio documents queued , ringing , in progress , completed , busy , failed , no answer , and canceled call states. Its Call resource reference also warns that completed means a connection was established and audio was transferred. A person, an IVR, or voicemail may have answered. "Completed" is therefore transport evidence, not a business verdict. Use five gates instead: Gate Minimum evidence A failure it exposes Reachability Correlated call ID plus connected or in progress status Queueing, no answer, invalid destination Turn timing User audio stopped at t1 ; assistant audio started at t2 Slow response hidden by a coherent transcript Interruption User interruption at t3 ; assistant audio stopped by t4 Agent continues speaking over the caller Tool effect Stable operation ID plus destination receipt Timeout followed by an unsafe blind retry Outcome Independent check of the promised result Normal call end with no booking, transfer, or cancellation Do not combine these into one score at first. A weighted average can let an excellent transcript hide a missing outcome. Keep the failed layer visible. Record audio boundaries, not just model latency The useful first response interval starts when caller audio is considered complete and ends when assistant audio actually begins playing: That is not the same as model time to first token. Speech endpointing, transcription, model inference, synthesis, buffering, and telephony transport all sit inside the caller's experience. Measuring only one internal stage can make a slow call look healthy. Interruption needs another paired observation: Vapi's server event documentation exposes separate status updates, speech updates, user interrupted , tool call, and end of call events. It notes that interruption can be paired with speech stop evidence. Other providers use different names, but the contract is portable: retain a call ID, turn ID where available, monotonic timestamps, event type, and a small set of non content fields. There is no universal good value for either interval. The executable fixture used for this article sets maxFirstResponseMs to 1200 and maxInterruptStopMs to 300 so the decision rule is inspectable. Those are example policy values, not industry standards. Calibrate with real network paths, languages, speaking styles, accessibility needs, and the cost of a false failure. A release test should also preserve uncertainty. If user speech stop is missing, do not calculate latency from a transcript timestamp. If the interruption event exists but assistant stop does not, return barge in failed or insufficient evidence according to the collection contract. Never manufacture a healthy interval from incomplete events. Replay a failure precedence before testing live calls The companion artifact contains eight content free call streams. Each event has a relative timestamp and only the fields needed for classification: Run it with: The executed fixture produced exact expected parity across eight verdicts: Precedence is important. Check reachability before timing because an unanswered call cannot have a meaningful first response. Check timing and interruption before tool effects because a caller may already have experienced a broken interaction. Reconcile the tool effect before evaluating terminal completion because retrying an uncertain side effect can duplicate work. Require the outcome last because it is the strongest claim. This order is an operating rule, not a severity ranking. A failed cancellation can have more impact than slow audio. Route severity from the workflow and user consequence after the evidence layer is known. Keep the tool receipt separate from the outcome receipt A voice agent often invokes a scheduling, payment, ticketing, or transfer tool. The model's tool result is not necessarily the destination's durable state. Give every side effecting attempt a stable operation ID. Retain the requested action, destination class, attempt number, response class, and a content free effect check. If transport times out after submission, reconcile that operation ID before retrying. A second request with a new identity may create a duplicate appointment or cancellation. Then verify the caller's promised result independently. A tool effect can be real while the user outcome is still wrong: an appointment exists under the wrong account, a transfer connected to an IVR instead of an operator, or a cancellation changed one item while the requested bundle remains active. The fixture's false complete case is deliberately inconvenient. It has a connected call, a 600 ms first response, a verified tool effect receipt, and a completed call status. It still fails because outcome.verified is absent. That is the exact case a transcript only gate is likely to miss. An LLM evaluator remains useful for qualities that are difficult to encode deterministically: whether the agent acknowledged frustration, explained a policy clearly, or followed an interaction style. Keep its verdict with the transcript evaluation layer. Do not let it overwrite reachability, timestamps, destination receipts, or external state. Ship the smallest privacy safe receipt The classifier does not need caller audio or transcript text. A minimal receipt can contain pseudonymous call and turn IDs, relative timestamps, event types, terminal state, hashed policy version, operation ID, and boolean effect and outcome results. Keep recordings only when consent, retention policy, and debugging value justify them. Before release, run the fixed fixture to prove the classifier itself. Then run a small set of real calls across the carriers, regions, languages, and caller behaviors that matter. Review threshold distributions rather than tuning to one clean lab call. Finally, replay production failures as new regression cases without copying sensitive conversation content into the test suite. The limitation is straightforward: this artifact validates a decision rule, not speech recognition quality or any provider's uptime. It cannot choose thresholds for your callers. It does, however, prevent a polished conversation from being mistaken for verified work. Sidewisp is currently in private preview. Its public experience is an early access website and interactive demonstration; production agent health collection and recovery are not shipped in the current website repository. The product direction is a health layer around existing runtimes, not a voice platform, telephony provider, or autonomous fixer. If this five gate receipt matches the failures you need to catch, you can join the Sidewisp private preview and describe the voice agent runtime you operate.