Voice agents fail in ways that text tests never catch. The agent talks over the caller, answers slowly, or drifts off-script. Agent evaluation lets you pin that behavior down. You script a conversation. You assert what the agent should do. You get a pass/fail result plus latency and quality metrics, reproducibly, in CI. A simulated caller connects to your agent over a WebSocket. No phone number, no browser, and no SIP are involved. The caller speaks your scripted turns with text-to-speech. The evaluator transcribes and grades the agent’s replies. The run exits 0 only if every assertion passes. It writes a JSON report with per-turn metrics.
Evaluation runs against a real agent, over the same runtime path that a live call uses. It uses the same speech-to-text, language, and speech providers that your agent runs with. Thus a green run reflects real behavior, not a mock.

​
Write a scenario

A scenario is JSON. It sets who the caller is, which agent to test, the scripted turns, and the assertions on each turn and on the call as a whole.
Set agent_key to the agent you built in the console. Or set it to "" to smoke-test the default speech-to-text → language → speech pipeline with no agent configured. A turn is either scripted user text (spoken through the TTS provider) or a wav fixture (PCM16, mono or stereo, any sample rate). Use a wav fixture when you need a specific voice, accent, or noise profile.

​
Assertion types

Each turn carries expect assertions. The call carries call-level expectations, which the judge grades against the whole transcript.
​
{ ms: number }
Latency gate: the gap from the caller’s last speech frame to the agent’s first audio frame must be under ms. It catches a slow agent before your users do.
​
{ pattern: string }
Case-insensitive regex over the agent’s transcript for that turn. It is good for must-say phrases, disclosures, or IDs.
​
{ criterion: string }
LLM-as-judge: the configured judge model returns a strict { "pass": bool, "reason": string } verdict against your plain-English criterion. It fails closed. A malformed or flaky judge reply counts as a fail. Thus a bad judge can never silently green a run.
​
{ name: string, args_contains?: object }
Assert that the agent called a tool by the end of the turn. You can also assert certain argument values.
Preview — pairs with the runtime’s tool-call surface.

​
Metrics in the report

Every run writes a JSON report with per-turn timings. All timings are measured client-side: Pair these metrics with the live-call latency breakdown and MOS / jitter / loss to compare test-bench numbers against production.

​
Run an evaluation

1

Point at an agent

Build and save the agent in the console, so its config is live. Note its agent_key. To smoke-test with no agent, use agent_key: "". The default pipeline then runs off your provider keys.
2

Provide credentials

The simulated caller reuses the same provider clients that your agent uses. Thus it needs keys for the TTS, ASR, and judge providers in your scenario (e.g. Deepgram, ElevenLabs, OpenAI). Set the keys in the environment.
3

Run the evaluator

Point the evaluator at your engine and scenario. Run it. The evaluator connects, plays the scripted call, grades every assertion, and writes the report.
Today the evaluator ships as a standalone binary. You run it against a self-hosted or managed engine. The clutchcall eval wrapper is the convenience entry point. See Self-Hosted deployment to run it against your own engine.
4

Read the result

The process exits 0 only if every assertion passed. The JSON report lands in your output directory. The report holds the per-turn metrics above and, for each failed assertion, the judge’s reason.

​
Gate your CI on it

The run is a single command with a meaningful exit code. Thus you can drop it into CI to block a regression before it ships:
Keep a suite of scenarios for each agent. Write one scenario per critical flow (billing, refund, escalation), plus a latency-only smoke scenario. The agent_key: "" default pipeline gives you a zero-config smoke test that needs only provider keys. Thus CI can catch a broken transport or a provider outage without a saved agent.

Turn Detection & Barge-in

Tune what your latency and barge-in assertions test.

Tool Calling

Learn the tools that function_call assertions check for.

Runtime Configuration

Learn the agent config that an evaluation runs against.

Latency Breakdown

Compare bench latency to production.