inference data flow

A duplex speech-to-speech turn over one QUIC connection: caller audio in, model audio out, turn detection in the middle.

Add a real-time voice AI agent to your calls. Originate or answer a call, then attach an agent. ClutchCall bridges the audio both ways to a single speech-to-speech model. Caller audio goes up. The agent’s voice comes down. The modality uses the Voice path, so it reuses the same call that you already place. ClutchCall handles the hard parts of a natural voice agent: who has the floor, when the model can answer, and how to keep a backchannel (“mhm”, “right”, “ok”) from derailing the turn. That logic lives in ClutchCall, not in your code. You get one speech-to-speech model. It takes audio in and sends audio out directly, with no transcript hop in the middle. Compare this with a chained ASR → LLM → TTS pipeline: the model hears you as you speak, so it starts to answer sooner. Tone and prosody survive end-to-end, because there is no transcript boundary to flatten them. If you need a specific ASR/LLM/TTS vendor, ClutchCall also supports that cascade — see Turn Detection.

​
How the audio flows

The model leg reuses the Voice modality’s track convention. A call carries two audio tracks. The agent attach binds the model to both:
Frames flow continuously. On the uplink, ClutchCall streams caller audio to the model at 1x real-time as it arrives. The model perceives the utterance during the listening window, not in a burst after the window ends. On the downlink, the model’s audio deltas stream directly back to the caller as the model produces them.
There is no “send the whole utterance, then wait” round-trip. Continuous prefill plus the commit gate (below) gives a near-zero gap between end-of-speech and the first audio-out frame.

​
Turn-taking: the client-commit gate

A server-side VAD model decides on its own when your turn ended, and then it starts to talk. A speech-to-speech model has no server VAD. Thus ClutchCall makes the decision of when the user’s turn ended, locally. This is a feature, not a gap: it is the protocol-clean place to gate backchannels. ClutchCall runs a local turn detector over the inbound caller audio and drives the model with explicit signals:
1

Append (continuous)

ClutchCall appends caller audio frames to the model’s input buffer at 1x real-time. The model already “listens” while the caller talks.
2

Prime (open turn)

A non-final commit keeps a turn primed and does not request a reply yet. The listening window stays open.
3

Commit (end of turn)

When the turn detector decides that a real turn ended, a final commit flushes the tail. The model then generates its reply. Only a real turn produces a commit.

​
Backchannel suppression

Do not treat short continuers (“ok”, “mhm”, “yeah”, “right”) as a turn. The turn detector filters each speech burst shorter than response_min_speech_ms (default 600 ms). Such a burst never produces a final commit. The model never wakes up to answer it, and the agent keeps its floor.

​
Barge-in: hold-and-confirm

Sometimes the caller speaks over the agent. Naive logic cancels the agent on the first speech frame, so a muttered “mhm” mid-sentence kills the whole reply. ClutchCall instead arms a pending barge-in at speech onset. It fires the barge-in only after speech continues for barge_confirm_ms (default 300 ms). A backchannel ends before the window elapses: there is no cancel, and the agent keeps talking. A real interruption runs past the window: ClutchCall cancels the agent, approximately barge_confirm_ms later than a naive cut. That delay is the only cost.
On a transport with no echo cancellation (raw SIP/PSTN), the agent’s own audio bleeds back through the mic and can trigger barge-in by itself. Raise the during-TTS mic gate above the quiet gate for those legs. On a browser leg with client-side AEC, a lower during-TTS gate keeps barge-in fast. Both are turn-detector params.

​
Codecs

The model wants raw audio. The caller leg can use any codec. ClutchCall transcodes at the bridge. You select what the model expects, and ClutchCall handles the call leg for you.

​
The metric that matters: turn latency

The number that a caller feels is turn latency:
turn latency = end-of-user-speech (last inbound audio frame) → start-of-agent-speech (first outbound audio-out frame)
Report p50 / p95 / p99, not only the mean. The tail is the product story. Two properties of this modality drive turn latency down:
  • Continuous prefill. The model hears the utterance as it streams, so the final commit only flushes the tail. The first audio out can land within tens of milliseconds of end-of-speech.
  • QUIC under load and loss. Over a WAN-emulated link (50 ms RTT, ~1% loss), steady-state QUIC first-token latency is approximately half of TCP+TLS, with a much cleaner p99 tail. There is no head-of-line blocking when a packet drops mid-turn.

​
Tool calling

A speech-to-speech agent is not only a voice. It can act. ClutchCall exposes function (tool) calling to the model:
  1. You declare a set of tools on the agent.
  2. The model decides to call one tool mid-conversation.
  3. ClutchCall runs the tool.
  4. ClutchCall feeds the result back to the model, and the model speaks the answer.
The caller hears a natural reply. The lookup happened in the background. You declare tools on the agent. They are part of the agent config that the control plane stores; an inline AgentSpec mirrors the same shape. Each tool advertises a name, a description, and a JSON-schema for its arguments. This is what the model sees. ClutchCall runs the call. It feeds the structured result back as a tool_result that the model can reason over.

​
Tool kinds

​
Built-in vs operator tools

Every agent session automatically advertises a set of built-in telephony tools: transfer the call, place it on hold, send DTMF, and so on. The model can use them with no configuration. The tools that you declare on the agent are operator tools. On a name collision, the operator tool wins. Thus a tool with a built-in name overrides the built-in tool. silent controls whether the model narrates the call. With silent: false, the model can say “let me look that up”. With silent: true, it stays quiet until it has the result.
Tool invokes are synchronous within the model turn. A slow upstream eats the turn budget, and the caller hears dead air. Keep individual tools fast (sub-second is ideal). Or split the work across multiple turns instead of cascaded tool calls inside one turn.
The full operator spec lives in Agent DAGs. It covers every field of a tool entry, the auth shapes, and the template substitution rules. The SDK Methods page documents the inline ToolSpec that you attach with an agent.

​
How it works

You do not have to think about these items to ship an agent. But if you chase the tail latency, these items keep it low.
  • Your agent stays responsive because stages overlap. The model works on the audio that it already heard while the caller still talks. The connection model and the commit gate make that overlap safe.
  • A dropped packet never stalls the conversation. Uplink audio, downlink audio, and control signals (commit / cancel) use independent MoQT streams over one multiplexed QUIC connection. A loss on one stream does not hold up the others.
  • Reconnects stay cheap for callers at the edge. A tuned QUIC ingress terminates the caller’s connection and forwards to the model-serving fabric. The model’s routing brain (cache-aware / prefix-aware worker selection) stays untouched behind the ingress. 0-RTT resumption keeps reconnect churn low.
  • Latency holds even under load. The audio fast path runs a kernel-bypass NIC path with lock-free mcache / dcache rings on a thread-per-core reactor and io_uring. Per-turn transport overhead stays in the low single-digit milliseconds, even at hundreds of concurrent turns.

​
When to use it

Use inference when

You want a single speech-to-speech model in the loop and the lowest possible turn latency. You want prosody preserved end-to-end. You want ClutchCall to own turn-taking and barge-in.

Use the cascade when

You need a specific ASR vendor, a text LLM with tools/RAG, or a particular TTS voice. The Turn Detection path chains ASR → LLM → TTS with the same barge-in policy.