BYO ASR / LLM / TTS Models

Run a cascaded agent when you want to pick the best model for each stage instead of a single duplex model. The runtime chains three provider sessions per call:

caller audio (PCM16, 8 kHz)
   -> ASR   (speech -> text)          e.g. Deepgram
   -> LLM   (text  -> reply text)     e.g. OpenAI / Anthropic / Gemini / Ollama
   -> TTS   (text  -> speech)         e.g. ElevenLabs / Cartesia
   -> caller audio

You choose each provider independently in one agent-config.yml. You drop in the credentials. The ClutchCall Agent Runtime then opens a streaming session with each vendor on the caller’s shard. You write no code. The runtime assembles the pipeline from config.

Do you prefer a single low-latency duplex model? See BYO speech-to-speech. That path sets pipeline_mode: realtime and hands the whole turn to one provider (OpenAI Realtime, Gemini Live, and others). A cascade gives you more control and lets you swap each stage. Speech-to-speech gives you the lowest turn latency.

Set the pipeline

Choose your three providers

Pick one id from each table below. A bare cascade falls back to the defaults: deepgram for ASR, openai for LLM, and elevenlabs for TTS. Thus you can start and override only the stage you care about.

Add credentials

Each provider reads a per-tenant key. The runtime resolves the key at call setup (see Credentials). Set the keys for the providers you selected. The runtime refuses to start an agent that has no usable provider key.

Write agent-config.yml

Set pipeline_mode: cascaded and the per-stage provider/model/voice fields. The full example is below.

Save and place a test call

The control plane hydrates the config into the runtime. No rebuild is necessary. Place a call. Then watch the latency breakdown to see per-stage timing (ASR final, LLM first token, TTS first audio).

agent-config.yml

name: support-agent
pipeline_mode: cascaded          # ASR -> LLM -> TTS chain (vs. "realtime")

# --- Speech-to-text ---
asr_provider: deepgram
asr_model: nova-2                # forwarded onto the ASR socket

# --- Language model ---
llm_provider: openai
llm_model: gpt-4o-mini
temperature: 0.4
conversation_memory_messages: 20 # rolling history the runtime replays each turn
llm_max_retries: 2               # retry the primary provider on a failed turn...
llm_fallback_provider: anthropic # ...then fall back once to this provider

# --- Text-to-speech ---
tts_provider: elevenlabs
tts_voice: <voice-id>            # ElevenLabs voice_id / Cartesia UUID / Deepgram Aura voice

# --- Shared ---
language: en                     # ISO 639-1 hint; pins the agent's output language
system_prompt: >
  You are a friendly support agent for Acme. Keep replies short and spoken-friendly.

The runtime processes audio as PCM16 LE at 8 kHz end to end. The runtime feeds and drains every provider you pick at that rate. It opens Deepgram with encoding=linear16. Some providers natively want 16/24 kHz; the runtime resamples for them.

ASR providers

Set asr_provider. You can also set asr_model; the runtime forwards it as the provider’s model hint.

`asr_provider`	Vendor	Notes
`deepgram` (default)	Deepgram	Streaming, interim + final transcripts. Default model `nova-2`.
`elevenlabs`	ElevenLabs	Streaming ASR surface (shares your ElevenLabs key).

More streaming ASR ids exist. They give Indic and regional coverage, and self-hosted transcription on a co-located model server. See Self-hosted inference. Deepgram is the recommended default for English telephony.

LLM providers

Set llm_provider and llm_model. Tool calling runs in the LLM stage. See Tool calling.

OpenAI
Anthropic
Gemini
Ollama (self-hosted)

llm_provider: openai      # default when llm_provider is unset
llm_model: gpt-4o-mini

llm_provider: anthropic
llm_model: claude-sonnet-4-5

This provider honors in-flight cancellation. A barge-in stops the running request, so you do not pay for a discarded turn.

llm_provider: gemini
llm_model: gemini-2.5-flash

llm_provider: ollama
llm_model: llama3.1

This provider points at a local Ollama endpoint (OLLAMA_ENDPOINT, default 127.0.0.1:11434). No API key is necessary. Use this option to keep the language model on your own infrastructure while you use cloud ASR/TTS.

If the primary provider fails after llm_max_retries, llm_fallback_provider runs a different provider entirely. Thus a vendor incident degrades one turn and does not drop the call. This is a resilience path, not a load balancer.

TTS providers

Set tts_provider. Put the provider’s voice id in tts_voice.

`tts_provider`	Vendor	Voice field
`elevenlabs` (default)	ElevenLabs	`tts_voice` = ElevenLabs `voice_id` (streaming, low first-audio).
`cartesia`	Cartesia (Sonic)	`tts_voice` = Cartesia voice UUID (streaming WebSocket).
`cartesia-http`	Cartesia	Same voices, one-shot HTTP fallback (no streaming).
`deepgram`	Deepgram (Aura)	`tts_voice` = Aura voice, e.g. `aura-asteria-en`.
`openai`	OpenAI	Reuses your OpenAI key; one-shot per utterance.

Streaming TTS providers emit audio as the LLM text arrives. This keeps first-audio low. Sometimes a provider cannot open a streaming session, for example because of a missing voice or a bad key. Then the runtime falls back to a one-shot synthesis for that turn, so the caller still hears a reply.

Credentials

Provider keys are per-tenant. Set them in the ClutchCall console at agent.clutchcall.dev. The control plane resolves the keys at call setup, seals them at rest, and injects them into the pipeline. The keys are never baked into agent-config.yml. For single-tenant or local development, the runtime also reads keys from the environment as a fallback:

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export DEEPGRAM_API_KEY=...
export ELEVENLABS_API_KEY=...
export CARTESIA_API_KEY=...

# Ollama needs no key — just the endpoint
export OLLAMA_ENDPOINT=127.0.0.1:11434

If an agent’s selected providers have no usable key, the runtime refuses the agent at session creation with a clean error. The agent does not start and then fail mid-call. Check that each stage’s key is present before you route live traffic to the agent.

What runs around the cascade

Turn detection. A cascade uses local, on-device VAD by default. The runtime owns endpointing and barge-in; it does not use a cloud provider’s server VAD. Tune the thresholds, the barge-in hold window, and the idle watchdog on the turn detection page.
Language. The language hint prepends a hard directive to the system prompt. The model then answers in your market’s language and does not mirror the caller.
Session limits. Connect, idle, and max-session deadlines apply per stage. See Session lifecycle.

Backchannel suppression in a cascade is acoustic. It uses utterance duration plus a hold-and-confirm barge window. Short “mhm”/“ok” replies do not interrupt the agent. Lexical gating (“stop” vs “ok”) is not built. Plan around the acoustic behavior. See turn detection.

Next steps

Runtime configuration

This reference lists every knob: turn detection, budgets, guardrails, and codecs.

Speech-to-speech

Trade per-stage control for the lowest turn latency with a single duplex model.

Tool calling

Give the LLM stage HTTP, MCP, and client tools.

Latency breakdown

See per-stage ASR / LLM / TTS timing on a live call.

​Set the pipeline

​agent-config.yml

​ASR providers

​LLM providers

​TTS providers

​Credentials

​What runs around the cascade

​Next steps

Runtime configuration

Speech-to-speech

Tool calling

Latency breakdown

Set the pipeline

agent-config.yml

ASR providers

LLM providers

TTS providers

Credentials

What runs around the cascade

Next steps