How Does an AI Voice Agent Actually Work? A Technical Teardown (STT → LLM → TTS)

An AI voice agent works by chaining three models over a live audio stream: speech-to-text (STT) turns the caller's voice into text, a large language model (LLM) decides what to say, and text-to-speech (TTS) turns that reply back into audio — with telephony carrying the signal at every step. Built well, the full round trip lands somewhere between roughly 450ms and 1.2 seconds, and costs between $0.05 and $0.25 per minute depending on which vendors sit behind each stage (Telnyx, 2026; Klariqo, 2026).

For venture-backed founders and CTOs scoping an AI phone agent — support, outbound sales, scheduling — the pipeline is the product. Get the latency wrong and callers hang up mid-sentence; get the cost math wrong and your unit economics break the moment you scale past a pilot. This teardown walks through each stage of the stack, shows where the milliseconds and cents actually go, and sets out when it's worth building a custom pipeline instead of paying a platform markup on top of it.

What's actually happening inside the STT → LLM → TTS pipeline?

Every turn of a voice AI conversation runs through four systems in sequence: telephony (the phone line itself), STT (transcribing the caller), an LLM (composing a reply) and TTS (speaking it back). None of these come from a single vendor by default — most production stacks stitch together separate providers for each stage, and that seam is exactly where extra latency and cost creep in (Retell AI, 2026).

Stage	What it does	Typical latency
Telephony / SIP	Carries the call audio between caller and your stack	30–150ms
Voice activity detection & turn-taking	Decides when the caller has actually stopped talking	150–300ms
STT (speech-to-text)	Transcribes the audio into text	50–225ms
LLM time-to-first-token	Generates the reply	150–650ms
TTS time-to-first-audio	Converts the reply back into speech	80–200ms

Turn-taking — deciding a caller has actually finished a sentence, not just paused for breath — is usually the single biggest variable in the stack, and the reason naive implementations interrupt people or leave awkward silences (Telnyx, 2026).

Where does the latency actually go?

The gap between a co-located, single-vendor stack and a bolted-together multi-vendor one is large. Telnyx's own architecture benchmark puts a co-located pipeline at roughly 450ms total — 45ms network, 100ms STT, 225ms LLM inference, 80ms TTS — against 1,210ms for a typical multi-vendor stack running the same call (Telnyx, 2026). Independent testing backs this up: a March 2026 study of 500 production calls found Retell averaging 680ms median round-trip latency (920ms p95), Vapi at 720ms (1,050ms p95), and Bland at 850ms (1,180ms p95) (Telnyx, 2026).

As a rule of thumb, conversations start to feel unnatural once total round-trip latency crosses roughly 700ms — callers notice the pause even if they can't name it. That threshold is why teams obsess over streaming: partial STT transcripts every ~50ms, LLM tokens streamed as they're generated rather than waiting for a full response, and TTS audio streamed in small chunks rather than rendered whole (Retell AI, 2026).

How much does an AI voice agent actually cost per minute?

Cost follows the same four-stage shape as latency. Speech-to-text is the cheapest line item, typically $0.003–0.008 per minute, with Deepgram Nova-3 often cited around $0.0077/min. Raw LLM inference is also small on paper — under $0.01/min for most models — but a single conversational turn commonly triggers around five separate model requests (tool calls, guardrail checks, response generation), which is where budgets quietly balloon (Klariqo, 2026).

Component	Typical cost per minute
Telephony (e.g. Twilio outbound + media stream)	~$0.020
Speech-to-text (e.g. Deepgram Nova-3)	$0.003–0.008
LLM inference (raw)	<$0.01, but ~5 requests per turn
Text-to-speech	$0.036 (Deepgram Aura-2) to $0.12–0.25 (ElevenLabs)
Platform orchestration fee (e.g. Vapi)	~$0.05

Stack those together and a realistic blended cost lands around $0.12–0.24 per minute even before you factor in an expensive TTS tier or high LLM usage per turn (CloudTalk, 2026). At 50,000 minutes a month — a modest but real production volume — that's the difference between roughly $6,000 and $12,000, before headcount savings are even counted.

Why doesn't the advertised "$0.05/min" rate match your bill?

Because it's rarely the full stack. Platform headline pricing usually covers orchestration only — not the STT, LLM and TTS vendors running underneath it. One detailed teardown of Vapi found its advertised $0.05/min becomes $0.14–0.22/min once every component is included: a 2.8–4.4x gap between the marketing number and the invoice (Klariqo, 2026). None of this is dishonest, exactly — it's just that "voice AI requires four components to work," and most published rates quote one of them (Klariqo, 2026). If you're pricing a build, ask any vendor or agency for the fully blended per-minute number, not the headline figure.

How big is this market, and why is everyone racing to ship?

The pull toward voice AI is a genuine market shift, not just hype. The voice AI agents market was valued at roughly $2.4 billion in 2024 and is projected to reach $47.5 billion by 2034, a 34.8% CAGR (Ringly, 2026). The broader voice recognition market alone is estimated at $22.49 billion in 2026 (Ringly, 2026). The economics explain the pace: an AI-handled call costs roughly $0.40, against $7–12 for a human agent handling the same call — a 90–95% reduction that's hard for any support or sales org to ignore once the latency and reliability bar is cleared (Ringly, 2026).

Build vs buy: when should you build a custom pipeline?

For most teams testing an idea, a platform (Vapi, Retell, Bland) is the right starting point — you trade a per-minute markup for weeks of engineering time you don't have to spend. The calculus flips once you have real call volume and a specific latency or cost target the platform can't hit: at that point, picking your own STT/LLM/TTS vendors, co-locating them, and owning the orchestration layer directly usually pays for itself within a few months of production minutes.

This is the same build-vs-buy trade-off we cover in more general terms in our breakdown of what it costs to build an AI MVP in 2026 — the short version is that de-risking the architecture before committing engineering time saves far more than it costs. At UZO Lab we build exactly this kind of stack under our AI Agents & Agentic Workflows and LLM Integration & AI Product Features work, and we typically start with a short, fixed-scope AI Prototyping & R&D Sprint (day rate from £500, or £2,000 for a full week) to prove the latency and cost numbers on your actual use case before any bigger commitment. Ongoing builds move to a Growth retainer from £5,000/month.

FAQ

What's the single biggest lever for reducing voice AI latency?

Turn-taking and LLM time-to-first-token together dominate the latency budget, so the two highest-leverage fixes are tuning voice-activity-detection sensitivity and streaming LLM tokens as they generate rather than waiting for a complete response — this alone can cut several hundred milliseconds off a naive implementation (Retell AI, 2026).

Is it cheaper to build a custom voice AI stack or use a platform like Vapi or Retell?

Below roughly 10,000–20,000 minutes a month, a platform's markup is usually cheaper than the engineering time a custom stack demands. Above that, owning your STT/LLM/TTS selection directly typically undercuts platform pricing, since the platform fee alone can run ~$0.05/min on top of the underlying vendor costs (CloudTalk, 2026).

What's a realistic end-to-end latency target for a production voice agent?

Aim for under 700ms round-trip; co-located, single-vendor architectures can reach ~450ms, while stitched multi-vendor stacks commonly land above 1.2 seconds (Telnyx, 2026).

How much should I budget per minute for a production voice AI agent?

Plan for a blended $0.12–0.25 per minute once telephony, STT, LLM and TTS are all accounted for — not the $0.05/min headline rate most platforms lead with (Klariqo, 2026).

Do I need a specialist AI agency to build a voice AI product?

Not necessarily for a first prototype, but the latency and cost trade-offs above are easy to get wrong on a first attempt. A short, fixed-scope sprint with a team that's shipped voice pipelines before is usually the fastest way to find out whether your use case is viable before committing to a full build.

Ready to see what this looks like for your product? Book a free 30-minute workshop and we'll map the latency and cost budget for your specific use case.

Sources: Telnyx, "Voice AI Agents Compared on Latency: 2026 Benchmarks"; Retell AI, "How Real-Time Voice AI Actually Works"; Ringly, "47 Voice AI Statistics for 2026"; CloudTalk, "Vapi AI Pricing 2026"; Klariqo, "AI Voice Agent Pricing 2026".