VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D9: AI Agents & Tool Use · Voice Agents — Real-Time Speech In and Out, Under an 800 ms Budget
Phase 5/Domain 9/Lesson 6

Voice Agents — Real-Time Speech In and Out, Under an 800 ms Budget

15 min read·Advanced· Unrolled Pro
Voice Agents — Real-Time Speech In and Out, Under an 800 ms Budget

You already know: agent loops and tool calling (D9.1, D9.3); speech recognition and synthesis basics (D18.3); time to first token and streaming (D12).

Why this matters for Atlas. In text, a 3-second answer feels fine. In conversation, a pause longer than about a second feels broken, and people start talking again, which the agent then has to handle. A voice agent is an agent whose every design choice is shaped by latency and turn-taking: stream every stage, decide quickly but correctly when the caller has finished, stop speaking the moment they interrupt, and confirm before doing anything irreversible because the caller can't see a screen. This lesson builds that, piece by piece.


1. The loop and the latency budget

The question. How fast does a voice agent have to respond, and where does the time go?

Start from nothing. In human conversation, the gap between turns is typically a few hundred milliseconds. A practical target for a voice agent is under about 800 ms from the caller stopping to the agent starting to speak. The classic cascade has four stages:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Voice Agents — Real-Time Speech In and Out, Under an 800 ms Budget"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 8 of 9: Let it actUse this lesson in the project: Give Atlas tools, memory and a loop, then put a gateway between the model and the tools so that a fooled model still can't do damage.