VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Disaggregated Serving — Splitting Prefill and Decode Across Machines
Phase 6/Domain 12/Lesson 11

Disaggregated Serving — Splitting Prefill and Decode Across Machines

25 min read·Staff+ Tier· Unrolled Pro
Disaggregated Serving — Splitting Prefill and Decode Across Machines

You already know: prefill is compute-bound and decode memory-bound (Inference fundamentals); chunked prefill reduces but doesn't remove their interference (Batching & scheduling); KV cache size per token (KV cache engineering); goodput under SLOs (Measurement & capacity); network bandwidth and latency (D1 Networking).

Why this matters for Atlas. On a shared GPU, prefill and decode fight: a prefill step delays every decoding sequence (TPOT spikes), and decode-heavy batches delay new prompts (TTFT grows). Every scheduler knob trades one against the other. Disaggregated serving stops the fight by running them on separate GPU pools: prefill workers process prompts and ship the resulting KV cache over a fast network to decode workers, which generate the tokens. Each pool can be sized, batched and even built from different hardware for its own phase. This is how several of the largest LLM services run today. This lesson explains when the split pays, what the KV transfer costs, and how to size the two pools.


1. The interference problem, measured

The question. Chunked prefill helped. Why isn't it enough?

Start from nothing. With colocated serving, each step mixes some prefill tokens with the decode batch. Every prefill token in a step makes that step longer for all decoding sequences, so TPOT rises with prefill load. You can cap prefill per step (small chunks) to protect TPOT, but then long prompts take more steps and TTFT rises. One budget, two SLOs: you can't fully satisfy both when prompts are long and traffic is high.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Disaggregated Serving — Splitting Prefill and Decode Across Machines"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.