VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Inference Fundamentals — Prefill, Decode and the Roofline

Inference Fundamentals — Prefill, Decode and the Roofline

35 min read·Staff+ Tier· Unrolled Pro
Inference Fundamentals — Prefill, Decode and the Roofline

You already know: the transformer block, attention and next-token prediction (D5 Transformer architecture); writing and sampling from your own small GPT (D5 Build an LLM from scratch); latency vs throughput and tail latency (D1 Networking, D11 Model deployment); and the D11 bridge's result that single-stream decoding keeps the GPU's arithmetic units idle more than 99% of the time.

Why this matters for Atlas. Every LLM product's speed, cost and capacity come from one fact: a response is produced in two very different phases. Prefill processes the whole prompt at once and is limited by arithmetic. Decode generates one token at a time and is limited by memory bandwidth. Almost every inference optimization in this domain (KV caching, batching, quantization, speculative decoding, disaggregation) targets one phase or the mismatch between them. This lesson builds both phases in NumPy, measures them with the roofline model, defines the latency metrics you'll use for the rest of the domain, and covers sampling.


1. Two phases: read the prompt, then write the answer

The question. Why does a long prompt make Atlas slow to start, while a busy server makes it slow to type?

Start from nothing. Generation is a loop. To produce token t+1, the model needs the attention keys and values of every earlier token.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Inference Fundamentals — Prefill, Decode and the Roofline"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.