VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · KV Cache I — What It Stores, Exact Sizing, and Paging

KV Cache I — What It Stores, Exact Sizing, and Paging

35 min read·Staff+ Tier· Unrolled Pro
KV Cache I — What It Stores, Exact Sizing, and Paging

You already know: prefill builds the KV cache and decode reads it every step (Inference fundamentals); multi-head, multi-query and grouped-query attention (D5 Transformer architecture); virtual memory, pages and fragmentation, including the PagedAttention preview (D1 Operating systems); prompt caching from the API side (D7 Context engineering).

Why this matters for Atlas. After the weights, the KV cache is the largest consumer of GPU memory, and unlike the weights it grows with every token of every active request. It decides how many requests fit in a batch (throughput), how long a context you can serve, and how much a repeated system prompt costs. Serving engines spend most of their cleverness managing it: allocating it in pages, sharing it across requests with a common prefix, evicting or offloading it under pressure. This lesson explains what the cache holds, computes its size exactly for real architectures, and builds the paged allocator every modern engine uses. Lesson II (KV Cache II) covers sharing it across requests, what happens when it runs out, quantizing it, planning capacity and operating it.

Where you'll use this. Answering "how many concurrent users can this GPU serve at 8K context?" before buying hardware. Understanding why a model with grouped-query attention serves several times more users than one without. Reading a serving engine's startup log ("# GPU blocks: 25,000") and knowing what it means for your traffic.


1. What is cached, and why it's worth it

The question. Why store keys and values at all, instead of recomputing them?

Start from nothing. At decode step t, attention needs the keys and values of all t earlier tokens in every layer. Without a cache you'd recompute them from scratch each step: generating n tokens would cost about n²/2 token-forwards instead of n. With a cache, each token's K and V are computed once (in prefill or in its own decode step) and read back afterwards. Queries are never cached: each is used once. The trade: compute saved for memory spent.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "KV Cache I — What It Stores, Exact Sizing, and Paging"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.