VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Tiered KV Cache — HBM, CPU Memory and SSD for Long Contexts and Reuse
Phase 6/Domain 12/Lesson 12

Tiered KV Cache — HBM, CPU Memory and SSD for Long Contexts and Reuse

15 min read·Staff+ Tier· Unrolled Pro
Tiered KV Cache — HBM, CPU Memory and SSD for Long Contexts and Reuse

You already know: KV sizes, paging, prefix caching and preemption by swap or recompute (KV cache engineering); the memory hierarchy and paging to disk (D1 Operating systems); radix-tree prefix caches and cache-aware routing (Inference frameworks); moving KV between machines (Disaggregated serving).

Why this matters for Atlas. GPU memory is the most expensive and smallest place to keep a KV cache, so a GPU-only prefix cache forgets quickly. Yet the reuse opportunity is enormous: multi-turn conversations, agents that call the model dozens of times on a growing context, shared documents many students ask about. A tiered KV cache keeps hot KV in GPU HBM, warm KV in CPU DRAM, and cold KV on local SSD (or a shared store across machines), and decides per request whether loading cached KV is faster than recomputing it. This is the memory-hierarchy thread at its most direct, and it's what turns "every return visit repays the whole prefill" into "only the new message is computed".


1. The hierarchy and its numbers

The question. What does each tier offer, and how much KV can it hold?

Start from nothing. Each step down is larger, cheaper and slower:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Tiered KV Cache — HBM, CPU Memory and SSD for Long Contexts and Reuse"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.