VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · KV Cache II — Prefix Sharing, Preemption, Quantization, Capacity and Operations

KV Cache II — Prefix Sharing, Preemption, Quantization, Capacity and Operations

35 min read·Staff+ Tier· Unrolled Pro
KV Cache II — Prefix Sharing, Preemption, Quantization, Capacity and Operations

You already know: what the KV cache stores, how to size it, and how paging allocates it (KV Cache I); quantization basics (D12 Quantization is next, this lesson previews the cache part); prompt caching from the API side (D7).

Why this matters for Atlas. Lesson I made the cache fit. This lesson makes it efficient and dependable under real traffic: sharing blocks across requests with a common prefix, deciding what to do when memory runs out, quantizing the cache to fit twice the tokens, planning capacity from a config and the engine's settings, and operating it with the right metrics so you see trouble before users do.

Where you'll use this. Cutting time to first token by restructuring prompts so the cache hits. Choosing vLLM or SGLang settings (memory utilization, maximum sequences, KV dtype, prefix caching) for a deployment. Deciding whether to buy more GPUs or switch to an FP8 cache. Building the dashboard and alerts for a model server.


1. Prefix caching: share what requests have in common

The question. Every Atlas request starts with the same 3,000-token system prompt and tool definitions. Why compute and store its KV cache thousands of times?

Start from nothing. Two requests that begin with the same tokens produce identical keys and values for that prefix (the cache of a token depends only on the tokens before it). So the engine can hash each full block of tokens together with everything before it, keep blocks in a cache keyed by that hash, and when a new request arrives, reuse every leading block whose hash is already present: no prefill compute and no extra memory for the shared part. Blocks are reference-counted: a shared block is freed only when no request uses it. This is the server-side version of the prompt caching that API providers bill at a discount (D7), and it's why "stable content first, variable content last" matters.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "KV Cache II — Prefix Sharing, Preemption, Quantization, Capacity and Operations"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.