VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · KV Cache III — Long Contexts, Attention Sinks and Eviction

KV Cache III — Long Contexts, Attention Sinks and Eviction

20 min read·Staff+ Tier· Unrolled Pro
KV Cache III — Long Contexts, Attention Sinks and Eviction

You already know: what the KV cache stores and its size per token (KV cache I); prefix sharing, preemption and KV quantization (KV cache II); softmax and attention (D5 Transformer architecture).

Why this matters for Atlas. Paging removes wasted KV memory; it cannot shrink what a long context genuinely needs. At 128K tokens a 70B model's cache for one request is about 43 GB, so long-context serving runs into a memory wall: few concurrent users and slow decode steps. One response is to evict parts of the cache and keep a fixed budget. Doing that naively breaks models in a surprising way, because of attention sinks. This lesson builds the arithmetic of the wall, shows why dropping the first tokens is catastrophic, and implements the two classic policies, StreamingLLM (sinks plus a rolling window) and H2O (sinks, window and heavy hitters), along with when not to evict at all.

Where you'll use this. Sizing long-context products before launch; choosing between eviction, KV quantization and memory tiers; reviewing a proposal to "just use a sliding window"; evaluating long-document quality honestly.


1. The long-context memory wall

The question. How many 128K-token conversations can an 8-GPU H100 node serve with a 70B model?

Start from nothing. KV bytes per token for the 70B GQA shape: 2 × 80 layers × 8 KV heads × 128 dimensions × 2 bytes = 327,680 bytes (320 KiB). Multiply by context length:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "KV Cache III — Long Contexts, Attention Sinks and Eviction"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.