You already know: what the KV cache stores and its size per token (KV cache I); prefix sharing, preemption and KV quantization (KV cache II); softmax and attention (D5 Transformer architecture).
Why this matters for Atlas. Paging removes wasted KV memory; it cannot shrink what a long context genuinely needs. At 128K tokens a 70B model's cache for one request is about 43 GB, so long-context serving runs into a memory wall: few concurrent users and slow decode steps. One response is to evict parts of the cache and keep a fixed budget. Doing that naively breaks models in a surprising way, because of attention sinks. This lesson builds the arithmetic of the wall, shows why dropping the first tokens is catastrophic, and implements the two classic policies, StreamingLLM (sinks plus a rolling window) and H2O (sinks, window and heavy hitters), along with when not to evict at all.
Where you'll use this. Sizing long-context products before launch; choosing between eviction, KV quantization and memory tiers; reviewing a proposal to "just use a sliding window"; evaluating long-document quality honestly.
1. The long-context memory wall
The question. How many 128K-token conversations can an 8-GPU H100 node serve with a 70B model?
Start from nothing. KV bytes per token for the 70B GQA shape: 2 × 80 layers × 8 KV heads × 128 dimensions × 2 bytes = 327,680 bytes (320 KiB). Multiply by context length: