You already know: what the KV cache stores, how to size it, and how paging allocates it (KV Cache I); quantization basics (D12 Quantization is next, this lesson previews the cache part); prompt caching from the API side (D7).
Why this matters for Atlas. Lesson I made the cache fit. This lesson makes it efficient and dependable under real traffic: sharing blocks across requests with a common prefix, deciding what to do when memory runs out, quantizing the cache to fit twice the tokens, planning capacity from a config and the engine's settings, and operating it with the right metrics so you see trouble before users do.
Where you'll use this. Cutting time to first token by restructuring prompts so the cache hits. Choosing vLLM or SGLang settings (memory utilization, maximum sequences, KV dtype, prefix caching) for a deployment. Deciding whether to buy more GPUs or switch to an FP8 cache. Building the dashboard and alerts for a model server.
1. Prefix caching: share what requests have in common
The question. Every Atlas request starts with the same 3,000-token system prompt and tool definitions. Why compute and store its KV cache thousands of times?
Start from nothing. Two requests that begin with the same tokens produce identical keys and values for that prefix (the cache of a token depends only on the tokens before it). So the engine can hash each full block of tokens together with everything before it, keep blocks in a cache keyed by that hash, and when a new request arrives, reuse every leading block whose hash is already present: no prefill compute and no extra memory for the shared part. Blocks are reference-counted: a shared block is freed only when no request uses it. This is the server-side version of the prompt caching that API providers bill at a discount (D7), and it's why "stable content first, variable content last" matters.