Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.
This page builds the code. The ideas behind it are taught in the Library lessons listed at the end; read them first if a step uses something you haven't met. Inspect and copy the complete code in the Atlas Codebase Browser and run each step as you read.
Atlas can learn, chat, retrieve and act. This last part makes it cheap and fast enough to serve many students at once. Every idea from the inference domain shows up at small scale: why decoding is slow, how the KV cache fixes the waste, why batching is nearly free throughput, why continuous batching beats static batches, and how all of that becomes a price per token.
Diagram— prefill processes the prompt once and fills the KV cache; each decode step adds one token's keys and values; static batches wait for their longest request while continuous batching refills a slot as soon as a request finishes.
Step 1 · The KV cache
Part 4's generate runs the whole sequence through the model for every new token, recomputing keys and values for positions that haven't changed. The cache keeps each layer's keys and values; a decode step computes only the new token's and attends over the cache.
No causal mask is needed in the decode step: a single query at the newest position may attend to everything before it. The prompt itself goes through prefill once, with the usual causal mask, and fills the cache.
That's the preview. Build Atlas Parts 1 and 2 are free; the full walkthrough of Part 9 is part of Unrolled Pro.
Staff AI & Pro Tier Lesson
Unlock "Build Atlas, Part 9: Serve it fast"
The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.
Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied