Domain 12: LLM Inference Engineering
LLM inference engineering: prefill and decode, KV cache sizing and paging, batching, quantization, serving frameworks, capacity planning and cost per token.
Unlock all 17 lessons with Unrolled Pro
Full production code, quantitative capacity math, failure recovery playbooks, and interview coaching.
Lessons
17 lessonsInference Fundamentals — Prefill, Decode and the Roofline
Every LLM product's speed, cost and capacity come from one fact: a response is produced in two very different phases. Prefill processes the whole prompt at once and is limited by arithmetic.
KV Cache I — What It Stores, Exact Sizing, and Paging
After the weights, the KV cache is the largest consumer of GPU memory, and unlike the weights it grows with every token of every active request.
KV Cache II — Prefix Sharing, Preemption, Quantization, Capacity and Operations
Lesson I made the cache fit.
KV Cache III — Long Contexts, Attention Sinks and Eviction
Paging removes wasted KV memory; it cannot shrink what a long context genuinely needs.
Batching & Scheduling — Continuous Batching, Chunked Prefill and Fairness
Batching is how one GPU serves many students; scheduling decides *which* requests share each step.
Quantization — Fewer Bits per Number, Measured Honestly
Quantization stores weights (and sometimes activations and the KV cache) in fewer bits: 8, 4, even fewer, instead of 16.
Advanced Inference Optimization — Speculation, Fused Attention, Parallelism and MoE
Once batching and quantization are in place, the next gains come from changing how much useful work each expensive operation does. Speculative decoding produces several tokens per pass of the big model.
Inference Frameworks — Inside vLLM, SGLang, TensorRT-LLM and Friends
You rarely build an inference engine; you choose, configure and operate one.
Measurement & Capacity — Benchmarks That Don't Lie, and Breaking One GPU on Purpose
Capacity planning answers one question: how many GPUs do we need to serve this traffic within these latency targets, at what cost?
Observability for LLM Serving — Golden Signals, Prometheus and SLOs
Load tests tell you what the system can do; observability tells you what it is doing right now, for real users, and warns you before they notice.
Disaggregated Serving — Splitting Prefill and Decode Across Machines
On a shared GPU, prefill and decode fight: a prefill step delays every decoding sequence (TPOT spikes), and decode-heavy batches delay new prompts (TTFT grows). Every scheduler knob trades one against the other.
Tiered KV Cache — HBM, CPU Memory and SSD for Long Contexts and Reuse
GPU memory is the most expensive and smallest place to keep a KV cache, so a GPU-only prefix cache forgets quickly.
Inference Economics — What a Token Costs, and Which Levers Move It
Inference is where AI products spend most of their money once they have users.
Autoscaling LLM Servers — Queue-Depth Signals, KEDA and Cold Starts
Traffic swings several-fold over a day and spikes within minutes. Too little capacity and students queue; too much and the GPU bill doubles for nothing.
Routing and Token Budgets — Model Cascades, Rate Limits and Load Shedding
Autoscaling controls supply. This lesson controls demand: which model each request goes to, how many tokens each tenant may consume, how long each response may run, and who waits when the fleet is full.
The Inference Reading List — The Papers Behind Every Serving Engine
Engines change every few months; the ideas underneath them change slowly.
Bridge: From Serving Systems to the Silicon Underneath
Atlas's serving stack now batches, pages, quantizes, speculates and caches across tiers.