You already know: decode is memory-bound and batching raises intensity (Inference fundamentals); paging, prefix caching and scheduling (KV cache engineering, Batching & scheduling); quantized weights and caches (Quantization); mixture-of-experts layers (D5 Architecture families); LoRA adapters (D6 Parameter-efficient fine-tuning).
Why this matters for Atlas. Once batching and quantization are in place, the next gains come from changing how much useful work each expensive operation does. Speculative decoding produces several tokens per pass of the big model. FlashAttention stops attention from shuttling huge intermediate matrices through memory. CUDA graphs and kernel fusion remove overhead between tiny kernels. Tensor parallelism spreads one model over GPUs. MoE and multi-LoRA serving each have their own bottlenecks. This lesson explains each from first principles with runnable models, including a proof (by simulation) that speculative decoding doesn't change the output distribution.
1. Speculative decoding: guess cheaply, verify in parallel
The question. Decode leaves the GPU's arithmetic mostly idle. Can one big-model pass produce more than one token?
Start from nothing. Verifying is cheaper than generating: the big (target) model can score k proposed tokens in one forward pass, just like a short prefill, at almost the cost of decoding one token (it's memory-bound either way). So: