You already know: batching raises decode's arithmetic intensity (Inference fundamentals); KV memory limits how many sequences can run, and preemption is expensive (KV cache engineering); heaps and priority queues (D1 Data structures); server-side batching windows for classical models (D11 Model deployment).
Why this matters for Atlas. Batching is how one GPU serves many students; scheduling decides which requests share each step. LLM requests make this hard: they arrive at random, their outputs vary from 5 to 5,000 tokens, prefill and decode compete for the same GPU, and memory caps how many can run. The scheduler is where throughput, latency and fairness are traded against each other, every few milliseconds. This lesson simulates the schedulers that real engines use and measures exactly what each one buys.
1. Static batching wastes the GPU on finished requests
The question. Why did the first LLM servers get poor throughput even with batching?
Start from nothing. Static (request-level) batching collects a batch, runs prefill for all of them, then decodes until every sequence in the batch has finished. Sequences that finish early sit in the batch as padding, doing nothing, and new requests wait outside until the longest one ends. With output lengths spread from tens to thousands of tokens, most of the batch is idle most of the time.