VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Batching & Scheduling — Continuous Batching, Chunked Prefill and Fairness

Batching & Scheduling — Continuous Batching, Chunked Prefill and Fairness

30 min read·Staff+ Tier· Unrolled Pro
Batching & Scheduling — Continuous Batching, Chunked Prefill and Fairness

You already know: batching raises decode's arithmetic intensity (Inference fundamentals); KV memory limits how many sequences can run, and preemption is expensive (KV cache engineering); heaps and priority queues (D1 Data structures); server-side batching windows for classical models (D11 Model deployment).

Why this matters for Atlas. Batching is how one GPU serves many students; scheduling decides which requests share each step. LLM requests make this hard: they arrive at random, their outputs vary from 5 to 5,000 tokens, prefill and decode compete for the same GPU, and memory caps how many can run. The scheduler is where throughput, latency and fairness are traded against each other, every few milliseconds. This lesson simulates the schedulers that real engines use and measures exactly what each one buys.


1. Static batching wastes the GPU on finished requests

The question. Why did the first LLM servers get poor throughput even with batching?

Start from nothing. Static (request-level) batching collects a batch, runs prefill for all of them, then decodes until every sequence in the batch has finished. Sequences that finish early sit in the batch as padding, doing nothing, and new requests wait outside until the longest one ends. With output lengths spread from tens to thousands of tokens, most of the batch is idle most of the time.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Batching & Scheduling — Continuous Batching, Chunked Prefill and Fairness"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.