VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Measurement & Capacity — Benchmarks That Don't Lie, and Breaking One GPU on Purpose

Measurement & Capacity — Benchmarks That Don't Lie, and Breaking One GPU on Purpose

30 min read·Staff+ Tier· Unrolled Pro
Measurement & Capacity — Benchmarks That Don't Lie, and Breaking One GPU on Purpose

You already know: TTFT, TPOT and goodput (Inference fundamentals); scheduler knobs, chunked prefill and preemption (Batching & scheduling); engine configuration (Inference frameworks); percentiles, intervals and honest comparisons (D10 Evaluation foundations); system-level monitoring (D11 Model monitoring).

Why this matters for Atlas. Capacity planning answers one question: how many GPUs do we need to serve this traffic within these latency targets, at what cost? Every number in that answer comes from a benchmark, and LLM benchmarks are easy to get wrong in ways that overstate capacity by 2–5×: closed-loop load generators that never queue, unrealistic prompt lengths, averages instead of tails, and metrics like "tokens per second" with no latency attached. This lesson builds a correct load test, finds the knee of the latency curve, deliberately breaks one GPU to locate its failure cliffs, computes utilization metrics (MFU, MBU), and turns the results into a capacity plan.


1. Open-loop load: the coordinated omission trap

The question. Why did the load test say 40 requests per second when production fails at 12?

Start from nothing. A closed-loop load generator has N virtual users, each sending a request, waiting for the response, then sending the next. When the server slows down, the generator automatically slows down too: it never builds the queue real traffic would build, and the latency it measures leaves out the time requests would have spent waiting. This is coordinated omission. Real users arrive independently of how busy you are: an open-loop generator sends requests on a schedule (e.g. Poisson arrivals at a target rate) regardless of whether earlier ones finished, and measures latency from the scheduled send time.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Measurement & Capacity — Benchmarks That Don't Lie, and Breaking One GPU on Purpose"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.