VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Autoscaling LLM Servers — Queue-Depth Signals, KEDA and Cold Starts
Phase 6/Domain 12/Lesson 14

Autoscaling LLM Servers — Queue-Depth Signals, KEDA and Cold Starts

20 min read·Staff+ Tier· Unrolled Pro
Autoscaling LLM Servers — Queue-Depth Signals, KEDA and Cold Starts

You already know: queue depth, KV-cache usage and Little's law (Observability, Measurement & capacity); utilisation as the cost lever and reserved versus elastic capacity (Inference economics); Kubernetes deployments, pods and the Horizontal Pod Autoscaler (D17 Kubernetes for ML).

Why this matters for Atlas. Traffic swings several-fold over a day and spikes within minutes. Too little capacity and students queue; too much and the GPU bill doubles for nothing. Autoscaling looks solved for web services, but the usual recipe, scale on CPU or GPU utilisation, fails for LLM servers in both directions. This lesson explains why, builds the scaling rule on the engine's own signals, works through the Kubernetes Horizontal Pod Autoscaler (HPA) arithmetic with KEDA, and deals with the problem that dominates everything else: a new replica takes minutes to start.


1. Why GPU metrics mislead an autoscaler

The question. The web-service recipe says: scale when utilisation passes 70%. Why does that fail for an inference server?

Start from nothing. Two GPU metrics are easy to scale on, and both mislead:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Autoscaling LLM Servers — Queue-Depth Signals, KEDA and Cold Starts"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.