VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D17: Cloud & AI Platform · Serving Platforms — Autoscaling, Cold Starts and the LLM Gateway

Serving Platforms — Autoscaling, Cold Starts and the LLM Gateway

15 min read·Intermediate· Unrolled Pro
Serving Platforms — Autoscaling, Cold Starts and the LLM Gateway

You already know: deployment patterns, canaries and sticky splits (D11 Model deployment); engine anatomy, prefix caching and cache-aware routing (D12 Inference frameworks); goodput, the KV cliff and capacity plans (D12 Measurement & capacity); load shedding and graceful degradation (D14 Reliability); probes, HPA and the cluster autoscaler (Kubernetes for ML).

Why this matters for Atlas. A serving platform wraps model servers in everything production needs: a uniform deployment API, autoscaling that follows demand, safe rollouts, and a gateway in front that gives every application one way to call any model, with authentication, quotas, routing, fallbacks, caching and cost tracking. For LLMs, two problems dominate: scaling on the right signal (GPU servers saturate long before their CPUs notice) and cold starts (a new replica must load tens to hundreds of gigabytes of weights before serving). This lesson builds an autoscaler and shows why the signal matters, quantifies cold starts and the fixes, and builds the core of an LLM gateway.


1. Model serving platforms

The question. What does a serving platform add on top of a model server like vLLM?

Start from nothing. A model server (vLLM, SGLang, TensorRT-LLM, Triton, D12) runs one model on some GPUs. A serving platform manages many of them:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Serving Platforms — Autoscaling, Cold Starts and the LLM Gateway"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee