You already know: deployment patterns, canaries and sticky splits (D11 Model deployment); engine anatomy, prefix caching and cache-aware routing (D12 Inference frameworks); goodput, the KV cliff and capacity plans (D12 Measurement & capacity); load shedding and graceful degradation (D14 Reliability); probes, HPA and the cluster autoscaler (Kubernetes for ML).
Why this matters for Atlas. A serving platform wraps model servers in everything production needs: a uniform deployment API, autoscaling that follows demand, safe rollouts, and a gateway in front that gives every application one way to call any model, with authentication, quotas, routing, fallbacks, caching and cost tracking. For LLMs, two problems dominate: scaling on the right signal (GPU servers saturate long before their CPUs notice) and cold starts (a new replica must load tens to hundreds of gigabytes of weights before serving). This lesson builds an autoscaler and shows why the signal matters, quantifies cold starts and the fixes, and builds the core of an LLM gateway.
1. Model serving platforms
The question. What does a serving platform add on top of a model server like vLLM?
Start from nothing. A model server (vLLM, SGLang, TensorRT-LLM, Triton, D12) runs one model on some GPUs. A serving platform manages many of them: