You already know: queue depth, KV-cache usage and Little's law (Observability, Measurement & capacity); utilisation as the cost lever and reserved versus elastic capacity (Inference economics); Kubernetes deployments, pods and the Horizontal Pod Autoscaler (D17 Kubernetes for ML).
Why this matters for Atlas. Traffic swings several-fold over a day and spikes within minutes. Too little capacity and students queue; too much and the GPU bill doubles for nothing. Autoscaling looks solved for web services, but the usual recipe, scale on CPU or GPU utilisation, fails for LLM servers in both directions. This lesson explains why, builds the scaling rule on the engine's own signals, works through the Kubernetes Horizontal Pod Autoscaler (HPA) arithmetic with KEDA, and deals with the problem that dominates everything else: a new replica takes minutes to start.
1. Why GPU metrics mislead an autoscaler
The question. The web-service recipe says: scale when utilisation passes 70%. Why does that fail for an inference server?
Start from nothing. Two GPU metrics are easy to scale on, and both mislead: