VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Observability for LLM Serving — Golden Signals, Prometheus and SLOs
Phase 6/Domain 12/Lesson 10

Observability for LLM Serving — Golden Signals, Prometheus and SLOs

20 min read·Staff+ Tier· Unrolled Pro
Observability for LLM Serving — Golden Signals, Prometheus and SLOs

You already know: TTFT, inter-token latency and goodput (Inference fundamentals); load testing and Little's law (Measurement & capacity); model and data monitoring (D11 Model monitoring); the KV-cache cliff (KV cache I and II).

Why this matters for Atlas. Load tests tell you what the system can do; observability tells you what it is doing right now, for real users, and warns you before they notice. LLM serving breaks the usual web-service habits: latency splits into two signals, the leading indicator is a queue and a memory pool rather than CPU, and incidents can start and end between two default metric scrapes. This lesson specialises the golden signals for LLM serving, then gets precise about the instruments: histogram buckets, scrape intervals, dashboards, alerts, SLOs and error budgets.

Where you'll use this. Building the first dashboard for a new model deployment; writing alert rules that page for user pain, not for noise; setting SLOs a team can actually run against.


1. A silent outage: why averages lie

The question. How can a dashboard stay green while users leave?

Start from nothing. Picture a chat product serving 200 requests per second. At 11:47 a few customers upload 12,000-token documents; a batch job then sends 50 long-context requests at once. The queue grows: 2, 12, 45, 120, 340 waiting requests over twenty minutes. p99 time to first token goes from 0.2 s to 22 s. The average latency, meanwhile, rises only from 180 ms to about 420 ms, because most requests are short and still finish quickly. Alerts on the mean never fire; the support queue fills first.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Observability for LLM Serving — Golden Signals, Prometheus and SLOs"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.