VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D12: Inference Serving & vLLM · Routing and Token Budgets — Model Cascades, Rate Limits and Load Shedding
Phase 6/Domain 12/Lesson 15

Routing and Token Budgets — Model Cascades, Rate Limits and Load Shedding

20 min read·Staff+ Tier· Unrolled Pro
Routing and Token Budgets — Model Cascades, Rate Limits and Load Shedding

You already know: cost per token and per task (Inference economics); queue depth, saturation and shedding (Observability, Measurement & capacity); autoscaling and cold starts (Autoscaling); routing in agent systems (D9 Agent architectures); token-bucket rate limiting (D1 Networking).

Why this matters for Atlas. Autoscaling controls supply. This lesson controls demand: which model each request goes to, how many tokens each tenant may consume, how long each response may run, and who waits when the fleet is full. These gateway decisions often save more than any kernel optimisation, and they are what keep one runaway client from degrading everyone else.


1. One large model for everything is expensive

The question. If the large model answers everything well, why not send everything to it?

Start from nothing. In many products most requests are simple: classify, extract, reformat, define, short factual lookups. With illustrative prices of about $5 per million tokens for a large model and $0.30 for a small one, sending a simple request to the large model costs about 17 times more for no quality gain, and it is slower: a small model has lower inter-token latency and shorter queues.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Routing and Token Budgets — Model Cascades, Rate Limits and Load Shedding"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 9 of 9: Serve it fastUse this lesson in the project: Add a KV cache to TinyGPT, measure what batching buys, simulate static against continuous batching on real traffic, and turn throughput into a cost per token.