You already know: LLM serving (batching, KV cache, memory), the LLM gateway and the semantic router. This case adds thousands of customer-specific adapters on one base model. LoRA (low-rank adaptation) adds a small set of extra weights per customer instead of a full model copy. The design questions: how to batch requests that need different adapters, where adapters live (GPU, host memory, storage), and how to keep tenants isolated.
How to read this lesson. No code, on purpose. Numbers marked assumed are assumptions you state out loud.
1. The prompt, and the questions that change the design
A mid-level answer: "load each customer's adapter into its own server". A staff candidate sees that the economics depend on sharing: most adapters are rarely used, so dedicated servers would sit idle. The hard problems are mixing adapters in one batch, memory tiers and cold starts, fair scheduling between tenants, and isolation and lifecycle (adapters are tied to a specific base model version).
Assumptions you state: