You already know: the LLM gateway (routing, quotas, logging), the semantic router and the serving cases. This case adds guardrails: checks on inputs, outputs and tool use that enforce safety and data policies for many products at once. The design questions: how to check cheaply enough to run on every request, how to check output that is still streaming, how to defend against prompt injection when a classifier will miss some, and what happens when the guardrail fails.
How to read this lesson. No code, on purpose. Numbers marked assumed are assumptions you state out loud; prices are illustrative.
1. The prompt, and the questions that change the design
A mid-level answer: "call a moderation model on the input and the output, and block if it says unsafe". A staff candidate asks: which products and users (a children's tutor and an internal security research tool need different policies), what a false block costs (refusing a legitimate medical question is a harm too), how output is delivered (streaming changes everything), and what happens when the check is unavailable (fail open or fail closed, decided in advance per category).
Assumptions you state: