You already know: every component an ML or AI system is built from (D1–D20), system shape and trade-offs (D21), and evaluation pipelines and experiments (D10). This lesson is the method used in every case in this bank: how to turn an eight-word prompt into a design that holds together, in the order an interviewer (or a design review) expects. The next two lessons go deeper on two steps: capacity estimation (lesson 02) and feature stores with training–serving parity (lesson 03). Every case after that applies the method in fourteen sections.
How to read this lesson. No code, on purpose. The running example is a lesson recommender for Atlas; numbers marked assumed are assumptions you state out loud.
1. What the interviewer is assessing
An ML system design interview is not a test of whether you know the latest model. It looks for four signals:
| Signal | What it looks like | Part of the method |
|---|---|---|
| Problem definition | You ask what "good" means, who uses it, what is out of scope | Clarify and frame |
| Quantitative reasoning | Your numbers change the design ("that's 18 million scorings a second, so we need a funnel") | Estimate |
| End-to-end design | Data, training, serving and logging connect into a loop; you go deep where it matters | Architecture and deep dive |
| Judgement | You state trade-offs, plan for failure, and say what you would build first | Evaluate, defend, plan |
Breadth first, then depth. A brilliant model inside a broken system scores lower than a plain model in a sound one. Sketch the whole system before you go deep anywhere.
Senior versus staff. A senior answer is correct and complete; a staff answer also explains why each choice, what would make you reverse it, and how the system evolves and fails.
Quick check
A candidate spends 35 minutes on a sophisticated sequence model for recommendations and runs out of time before discussing data, evaluation or serving. What is the main weakness?
Show the answer
B. Time allocation and missing end-to-end thinking: the interviewer could not assess problem definition, numbers, data, evaluation, serving or failure, which carry most of the signal — Depth without breadth leaves most of the assessed signals unseen.
2. Clarify: requirements and the metric ladder
Ask before you design. Who are the users? What decision does the system make? What happens today? What scale (users, items, requests)? What latency? What must never happen? What is explicitly out of scope?
Build a metric ladder:
| Layer | Example for the lesson recommender | Why |
|---|---|---|
| Business | Course completion; paid retention | What the company wants |
| Product (online) | Lessons completed after a recommendation; 30-day return | What users do that reflects value |
| Model (offline) | Candidate recall; graded ranking quality (NDCG) | What you can measure before launch |
| Guardrails | Latency under 100 ms; prerequisites respected; fairness across groups | What must not get worse |
Say which online metric each offline metric predicts. Offline metrics are proxies; if the proxy is wrong (clicks instead of completions), the system optimises the wrong thing.
Name non-goals. "Not designing the content pipeline or the payment flow" saves time and shows scope control.
Quick check
Why should a metric ladder include guardrails such as latency, cost and fairness alongside the main success metric?
Show the answer
B. A change can improve the main metric while breaking something that matters (slower pages, higher cost, worse outcomes for one group); guardrails make those regressions visible and block launches that cause them — Optimising one number invites hidden damage elsewhere. Guardrails catch it.
3. Frame the ML problem
What decision does the prediction drive? "Which ten lessons to show on the home page" is a ranking problem; "whether to send a reminder" is a classification with a cost per action.
Define the label, and when it is known. For the recommender: completed (best), started, clicked (weakest). Completion is known days later; clicks in seconds. The label is a product decision as much as a modelling one.
Inputs at prediction time. Only what exists when the prediction is made: no leakage from the future (lesson 03).
Name the biases:
- Position bias: items shown first get more clicks.
- Selection bias: you only observe outcomes for what you showed.
- Delay: labels arrive late (completion, chargebacks, defaults).
- Feedback loops: the model's choices shape tomorrow's training data.
Baseline. Start from the simplest thing that could work (popular lessons per topic and level) and say how the ML version will beat it.
Quick check
For a lesson recommender, why might 'lesson completed' be a better label than 'lesson clicked', despite arriving later?
Show the answer
B. Clicks reward catchy titles and thumbnails, while completion reflects learning value, which is what the business and product metrics care about; the delay is a cost worth paying, with clicks as an auxiliary signal — Choose the label closest to the real goal you can measure, and use faster signals to help.
4. Estimate: numbers that pick the architecture
Turn users into load (lesson 02 has the full method): requests per second at peak, items per request, storage, model compute, cost.
Example (assumed). Atlas has about 120,000 lessons and about 150 home-feed requests per second at peak. Scoring every lesson with a heavy ranker would need 150 × 120,000 = 18 million heavy scorings per second, each taking milliseconds of accelerator time: impossible within budget.
The decision. A multi-stage funnel: cheap candidate generation retrieves about 500 lessons; a heavier ranker orders them (150 × 500 = 75,000 scorings per second); a re-ranker applies rules and diversity; 10 are shown. The funnel is the backbone of recommendation, search and ads, and it came from one multiplication.
Only estimate what changes the design. Storage for lesson thumbnails rarely does; model memory and requests per second usually do.
Quick check
Why does the estimate of 18 million heavy scorings per second lead directly to a multi-stage funnel?
Show the answer
A. Because no affordable fleet can run a heavy model on every item for every request within the latency budget; a cheap first stage cuts candidates to hundreds so the expensive model runs only where it matters — Good estimates end in a design decision. This one decides the shape of the whole system.
5. Architecture: the loop from data to serving and back
Draw the whole loop first:
- Logging: impressions (what was shown, where) and outcomes (started, completed).
- Pipelines: stream for fresh signals, batch for history.
- Feature store: one definition, offline for training and online for serving (lesson 03).
- Training and evaluation with a model registry and release gates.
- Serving: candidate generation → ranking → re-ranking with rules → response within the latency budget.
- Back to logging: what you log at serving is what you can learn from tomorrow.
Name the boundaries. Which parts are owned by which team, which data contracts exist, where privacy and fairness reviews happen.
Keep hard rules outside models. Prerequisites, access by plan, safety filters: deterministic layers that every fallback keeps.
Quick check
Why should hard rules such as 'never recommend a lesson whose prerequisites the student hasn't met' live outside the ranking model?
Show the answer
D. A learned model can trade the rule off against predicted engagement, and fallbacks may bypass the model entirely; a deterministic rule layer guarantees the requirement on every path — Guarantees belong in deterministic code paths. Models optimise; rules constrain.
6. Deep dive: go deep where the risk is
After the whole loop is on the board, choose one or two parts to go deep on, and say why.
- The model: architecture, features, objectives. Familiar and tempting, but rarely where real systems fail.
- Data and labels: leakage, delayed labels, position and selection bias, feedback loops, point-in-time joins. This is where most production ML fails.
- Serving: latency budgets per stage, caching, freshness, cost per request.
Pick the riskiest part of this prompt. For fraud, labels arrive 30–90 days late: go deep on labels. For an LLM assistant, cost and latency dominate: go deep on serving and routing. For a feed, the objective and feedback loops: go deep on labels and evaluation.
Say the sentence: "I'll go deep on the labels, because they arrive two months late and are biased by our own decisions." That sentence alone is a staff signal.
Quick check
In a fraud detection design, which deep dive usually carries the most signal, and why?
Show the answer
C. Labels and evaluation: chargebacks arrive weeks later, adversaries change behaviour, and random splits leak the future, so how you get labels and evaluate honestly decides whether the system works — The best deep dive targets the part most likely to make the system fail.
7. Evaluate before and after launch
Offline. Time-based splits (train on the past, test on later data), metrics tied to the ladder, slices by segment (new users, rare categories, groups), comparison with the baseline, calibration where probabilities are used.
Stress. Abuse and adversaries, data outages, extreme inputs, load tests.
Online. A/B tests with guardrails. Ask whether units interfere: users who share drivers, budgets, inventory or social connections affect each other, so randomise bigger units (markets, time windows, graph clusters). Keep long-term holdouts for slow effects.
Gates. A change moves from offline to a small test to a ramp, with criteria written down before the test.
Quick check
A new recommendation model wins offline by 4% on logged data but shows no change in an A/B test. What should you suspect first?
Show the answer
A. The offline evaluation learned the logging bias (it rewards ranking what was already shown first), so its gain doesn't transfer; correct for position and selection in offline evaluation and trust the online result — Offline evaluation on logged data inherits the old system's biases. Online tests are the truth.
8. Defend: the trade-off ledger
Every case in this bank includes a trade-off ledger. Its columns:
| Decision | Chosen | Rejected | Why | Reverse if… |
|---|---|---|---|---|
| Label | Completion-weighted | Clicks | Aligns with learning | Completion signal too sparse |
| Funnel | Two-stage retrieval + ranking | Score everything | 18 M scorings/s | Catalogue shrinks to hundreds |
| Freshness | Streaming session features | Daily batch only | In-session intent | Sessions are rare |
The "reverse if" column is what makes a ledger staff-level: it shows you know the conditions under which your choice stops being right.
Spend design time where reversal is expensive. Objectives, labels, logging and privacy commitments are hard to change later; thresholds and cache sizes are easy.
Quick check
Why is the 'reverse if…' column the most important part of a trade-off ledger?
Show the answer
D. It shows you understand the assumptions behind each choice and the evidence that would change your mind, which is what separates a defensible decision from a preference — Judgement is knowing when you'd be wrong. The column makes that explicit.
9. Failure: degrade, don't collapse
Plan the degradation ladder. For each dependency, what happens when it fails? Each rung should be simpler and safer, never unsafe.
State the invariants that hold on every rung: safety and policy filters, privacy, honest messages to users, budgets and caps.
Write one incident. "A logging change dropped 15% of completion events; the model learned that completion was rarer and down-ranked long lessons; caught by a label-rate monitor two days later." An incident shows you think about how real systems fail, and what monitoring and gates would have caught it.
Common failure patterns across the bank: silent data breakage (units, schema, time zones), stale features that look valid, defaults that look like real values, feedback loops narrowing what the system shows, mismatched model and index versions.
Quick check
Why is 'return nothing' sometimes the right degraded behaviour (as in autocomplete or ad bidding)?
Show the answer
C. When any answer risks harm (an unsafe suggestion, an unpriced bid, a double booking), saying nothing costs little and breaks nothing, while a wrong answer can cause real damage; the right fallback depends on the cost of each kind of mistake — Fallbacks should be chosen by the cost of mistakes, not by habit.
10. A 45-minute plan
| Minutes | Step | Output on the board |
|---|---|---|
| 0–5 | Clarify | Users, decision, scale, latency, non-goals |
| 5–9 | Metrics | The ladder with guardrails |
| 9–14 | Estimate | Two or three numbers that change the design |
| 14–22 | v1 architecture | The loop from logs to serving |
| 22–38 | Deep dives | One or two risky parts, in depth |
| 38–45 | Failure, evolution, trade-offs | Ladder, v1→v3, ledger rows |
Common traps:
- Jumping to the model before clarifying.
- Numbers that change nothing.
- Drawing a pipeline instead of a loop (no logging, no feedback).
- Going deep on the model by default.
- Offline metrics with no online link.
- No failure plan; no "reverse if".
How the cases are written. Every case in this bank follows these steps in fourteen sections: questions, metrics, numbers, v1 architecture, three deep dives, evaluation, trade-off ledger, failure and an incident, evolution, cross-questions with curveballs, senior versus staff, and traps with a timeline. Practise each case aloud against this plan.
Quick check
At minute 30 of a 45-minute interview you are still on the first deep dive. What should you do?
Show the answer
A. Summarise the deep dive's key decision in a sentence, then move on to evaluation, failure and trade-offs, offering to return if there's time, because the remaining steps carry signal the interviewer hasn't seen yet — Time management is part of the assessment. Cover all signals, then add depth.
Check yourself
- What four signals does a system design interview assess?
- Build a metric ladder for a lesson recommender.
- What decisions go into framing the ML problem, and which biases should you name?
- Why does the scale estimate lead to a multi-stage funnel?
- How do you choose where to go deep?
- What makes a trade-off ledger staff-level?
Answers
- Problem definition, quantitative reasoning, end-to-end design, and judgement about trade-offs and failure.
- Business: completion and retention; online: lessons completed after a recommendation, 30-day return; offline: candidate recall and graded NDCG; guardrails: latency, prerequisites, fairness.
- The decision the prediction drives, the label and when it is known, inputs available at prediction time, and a baseline; name position bias, selection bias, label delay and feedback loops.
- Scoring every item with a heavy model (about 18 million scorings per second here) is unaffordable within the latency budget; cheap retrieval followed by heavier ranking on hundreds of items fits.
- Choose the part most likely to make this system fail (often labels and data, sometimes serving cost), and say why.
- The "reverse if…" column: the conditions under which each choice stops being right.
This unlocks: Capacity Estimation and Sizing.