VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D22: AI System Design Bank · The System Design Method — From a Vague Prompt to a Defensible ML System at Staff Level

The System Design Method — From a Vague Prompt to a Defensible ML System at Staff Level

15 min read·Staff+ Tier·Free lesson
The System Design Method — From a Vague Prompt to a Defensible ML System at Staff Level

You already know: every component an ML or AI system is built from (D1–D20), system shape and trade-offs (D21), and evaluation pipelines and experiments (D10). This lesson is the method used in every case in this bank: how to turn an eight-word prompt into a design that holds together, in the order an interviewer (or a design review) expects. The next two lessons go deeper on two steps: capacity estimation (lesson 02) and feature stores with training–serving parity (lesson 03). Every case after that applies the method in fourteen sections.

How to read this lesson. No code, on purpose. The running example is a lesson recommender for Atlas; numbers marked assumed are assumptions you state out loud.


1. What the interviewer is assessing

four kinds of signal (problem definition, numbers, end-to-end design, judgement), each linked to the part of the method that shows it.
Diagram— four kinds of signal (problem definition, numbers, end-to-end design, judgement), each linked to the part of the method that shows it.

An ML system design interview is not a test of whether you know the latest model. It looks for four signals:

Signal What it looks like Part of the method
Problem definition You ask what "good" means, who uses it, what is out of scope Clarify and frame
Quantitative reasoning Your numbers change the design ("that's 18 million scorings a second, so we need a funnel") Estimate
End-to-end design Data, training, serving and logging connect into a loop; you go deep where it matters Architecture and deep dive
Judgement You state trade-offs, plan for failure, and say what you would build first Evaluate, defend, plan

Breadth first, then depth. A brilliant model inside a broken system scores lower than a plain model in a sound one. Sketch the whole system before you go deep anywhere.

Senior versus staff. A senior answer is correct and complete; a staff answer also explains why each choice, what would make you reverse it, and how the system evolves and fails.

Quick check

A candidate spends 35 minutes on a sophisticated sequence model for recommendations and runs out of time before discussing data, evaluation or serving. What is the main weakness?

  1. The model was too simple
  2. Time allocation and missing end-to-end thinking: the interviewer could not assess problem definition, numbers, data, evaluation, serving or failure, which carry most of the signal
  3. Not enough maths
  4. Choosing the wrong framework
Show the answer

B. Time allocation and missing end-to-end thinking: the interviewer could not assess problem definition, numbers, data, evaluation, serving or failure, which carry most of the signal — Depth without breadth leaves most of the assessed signals unseen.


2. Clarify: requirements and the metric ladder

a four-layer metric ladder: business outcome, online product metric, offline model metric and guardrails.
Diagram— a four-layer metric ladder: business outcome, online product metric, offline model metric and guardrails.

Ask before you design. Who are the users? What decision does the system make? What happens today? What scale (users, items, requests)? What latency? What must never happen? What is explicitly out of scope?

Build a metric ladder:

Layer Example for the lesson recommender Why
Business Course completion; paid retention What the company wants
Product (online) Lessons completed after a recommendation; 30-day return What users do that reflects value
Model (offline) Candidate recall; graded ranking quality (NDCG) What you can measure before launch
Guardrails Latency under 100 ms; prerequisites respected; fairness across groups What must not get worse

Say which online metric each offline metric predicts. Offline metrics are proxies; if the proxy is wrong (clicks instead of completions), the system optimises the wrong thing.

Name non-goals. "Not designing the content pipeline or the payment flow" saves time and shows scope control.

Quick check

Why should a metric ladder include guardrails such as latency, cost and fairness alongside the main success metric?

  1. To make the slide longer
  2. A change can improve the main metric while breaking something that matters (slower pages, higher cost, worse outcomes for one group); guardrails make those regressions visible and block launches that cause them
  3. Guardrails replace the main metric
  4. Interviewers require four rows
Show the answer

B. A change can improve the main metric while breaking something that matters (slower pages, higher cost, worse outcomes for one group); guardrails make those regressions visible and block launches that cause them — Optimising one number invites hidden damage elsewhere. Guardrails catch it.


3. Frame the ML problem

framing steps: the decision, the label and when it is known, the inputs available at prediction time, the biases, and a simple baseline; side panel of hard parts.
Diagram— framing steps: the decision, the label and when it is known, the inputs available at prediction time, the biases, and a simple baseline; side panel of hard parts.

What decision does the prediction drive? "Which ten lessons to show on the home page" is a ranking problem; "whether to send a reminder" is a classification with a cost per action.

Define the label, and when it is known. For the recommender: completed (best), started, clicked (weakest). Completion is known days later; clicks in seconds. The label is a product decision as much as a modelling one.

Inputs at prediction time. Only what exists when the prediction is made: no leakage from the future (lesson 03).

Name the biases:

  • Position bias: items shown first get more clicks.
  • Selection bias: you only observe outcomes for what you showed.
  • Delay: labels arrive late (completion, chargebacks, defaults).
  • Feedback loops: the model's choices shape tomorrow's training data.

Baseline. Start from the simplest thing that could work (popular lessons per topic and level) and say how the ML version will beat it.

Quick check

For a lesson recommender, why might 'lesson completed' be a better label than 'lesson clicked', despite arriving later?

  1. Completion is easier to log
  2. Clicks reward catchy titles and thumbnails, while completion reflects learning value, which is what the business and product metrics care about; the delay is a cost worth paying, with clicks as an auxiliary signal
  3. Clicks are illegal to use
  4. Completion has no bias
Show the answer

B. Clicks reward catchy titles and thumbnails, while completion reflects learning value, which is what the business and product metrics care about; the delay is a cost worth paying, with clicks as an auxiliary signal — Choose the label closest to the real goal you can measure, and use faster signals to help.


4. Estimate: numbers that pick the architecture

Charts: items per request at each stage (120,000 lessons, 500 candidates, 10 shown) and heavy scorings per second (about 18 million if everything is scored versus about 75,000 with a funnel).
Diagram— Charts: items per request at each stage (120,000 lessons, 500 candidates, 10 shown) and heavy scorings per second (about 18 million if everything is scored versus about 75,000 with a funnel).

Turn users into load (lesson 02 has the full method): requests per second at peak, items per request, storage, model compute, cost.

Example (assumed). Atlas has about 120,000 lessons and about 150 home-feed requests per second at peak. Scoring every lesson with a heavy ranker would need 150 × 120,000 = 18 million heavy scorings per second, each taking milliseconds of accelerator time: impossible within budget.

The decision. A multi-stage funnel: cheap candidate generation retrieves about 500 lessons; a heavier ranker orders them (150 × 500 = 75,000 scorings per second); a re-ranker applies rules and diversity; 10 are shown. The funnel is the backbone of recommendation, search and ads, and it came from one multiplication.

Only estimate what changes the design. Storage for lesson thumbnails rarely does; model memory and requests per second usually do.

Quick check

Why does the estimate of 18 million heavy scorings per second lead directly to a multi-stage funnel?

  1. Because no affordable fleet can run a heavy model on every item for every request within the latency budget; a cheap first stage cuts candidates to hundreds so the expensive model runs only where it matters
  2. Because funnels are fashionable
  3. Because the catalogue is too small
  4. Because ranking models are inaccurate
Show the answer

A. Because no affordable fleet can run a heavy model on every item for every request within the latency budget; a cheap first stage cuts candidates to hundreds so the expensive model runs only where it matters — Good estimates end in a design decision. This one decides the shape of the whole system.


5. Architecture: the loop from data to serving and back

two planes. Learning: event logs, stream and batch pipelines, a feature store, training and evaluation with a registry; a side box for governance. Serving: request, candidates, ranking, re-ranking, response and logging of what was shown and what happened.
Diagram— two planes. Learning: event logs, stream and batch pipelines, a feature store, training and evaluation with a registry; a side box for governance. Serving: request, candidates, ranking, re-ranking, response and logging of what was shown and what happened.

Draw the whole loop first:

  1. Logging: impressions (what was shown, where) and outcomes (started, completed).
  2. Pipelines: stream for fresh signals, batch for history.
  3. Feature store: one definition, offline for training and online for serving (lesson 03).
  4. Training and evaluation with a model registry and release gates.
  5. Serving: candidate generation → ranking → re-ranking with rules → response within the latency budget.
  6. Back to logging: what you log at serving is what you can learn from tomorrow.

Name the boundaries. Which parts are owned by which team, which data contracts exist, where privacy and fairness reviews happen.

Keep hard rules outside models. Prerequisites, access by plan, safety filters: deterministic layers that every fallback keeps.

Quick check

Why should hard rules such as 'never recommend a lesson whose prerequisites the student hasn't met' live outside the ranking model?

  1. Models cannot read prerequisites
  2. Interviewers prefer rules
  3. Rules are faster to train
  4. A learned model can trade the rule off against predicted engagement, and fallbacks may bypass the model entirely; a deterministic rule layer guarantees the requirement on every path
Show the answer

D. A learned model can trade the rule off against predicted engagement, and fallbacks may bypass the model entirely; a deterministic rule layer guarantees the requirement on every path — Guarantees belong in deterministic code paths. Models optimise; rules constrain.


6. Deep dive: go deep where the risk is

three places to spend deep-dive time: the model (impressive, often misplaced), data and labels (where systems fail; often the right choice), and serving (latency, cost, freshness).
Diagram— three places to spend deep-dive time: the model (impressive, often misplaced), data and labels (where systems fail; often the right choice), and serving (latency, cost, freshness).

After the whole loop is on the board, choose one or two parts to go deep on, and say why.

  • The model: architecture, features, objectives. Familiar and tempting, but rarely where real systems fail.
  • Data and labels: leakage, delayed labels, position and selection bias, feedback loops, point-in-time joins. This is where most production ML fails.
  • Serving: latency budgets per stage, caching, freshness, cost per request.

Pick the riskiest part of this prompt. For fraud, labels arrive 30–90 days late: go deep on labels. For an LLM assistant, cost and latency dominate: go deep on serving and routing. For a feed, the objective and feedback loops: go deep on labels and evaluation.

Say the sentence: "I'll go deep on the labels, because they arrive two months late and are biased by our own decisions." That sentence alone is a staff signal.

Quick check

In a fraud detection design, which deep dive usually carries the most signal, and why?

  1. The neural network architecture
  2. The colour scheme of the review tool
  3. Labels and evaluation: chargebacks arrive weeks later, adversaries change behaviour, and random splits leak the future, so how you get labels and evaluate honestly decides whether the system works
  4. Database indexing
Show the answer

C. Labels and evaluation: chargebacks arrive weeks later, adversaries change behaviour, and random splits leak the future, so how you get labels and evaluate honestly decides whether the system works — The best deep dive targets the part most likely to make the system fail.


7. Evaluate before and after launch

three evaluation loops: offline with time splits, ladder metrics, slices and baselines; stress with abuse, outages, extremes and load; online with guarded experiments, interference checks, holdouts and monitoring; gates from offline to small test, ramp and holdout.
Diagram— three evaluation loops: offline with time splits, ladder metrics, slices and baselines; stress with abuse, outages, extremes and load; online with guarded experiments, interference checks, holdouts and monitoring; gates from offline to small test, ramp and holdout.

Offline. Time-based splits (train on the past, test on later data), metrics tied to the ladder, slices by segment (new users, rare categories, groups), comparison with the baseline, calibration where probabilities are used.

Stress. Abuse and adversaries, data outages, extreme inputs, load tests.

Online. A/B tests with guardrails. Ask whether units interfere: users who share drivers, budgets, inventory or social connections affect each other, so randomise bigger units (markets, time windows, graph clusters). Keep long-term holdouts for slow effects.

Gates. A change moves from offline to a small test to a ramp, with criteria written down before the test.

Quick check

A new recommendation model wins offline by 4% on logged data but shows no change in an A/B test. What should you suspect first?

  1. The offline evaluation learned the logging bias (it rewards ranking what was already shown first), so its gain doesn't transfer; correct for position and selection in offline evaluation and trust the online result
  2. The A/B test is broken
  3. The model needs more layers
  4. Users can't see the change
Show the answer

A. The offline evaluation learned the logging bias (it rewards ranking what was already shown first), so its gain doesn't transfer; correct for position and selection in offline evaluation and trust the online result — Offline evaluation on logged data inherits the old system's biases. Online tests are the truth.


8. Defend: the trade-off ledger

typical decisions placed by cost to reverse and confidence today: the objective and labels, logging and data contracts, and privacy commitments are costly to reverse; the architecture shape moderately; thresholds, cache sizes and refresh cadence are cheap to tune. Qualitative, not measured.
Diagram— typical decisions placed by cost to reverse and confidence today: the objective and labels, logging and data contracts, and privacy commitments are costly to reverse; the architecture shape moderately; thresholds, cache sizes and refresh cadence are cheap to tune. Qualitative, not measured.

Every case in this bank includes a trade-off ledger. Its columns:

Decision Chosen Rejected Why Reverse if…
Label Completion-weighted Clicks Aligns with learning Completion signal too sparse
Funnel Two-stage retrieval + ranking Score everything 18 M scorings/s Catalogue shrinks to hundreds
Freshness Streaming session features Daily batch only In-session intent Sessions are rare

The "reverse if" column is what makes a ledger staff-level: it shows you know the conditions under which your choice stops being right.

Spend design time where reversal is expensive. Objectives, labels, logging and privacy commitments are hard to change later; thresholds and cache sizes are easy.

Quick check

Why is the 'reverse if…' column the most important part of a trade-off ledger?

  1. It makes the table wider
  2. It replaces the 'why' column
  3. Interviewers count columns
  4. It shows you understand the assumptions behind each choice and the evidence that would change your mind, which is what separates a defensible decision from a preference
Show the answer

D. It shows you understand the assumptions behind each choice and the evidence that would change your mind, which is what separates a defensible decision from a preference — Judgement is knowing when you'd be wrong. The column makes that explicit.


9. Failure: degrade, don't collapse

a generic degradation ladder: full system, a lighter model when the heavy model is down, last good features when features are stale, cached results when candidates are down, and a safe default when everything is down; safety, privacy and honest messages hold on every rung.
Diagram— a generic degradation ladder: full system, a lighter model when the heavy model is down, last good features when features are stale, cached results when candidates are down, and a safe default when everything is down; safety, privacy and honest messages hold on every rung.

Plan the degradation ladder. For each dependency, what happens when it fails? Each rung should be simpler and safer, never unsafe.

State the invariants that hold on every rung: safety and policy filters, privacy, honest messages to users, budgets and caps.

Write one incident. "A logging change dropped 15% of completion events; the model learned that completion was rarer and down-ranked long lessons; caught by a label-rate monitor two days later." An incident shows you think about how real systems fail, and what monitoring and gates would have caught it.

Common failure patterns across the bank: silent data breakage (units, schema, time zones), stale features that look valid, defaults that look like real values, feedback loops narrowing what the system shows, mismatched model and index versions.

Quick check

Why is 'return nothing' sometimes the right degraded behaviour (as in autocomplete or ad bidding)?

  1. It is easier to code
  2. Users prefer empty screens
  3. When any answer risks harm (an unsafe suggestion, an unpriced bid, a double booking), saying nothing costs little and breaks nothing, while a wrong answer can cause real damage; the right fallback depends on the cost of each kind of mistake
  4. It saves compute
Show the answer

C. When any answer risks harm (an unsafe suggestion, an unpriced bid, a double booking), saying nothing costs little and breaks nothing, while a wrong answer can cause real damage; the right fallback depends on the cost of each kind of mistake — Fallbacks should be chosen by the cost of mistakes, not by habit.


10. A 45-minute plan

a suggested split of a 45-minute interview: clarify 5, metrics 4, estimates 5, v1 architecture 8, deep dives 16, failure and evolution 7.
Chart— a suggested split of a 45-minute interview: clarify 5, metrics 4, estimates 5, v1 architecture 8, deep dives 16, failure and evolution 7.
Minutes Step Output on the board
0–5 Clarify Users, decision, scale, latency, non-goals
5–9 Metrics The ladder with guardrails
9–14 Estimate Two or three numbers that change the design
14–22 v1 architecture The loop from logs to serving
22–38 Deep dives One or two risky parts, in depth
38–45 Failure, evolution, trade-offs Ladder, v1→v3, ledger rows

Common traps:

  • Jumping to the model before clarifying.
  • Numbers that change nothing.
  • Drawing a pipeline instead of a loop (no logging, no feedback).
  • Going deep on the model by default.
  • Offline metrics with no online link.
  • No failure plan; no "reverse if".

How the cases are written. Every case in this bank follows these steps in fourteen sections: questions, metrics, numbers, v1 architecture, three deep dives, evaluation, trade-off ledger, failure and an incident, evolution, cross-questions with curveballs, senior versus staff, and traps with a timeline. Practise each case aloud against this plan.

Quick check

At minute 30 of a 45-minute interview you are still on the first deep dive. What should you do?

  1. Summarise the deep dive's key decision in a sentence, then move on to evaluation, failure and trade-offs, offering to return if there's time, because the remaining steps carry signal the interviewer hasn't seen yet
  2. Keep going until it's perfect
  3. Start over
  4. Skip evaluation entirely
Show the answer

A. Summarise the deep dive's key decision in a sentence, then move on to evaluation, failure and trade-offs, offering to return if there's time, because the remaining steps carry signal the interviewer hasn't seen yet — Time management is part of the assessment. Cover all signals, then add depth.


Check yourself

  1. What four signals does a system design interview assess?
  2. Build a metric ladder for a lesson recommender.
  3. What decisions go into framing the ML problem, and which biases should you name?
  4. Why does the scale estimate lead to a multi-stage funnel?
  5. How do you choose where to go deep?
  6. What makes a trade-off ledger staff-level?
Answers
  1. Problem definition, quantitative reasoning, end-to-end design, and judgement about trade-offs and failure.
  2. Business: completion and retention; online: lessons completed after a recommendation, 30-day return; offline: candidate recall and graded NDCG; guardrails: latency, prerequisites, fairness.
  3. The decision the prediction drives, the label and when it is known, inputs available at prediction time, and a baseline; name position bias, selection bias, label delay and feedback loops.
  4. Scoring every item with a heavy model (about 18 million scorings per second here) is unaffordable within the latency budget; cheap retrieval followed by heavier ranking on hundreds of items fits.
  5. Choose the part most likely to make this system fail (often labels and data, sometimes serving cost), and say why.
  6. The "reverse if…" column: the conditions under which each choice stops being right.

This unlocks: Capacity Estimation and Sizing.