You already know: sampling, confidence intervals and hypothesis tests (D2 Probability & statistics); train/test splits and leakage (D3 ML evaluation); building an eval set, paired comparisons and judge bias for prompts (D7 Prompt evaluation); retrieval metrics and bootstrap comparisons (D8 RAG evaluation); and the bridge's warning that 20 cases decide nothing.
Why this matters for Atlas. Every decision about an AI system (ship this model? keep this prompt? is the agent getting worse?) is a measurement. Bad measurement is worse than none, because it gives false confidence: a "win" that is noise, an average that hides a broken segment, a human rating nobody agrees on. This lesson is the statistics and design that make an evaluation trustworthy: what to measure, how to build the dataset, how many cases you need, how to compare two systems properly, how to know your human labels mean something, and how not to fool yourself when you track twenty metrics at once.
1. Decide what you're measuring, and slice it
The question. Atlas's overall answer quality went up 2 points. Can anything have gotten worse?
Start from nothing. An evaluation is three choices: