You already know: confidence intervals, paired tests, kappa and multiple comparisons (Evaluation foundations); pretraining data pipelines and deduplication (D5 LLM pretraining); LLM-as-judge and its biases at the prompt level (D7 Prompt evaluation); reward models and Goodhart's law (D6 Preference optimization).
Why this matters for Atlas. Choosing a model, a prompt or a fine-tune means trusting numbers that someone produced. Public benchmarks are useful and routinely misleading: they saturate, leak into training data, and measure something adjacent to your task. Your own evaluations depend on graders, often an LLM judge, which has biases you must measure before you trust it. This lesson covers what benchmarks can and can't tell you, how to detect contamination, how to score free-form outputs (exact match, pass@k, rubrics), and how to build an LLM judge whose verdicts match humans. It ends with the metrics people forget: safety, latency and cost per successful task.
1. What a benchmark measures, and when it stops measuring
The question. Model X scores 89% on a benchmark and model Y 87%. What have you learned?
Start from nothing. A benchmark is a fixed dataset plus a scoring rule, run the same way across models so scores are comparable. Common families: