You already know: golden sets, paired tests and multiple-comparison corrections (Evaluation foundations); graders and calibrated judges (LLM evaluation); environment-based agent evals and pass^k (Agent evaluation); traces and replay (Advanced agent engineering); prompt evaluation as a release gate (D7 Prompt evaluation).
Why this matters for Atlas. Evaluations that someone runs by hand before a big launch catch big problems occasionally. Production AI teams treat evaluation like CI: every change to a prompt, model, retrieval setting or tool runs the same pipeline automatically; releases pass a gate with explicit, statistically sound criteria; changes that pass offline are rolled out gradually and compared online; and live traffic is sampled, graded and monitored continuously, with failures flowing back into the test set. This lesson builds that pipeline end to end: reproducible eval runs, a release gate, an honest A/B test, and online monitoring.
1. Evaluation-driven development
The question. In what order should you build an AI feature and its evaluation?
Start from nothing. Evaluation-first, the way test-driven development works for code: