You already know: eval-set sizing, paired comparisons and LLM judges (D7 Prompt evaluation); confidence intervals and the bootstrap (D2 Statistics); every component of a RAG pipeline (this domain).
Why this matters for Atlas. A wrong RAG answer has two possible causes: retrieval never found the right passage, or the model had it and still got the answer wrong. They need completely different fixes, and an end-to-end "accuracy" number can't tell them apart. RAG evaluation therefore measures each stage on its own: retrieval with ranking metrics against labelled relevant documents, and generation with groundedness (is every claim supported by the sources?) and correctness. This lesson builds the metrics, a failure-attribution report, and the statistics to decide whether a change really helped.
1. Retrieval metrics: did we find it, and how high?
The question. Given the documents that answer each question, how do you score a ranked list?
Start from nothing. First you need relevance labels (qrels): for each test question, which documents answer it, optionally graded (2 = answers it, 1 = partly helpful). Then four standard metrics: