You already know: train/test splits, metrics and leakage (D3 ML evaluation); confidence intervals, the bootstrap and hypothesis tests (D2 Statistics); prompt templates and structured outputs (Prompt fundamentals); constrained decoding and self-consistency (Advanced techniques).
Why this matters for Atlas. Every prompt change is a code change to the product, made in a language with no compiler and no type checker. Without evaluation you're shipping on vibes: a few hand-picked examples that look better, while other cases quietly break. With a good evaluation, prompt work becomes ordinary engineering. You have a test suite, a metric, a confidence interval, a release gate, and a record of what changed and why. Most of the reliability of LLM products comes from this unglamorous loop.
1. Building the evaluation set
The question. What should the test cases be, and how many do you need?
Start from nothing. An evaluation set is a list of inputs with a way to judge each output. Good sets have three ingredients: