You already know: what makes published evidence weak (Reading papers); hypothesis tests, bootstrap and multiple comparisons (D2); experiment tracking and pipelines (D11); compute and cost of training runs (D15).
Why this matters for Atlas. Research compute is the most expensive thing an AI team spends, and most of it is wasted on experiments that can't answer their question: no baseline at equal budget, one seed, three things changed at once, or a big run launched before small runs showed the trend. This lesson is the workflow that research engineers at frontier labs use to make each GPU-hour count: write the hypothesis and decision rule first, design ablations, quantify seed variance, test significance correctly across many comparisons, and use small-scale scaling studies to predict (and de-risk) large runs.
1. Start from a falsifiable hypothesis
The question. What should exist before the first run?
Start from nothing. A one-page experiment plan: the hypothesis ("adding retrieved passages during fine-tuning improves grounded-answer accuracy on our tutor eval by ≥3 points at equal training compute"), the primary metric and eval set (fixed in advance), the baselines (including the strongest simple one), the comparisons and ablations, the number of seeds, the analysis (paired test, threshold), and the decision rule (what result makes us adopt, abandon or investigate further). Writing this first prevents the most common failure: running experiments, then choosing the metric and comparison that look best (the "garden of forking paths").