You already know: pretraining objectives, tokenization and packing (D5 LLM pretraining); SFT data, chat templates and loss masking (D6 Supervised fine-tuning); preference pairs (D6 Preference optimization); synthetic data and model collapse (D6 Synthetic data); deduplication, filtering and decontamination (Data processing & quality); table formats and lineage (Data ingestion & storage).
Why this matters for Atlas. A training dataset is a design: which sources, in what proportions, repeated how many times, formatted how, held out how, and documented how. For pretraining, the mixture shapes what the model is good at (more code improves reasoning; more maths improves maths; too little of a language and the model can't speak it). For post-training, a few thousand carefully constructed examples matter more than millions of noisy ones. And unless the dataset is versioned and documented, no experiment can be reproduced or compared. This lesson covers mixture design with temperature sampling, how much repetition small high-quality sources can take, building instruction and preference datasets, and dataset versioning and documentation.
1. Mixtures: how much of each source
The question. Sources differ in size by a factor of 4,500. Should the model see them in proportion to their size?
Start from nothing. Sampling proportionally means the model sees almost nothing but web text: Atlas's own lessons would be 0.02% of training. Sampling uniformly across sources means repeating the small ones hundreds of times. The standard compromise is temperature sampling: with source sizes nᵢ, sample source i with probability ∝ nᵢ^(1/T). T = 1 is proportional; larger T flattens toward uniform. Mixtures in practice are set by a mix of heuristics (upweight code, maths, textbooks; cap repetitions) and ablations at small scale (train small models on candidate mixtures, compare downstream evals, extrapolate), plus methods that learn weights automatically (DoReMi, data mixing laws).