You already know: next-token cross-entropy and perplexity (D2 Information theory; NLP foundations §4); AdamW with warmup + cosine decay (D2 Optimization §4–5); BF16, accumulation, checkpointing (D4 PyTorch §5); the decoder-only architecture (Transformer architecture).
Why this matters for Atlas. Pretraining is where a model gets almost all of its knowledge and ability — and almost all of its cost. Three questions dominate: What data? (quality beats quantity, and leakage of test sets ruins evaluations.) How much compute? (measured in FLOPs, set by budget.) How should compute be split between model size and data size? (scaling laws.) Even if you never pretrain a frontier model, these ideas explain model behaviour and guide fine-tuning and evaluation.
1. The data pipeline: from raw text to training tokens
The question. The web has petabytes of text, much of it duplicated, spammy or unusable. What actually goes into a model?
Start from nothing. A typical pretraining data pipeline: