VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D5: LLM Architecture · LLM Pretraining — Data, Compute and Scaling Laws
Phase 4/Domain 5/Lesson 5

LLM Pretraining — Data, Compute and Scaling Laws

15 min read·Advanced· Unrolled Pro
LLM Pretraining — Data, Compute and Scaling Laws

You already know: next-token cross-entropy and perplexity (D2 Information theory; NLP foundations §4); AdamW with warmup + cosine decay (D2 Optimization §4–5); BF16, accumulation, checkpointing (D4 PyTorch §5); the decoder-only architecture (Transformer architecture).

Why this matters for Atlas. Pretraining is where a model gets almost all of its knowledge and ability — and almost all of its cost. Three questions dominate: What data? (quality beats quantity, and leakage of test sets ruins evaluations.) How much compute? (measured in FLOPs, set by budget.) How should compute be split between model size and data size? (scaling laws.) Even if you never pretrain a frontier model, these ideas explain model behaviour and guide fine-tuning and evaluation.


1. The data pipeline: from raw text to training tokens

The question. The web has petabytes of text, much of it duplicated, spammy or unusable. What actually goes into a model?

Start from nothing. A typical pretraining data pipeline:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "LLM Pretraining — Data, Compute and Scaling Laws"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 4 of 9: From text to GPTUse this lesson in the project: Train a byte-level BPE tokenizer and a 940K-parameter GPT on the lessons, on a laptop CPU, and read its loss curve honestly.