VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D16: AI Data Engineering · Datasets for LLMs — Mixtures, Repetition, Instruction Data and Versioning

Datasets for LLMs — Mixtures, Repetition, Instruction Data and Versioning

15 min read·Intermediate· Unrolled Pro
Datasets for LLMs — Mixtures, Repetition, Instruction Data and Versioning

You already know: pretraining objectives, tokenization and packing (D5 LLM pretraining); SFT data, chat templates and loss masking (D6 Supervised fine-tuning); preference pairs (D6 Preference optimization); synthetic data and model collapse (D6 Synthetic data); deduplication, filtering and decontamination (Data processing & quality); table formats and lineage (Data ingestion & storage).

Why this matters for Atlas. A training dataset is a design: which sources, in what proportions, repeated how many times, formatted how, held out how, and documented how. For pretraining, the mixture shapes what the model is good at (more code improves reasoning; more maths improves maths; too little of a language and the model can't speak it). For post-training, a few thousand carefully constructed examples matter more than millions of noisy ones. And unless the dataset is versioned and documented, no experiment can be reproduced or compared. This lesson covers mixture design with temperature sampling, how much repetition small high-quality sources can take, building instruction and preference datasets, and dataset versioning and documentation.


1. Mixtures: how much of each source

The question. Sources differ in size by a factor of 4,500. Should the model see them in proportion to their size?

Start from nothing. Sampling proportionally means the model sees almost nothing but web text: Atlas's own lessons would be 0.02% of training. Sampling uniformly across sources means repeating the small ones hundreds of times. The standard compromise is temperature sampling: with source sizes nᵢ, sample source i with probability ∝ nᵢ^(1/T). T = 1 is proportional; larger T flattens toward uniform. Mixtures in practice are set by a mix of heuristics (upweight code, maths, textbooks; cap repetitions) and ablations at small scale (train small models on candidate mixtures, compare downstream evals, extrapolate), plus methods that learn weights automatically (DoReMi, data mixing laws).

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Datasets for LLMs — Mixtures, Repetition, Instruction Data and Versioning"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee