You already know: parsing and chunking documents (D8 Document ingestion); validation checks and PSI (D11 ML pipelines); MapReduce, shuffles and skew (D14 Distributed compute); pretraining data pipelines and why dedup matters (D5 LLM pretraining); the D15 bridge's demonstration that deduplicated data covers far more knowledge per token.
Why this matters for Atlas. Raw data is mostly not what you want to train on. Web crawls are dominated by boilerplate, duplicates and low-quality text; operational data has nulls, inconsistent encodings and leaked personal information; evaluation data leaks into training. Data processing turns raw inputs into training data through a pipeline of normalization, deduplication, quality filtering, PII removal and decontamination, each designed to run over billions of documents. Published results (RefinedWeb, FineWeb, DCLM) show that better filtering of the same raw crawl produces markedly better models at the same compute. This lesson builds each stage, including MinHash with locality-sensitive hashing for near-duplicate detection, and measures their trade-offs.
1. Normalize before you compare anything
The question. Why do "café" and "café" hash differently, and why does it matter for deduplication?
Start from nothing. Text that looks identical can differ in bytes: Unicode composed vs decomposed characters (é as one code point or e + combining accent), different whitespace, curly vs straight quotes, HTML entities, zero-width characters. If you deduplicate or filter on raw bytes, identical content slips through and filters misfire. Normalization (Unicode NFC/NFKC, whitespace collapsing, entity decoding, stripping control characters) comes first, and the same normalization must be applied anywhere text is compared later (dedup, decontamination, PII matching).