VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D16: AI Data Engineering · Data Processing & Quality — Cleaning, Deduplication, Filtering and PII at Scale

Data Processing & Quality — Cleaning, Deduplication, Filtering and PII at Scale

20 min read·Intermediate· Unrolled Pro
Data Processing & Quality — Cleaning, Deduplication, Filtering and PII at Scale

You already know: parsing and chunking documents (D8 Document ingestion); validation checks and PSI (D11 ML pipelines); MapReduce, shuffles and skew (D14 Distributed compute); pretraining data pipelines and why dedup matters (D5 LLM pretraining); the D15 bridge's demonstration that deduplicated data covers far more knowledge per token.

Why this matters for Atlas. Raw data is mostly not what you want to train on. Web crawls are dominated by boilerplate, duplicates and low-quality text; operational data has nulls, inconsistent encodings and leaked personal information; evaluation data leaks into training. Data processing turns raw inputs into training data through a pipeline of normalization, deduplication, quality filtering, PII removal and decontamination, each designed to run over billions of documents. Published results (RefinedWeb, FineWeb, DCLM) show that better filtering of the same raw crawl produces markedly better models at the same compute. This lesson builds each stage, including MinHash with locality-sensitive hashing for near-duplicate detection, and measures their trade-offs.


1. Normalize before you compare anything

The question. Why do "café" and "café" hash differently, and why does it matter for deduplication?

Start from nothing. Text that looks identical can differ in bytes: Unicode composed vs decomposed characters (é as one code point or e + combining accent), different whitespace, curly vs straight quotes, HTML entities, zero-width characters. If you deduplicate or filter on raw bytes, identical content slips through and filters misfire. Normalization (Unicode NFC/NFKC, whitespace collapsing, entity decoding, stripping control characters) comes first, and the same normalization must be applied anywhere text is compared later (dedup, decontamination, PII matching).

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Data Processing & Quality — Cleaning, Deduplication, Filtering and PII at Scale"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee