You already know: why word-level and character-level vocabularies both fail (Bridge D4 → D5); PCA/SVD finds low-dimensional structure (D3 Unsupervised §4, D2 Linear algebra §6); cross-entropy and perplexity (D2 Information theory §3, §5).
Why this matters for Atlas. Every LLM pipeline begins the same way: text → tokens → token IDs → embedding vectors, and every LLM is trained on the same task: predict the next token. Tokenization quietly decides cost (you pay per token), context length, multilingual quality and even arithmetic ability. Understanding these foundations explains many "weird" LLM behaviours.
1. Tokenization and byte-pair encoding (BPE)
The question. How do we choose units between "characters" and "words"?
Start from nothing. Byte-pair encoding learns a vocabulary from data: