VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D5: LLM Architecture · NLP Foundations — From Text to Tokens to Predictions
Phase 4/Domain 5/Lesson 1

NLP Foundations — From Text to Tokens to Predictions

15 min read·Advanced· Unrolled Pro
NLP Foundations — From Text to Tokens to Predictions

You already know: why word-level and character-level vocabularies both fail (Bridge D4 → D5); PCA/SVD finds low-dimensional structure (D3 Unsupervised §4, D2 Linear algebra §6); cross-entropy and perplexity (D2 Information theory §3, §5).

Why this matters for Atlas. Every LLM pipeline begins the same way: text → tokens → token IDs → embedding vectors, and every LLM is trained on the same task: predict the next token. Tokenization quietly decides cost (you pay per token), context length, multilingual quality and even arithmetic ability. Understanding these foundations explains many "weird" LLM behaviours.


1. Tokenization and byte-pair encoding (BPE)

The question. How do we choose units between "characters" and "words"?

Start from nothing. Byte-pair encoding learns a vocabulary from data:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "NLP Foundations — From Text to Tokens to Predictions"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 4 of 9: From text to GPTUse this lesson in the project: Train a byte-level BPE tokenizer and a 940K-parameter GPT on the lessons, on a laptop CPU, and read its loss curve honestly.