VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
Build Atlas

Part 4 of 9 · project Unrolled Pro

From text to GPT

Train a byte-level BPE tokenizer and a 940K-parameter GPT on the lessons, on a laptop CPU, and read its loss curve honestly.

Before you start

This page builds the code. The ideas behind it are taught in the Library lessons listed at the end; read them first if a step uses something you haven't met. Inspect and copy the complete code in the Atlas Codebase Browser and run each step as you read.

This is the part where Atlas gets a language model of its own. Two pieces: a tokenizer that turns text into ids a model can learn from, and TinyGPT, a decoder-only transformer with the same architecture as GPT-2, shrunk to 940,800 parameters so it trains in about a minute and a half on a laptop. Then comes the part that matters most: reading the loss curve and keeping the right weights.

text becomes BPE token ids, then embeddings plus positions, then four transformer blocks of causal self-attention and an MLP with residual connections, then a final norm and a head tied to the embedding matrix that predicts the next token.
Diagram— text becomes BPE token ids, then embeddings plus positions, then four transformer blocks of causal self-attention and an MLP with residual connections, then a final norm and a head tied to the embedding matrix that predicts the next token.

Step 1 · A byte-level BPE tokenizer

Characters make sequences long; whole words make the vocabulary explode and fail on unseen words. Byte-pair encoding sits in between. Start from the 256 possible bytes, so any text in any language can be encoded. Then repeatedly find the most frequent adjacent pair and merge it into a new token.

Two details make it fast and sensible. Text is first cut into pieces that keep their leading space (" the", " Python"), like GPT tokenizers, and merges never cross a piece. And each distinct piece is counted once with its frequency, instead of rescanning 180K characters for every merge.

That's the preview. Build Atlas Parts 1 and 2 are free; the full walkthrough of Part 4 is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Build Atlas, Part 4: From text to GPT"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee

Go deeper in the Library

The lessons that teach the ideas this part uses, in the order to read them.