This is the part where Atlas gets a language model of its own. Two pieces: a tokenizer that turns text into ids a model can learn from, and TinyGPT, a decoder-only transformer with the same architecture as GPT-2, shrunk to 940,800 parameters so it trains in about a minute and a half on a laptop. Then comes the part that matters most: reading the loss curve and keeping the right weights.
Step 1 · A byte-level BPE tokenizer
Characters make sequences long; whole words make the vocabulary explode and fail on unseen words. Byte-pair encoding sits in between. Start from the 256 possible bytes, so any text in any language can be encoded. Then repeatedly find the most frequent adjacent pair and merge it into a new token.
Two details make it fast and sensible. Text is first cut into pieces that keep their leading space (" the", " Python"), like GPT tokenizers, and merges never cross a piece. And each distinct piece is counted once with its frequency, instead of rescanning 180K characters for every merge.