Domain 5: LLM Architecture & Engineering
How LLMs work: tokenization, attention, RoPE and grouped-query attention, the transformer block, architecture families, pretraining and building a GPT.
Unlock all 7 lessons with Unrolled Pro
Full production code, quantitative capacity math, failure recovery playbooks, and interview coaching.
Lessons
7 lessonsNLP Foundations — From Text to Tokens to Predictions
Every LLM pipeline begins the same way: text → tokens → token IDs → embedding vectors, and every LLM is trained on the same task: predict the next token.
The Transformer I — Attention, Positions and Heads from First Principles
Every modern language model, from a 1M-parameter TinyGPT to the frontier, is the same stack: embeddings, then N identical blocks of attention and MLP, then a projection back to the vocabulary.
The Transformer II — The Block, the Output, the Arithmetic and Production Practice
Lesson I built attention.
LLM Architecture Families — Trade-offs, Not Model Names
Model names change every month; the design choices underneath don't. Each family makes a different trade between understanding vs generating, parameters vs compute per token, and context length vs memory.
LLM Pretraining — Data, Compute and Scaling Laws
Pretraining is where a model gets almost all of its knowledge and ability — and almost all of its cost. Three questions dominate: What data? (quality beats quantity, and leakage of test sets ruins evaluations.
Build an LLM from Scratch
Every idea in this domain has been shown in pieces. This lesson puts them together into one script that trains a working GPT: data → tokenizer → model → training → checkpoint → sampling → evaluation.
Bridge: From a Model That Writes to a Model That Helps
Atlas now has a language model that writes fluent text. Ask it a question, though, and it carries on writing instead of answering.