You already know: BPE and next-token prediction (NLP foundations); causal attention, multi-head attention, pre-norm blocks and tied embeddings (Transformer architecture); data pipelines, perplexity and the training recipe (LLM pretraining); the PyTorch training loop, AdamW, warmup + cosine, clipping and checkpointing (D4 PyTorch engineering).
Why this matters for Atlas. Every idea in this domain has been shown in pieces. This lesson puts them together into one script that trains a working GPT: data → tokenizer → model → training → checkpoint → sampling → evaluation. It is small (about 0.8M parameters, trained for about a minute on a laptop CPU), but it is the same program used to train production models. The differences are scale, data and engineering, all of which later domains cover. Once you have built this, nothing about an LLM is a black box.
The plan.
| Stage | What we build | Real-world counterpart |
|---|---|---|
| 1. Data | A corpus of Python standard-library docstrings, de-duplicated, train/val split | Web-scale crawl + filtering (LLM pretraining §1) |
| 2. Tokenizer | Character-level encode/decode | BPE with 32k–200k tokens (NLP foundations §1) |
| 3. Model | Embeddings, positions, causal multi-head attention, MLP, LayerNorm, residuals | The same block, wider and deeper |
| 4. Initialization | Small-std weights so the starting loss is ln V | The same rule |
| 5. Training | AdamW, warmup + cosine, clipping, validation | The same loop, distributed (D15) |
| 6. Checkpoint & sample | Save the best weights; temperature + top-k sampling | Serving stacks (D12) |
| 7. Evaluate | Perplexity against a baseline; the train/val gap | Benchmark suites (D10) |
1. Dataset preparation
The question. We need text that a laptop can learn from in a minute: big enough to have real structure, small enough to fit in one pass.
Start from nothing. A language model learns whatever its data looks like. We want text that is (a) consistent in style, so a small model can pick up patterns, (b) free to use, and (c) available offline. Python's own docstrings fit: every installed Python ships with them, they read like technical English, and they contain code-like fragments (`str`, >>>, indentation).