You already know: MLPs, activations, normalization and residual blocks (Neural network fundamentals); attention as softmax(QKᵀ/√d)V (D2 Linear algebra §5).
Why this matters for Atlas. An MLP treats its input as a flat list and ignores structure. But images are grids (nearby pixels matter), events are sequences (order matters), knowledge is a graph (connections matter). Each architecture builds in an assumption — an inductive bias — about the data's shape. The transformer is the one that took over language, and then nearly everything else.
1. Convolutional networks (CNNs): patterns anywhere in a grid
The question. A "warning icon" means the same thing in the top-left or bottom-right of a screenshot. How do we avoid learning it separately for every position?
Start from nothing. A convolution slides a small filter (say 3×3 weights) across the image and computes a dot product at every position, producing a feature map. Two built-in assumptions: