You already know: linear and logistic regression, overfitting and regularization (Supervised learning); Bayes' rule (Probability); Lagrange multipliers and convexity (Convexity and Lagrange); distances and dot products (Linear algebra).
Why this matters for Atlas. These three methods are built on three different ideas: similarity (kNN: you're like your neighbours), probability (Naive Bayes: which class most likely generated these features?) and geometry (SVM: the widest possible gap between classes). Each idea survives in modern systems: nearest-neighbour search powers RAG and recommendations, Naive Bayes-style counting powers fast text filters, and margins and kernels underlie contrastive learning and attention. This lesson builds each from scratch, shows where it shines and fails, and compares them on the same problems.
1. k-nearest neighbours: predict like your neighbours
The question. What's the simplest classifier that can learn any boundary?
Start from nothing. Store the training set. To classify a new point, find its k nearest training points (by some distance) and take a majority vote (or average, for regression). There's no training step; all the cost is at prediction time, which is why large-scale kNN relies on approximate nearest-neighbour indexes (D8). k controls the bias–variance trade-off: k = 1 memorizes noise, large k smooths over real structure. Features must be on comparable scales, and in high dimensions distances concentrate (everything is about equally far away), the curse of dimensionality, unless the data lives on a lower-dimensional structure.