VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
Build Atlas

Part 2 of 9 · project

Learning from data

A first model that learns: TF-IDF features, softmax regression in NumPy, and an evaluation that can't cheat.

Before you start

This page builds the code. The ideas behind it are taught in the Library lessons listed at the end; read them first if a step uses something you haven't met. Inspect and copy the complete code in the Atlas Codebase Browser and run each step as you read.

Before Atlas gets a neural network, it gets a baseline: the simplest model that could work, measured honestly. The task is small and real. Given a section of a lesson, which topic is it about: Python the language, Python performance, or systems? Everything here is NumPy you can read top to bottom. The model is not the important part. The evaluation is, because every later part of Atlas is judged with the same discipline.

sections from training lessons fit a TF-IDF vocabulary and a softmax regression; five whole lessons are held out and only used once, for the final score, next to the majority baseline.
Diagram— sections from training lessons fit a TF-IDF vocabulary and a softmax regression; five whole lessons are held out and only used once, for the final score, next to the majority baseline.

Step 1 · Hold out whole lessons

The first decision is the one that makes the numbers believable. If sections of one lesson land in both training and test, the model can recognise the lesson's own vocabulary ("dict", "hash", "bucket") instead of learning the topic. So five lessons are held out entirely, and no part of them is seen during training or tuning.

python
TEST_LESSONS = {"py-classes-types", "py-dicts-sets", "py-profiling", "py-async-concurrency", "databases-and-model-registry"}

That leaves 112 sections to train on and 37 to test on.

Step 2 · Features: TF-IDF

Each section becomes a vector with one entry per word. Counts are damped with log1p (the tenth "the" adds little), weighted by inverse document frequency (rare words say more about the topic), and each row is scaled to length 1 so long sections don't dominate.

python
def transform(self, docs: list) -> np.ndarray:
    X = np.zeros((len(docs), len(self.index)))
    for r, d in enumerate(docs):
        for w, c in Counter(tokenize(d)).items():
            j = self.index.get(w)
            if j is not None:
                X[r, j] = c
    X = np.log1p(X) * self.idf
    return X / np.maximum(np.linalg.norm(X, axis=1, keepdims=True), 1e-12)

The vocabulary and the IDF weights are fit on training text only. Fitting them on everything would let test lessons shape the features: a quieter form of the same leak.

Step 3 · Softmax regression, by hand

One weight matrix W maps features to three scores; softmax turns scores into probabilities; cross-entropy measures how much probability went to the right topic. The gradient of the mean cross-entropy with respect to the scores is simply P - Y (predicted minus one-hot truth), which makes the whole training loop four lines.

python
def fit(self, X, y, k):
    n, d = X.shape
    self.W, self.b = np.zeros((d, k)), np.zeros(k)
    Y = np.eye(k)[y]
    for _ in range(self.steps):
        P = softmax(X @ self.W + self.b)
        G = (P - Y) / n                                   # d(loss)/d(scores)
        self.W -= self.lr * (X.T @ G + self.l2 * self.W)  # chain rule, plus L2
        self.b -= self.lr * G.sum(axis=0)
    return self

softmax subtracts each row's maximum before exp, so large scores can't overflow.

Step 4 · Tune without touching the test set

The L2 strength needs choosing. Choosing it by test score would turn the test set into training data. Instead, the training lessons are split into four groups, each group is held out in turn (whole lessons again, mirroring the real test), and the L2 with the best average macro-F1 wins.

python
def choose_l2(texts, y, slugs, grid=(1e-3, 1e-2, 3e-2, 1e-1)):
    lessons = sorted(set(slugs))
    folds = [lessons[i::4] for i in range(4)]
    ...
    return max(scores, key=scores.get), scores

Step 5 · Report against the dumbest possible model

Accuracy alone flatters a model when one topic is common. Always predicting the most frequent training topic is the bar to beat, and macro-F1 (the average F1 over topics) exposes a model that ignores the rare ones.

Run it

bash
python3 scripts/part2_baseline.py
text
train 112 sections · test 37 sections from held-out lessons
L2 chosen by grouped cross-validation on training lessons: 0.001
  (CV macro-F1 per L2: {0.001: 0.432, 0.01: 0.264, 0.03: 0.23, 0.1: 0.209})
training loss 1.099 -> 0.126

                        accuracy  macro-F1
majority baseline          37.8%     0.183
softmax regression         45.9%     0.352

confusion (rows = true, cols = predicted): ['python-language', 'python-performance', 'systems']
[[14  0  0]
 [13  0  1]
 [ 6  0  3]]

The model beats the baseline, and that's all it does. The confusion matrix says why: it never predicts python-performance. Those sections use the same vocabulary as the language lessons (functions, lists, loops), and 112 sections aren't enough for a linear model to find the difference. The training loss fell to 0.126 while test accuracy is 46%: the model fits its training sections far better than it generalises.

This is the honest starting point. Part 3 asks whether a neural network does better on the same split.

Check it works

bash
python3 -m pytest -q tests -k "softmax or tfidf or macro"

The tests check that softmax regression separates two clean clusters perfectly while its loss falls, that TF-IDF rows have length 1, and that macro-F1 is 1.0 for perfect predictions and low for a majority guess.

Checkpoint. A baseline, and an evaluation you can trust: no lesson appears in both training and test, and tuning never sees the test set.

View Source Code

Go deeper in the Library

The lessons that teach the ideas this part uses, in the order to read them.