Before Atlas gets a neural network, it gets a baseline: the simplest model that could work, measured honestly. The task is small and real. Given a section of a lesson, which topic is it about: Python the language, Python performance, or systems? Everything here is NumPy you can read top to bottom. The model is not the important part. The evaluation is, because every later part of Atlas is judged with the same discipline.
Step 1 · Hold out whole lessons
The first decision is the one that makes the numbers believable. If sections of one lesson land in both training and test, the model can recognise the lesson's own vocabulary ("dict", "hash", "bucket") instead of learning the topic. So five lessons are held out entirely, and no part of them is seen during training or tuning.
TEST_LESSONS = {"py-classes-types", "py-dicts-sets", "py-profiling", "py-async-concurrency", "databases-and-model-registry"}
That leaves 112 sections to train on and 37 to test on.
Step 2 · Features: TF-IDF
Each section becomes a vector with one entry per word. Counts are damped with log1p (the tenth "the" adds little), weighted by inverse document frequency (rare words say more about the topic), and each row is scaled to length 1 so long sections don't dominate.
def transform(self, docs: list) -> np.ndarray:
X = np.zeros((len(docs), len(self.index)))
for r, d in enumerate(docs):
for w, c in Counter(tokenize(d)).items():
j = self.index.get(w)
if j is not None:
X[r, j] = c
X = np.log1p(X) * self.idf
return X / np.maximum(np.linalg.norm(X, axis=1, keepdims=True), 1e-12)
The vocabulary and the IDF weights are fit on training text only. Fitting them on everything would let test lessons shape the features: a quieter form of the same leak.
Step 3 · Softmax regression, by hand
One weight matrix W maps features to three scores; softmax turns scores into probabilities; cross-entropy measures how much probability went to the right topic. The gradient of the mean cross-entropy with respect to the scores is simply P - Y (predicted minus one-hot truth), which makes the whole training loop four lines.
def fit(self, X, y, k):
n, d = X.shape
self.W, self.b = np.zeros((d, k)), np.zeros(k)
Y = np.eye(k)[y]
for _ in range(self.steps):
P = softmax(X @ self.W + self.b)
G = (P - Y) / n # d(loss)/d(scores)
self.W -= self.lr * (X.T @ G + self.l2 * self.W) # chain rule, plus L2
self.b -= self.lr * G.sum(axis=0)
return self
softmax subtracts each row's maximum before exp, so large scores can't overflow.
Step 4 · Tune without touching the test set
The L2 strength needs choosing. Choosing it by test score would turn the test set into training data. Instead, the training lessons are split into four groups, each group is held out in turn (whole lessons again, mirroring the real test), and the L2 with the best average macro-F1 wins.
def choose_l2(texts, y, slugs, grid=(1e-3, 1e-2, 3e-2, 1e-1)):
lessons = sorted(set(slugs))
folds = [lessons[i::4] for i in range(4)]
...
return max(scores, key=scores.get), scores
Step 5 · Report against the dumbest possible model
Accuracy alone flatters a model when one topic is common. Always predicting the most frequent training topic is the bar to beat, and macro-F1 (the average F1 over topics) exposes a model that ignores the rare ones.
Run it
python3 scripts/part2_baseline.py
train 112 sections · test 37 sections from held-out lessons
L2 chosen by grouped cross-validation on training lessons: 0.001
(CV macro-F1 per L2: {0.001: 0.432, 0.01: 0.264, 0.03: 0.23, 0.1: 0.209})
training loss 1.099 -> 0.126
accuracy macro-F1
majority baseline 37.8% 0.183
softmax regression 45.9% 0.352
confusion (rows = true, cols = predicted): ['python-language', 'python-performance', 'systems']
[[14 0 0]
[13 0 1]
[ 6 0 3]]
The model beats the baseline, and that's all it does. The confusion matrix says why: it never predicts python-performance. Those sections use the same vocabulary as the language lessons (functions, lists, loops), and 112 sections aren't enough for a linear model to find the difference. The training loss fell to 0.126 while test accuracy is 46%: the model fits its training sections far better than it generalises.
This is the honest starting point. Part 3 asks whether a neural network does better on the same split.
Check it works
python3 -m pytest -q tests -k "softmax or tfidf or macro"
The tests check that softmax regression separates two clean clusters perfectly while its loss falls, that TF-IDF rows have length 1, and that macro-F1 is 1.0 for perfect predictions and low for a majority guess.
Checkpoint. A baseline, and an evaluation you can trust: no lesson appears in both training and test, and tuning never sees the test set.