Problem. Given logits Z (n × C, one row of scores per example) and integer labels y (n), compute
softmax(z)_c = e^(z_c) / Σ_j e^(z_j)
loss = −mean over rows of log softmax(z_i)[y_i] (cross-entropy)
∂loss/∂Z = (softmax(Z) − onehot(y)) / n
for any logits a model might produce, including very large or very negative ones, with no inf, no nan and no log(0).
The math is three lines. The interview question is what happens at Z = 1000: e^1000 overflows to inf, and inf / inf is nan. And
at Z = −1000 every e^z rounds to 0, so softmax is 0 / 0. Two identities fix everything: softmax is unchanged by subtracting a
constant from a row, and log-softmax can be computed directly with log-sum-exp, never taking log of a rounded probability.
ML context. This is the loss of every classifier and every language model (C = vocabulary size, one row per token). Training
diverging with nan is very often this computation done naively, or a mask setting logits to −∞ for a whole row (lesson 15.7).
Frameworks fuse softmax and cross-entropy for exactly the reasons in this lesson.