VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 15: The ML coding round

Lesson 15.6 of 12 in this module Unrolled Pro

Softmax and cross-entropy, numerically stable

Turn a batch of logits into probabilities with softmax and compute the cross-entropy loss and its gradient, without overflow or log(0). From per-row loops with exp that overflows at logits around 710, to subtracting each row's maximum (softmax doesn't change), computing log-softmax with log-sum-exp so the loss never takes log of a rounded zero, and the gradient (softmax − one-hot)/n verified against finite differences. Then padding tokens to ignore, label smoothing, temperature, class weights, and vocabularies too large to materialise.

about 10 min 3 Python examples 2 quick checks

Problem. Given logits Z (n × C, one row of scores per example) and integer labels y (n), compute

text
softmax(z)_c = e^(z_c) / Σ_j e^(z_j)
loss = −mean over rows of log softmax(z_i)[y_i]                  (cross-entropy)
∂loss/∂Z = (softmax(Z) − onehot(y)) / n

for any logits a model might produce, including very large or very negative ones, with no inf, no nan and no log(0).

The math is three lines. The interview question is what happens at Z = 1000: e^1000 overflows to inf, and inf / inf is nan. And at Z = −1000 every e^z rounds to 0, so softmax is 0 / 0. Two identities fix everything: softmax is unchanged by subtracting a constant from a row, and log-softmax can be computed directly with log-sum-exp, never taking log of a rounded probability.

ML context. This is the loss of every classifier and every language model (C = vocabulary size, one row per token). Training diverging with nan is very often this computation done naively, or a mask setting logits to −∞ for a whole row (lesson 15.7). Frameworks fuse softmax and cross-entropy for exactly the reasons in this lesson.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Softmax and cross-entropy, numerically stable"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee