Problem. Implement batchnorm_forward(x, gamma, beta, running_mean, running_var, momentum, eps, training):
training: μ = mean of x over the batch, σ² = biased variance over the batch
x̂ = (x − μ) / √(σ² + ε)
y = γ · x̂ + β
running_mean ← (1 − momentum) · running_mean + momentum · μ
running_var ← (1 − momentum) · running_var + momentum · σ²_unbiased
inference: x̂ = (x − running_mean) / √(running_var + ε), y = γ · x̂ + β
for dense inputs x (N × D, statistics per feature over N) and convolutional inputs (N × C × H × W, statistics per channel over N, H and W).
The formula is short. What's tested: reducing over the right axes, computing the variance in a numerically safe way, updating running statistics only in training, using them (not batch statistics) at inference, and knowing why this breaks with tiny batches.
ML context. Batch norm is in most convolutional networks; its sibling layer norm (Change A) is in every transformer. The classic production bug is evaluating a model in training mode: predictions then depend on what else is in the batch. Another is a variance computed as E[x²] − E[x]², which goes negative or loses all digits for features with a large mean.