Loss drops for 20 steps, then jumps up and becomes NaN. Gradients are not clipped. What is the first thing to try?
Answer: B. Lower the learning rate (add warmup and gradient clipping)
Steps that are too large overshoot the minimum and diverge. A smaller learning rate, a warmup period and gradient clipping are the standard fixes.