You already know: derivatives, partial derivatives, gradients and the chain rule (Calculus); matrices as transformations and eigenvalues (Linear algebra, Linear systems).
Why this matters for Atlas. Backpropagation is the chain rule applied to functions whose inputs and outputs are vectors, so each step multiplies by a Jacobian matrix. Frameworks never form these matrices; they compute vector–Jacobian products, which is why backprop costs about the same as the forward pass. Taylor series explain why gradient descent works (locally a function is linear), how far a step can go (curvature), and where second-order methods and learning-rate limits come from. This lesson makes both concrete and checks them numerically.
1. The Jacobian: derivatives of vector functions
The question. What is the derivative of a function that outputs a vector?
Start from nothing. For f: ℝⁿ → ℝᵐ, the Jacobian J is the m×n matrix of partial derivatives Jᵢⱼ = ∂fᵢ/∂xⱼ: row i is the gradient of output i. It's the best linear approximation of f near x: f(x + δ) ≈ f(x) + Jδ. For a linear layer y = Wx, J = W. For an elementwise function like ReLU or tanh, J is diagonal. For softmax, J = diag(p) − ppᵀ. Checking a Jacobian with finite differences is the standard way to verify hand-written derivatives.