Problem. Given X (n × d) and targets y (n), find weights w and bias b minimising the mean squared error
MSE(w, b) = mean( (X w + b − y)² )
Implement (1) the closed form and (2) gradient descent, and say when to use which.
Setting the gradient to zero gives the normal equations XᵀX w = Xᵀy (with a column of ones appended for the bias). The textbook answer is w = (XᵀX)⁻¹ Xᵀy. The interview answer is: never form that inverse. Solve the least-squares problem with a factorisation (QR or SVD), which stays accurate when features are nearly collinear, and switch to gradient descent when d or n is too large to factorise.
ML context. Linear regression is the baseline regressor, the last layer of many networks, the solver inside linear probes on frozen embeddings, and the building block of ridge, kernel and Bayesian regression. Its two solution styles, a direct solve vs iterative optimisation, are the same trade-off you face for every model: exact and O(d³), or approximate and O(n · d) per step.