You already know: log-likelihood and its gradient (D2 Calculus, Probability); variance of estimators (D2 Statistics); sequence log-probs and the KL-regularized objective (Preference optimization); sampling with temperature (Build an LLM from scratch).
Why this matters for Atlas. SFT imitates and DPO learns from fixed comparisons. Reinforcement learning lets the model learn from its own attempts: it generates answers, a reward says how good each one was, and the model shifts probability toward what worked. When the reward can be checked automatically — a maths answer, a passing unit test, a valid proof — RL can improve the model beyond any demonstration data. This is how today's reasoning models were trained (DeepSeek-R1, OpenAI's o-series), and it is the foundation for training agents (D9).
1. Generating text is a decision process
The question. RL comes from robotics and games. Where are the states, actions and rewards in text generation?
Start from nothing. A Markov decision process (MDP) has states, actions, transitions and rewards. Mapped onto an LLM: