You already know: value functions, TD errors and bootstrapping (RL I and II); the policy-gradient idea and PPO in the LLM setting, briefly (D6 RL for LLMs is the next lesson in this track and builds on this one).
Why this matters for Atlas. Value-based methods (Q-learning) pick actions by maximizing estimated values, which is awkward for huge or continuous action spaces like "the next token". Policy-gradient methods optimize the policy directly: increase the probability of actions that led to better-than-expected outcomes. Their weakness is variance; the fixes are baselines, critics and generalized advantage estimation (GAE). Their other weakness is step size: one bad update can wreck a policy, and PPO's clipping is the standard guard. This lesson measures each fix on a small control problem.
1. Policy gradients and the variance problem
The question. How do you take a gradient of expected reward when the reward isn't differentiable?
Start from nothing. The policy-gradient theorem (REINFORCE) gives ∇J(θ) = E[Σₜ ∇log π_θ(aₜ|sₜ) · Gₜ]: push up the log-probability of each action in proportion to the return that followed. It needs no model and no differentiable reward, just samples. But Gₜ is a sum of many random rewards, so single-batch estimates are very noisy. Subtracting a baseline b(sₜ) that doesn't depend on the action leaves the expected gradient unchanged and can cut variance enormously; the natural choice is the value V(sₜ), giving the advantage Aₜ = Gₜ − V(sₜ): "how much better than expected was this?". Actor-critic methods learn V (the critic) alongside the policy (the actor). GAE blends TD errors δₜ = rₜ + γV(sₜ₊₁) − V(sₜ) over several steps with a parameter λ, trading bias (λ → 0, pure one-step TD) against variance (λ → 1, Monte Carlo).