VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D6: Post-training & Alignment · Reinforcement Learning III — Policy Gradients, Baselines, Actor-Critic, GAE and PPO From Scratch
Phase 4/Domain 6/Lesson 6

Reinforcement Learning III — Policy Gradients, Baselines, Actor-Critic, GAE and PPO From Scratch

15 min read·Advanced· Unrolled Pro
the actor (policy) and critic (value function) loop: rollouts → returns and TD errors → GAE advantages → policy update; a histogram of gradient estimates that narrows from REINFORCE to baseline to GAE; PPO's clipped objective as a function of the probability ratio, flat outside [1 − ε, 1 + ε].
Diagram— the actor (policy) and critic (value function) loop: rollouts → returns and TD errors → GAE advantages → policy update; a histogram of gradient estimates that narrows from REINFORCE to baseline to GAE; PPO's clipped objective as a function of the probability ratio, flat outside [1 − ε, 1 + ε].

You already know: value functions, TD errors and bootstrapping (RL I and II); the policy-gradient idea and PPO in the LLM setting, briefly (D6 RL for LLMs is the next lesson in this track and builds on this one).

Why this matters for Atlas. Value-based methods (Q-learning) pick actions by maximizing estimated values, which is awkward for huge or continuous action spaces like "the next token". Policy-gradient methods optimize the policy directly: increase the probability of actions that led to better-than-expected outcomes. Their weakness is variance; the fixes are baselines, critics and generalized advantage estimation (GAE). Their other weakness is step size: one bad update can wreck a policy, and PPO's clipping is the standard guard. This lesson measures each fix on a small control problem.


1. Policy gradients and the variance problem

The question. How do you take a gradient of expected reward when the reward isn't differentiable?

Start from nothing. The policy-gradient theorem (REINFORCE) gives ∇J(θ) = E[Σₜ ∇log π_θ(aₜ|sₜ) · Gₜ]: push up the log-probability of each action in proportion to the return that followed. It needs no model and no differentiable reward, just samples. But Gₜ is a sum of many random rewards, so single-batch estimates are very noisy. Subtracting a baseline b(sₜ) that doesn't depend on the action leaves the expected gradient unchanged and can cut variance enormously; the natural choice is the value V(sₜ), giving the advantage Aₜ = Gₜ − V(sₜ): "how much better than expected was this?". Actor-critic methods learn V (the critic) alongside the policy (the actor). GAE blends TD errors δₜ = rₜ + γV(sₜ₊₁) − V(sₜ) over several steps with a parameter λ, trading bias (λ → 0, pure one-step TD) against variance (λ → 1, Monte Carlo).

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Reinforcement Learning III — Policy Gradients, Baselines, Actor-Critic, GAE and PPO From Scratch"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee