VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D6: Post-training & Alignment · Reinforcement Learning for LLMs — Policy Gradients, PPO and GRPO
Phase 4/Domain 6/Lesson 7

Reinforcement Learning for LLMs — Policy Gradients, PPO and GRPO

20 min read·Advanced· Unrolled Pro
Reinforcement Learning for LLMs — Policy Gradients, PPO and GRPO

You already know: log-likelihood and its gradient (D2 Calculus, Probability); variance of estimators (D2 Statistics); sequence log-probs and the KL-regularized objective (Preference optimization); sampling with temperature (Build an LLM from scratch).

Why this matters for Atlas. SFT imitates and DPO learns from fixed comparisons. Reinforcement learning lets the model learn from its own attempts: it generates answers, a reward says how good each one was, and the model shifts probability toward what worked. When the reward can be checked automatically — a maths answer, a passing unit test, a valid proof — RL can improve the model beyond any demonstration data. This is how today's reasoning models were trained (DeepSeek-R1, OpenAI's o-series), and it is the foundation for training agents (D9).


1. Generating text is a decision process

The question. RL comes from robotics and games. Where are the states, actions and rewards in text generation?

Start from nothing. A Markov decision process (MDP) has states, actions, transitions and rewards. Mapped onto an LLM:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Reinforcement Learning for LLMs — Policy Gradients, PPO and GRPO"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 5 of 9: Make it helpfulUse this lesson in the project: Teach TinyGPT to answer questions with a chat template and a loss mask, then compare full fine-tuning with LoRA on unseen questions.