VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D6: Post-training & Alignment · RL in Practice — Environments, Verifiers, Reward Hacking, Reading RL Curves, and a GRPO Capstone
Phase 4/Domain 6/Lesson 8

RL in Practice — Environments, Verifiers, Reward Hacking, Reading RL Curves, and a GRPO Capstone

15 min read·Advanced· Unrolled Pro
an RL environment for LLMs (task sampler → policy generates an answer or tool-use trajectory in a sandbox → verifier or reward model scores it → advantages → update), with failure points marked: a leaky verifier, reward hacking, zero-variance groups. Beside it, four training curves to read together: reward, true held-out accuracy, policy entropy, and the fraction of groups with no learning signal.
Diagram— an RL environment for LLMs (task sampler → policy generates an answer or tool-use trajectory in a sandbox → verifier or reward model scores it → advantages → update), with failure points marked: a leaky verifier, reward hacking, zero-variance groups. Beside it, four training curves to read together: reward, true held-out accuracy, policy entropy, and the fraction of groups with no learning signal.

You already know: policy gradients, PPO and GRPO for LLMs (RL for LLMs, RL III); evaluation discipline and contamination (D10); agent sandboxes and tool security (D19).

Why this matters for Atlas. Reinforcement learning optimizes whatever the reward says, not what you meant. With verifiable rewards (maths answers, unit tests, environment states), that's powerful: no human labels, unlimited practice. But every shortcut the verifier allows, the policy will eventually find (reward hacking), learned reward models are exploited even more readily (over-optimization), and RL runs fail quietly: rewards rise while real ability doesn't, or learning stalls because the reward signal vanishes. This lesson covers how to build environments and verifiers, how to detect hacking, which curves to watch, and finishes with a from-scratch GRPO run that shows each failure mode on a small arithmetic task.


1. Environments and verifiers

The question. What makes a good RL environment for a language model?

Start from nothing. An environment provides tasks (prompts with hidden ground truth or a checkable goal), an interaction protocol (single answer, or multi-turn tool use in a sandbox), and a reward. Good verifiers are: correct (accept all right answers, including equivalent forms: "1/2", "0.5", "\frac{1}{2}"), strict (reject everything else, including answers that merely contain the right string), robust to exploits (can't be satisfied by printing many candidates, editing the tests, or crashing the checker), cheap and deterministic, and decontaminated from evaluation sets. For agents, run code and tools in a sandbox with time and resource limits (D19), and check the final state (was the file actually fixed? did the test suite pass unmodified?), not the agent's claims.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "RL in Practice — Environments, Verifiers, Reward Hacking, Reading RL Curves, and a GRPO Capstone"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee