You already know: policy gradients, PPO and GRPO for LLMs (RL for LLMs, RL III); evaluation discipline and contamination (D10); agent sandboxes and tool security (D19).
Why this matters for Atlas. Reinforcement learning optimizes whatever the reward says, not what you meant. With verifiable rewards (maths answers, unit tests, environment states), that's powerful: no human labels, unlimited practice. But every shortcut the verifier allows, the policy will eventually find (reward hacking), learned reward models are exploited even more readily (over-optimization), and RL runs fail quietly: rewards rise while real ability doesn't, or learning stalls because the reward signal vanishes. This lesson covers how to build environments and verifiers, how to detect hacking, which curves to watch, and finishes with a from-scratch GRPO run that shows each failure mode on a small arithmetic task.
1. Environments and verifiers
The question. What makes a good RL environment for a language model?
Start from nothing. An environment provides tasks (prompts with hidden ground truth or a checkable goal), an interaction protocol (single answer, or multi-turn tool use in a sandbox), and a reward. Good verifiers are: correct (accept all right answers, including equivalent forms: "1/2", "0.5", "\frac{1}{2}"), strict (reject everything else, including answers that merely contain the right string), robust to exploits (can't be satisfied by printing many candidates, editing the tests, or crashing the checker), cheap and deterministic, and decontaminated from evaluation sets. For agents, run code and tools in a sandbox with time and resource limits (D19), and check the final state (was the file actually fixed? did the test suite pass unmodified?), not the agent's claims.