VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D6: Post-training & Alignment · Preference Optimization — Reward Models, RLHF and DPO
Phase 4/Domain 6/Lesson 3

Preference Optimization — Reward Models, RLHF and DPO

20 min read·Advanced· Unrolled Pro
Preference Optimization — Reward Models, RLHF and DPO

You already know: the sigmoid and logistic regression (D3 Supervised learning); maximum likelihood (D2 Probability); KL divergence (D2 Information theory); SFT and loss masking (Supervised fine-tuning); LoRA (Parameter-efficient fine-tuning).

Why this matters for Atlas. SFT teaches the model to imitate demonstrations, so it can never be better than the demonstrations. And for many qualities — clarity, tone, safety, "is this actually helpful?" — it is much easier for a person to compare two answers than to write the perfect one. Preference optimization turns those comparisons into training signal. It is how InstructGPT, ChatGPT, Claude and Llama were aligned, and the same tools return in D10 (LLM-as-a-judge) and D19 (safety training).


1. From comparisons to scores: the Bradley–Terry model

The question. People give us "A is better than B". How do we turn that into a number for each answer?

Start from nothing. Suppose each answer i has a hidden quality score sᵢ, and the chance a person prefers i over j depends only on the difference:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Preference Optimization — Reward Models, RLHF and DPO"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 5 of 9: Make it helpfulUse this lesson in the project: Teach TinyGPT to answer questions with a chat template and a loss mask, then compare full fine-tuning with LoRA on unseen questions.