You already know: the sigmoid and logistic regression (D3 Supervised learning); maximum likelihood (D2 Probability); KL divergence (D2 Information theory); SFT and loss masking (Supervised fine-tuning); LoRA (Parameter-efficient fine-tuning).
Why this matters for Atlas. SFT teaches the model to imitate demonstrations, so it can never be better than the demonstrations. And for many qualities — clarity, tone, safety, "is this actually helpful?" — it is much easier for a person to compare two answers than to write the perfect one. Preference optimization turns those comparisons into training signal. It is how InstructGPT, ChatGPT, Claude and Llama were aligned, and the same tools return in D10 (LLM-as-a-judge) and D19 (safety training).
1. From comparisons to scores: the Bradley–Terry model
The question. People give us "A is better than B". How do we turn that into a number for each answer?
Start from nothing. Suppose each answer i has a hidden quality score sᵢ, and the chance a person prefers i over j depends only on the difference: