You already know: matrix rank and the SVD (D2 Linear algebra); Adam keeps two extra numbers per parameter (D2 Optimization); requires_grad, BF16 and memory budgets (D4 PyTorch engineering); SFT and catastrophic forgetting (Supervised fine-tuning).
Why this matters for Atlas. Full fine-tuning updates every weight, so it needs memory for weights, gradients and optimizer state for all 8 billion parameters, and it produces a full new copy of the model. Parameter-efficient fine-tuning (PEFT) freezes the pretrained model and trains a tiny number of new parameters instead — typically 0.1–1%. That makes fine-tuning fit on a single GPU, makes each customization a file of megabytes instead of gigabytes, and lets one server host hundreds of customizations at once. LoRA, the dominant PEFT method, is what you will actually use for most fine-tuning work.
1. The idea: the update is low-rank
The question. Fine-tuning changes a weight matrix W into W + ΔW. Does ΔW really need all d × d numbers?
Start from nothing. A 4096 × 4096 weight matrix has 16.8 million entries. If fine-tuning only needs to adjust the model's behaviour in a few "directions", the change ΔW should be low-rank: expressible as the product of a tall, thin matrix and a short, wide one. LoRA (Low-Rank Adaptation; Hu et al., 2021) builds that assumption in: