VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D6: Post-training & Alignment · Parameter-Efficient Fine-Tuning — LoRA, QLoRA and Adapters
Phase 4/Domain 6/Lesson 2

Parameter-Efficient Fine-Tuning — LoRA, QLoRA and Adapters

20 min read·Advanced· Unrolled Pro
Parameter-Efficient Fine-Tuning — LoRA, QLoRA and Adapters

You already know: matrix rank and the SVD (D2 Linear algebra); Adam keeps two extra numbers per parameter (D2 Optimization); requires_grad, BF16 and memory budgets (D4 PyTorch engineering); SFT and catastrophic forgetting (Supervised fine-tuning).

Why this matters for Atlas. Full fine-tuning updates every weight, so it needs memory for weights, gradients and optimizer state for all 8 billion parameters, and it produces a full new copy of the model. Parameter-efficient fine-tuning (PEFT) freezes the pretrained model and trains a tiny number of new parameters instead — typically 0.1–1%. That makes fine-tuning fit on a single GPU, makes each customization a file of megabytes instead of gigabytes, and lets one server host hundreds of customizations at once. LoRA, the dominant PEFT method, is what you will actually use for most fine-tuning work.


1. The idea: the update is low-rank

The question. Fine-tuning changes a weight matrix W into W + ΔW. Does ΔW really need all d × d numbers?

Start from nothing. A 4096 × 4096 weight matrix has 16.8 million entries. If fine-tuning only needs to adjust the model's behaviour in a few "directions", the change ΔW should be low-rank: expressible as the product of a tall, thin matrix and a short, wide one. LoRA (Low-Rank Adaptation; Hu et al., 2021) builds that assumption in:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Parameter-Efficient Fine-Tuning — LoRA, QLoRA and Adapters"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 5 of 9: Make it helpfulUse this lesson in the project: Teach TinyGPT to answer questions with a chat template and a loss mask, then compare full fine-tuning with LoRA on unseen questions.