VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D15: Distributed Training · Training Memory Optimization — ZeRO, FSDP, Checkpointing and Mixed Precision

Training Memory Optimization — ZeRO, FSDP, Checkpointing and Mixed Precision

15 min read·Staff+ Tier· Unrolled Pro
Training Memory Optimization — ZeRO, FSDP, Checkpointing and Mixed Precision

You already know: Adam keeps two moment estimates per parameter (D2 Optimization); mixed precision and the 16 bytes/parameter of training state (D5 LLM pretraining, D14 bridge); BF16 vs FP16 range (D12 Quantization); data parallelism and all-gather/reduce-scatter (Parallelism strategies, Collective communication).

Why this matters for Atlas. Training memory has four parts: parameters, gradients, optimizer state and activations. Data parallelism replicates the first three on every GPU, wasting most of the cluster's memory on identical copies. ZeRO and FSDP shard them instead, so N GPUs hold one copy between them. Activation checkpointing trades recomputation for activation memory. Mixed precision halves bytes without losing accuracy. Together they decide the largest model and batch you can train on given hardware, often by 8× or more. This lesson derives each saving, implements sharded training in NumPy to show it's exact, and quantifies the compute and communication each technique costs.


1. Where the memory goes

The question. What exactly fills a GPU during training?

Start from nothing. With mixed-precision training and Adam, per parameter:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Training Memory Optimization — ZeRO, FSDP, Checkpointing and Mixed Precision"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee