VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D15: Distributed Training · Parallelism Strategies — Data, Tensor, Pipeline, Sequence and Expert

Parallelism Strategies — Data, Tensor, Pipeline, Sequence and Expert

15 min read·Staff+ Tier· Unrolled Pro
Parallelism Strategies — Data, Tensor, Pipeline, Sequence and Expert

You already know: training FLOPs (6 × params × tokens) and mixed-precision training (D5 LLM pretraining); the training loop in PyTorch (D4); tensor parallelism and MoE at inference (D12 Advanced optimization); stragglers and lockstep work (D14 Distributed compute); the D14 bridge's memory and time arithmetic.

Why this matters for Atlas. No single GPU can hold or train a frontier model, so training splits the work along one or more dimensions: the data (each GPU sees different examples), the tensors inside each layer, the layers themselves (pipeline), the sequence, or the experts of an MoE model. Each split saves memory or time in a different way and costs a different kind of communication. Real training runs combine them ("3D" or "4D" parallelism), chosen to fit the model in memory and keep the expensive communication on the fastest links. This lesson builds each strategy in NumPy, proves it computes the same thing as one big GPU would, and quantifies what it costs.


1. Data parallelism: same model, different data

The question. The model fits on one GPU but training is too slow. What's the simplest way to use 8 GPUs?

Start from nothing. Give every GPU a full copy of the model and a different slice of each batch. Each computes gradients on its slice; the gradients are averaged across GPUs (an all-reduce, next lesson); every GPU applies the same averaged gradient, so the copies stay identical. Mathematically it's the same as training on the whole batch on one GPU, because the gradient of a mean loss is the mean of the per-slice gradients. PyTorch DDP does exactly this, overlapping the all-reduce with the backward pass.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Parallelism Strategies — Data, Tensor, Pipeline, Sequence and Expert"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee