VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D15: Distributed Training · Collective Communication — All-Reduce, All-Gather and the Networks Underneath

Collective Communication — All-Reduce, All-Gather and the Networks Underneath

15 min read·Staff+ Tier· Unrolled Pro
Collective Communication — All-Reduce, All-Gather and the Networks Underneath

You already know: data, tensor, pipeline and expert parallelism and what each must exchange (Parallelism strategies); tree reductions and scans (D13 Parallel computation); NCCL's place in the library stack (D13 Libraries & tools); NVLink, PCIe and InfiniBand bandwidths (D13 GPU architecture); latency vs bandwidth (D1 Networking).

Why this matters for Atlas. Every distributed training step is computation punctuated by collectives: operations in which a group of GPUs exchanges data in a fixed pattern. Data parallelism needs all-reduce (or reduce-scatter plus all-gather), FSDP needs all-gather and reduce-scatter, tensor parallelism needs all-reduce per layer, MoE needs all-to-all. Their cost, and whether it's hidden behind computation, often decides whether a training run reaches 50% MFU or 20%. This lesson defines the collectives, builds the ring all-reduce and proves it's bandwidth-optimal, gives the α–β cost model used to reason about them, and covers the hardware they run on.


1. The collectives

The question. What exactly do "all-reduce" and "all-gather" do?

Start from nothing. N ranks (GPUs), each holding a buffer:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Collective Communication — All-Reduce, All-Gather and the Networks Underneath"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee