You already know: data, tensor, pipeline and expert parallelism and what each must exchange (Parallelism strategies); tree reductions and scans (D13 Parallel computation); NCCL's place in the library stack (D13 Libraries & tools); NVLink, PCIe and InfiniBand bandwidths (D13 GPU architecture); latency vs bandwidth (D1 Networking).
Why this matters for Atlas. Every distributed training step is computation punctuated by collectives: operations in which a group of GPUs exchanges data in a fixed pattern. Data parallelism needs all-reduce (or reduce-scatter plus all-gather), FSDP needs all-gather and reduce-scatter, tensor parallelism needs all-reduce per layer, MoE needs all-to-all. Their cost, and whether it's hidden behind computation, often decides whether a training run reaches 50% MFU or 20%. This lesson defines the collectives, builds the ring all-reduce and proves it's bandwidth-optimal, gives the α–β cost model used to reason about them, and covers the hardware they run on.
1. The collectives
The question. What exactly do "all-reduce" and "all-gather" do?
Start from nothing. N ranks (GPUs), each holding a buffer: