You already know: parallelism layouts and collectives (Parallelism strategies, Collective communication); sharded states (Memory optimization); stragglers (D14 Distributed compute); SLOs, redundancy and incident response (D14 Reliability engineering); object storage and erasure coding (D14 Distributed storage); data pipelines (D5 Pretraining).
Why this matters for Atlas. At thousands of GPUs, failure is not an exception: it is the steady state. Every component (GPUs, memory, NICs, cables, switches, hosts, storage) fails at some rate, and a synchronous training job stops when any one does. The infrastructure's job is to maximize goodput: the fraction of paid GPU-hours that produce useful training progress. That depends on how often failures happen, how fast they're detected, how much work each one destroys (checkpoint interval), how quickly the job restarts, and whether stragglers and data stalls silently slow every step. This lesson quantifies each of those and the engineering that controls them.
1. Failures at scale are certain
The question. If one GPU fails once every few years, how often does a 4,096-GPU job see a failure?
Start from nothing. Failures of independent components add up. If each GPU (with its share of host, NIC and network) has a mean time between failures (MTBF) of M hours, a job using N of them sees a failure every M / N hours on average. Published large runs bear this out: Meta's Llama 3 405B training on 16,384 GPUs recorded 419 unexpected interruptions over 54 days, about one every three hours, most caused by GPU and HBM faults.