You already know: every technique these frameworks package: data, tensor, pipeline, context and expert parallelism (Parallelism strategies), collectives and overlap (Collective communication), ZeRO/FSDP, checkpointing and mixed precision (Memory optimization), elastic restarts and distributed checkpoints (Training infrastructure), torch.compile and fused kernels (D13).
Why this matters for Atlas. You don't implement ring all-reduce or ZeRO-3 for a real run; you configure a framework that does. The frameworks differ in which parallelisms they support well, how much they change your model code, how they scale, and how fast they adopt new hardware. Choosing well saves months; choosing badly means rewriting the training loop halfway through a project. This lesson maps the landscape, shows the minimal code for the PyTorch-native path (DDP and FSDP2), explains where DeepSpeed, Megatron-LM and JAX fit, and ends with a configuration planner that encodes the decision process.
1. The landscape
The question. Which framework does what?