You already know: SMs, warps and latency hiding (GPU architecture); pointers, memory allocation and RAII in C++ (D1 C++ & memory); asynchronous programming with an event loop (D1 Async concurrency).
Why this matters for Atlas. CUDA is the programming model under PyTorch, cuBLAS, FlashAttention, vLLM's kernels and Triton's output. Even if you mostly write Python, reading a kernel, understanding a profiler trace or debugging "illegal memory access" requires the model: a kernel runs as a grid of thread blocks, each thread computes its own index and handles its own data, memory moves explicitly between host and device, and work is queued asynchronously on streams. This lesson builds that model step by step, with CUDA C++ shown as it is written and Python simulations you can run anywhere to check the logic.
1. A kernel is a function run by many threads
The question. How does one function process ten million elements on a GPU?
Start from nothing. You write the code for one thread. At launch you say how many threads to run, organized as a grid of blocks, each block holding up to 1,024 threads. Each thread reads built-in variables (blockIdx, blockDim, threadIdx) to compute which element it owns, and checks that the index is in range (the grid is usually rounded up to whole blocks).