VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D13: GPU & CUDA Programming
Phase 6: GPU & LLM Inference Engineering/DOMAIN 13

Domain 13: GPU Engineering & CUDA

Staff+ TierUnrolled Pro

GPU programming for AI: GPU architecture, the CUDA model, the memory hierarchy, parallel patterns, performance engineering, CUDA libraries and Triton.

Estimated study time: about 140 hours
Access: Unrolled Pro
Unrolled Pro Domain

Unlock all 8 lessons with Unrolled Pro

Full production code, quantitative capacity math, failure recovery playbooks, and interview coaching.

Get Pro from ₹167/mo (~$2/mo)14-day refund guarantee

Lessons

8 lessons
01

GPU Architecture — Thousands of Simple Cores and a Memory System to Feed Them

Pro 15m

Every model in this course runs on GPUs, and every performance number in Domain 12 came from two datasheet figures.

✓ Runnable code✓ Exercise with solutionStart Lesson
02

The CUDA Programming Model — Grids, Blocks, Threads and Streams

Pro 15m

CUDA is the programming model under PyTorch, cuBLAS, FlashAttention, vLLM's kernels and Triton's output.

✓ Runnable code✓ Exercise with solutionStart Lesson
03

The GPU Memory Hierarchy — Coalescing, Shared Memory and Bank Conflicts

Pro 15m

Most kernels in an LLM are memory-bound, so the memory system decides their speed.

✓ Runnable code✓ Exercise with solutionStart Lesson
04

Parallel Computation — Reductions, Scans, Atomics and Tiled Matmul

Pro 20m

A handful of parallel patterns cover most GPU kernels in deep learning: map (elementwise ops), reduce (sums, maxima: norms, softmax, loss), scan (prefix sums: top-p sampling, stream compaction, MoE token packing), histogram/scatter with atomics (counting tokens per expert), and tiled matrix multiplication.

✓ Runnable code✓ Exercise with solutionStart Lesson
05

GPU Performance Engineering — Occupancy, Profiling and the Roofline in Practice

Pro 15m

Performance work without measurement is guessing, and GPU guesses are usually wrong because there are three different ways to be slow: memory-bound (not enough bytes per second), compute-bound (not enough FLOPs per second), and latency/overhead-bound (the GPU waits: launch gaps, too few warps to hide latency, synchronization, CPU stalls).

✓ Runnable code✓ Exercise with solutionStart Lesson
06

CUDA Libraries & Tools — cuBLAS, cuDNN, CUTLASS, NCCL and When Not to Write a Kernel

Pro 15m

NVIDIA and the open-source community have spent decades tuning libraries for the operations deep learning uses most: matrix multiplication, convolution, attention, collectives, sorting and scanning.

✓ Runnable code✓ Exercise with solutionStart Lesson
07

Triton Programming — Fast Fused Kernels in Python

Pro 20m

Triton (OpenAI, now an open-source compiler that PyTorch 2 uses internally) lets you write GPU kernels in Python at the level of blocks of data instead of individual threads.

✓ Runnable code✓ Exercise with solutionStart Lesson
08

Bridge: From One GPU to Many Machines

Pro 10m

Atlas's kernels are near the roofline and one GPU is used well. But Atlas now runs on 60 GPU servers, a queue of nightly jobs, three databases and a vector store, in two regions.

✓ Runnable codeStart Lesson
Domain 13: GPU & CUDA Programming · Unrolled