VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D13: GPU & CUDA Programming · Parallel Computation — Reductions, Scans, Atomics and Tiled Matmul

Parallel Computation — Reductions, Scans, Atomics and Tiled Matmul

20 min read·Staff+ Tier· Unrolled Pro
Parallel Computation — Reductions, Scans, Atomics and Tiled Matmul

You already know: warps, blocks, barriers and shared memory (CUDA programming model); coalescing, reuse and bank conflicts (GPU memory hierarchy); Big-O and why work and depth both matter (D1 Data structures); matrix multiplication (D2 Linear algebra).

Why this matters for Atlas. A handful of parallel patterns cover most GPU kernels in deep learning: map (elementwise ops), reduce (sums, maxima: norms, softmax, loss), scan (prefix sums: top-p sampling, stream compaction, MoE token packing), histogram/scatter with atomics (counting tokens per expert), and tiled matrix multiplication. Each has a sequential algorithm that is wrong for a GPU and a parallel algorithm that is right. This lesson builds the parallel versions, measures their work (total operations) and depth (parallel steps), and shows the GPU-specific tricks (warp shuffles, privatization, tiling) that make them fast.


1. Work and depth

The question. Summing a million numbers takes a million steps sequentially. How many steps can it take in parallel?

Start from nothing. Two measures describe a parallel algorithm:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Parallel Computation — Reductions, Scans, Atomics and Tiled Matmul"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee