VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D20: Research Engineering · Interpretability — Probes, Activation Patching, Superposition and Sparse Autoencoders

Interpretability — Probes, Activation Patching, Superposition and Sparse Autoencoders

15 min read·Advanced· Unrolled Pro
Interpretability — Probes, Activation Patching, Superposition and Sparse Autoencoders

You already know: transformers, the residual stream and attention heads (D5); linear algebra, projections and PCA (D2); training small networks from scratch (D4); experimental controls and ablations (Research workflow).

Why this matters for Atlas. Evaluations say whether a model fails; interpretability asks why, by studying its internal activations. Three tools cover most practical work. Probes test whether information is linearly readable from a layer. Activation patching tests whether a component causally matters for a behaviour by transplanting its activation between runs. Sparse autoencoders (SAEs) unpack dense activations into many interpretable features, working around superposition (models packing more features than they have dimensions). This lesson builds each in NumPy on small networks where the ground truth is known, including the controls that keep conclusions honest.


1. Probes, and the control that makes them meaningful

The question. Does a layer "know" some property of the input?

Start from nothing. A probe is a simple classifier (usually linear) trained to predict a property from a layer's activations. High probe accuracy shows the information is linearly decodable there. It does not show the model uses it: a wide enough layer re-encodes much of its input whether trained or not. So every probe needs a control: the same probe on an untrained (random-weight) network, or on a control task with random labels (Hewitt & Liang). The interesting signal is what the trained model represents beyond the control.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Interpretability — Probes, Activation Patching, Superposition and Sparse Autoencoders"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee