VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D4: Deep Learning · Neural Network Fundamentals — Layers That Learn Their Own Features
Phase 3/Domain 4/Lesson 1

Neural Network Fundamentals — Layers That Learn Their Own Features

15 min read·Intermediate·Free lesson
Neural Network Fundamentals — Layers That Learn Their Own Features

You already know: a layer is W x + b (D2 Linear algebra §3); backprop computes gradients through a graph (D2 Calculus §4); vanishing gradients and clipping (D2 Calculus §5); a 2-layer network solved XOR where a linear model couldn't (Bridge D3 → D4).

Why this matters for Atlas. Every LLM is a very deep neural network. Its building blocks — linear layers, activations, normalization, residual connections, careful initialization — are all on this page. Get these right and a 100-layer network trains smoothly; get one wrong and it silently fails to learn.

PyTorch examples run on CPU in a few seconds (pip install torch).


1. From a perceptron to a multilayer network

The question. What is a "neuron", and why stack them?

Start from nothing. A single neuron computes a weighted sum and passes it through a nonlinear function: a = φ(w·x + b). A layer is many neurons side by side — exactly a matrix multiply plus a nonlinearity: h = φ(W x + b). A multilayer perceptron (MLP) stacks layers: ŷ = W₃ φ(W₂ φ(W₁ x + b₁) + b₂) + b₃.

Why the nonlinearity? Without it, stacked layers collapse into one matrix (W₃W₂W₁ is just another matrix) — no more powerful than linear regression. With it, each layer can bend and fold the space, and with enough hidden units an MLP can approximate any reasonable function (the universal approximation theorem). Depth lets it build features from features: edges → shapes → objects, or characters → words → meaning.

input layer (3 circles) → hidden layer (5) → hidden layer (5) → output (2), fully connected; one neuron enlarged to show Σ wᵢxᵢ + b → φ.
Diagram— input layer (3 circles) → hidden layer (5) → hidden layer (5) → output (2), fully connected; one neuron enlarged to show Σ wᵢxᵢ + b → φ.

Real-life analogy. An assembly line of specialists: the first station sorts raw parts, the next assembles sub-components, the last builds the product. No single station does everything.

Worked example — the same MLP in NumPy (forward pass) and PyTorch (training):

python
import numpy as np
import torch
from torch import nn

torch.manual_seed(0)
rng = np.random.default_rng(0)

# Two interleaved spirals: hopeless for a linear model
n = 1000
t = rng.uniform(0, 3 * np.pi, n)
cls = rng.integers(0, 2, n)
X = np.column_stack([t * np.cos(t + np.pi * cls), t * np.sin(t + np.pi * cls)]) / 10
X += rng.normal(0, 0.02, X.shape)

model = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 2))
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(cls)
opt = torch.optim.AdamW(model.parameters(), lr=1e-2)
for step in range(1500):
    loss = nn.functional.cross_entropy(model(Xt), yt)
    opt.zero_grad(); loss.backward(); opt.step()
acc = (model(Xt).argmax(1) == yt).float().mean().item()
print(f"MLP accuracy on spirals: {acc:.3f}  (final loss {loss.item():.3f})")

# The forward pass is just matrix multiplies + ReLU — redo it in NumPy with the learned weights
W = [layer.weight.detach().numpy() for layer in model if isinstance(layer, nn.Linear)]
b = [layer.bias.detach().numpy() for layer in model if isinstance(layer, nn.Linear)]
h = np.maximum(0, X @ W[0].T + b[0]); h = np.maximum(0, h @ W[1].T + b[1]); logits = h @ W[2].T + b[2]
print("NumPy forward pass agrees:", np.allclose(logits, model(Xt).detach().numpy(), atol=1e-5))

Common trap. "More parameters = better." A huge MLP memorizes small datasets; the evaluation discipline from D3 still decides.


2. Activation functions

The question. Which nonlinearity should go between layers?

Start from nothing. The activation decides how signals and gradients flow:

Activation Formula Gradient behaviour Used in
Sigmoid 1 / (1 + e^{−x}) ≤ 0.25, vanishes for large |x| Output layer for probabilities, gates
Tanh (eˣ − e^{−x}) / (eˣ + e^{−x}) ≤ 1, vanishes for large |x| RNN/LSTM internals
ReLU max(0, x) 1 for x > 0, 0 otherwise ("dead" units possible) CNNs, many MLPs
GELU x · Φ(x) (Φ = normal CDF) smooth, slightly negative for small x < 0 BERT, GPT-2/3, many transformers
SiLU / Swish x · σ(x) smooth, like GELU LLaMA-family (inside SwiGLU)

ReLU fixed the vanishing-gradient problem of sigmoid/tanh in deep networks; GELU and SiLU are smooth versions that train slightly better in transformers.

the five activation curves on one plot from x = −4 to 4; below, their derivatives.
Chart— the five activation curves on one plot from x = −4 to 4; below, their derivatives.

Worked example — gradients through a stack of 20 layers:

python
import torch

torch.manual_seed(0)
for name, act in [("sigmoid", torch.sigmoid), ("tanh", torch.tanh),
                  ("relu", torch.relu), ("gelu", torch.nn.functional.gelu), ("silu", torch.nn.functional.silu)]:
    x = torch.randn(256, 128, requires_grad=True)
    h = x
    for _ in range(20):
        W = torch.randn(128, 128) * (2 / 128) ** 0.5          # He-style init (section 4)
        h = act(h @ W)
    h.sum().backward()
    print(f"{name:7s}: input-gradient size after 20 layers = {x.grad.abs().mean():.2e}")

Output: sigmoid ≈ 7e-11 (vanished), tanh ≈ 1.4, ReLU ≈ 1.8, GELU ≈ 0.7, SiLU ≈ 0.05. Sigmoid's gradient all but disappears through 20 layers; tanh, ReLU and GELU keep it alive at this initialization, and SiLU shrinks it somewhat. Whether gradients survive depends on the activation and the weight scale together — which is why initialization (section 4), normalization (section 5) and residual connections (section 6) are used together in real networks.


3. Loss functions and their numerics

The question. Which loss should a network minimize, and why do frameworks take raw scores (logits) instead of probabilities?

Start from nothing.

  • Regression: mean squared error (MSE), or Huber loss (quadratic for small errors, linear for large — robust to outliers).
  • Binary classification: binary cross-entropy. Use BCEWithLogitsLoss — it fuses sigmoid + log in a numerically stable way.
  • Multi-class (and next-token prediction): cross-entropy on logits (F.cross_entropy), which fuses softmax + log via the log-sum-exp trick. Computing softmax and then log separately can overflow (e^{1000}) or produce log(0) = −∞.
two pipelines — "softmax then log" with a red overflow warning at large logits, and "log-softmax (log-sum-exp)" with a green check.
Diagram— two pipelines — "softmax then log" with a red overflow warning at large logits, and "log-softmax (log-sum-exp)" with a green check.

Worked example.

python
import torch
import torch.nn.functional as F

logits = torch.tensor([[1000.0, 0.0, -1000.0]])
target = torch.tensor([1])

naive = -torch.log(torch.softmax(logits, dim=1)[0, target])      # softmax underflows to 0 -> log(0)
stable = F.cross_entropy(logits, target)                          # uses log-sum-exp internally
print("naive :", naive.item())                                    # inf
print("stable:", stable.item())                                   # 1000.0

# log-sum-exp by hand: log(sum(e^z)) = m + log(sum(e^(z - m)))
m = logits.max()
lse = m + torch.log(torch.exp(logits - m).sum())
print("by hand:", (lse - logits[0, 1]).item())                    # 1000.0

4. Initialization: starting at the right scale

The question. If every weight starts at 0, or at a large random value, what goes wrong?

Start from nothing.

  • All zeros: every neuron in a layer computes the same thing and gets the same gradient — they never become different (symmetry).
  • Too large / too small random values: activations grow or shrink by a constant factor per layer, so after 50 layers they explode or vanish.

The fix is to choose the random scale so the variance of activations stays roughly constant across layers. With fan_in inputs per neuron:

  • Xavier/Glorot: Var(w) = 2 / (fan_in + fan_out) — for tanh/sigmoid.
  • He/Kaiming: Var(w) = 2 / fan_in — for ReLU (which zeroes half its inputs, so it needs twice the variance).
activation standard deviation per layer (log scale) for 50 layers — "too small" falling to 0, "too large" exploding, "He init" flat near 1.
Chart— activation standard deviation per layer (log scale) for 50 layers — "too small" falling to 0, "too large" exploding, "He init" flat near 1.

Worked example.

python
import torch

torch.manual_seed(0)
x0 = torch.randn(512, 256)
for label, scale in [("std 0.01", lambda n: 0.01), ("std 0.1", lambda n: 0.1), ("He: sqrt(2/n)", lambda n: (2 / n) ** 0.5)]:
    h = x0
    stds = []
    for layer in range(30):
        W = torch.randn(256, 256) * scale(256)
        h = torch.relu(h @ W)
        stds.append(h.std().item())
    print(f"{label:14s} activation std at layers 1/10/30: {stds[0]:.2e} {stds[9]:.2e} {stds[29]:.2e}")

PyTorch's nn.Linear already uses a sensible default (Kaiming-uniform variant); transformers often use small normal init (std ≈ 0.02) plus scaled residual branches.


5. Normalization: BatchNorm, LayerNorm, RMSNorm

The question. Even with good initialization, activations drift during training. How do deep networks keep them in range?

Start from nothing. A normalization layer rescales activations to mean ≈ 0 and variance ≈ 1, then applies a learned scale (γ) and shift (β):

  • BatchNorm — normalizes each feature across the batch. Great for CNNs; awkward for small batches and variable-length sequences, and behaves differently in training vs inference.
  • LayerNorm — normalizes across the features of each token independently. The standard in transformers (works with any batch size).
  • RMSNorm — LayerNorm without the mean subtraction: x / √(mean(x²) + ε) · γ. Cheaper, works as well; used in LLaMA and most modern LLMs.
a (batch × features) grid; BatchNorm statistics computed down each column, LayerNorm/RMSNorm across each row.
Diagram— a (batch × features) grid; BatchNorm statistics computed down each column, LayerNorm/RMSNorm across each row.

Worked example — implementing LayerNorm and RMSNorm and checking them:

python
import torch

torch.manual_seed(0)
x = torch.randn(4, 8) * 5 + 3                    # 4 tokens, 8 features, off-centre and wide

def layer_norm(x, gamma, beta, eps=1e-5):
    mean = x.mean(-1, keepdim=True)
    var = x.var(-1, keepdim=True, unbiased=False)
    return (x - mean) / torch.sqrt(var + eps) * gamma + beta

def rms_norm(x, gamma, eps=1e-6):
    return x / torch.sqrt((x ** 2).mean(-1, keepdim=True) + eps) * gamma

gamma, beta = torch.ones(8), torch.zeros(8)
ours = layer_norm(x, gamma, beta)
ref = torch.nn.LayerNorm(8, elementwise_affine=False)(x)
print("matches nn.LayerNorm:", torch.allclose(ours, ref, atol=1e-5))
print("per-token mean ≈ 0:", ours.mean(-1).abs().max().item() < 1e-5, "| std ≈ 1:", ours.std(-1, unbiased=False).round(decimals=3).tolist())
print("RMSNorm per-token RMS:", torch.sqrt((rms_norm(x, gamma) ** 2).mean(-1)).round(decimals=3).tolist())

6. Regularization and residual connections

The question. How do we stop a big network from memorizing — and how do 100-layer networks train at all?

Start from nothing.

  • Dropout — during training, randomly zero a fraction p of activations (and scale the rest by 1/(1−p)), so the network can't rely on any single unit; at inference, dropout is off. Also: weight decay (AdamW), data augmentation, early stopping.
  • Residual connections — instead of h = f(x), compute h = x + f(x). The layer only has to learn a change to its input, and gradients get a direct "highway" backwards (the derivative of x + f(x) with respect to x includes a 1). This is what made very deep networks (ResNet, 2015) and every transformer trainable.

Transformers combine them as: x = x + Dropout(Attention(Norm(x))), then x = x + Dropout(MLP(Norm(x))) — the "pre-norm residual block".

a residual block — input x splits into a skip arrow straight to a + node and a branch Norm → f → Dropout that rejoins at the +.
Diagram— a residual block — input x splits into a skip arrow straight to a + node and a branch Norm → f → Dropout that rejoins at the +.

Worked example — gradient flow with and without residuals, and dropout's train/eval difference:

python
import torch
from torch import nn

torch.manual_seed(0)

def first_layer_grad(residual, depth=50, width=64):
    layers = [nn.Sequential(nn.Linear(width, width), nn.Tanh()) for _ in range(depth)]
    x = torch.randn(32, width)
    h = x
    for f in layers:
        h = h + f(h) if residual else f(h)
    h.pow(2).mean().backward()
    return layers[0][0].weight.grad.abs().mean().item()

print(f"50 plain layers    : first-layer gradient {first_layer_grad(False):.2e}")
print(f"50 residual layers : first-layer gradient {first_layer_grad(True):.2e}")

drop = nn.Dropout(p=0.5)
x = torch.ones(1, 10)
drop.train(); print("train mode:", drop(x))      # about half zeroed, survivors scaled to 2.0
drop.eval();  print("eval mode :", drop(x))      # unchanged

Common trap. Forgetting model.eval() at inference: dropout keeps dropping and BatchNorm uses batch statistics, so predictions become random and batch-dependent.

In production / interviews. Interview question: "Why do transformers use pre-norm residual blocks with LayerNorm/RMSNorm rather than BatchNorm?" (Batch-size independence, variable-length sequences, stable gradients through residual highways in very deep stacks.)

Quick check

A deep ReLU network initialized with weights from N(0, 1) produces activations that explode layer by layer. What's the fix?

  1. Remove biases
  2. Use a larger batch
  3. Use sigmoid
  4. Scale initial weights by sqrt(2 / fan_in) (He initialization) so activation variance stays roughly constant across layers
Show the answer

D. Scale initial weights by sqrt(2 / fan_in) (He initialization) so activation variance stays roughly constant across layers — Each layer multiplies variance by fan_in × Var(w)/2 for ReLU; He initialization makes that factor about 1.


Check yourself

  1. Recall: Why must there be a nonlinearity between linear layers?
  2. Apply: You're stacking 40 ReLU layers. Which initialization?
  3. Predict: What does F.cross_entropy return for logits [0, 0, 0, 0] and any target?
Answers
  1. Without it, the composition of linear layers is a single linear map — no extra expressive power.
  2. He/Kaiming (variance 2/fan_in), ideally with residual connections and normalization.
  3. ln 4 ≈ 1.386 — a uniform distribution over 4 classes gives each probability ¼.

Exercise

Build a 12-layer MLP classifier in PyTorch for the spiral data with a configurable block: Linear → (optional) LayerNorm → activation → (optional) Dropout, with an optional residual connection. Train four variants (plain, +residual, +residual+LayerNorm, +all with dropout 0.1) for the same number of steps and compare final training loss and validation accuracy.

Solution sketch
python
import numpy as np
import torch
from torch import nn

torch.manual_seed(0); rng = np.random.default_rng(0)
n = 2000
t = rng.uniform(0, 3 * np.pi, n); c = rng.integers(0, 2, n)
X = np.column_stack([t * np.cos(t + np.pi * c), t * np.sin(t + np.pi * c)]) / 10 + rng.normal(0, 0.03, (n, 2))
X, y = torch.tensor(X, dtype=torch.float32), torch.tensor(c)
tr, va = slice(0, 1600), slice(1600, None)

class Block(nn.Module):
    def __init__(self, w, norm, drop, residual):
        super().__init__()
        self.lin, self.norm = nn.Linear(w, w), nn.LayerNorm(w) if norm else nn.Identity()
        self.drop, self.residual = nn.Dropout(drop), residual
    def forward(self, x):
        out = self.drop(nn.functional.gelu(self.norm(self.lin(x))))
        return x + out if self.residual else out

def make(norm=False, drop=0.0, residual=False, depth=12, w=64):
    return nn.Sequential(nn.Linear(2, w), *[Block(w, norm, drop, residual) for _ in range(depth)], nn.Linear(w, 2))

for name, cfg in {"plain": {}, "+residual": dict(residual=True), "+res+norm": dict(residual=True, norm=True),
                  "+res+norm+dropout": dict(residual=True, norm=True, drop=0.1)}.items():
    torch.manual_seed(0); m = make(**cfg); opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
    for _ in range(600):
        m.train(); loss = nn.functional.cross_entropy(m(X[tr]), y[tr]); opt.zero_grad(); loss.backward(); opt.step()
    m.eval(); acc = (m(X[va]).argmax(1) == y[va]).float().mean().item()
    print(f"{name:18s} train loss {loss.item():.3f}  val acc {acc:.3f}")

At 12 layers with AdamW, all four variants reach ≈97% — adaptive optimizers and GELU are forgiving at this depth. Set depth=50 and the plain network stalls at loss 0.693 (= ln 2, a coin flip) and ~49% accuracy, while every residual variant still reaches ~97%: the benefit of these components grows with depth.


This unlocks:

  • Major architectures — CNNs, RNNs and transformers built from these blocks.
  • PyTorch engineering — what nn.Module, autograd and mixed precision do under the hood.
  • D5 Transformer architecture — pre-norm residual blocks with RMSNorm and SwiGLU.

Sources: Goodfellow et al., Deep Learning (ch. 6–8); Glorot & Bengio (2010) and He et al. (2015) on initialization; Ioffe & Szegedy, Batch Normalization (2015); Ba et al., Layer Normalization (2016); Zhang & Sennrich, RMSNorm (2019); Srivastava et al., Dropout (2014); He et al., Deep Residual Learning (2015); Hendrycks & Gimpel, GELU (2016).

Build Atlas · Part 3 of 9: Neural networks from scratchUse this lesson in the project: Build backpropagation yourself, prove it against finite differences, then write the training loop every later part reuses.