Backpropagation

Run it first

The interactive playground below traces forward evaluation and reverse-mode gradient propagation through a 2-2-1 network. Step forward to compute preactivations and outputs; step backward to inspect how adjoint errors propagate and accumulate into weight sensitivities.

Enable JavaScript to use this local playground. The explanation and derivations below work without it.

Forward-mode vs. reverse-mode computational complexity

Consider differentiating a general mapping $f : ℝ^{n} \to ℝ^{m}$ whose Jacobian matrix has dimension $m \times n$ :

Forward-Mode Automatic Differentiation computes directional derivatives (Jacobian-Vector Products, JVPs). Evaluating the full Jacobian requires seeding the forward pass with n standard basis vectors: $O (n)$ forward sweeps.
Reverse-Mode Automatic Differentiation propagates adjoint sensitivities backwards (Vector-Jacobian Products, VJPs). Seeding the backward pass at the output requires m backward sweeps: $O (m)$ sweeps.

In deep learning, the objective function produces a single scalar loss: $m = 1$ . The parameter space contains $n = P$ weights (millions or billions). Forward-mode autodiff would require P forward passes per training step—an impossible computational cost. Reverse-mode backpropagation computes the exact analytical gradient for all P parameters simultaneously in a single reverse sweep ( $m = 1$ ) at the cost of approximately two forward passes.

The multivariate chain rule: summing over fan-out paths

When an intermediate variable x fans out and influences multiple downstream operations $y_{1}, y_{2}, \dots, y_{k}$ , the total derivative of the scalar loss L with respect to x is the sum of partial derivatives across all outgoing dependency paths:

\frac{\partial L}{\partial x} = \sum_{j = 1}^{k} \frac{\partial L}{\partial y_{j}} \frac{\partial y_{j}}{\partial x}

Forward pass:                         Backward pass (Adjoint accumulation):
     ┌──▶ [ y1 ] ──▶                     ┌──◀ [ ∂L/∂y1 ] ◀──
[ x ]│                               [ x ]│
     └──▶ [ y2 ] ──▶                     └──◀ [ ∂L/∂y2 ] ◀──
                                          ∂L/∂x = (∂L/∂y1 · ∂y1/∂x) + (∂L/∂y2 · ∂y2/∂x)

In implementation, this multivariate rule requires that backward passes initialize parameter gradient buffers with zero and accumulate with addition (+=) as incoming signals arrive. Overwriting gradients instead of adding them drops downstream dependency branches.

Analytical trace of the 2-2-1 network

The playground above executes a complete 2-2-1 network with input $x \in ℝ^{2}$ , target y, and parameters $W \in ℝ^{2 \times 2}, b \in ℝ^{2}, v \in ℝ^{2}, c \in ℝ$ :

1. Forward equations

z_{1} = W x + b, h = \tanh (z_{1})

z_{2} = v^{T} h + c, \hat{y} = σ (z_{2}) = \frac{1}{1 + e^{- z_{2}}}, L = {(\hat{y} - y)}^{2}

2. Backward equations

δ_{out} = \frac{\partial L}{\partial z_{2}} = \frac{\partial L}{\partial \hat{y}} σ^{'} (z_{2}) = 2 (\hat{y} - y) \cdot \hat{y} (1 - \hat{y})

\frac{\partial L}{\partial v} = δ_{out} h, \frac{\partial L}{\partial c} = δ_{out}

\frac{\partial L}{\partial h} = δ_{out} v, δ_{hidden} = \frac{\partial L}{\partial z_{1}} = \frac{\partial L}{\partial h} ⊙ (1 - h^{2})

\frac{\partial L}{\partial W} = δ_{hidden} x^{T}, \frac{\partial L}{\partial b} = δ_{hidden}

Each weight gradient is the outer product of the local output sensitivity $δ$ and the incoming input activation x. Computing these derivatives requires storing all forward activations in memory until the reverse pass reaches their corresponding layer.

Memory vs. compute: activation checkpointing

In standard backpropagation, peak memory consumption scales linearly with model depth: $O (L \cdot B \cdot d)$ for L layers, batch size B, and activation width d. For frontier LLMs, caching intermediate activations across 80+ layers with 32K context windows rapidly exhausts GPU VRAM before model weights even load.

Activation checkpointing (Griewank, 1992; Chen et al., 2016)

Activation checkpointing discards intermediate activations during the forward pass, retaining only the inputs to coarse block boundaries (e.g. every $\sqrt{L}$ layers):

Forward Pass:
[ x0 ] ──▶ ( Block 1 ) ──▶ [ x1 (saved) ] ──▶ ( Block 2 ) ──▶ [ x2 (saved) ] ──▶ Loss

Backward Pass (Block 2):
1. Recompute intermediate activations inside Block 2 from saved x1
2. Backpropagate gradients through Block 2 back to x1
3. Discard Block 2 activations and repeat for Block 1 from saved x0

By re-evaluating the forward pass of one block at a time during the backward pass, activation memory drops from $O (L)$ to $O (\sqrt{L})$ , at the cost of running approximately one extra forward pass (+33% arithmetic compute).