Backpropagation is the method that tells a neural network how much each of its weights contributed to an error, so that training can nudge every weight in the direction that lowers the error. It works by applying the chain rule from calculus one layer at a time, starting at the loss and moving backward. The 1986 Nature paper by Rumelhart, Hinton and Williams made the approach widely known, and the same idea still sits inside every modern training framework.
The one rule you need: the chain rule
If a quantity depends on another quantity, which in turn depends on a third, the rate of change of the first with respect to the third is the product of the rates of change along the way. Written out, if y = f(u) and u = g(x), then dy/dx = (dy/du) × (du/dx).
A network is a long chain of such dependencies: inputs become weighted sums, sums pass through activation functions, activations feed the next layer, and the last layer produces a number the loss function scores. To learn how one early weight affects the loss, you multiply the local rates of change along the path from that weight to the loss. Backpropagation is simply a bookkeeping scheme that computes each local rate of change once and reuses it for every weight that needs it.
A tiny network
Take the smallest network that still shows the mechanism: one input x, one weight w, one bias b, a sigmoid activation, and a squared-error loss.
- Input x = 2, target t = 1.
- Parameters w = 0.5, b = 0.
- Pre-activation z = w × x + b.
- Output a = σ(z), where σ(z) = 1 / (1 + e^(−z)).
- Loss L = ½ (a − t)².
The goal is to find dL/dw and dL/db, the numbers gradient descent needs.
Step 1: the forward pass
Compute values from input to loss and keep them, because the backward pass reuses them.
- z = 0.5 × 2 + 0 = 1.0
- a = σ(1.0) = 1 / (1 + e^(−1)) ≈ 0.7311
- L = ½ × (0.7311 − 1)² = ½ × 0.0723 ≈ 0.0361
Step 2: the backward pass
Now apply the chain rule from the loss back to the parameters. Each factor is a local derivative.
- How the loss changes with the output: dL/da = a − t = 0.7311 − 1 = −0.2689.
- How the output changes with the pre-activation: da/dz = σ(z) × (1 − σ(z)) = 0.7311 × 0.2689 ≈ 0.1966.
- How the pre-activation changes with the weight: dz/dw = x = 2. With the bias: dz/db = 1.
Multiply along the path:
- dL/dz = dL/da × da/dz = −0.2689 × 0.1966 ≈ −0.0529
- dL/dw = dL/dz × dz/dw = −0.0529 × 2 ≈ −0.1057
- dL/db = dL/dz × dz/db = −0.0529 × 1 ≈ −0.0529
The negative signs mean that increasing w or b would lower the loss, which matches intuition: the output 0.73 is below the target 1.
Step 3: the update
Gradient descent moves each parameter against its gradient by a step set by the learning rate. With a learning rate of 0.5:
- w ← 0.5 − 0.5 × (−0.1057) ≈ 0.5529
- b ← 0 − 0.5 × (−0.0529) ≈ 0.0264
Running the forward pass again gives z = 0.5529 × 2 + 0.0264 ≈ 1.1322, a ≈ 0.7562 and a loss near 0.0297, down from 0.0361. One step moved the output toward the target, which is the whole idea. For the optimization side, see our glossary entry on gradient descent.
Why it scales to billions of weights
Notice what happened to dL/dz: it was computed once, then multiplied by dz/dw and dz/db separately. In a real network, the same shared quantity feeds the gradients of every weight in a layer, and the error signal passed backward to the previous layer is built from it too. This reuse is why the backward pass costs roughly the same order of effort as the forward pass, instead of requiring a fresh calculation for each of millions or billions of parameters.
In matrix form, a layer computes Z = W·A + b. The backward step multiplies the incoming gradient by the transpose of W to pass the signal down, and by the layer’s input to get the weight gradient. Frameworks such as PyTorch automate this with automatic differentiation: they record the operations of the forward pass and replay them in reverse, so you rarely write derivatives by hand.
Common pitfalls
- Vanishing gradients. The sigmoid’s derivative is at most 0.25, so in a deep chain, multiplying many such factors shrinks the signal toward zero. This is one reason modern networks favor other activations and add residual connections.
- Exploding gradients. If the local factors are larger than one, the product grows and training becomes unstable. Gradient clipping is a standard remedy.
- Forgetting to cache the forward values. The backward pass needs a and z. Training memory is largely the cost of storing these, which is why techniques that recompute them trade time for memory.
- Wrong loss pairing. With softmax outputs and cross-entropy loss, the gradient simplifies neatly; our piece on softmax and cross-entropy shows why.
Try it yourself
Change x to 3 or the target to 0 and redo the three factors. The signs of the gradients should flip when the output overshoots the target. If you want to check your arithmetic, write the same network in PyTorch with requires_grad=True and call loss.backward(); the printed gradients should match your hand calculation to a few decimals. For the broader list of topics worth studying, see the math you need for AI.
What to watch next
- How frameworks reduce the memory cost of storing activations during the backward pass.
- Research on training methods that avoid or approximate backpropagation, such as forward-only learning rules.
- New activation functions and normalization layers designed to keep gradients well behaved in very deep models.
- Hardware features that speed the transposed matrix multiplications at the heart of the backward pass.
Key takeaways
- Backpropagation is the chain rule applied layer by layer from the loss back to each weight.
- The forward pass stores intermediate values; the backward pass reuses them.
- In our example, one update lowered the loss from about 0.0361 to about 0.0297.
- Reusing shared gradient terms is what makes the cost grow with the model’s size only roughly linearly.
- Vanishing and exploding gradients come from multiplying many local factors.
Frequently asked questions
Is backpropagation the same as gradient descent?
No. Backpropagation computes the gradients; gradient descent, or a variant such as Adam, uses them to update the weights. They are two steps of one training loop.
Do I need to calculate backpropagation by hand?
Not in practice, since automatic differentiation in PyTorch, JAX and similar tools does it for you. Working through a small case once makes errors such as shape mismatches and vanishing gradients much easier to diagnose.
Who invented backpropagation?
The 1986 Nature paper by Rumelhart, Hinton and Williams popularized it for neural networks, but the underlying technique of reverse-mode differentiation was developed earlier in other fields.
