What this is
This film follows backpropagation through a single neuron, like chapter 2's but with a bias, one operation at a time. The rest of the page explains each step, then hands you the controls.
Chapter 1 showed what training does: guess, measure, adjust. Chapter 2 showed what the adjusting needs: one slope per weight, together called the gradient, which points uphill on the loss landscape so training can step the other way. In chapter 1 the Backward stage produced every one of those slopes at once. This chapter shows how. If you skipped the earlier chapters, the film and the next section are enough to follow along.
The question
A gradient answers one question: if I nudge this parameter up a little, how much does the loss change? (A network's parameters are its weights and biases; chapter 1's network has 17.) There is an obvious way to find out. Nudge the parameter, run the network again, and measure. That works, and the playground below uses it to double-check every answer. But it costs one extra run of the network for every single parameter, or two if you nudge both ways as the playground does.
Backpropagation gets every gradient for about the cost of three runs, however many parameters there are: one forward pass, then a backward pass that costs about two more. That one trick, more than any other, is why large networks can be trained at all. It is built from three simple tools.
Tool 1: the slope of one operation
Chapter 2 measured slopes by nudging the whole network. Backpropagation instead uses the slope of each small operation, which is always easy to know. Start with one operation on its own, such as tanh. Its slope at a point says how much the output moves when you nudge the input. A slope of 0.5 means a nudge of 0.01 in the input moves the output by about 0.005. Drag across the curve to see the slope change.
Every operation in a network has a simple slope rule. These six cover everything on this page:
| Operation | Slope | In words |
|---|---|---|
| a + b | 1 and 1 | A nudge to either input passes straight through. |
| a × b | b and a | Each input's slope is the other input. |
| tanh(a) | 1 − tanh(a)² | Steep near 0, flat far away. |
| σ(a) | σ(a)·(1 − σ(a)) | Steepest at a = 0, where σ = 0.5; flat when σ is near 0 or 1. |
| −log(a) | −1 / a | Very steep when a is close to 0. |
| a² | 2a | Grows with a; zero at a = 0. |
These are called local slopes, because each one looks only at its own inputs and output, and nothing else in the network.
Tool 2: the chain rule
Operations come in chains. If y moves 3 times as much as x, and z moves 0.5 times as much as y, then z moves 3 × 0.5 = 1.5 times as much as x. Slopes along a chain multiply, the same way exchange rates do: if 1 yen buys 0.007 dollars and 1 dollar buys 0.9 euros, then 1 yen buys 0.007 × 0.9 euros.
That is the whole chain rule. Backpropagation applies it to every path through the network at once, and reuses shared work, so nothing is computed twice.
Tool 3: the graph, three rules, and a tape
Any calculation can be drawn as a graph: numbers flow along wires into small operations. The film's neuron is six operations. Chapter 1's network, for a single dot, is about thirty. Backpropagation walks that graph backward with three rules:
- Start with 1 at the loss. Nudge L by a little and L changes by exactly that much, so ∂L/∂L = 1.
- Multiply and pass back. Each operation takes the gradient it has received, multiplies it by its local slope for each input, and sends the result to that input.
- Add what arrives. If a value is used in several places, its gradient is the sum of everything that comes back to it.
To walk backward you need to know the order things happened in. So during the forward pass, every operation writes itself onto a list called the tape. The backward pass reads the tape in reverse. Here is the complete algorithm in six short parts. Choose how you want it explained: in words, in math, or as code. The code is the JavaScript this site runs, or the same program in Python; the last part shows the same calculation in PyTorch. Chapters 1 and 2 fused σ and the loss into one step; here they stay separate so you can watch each.
The playground
Pick a graph and press Step. Each click does exactly one thing: one operation forward, or one operation's backward rule. The box below the graph says what happened, in words, in numbers and in code. Tap any starting number to change it, and use Nudge test to check autograd's answer the slow way.