What this is
In this film, a neuron with just two weights learns to separate two groups of dots. On the right is a window of settings of its two weights, centered on the best one and colored by loss. Training is the ball rolling downhill.
Chapter 1 described training as guess, measure, adjust. This chapter is about the measure and the adjust: what the loss is, why it looks the way it does, and how the gradient tells each weight which way to move. Chapter 3 then shows how backpropagation computes that gradient for a big network.
Why not just count mistakes?
The obvious way to score a network is accuracy, the share of dots it gets right. So why does chapter 1 train on a formula with a logarithm in it? Drag across the charts below. They show what happens to accuracy and to the loss as one weight, w₁, changes while the other stays fixed.
Accuracy only changes when the line jumps past a dot. Between jumps it is perfectly flat, so nudging a weight usually tells you nothing at all. The loss changes smoothly everywhere: every nudge says whether things got better or worse, and by how much. That is why networks are trained on a loss and judged on accuracy.
What the loss measures
For one dot, the loss looks only at the probability the network gave to the right answer. If the dot is blue, the loss is −log ŷ, where ŷ is the network's estimate that the dot is blue. Drag the network's confidence and compare it with the simpler alternative, squared error.
Three things make cross-entropy a good penalty. It is close to 0 for a confident right answer. It is 0.69 for a 50-50 shrug. And it grows without limit for a confident wrong answer. Squared error never goes above 1, and together with σ its push almost vanishes exactly when the network is confidently wrong, so the network learns slowly from its worst mistakes. The network's total loss is the average over all its dots.
The same thing in formulas
L = −[y·log ŷ + (1−y)·log(1−ŷ)]cross-entropy for one dot, y = 1 for blueL = (ŷ − y)²squared error for one dot∂L/∂z = ŷ − ycross-entropy's slope on the neuron's sum z; training pushes z the opposite way, by −(ŷ − y)∂L/∂z = 2(ŷ − y)·ŷ(1 − ŷ)squared error's slope: the ŷ(1 − ŷ) factor shrinks to 0 when ŷ is near 0 or 1The loss landscape
Now give the neuron two weights, w₁ and w₂, so it computes ŷ = σ(w₁x₁ + w₂x₂). Every pair of values is one point on a flat map, and every point has a loss: the average over all 160 dots. Lift every point to a height equal to its loss, and the map becomes ground, with hills where the neuron does badly and a valley where it does well. The best weights sit at the bottom of the valley.
The loss curve you dragged across earlier is one slice through this ground, cut along w₁. And the flat, colored map is the same ground seen from directly above, the way a hiking map shows height with color and contour lines. The film opens from above and then tilts, so you can watch one turn into the other.
The point on the map and the neuron are the same thing seen two ways. The point (w₁, w₂) is also an arrow from the center of the dot plot, and the neuron's line always runs at right angles to that arrow. Swing the point around the center and the line turns. Move it straight outward and the line stays put, but the neuron grows more confident, so its edge gets sharper.
The gradient: which way is down
Standing at one point, the network cannot see the valley. It can only feel the slope under its feet, like walking downhill in fog. It measures the slope one weight at a time: nudge w₁ and see how the loss changes, then nudge w₂. Those two slopes together are the gradient. Drawn on the map, the gradient is an arrow pointing straight uphill, and its length says how steep the ground is. Training repeats three steps:
- Measure the gradient where you stand.
- Step the opposite way: w ← w − η × gradient. The learning rate η sets the size of the step.
- Repeat until the ground is flat.
Here is the whole procedure for this neuron, in words, in math, or as a short program you could run yourself:
This is called gradient descent. Chapter 1's network has 17 parameters (12 weights and 5 biases), so its map has 17 dimensions and nobody can draw it. The rules are the same: the gradient is just a list of 17 slopes. Measuring them by nudging costs one or two runs of the network per parameter. Chapter 3 shows how backpropagation gets all of them at once.
The playground
Click anywhere on the ground to drop the ball, then press Run. Drag to turn the ground and look at it from any side. Try different learning rates. Then switch to squared error or to accuracy, and see why they make poor landscapes.