What this is
This film races two small networks on four rings of dots, then opens one up to show what its layers do.
Chapter 1's network had a single hidden layer: each neuron drew one straight edge, and the output added the edges up. This chapter asks what changes when you stack a second layer on top, and what that costs.
The question
Here are four rings of dots, alternating blue and orange. Two networks race on them. One has a single hidden layer of 32 neurons, 129 parameters in all. The other has two hidden layers of 8 neurons each: 105 parameters, fewer than the first. Same dots, same kind of starting weights, the same learning rate η = 1, and 3,000 steps of gradient descent each.
After 3,000 steps the single layer still has parts of the rings wrong; the two layers get every dot right in about 1,500 steps. That is not the luck of one draw. We ran both networks from 20 different random starts. Within 3,000 steps the single layer of 32 never reached 95% (its best was 91%), while the two layers of 8 reached 95% in 19 of the 20 runs. Given many more steps the single layer does get there: it reached 95% in all 20 runs, but needed a median of about 5,700 steps, nearly four times the two layers' 1,500. In principle, one hidden layer with enough neurons can approximate any boundary. In practice, stacking layers often makes shapes like these much faster to learn, with fewer parameters. Two ideas explain why, and a third shows how such a network is trained.
Tool 1: a layer gives every dot new coordinates
A layer takes each dot's numbers and hands the next layer new ones: one number per neuron. When a layer has exactly two neurons, its two numbers can be drawn as coordinates on a new plane. This network, with layers of 8, 8 and 2 neurons, was trained on chapter 1's Spiral. On the left are the dots where they start. On the right, the same dots at their coordinates in the last hidden layer, the two-neuron one. Drag the slider to watch them move from one to the other.
In the new coordinates the two arms no longer wind around each other, and the output neuron separates the colors with one straight line. That is what the layers before it did: they bent and folded the plane until a straight cut was enough. The faint grid shows the bending: straight lines on the left come out bent and squeezed on the right. tanh never goes past −1 or 1, so much of the grid is pressed against the edges.
Tool 2: lines, then curves, then shapes
Each layer builds on the one below it. Here is the two-layer network from the race after training. Every square shows one neuron's output across the whole plane, blue for positive and orange for negative.
The first layer can only draw straight edges, exactly like chapter 1's neurons. The second layer combines them: its neurons respond to regions with curved edges, such as bulges, bands and arcs. The output combines those regions into one shape that puts every ring on its correct side. Lines, then curves, then the shape.
Tool 3: the algorithm, layer by layer
Training a deep network uses the same chain rule as chapter 3, organized layer by layer instead of operation by operation. Here it is in five short parts. The code is the JavaScript this page actually runs, or the same program in Python.
The price of depth
Depth has a cost, and part 4 above shows where it comes from. On the way back, the error is multiplied by the slope of every bend it passes. With many layers those slopes multiply together. σ's slope is never more than 0.25, so on its way back through six σ layers the slopes alone multiply the error by at most 0.25⁶, about 1/4,000; the weights can shrink it further. This widget measures it on fresh networks with six hidden layers of six neurons, at the very start of training, averaged over 20 random starting weights.
This is called the vanishing-gradient problem, and σ is the worst case. tanh's slope reaches 1 near zero, and ReLU's slope is exactly 1 wherever a neuron is active, so with well-scaled starting weights both keep the first layer's gradients within a small factor of the output layer's at this depth, as the widget shows. That is why hidden layers today use ReLU or close relatives of it rather than σ, and why starting weights are scaled with care: in the widget, starting weights half as big make every layer's gradient shrink, with any of the three bends, and the first layer's falls further behind the output layer's.
The playground
Build a network: choose up to three hidden layers and their widths, pick a set of dots, and train. Every neuron's square is shown, layer by layer. Make the last hidden layer two neurons wide to see the plane it makes.