Listening: spectrograms and sound

The problem: getting a computer to hear

Someone says "seven" and you understand it at once, whoever says it. A computer gets only what the microphone measures: thousands of numbers every second (16,000 in the recordings here). Turning those numbers into "seven" is speech recognition, one part of audio recognition. Recognizing a few chosen words, the job here, is called keyword spotting.

Why recognize sound?

Because people talk, and many sounds matter. Phones and speakers listen for a wake word, such as "Hey Siri", with a small network that runs all the time and wakes the rest of the system only when it hears that word. Speech recognition writes captions for people who cannot hear, lets you control devices with your hands busy, and transcribes meetings. Sound recognition identifies birds from their songs and warns when a machine starts to sound wrong. The recordings in this chapter come from Speech Commands, a collection Google first released in 2017, made to train exactly this kind of small keyword spotter.

Why is it hard?

A spoken word is never the same twice. Voices are high or low, people speak fast or slowly, with different accents, close to the microphone or far away, in quiet rooms or noisy ones. These are real recordings of single spoken digits from Speech Commands, each by a different person. Each square is a spectrogram, a picture of the sound that the rest of this chapter explains: time runs from left to right, pitch from bottom to top, and the stronger the color, the louder. Click one of the three marked ▶ to hear it.

As with handwriting in chapter 5, the simplest idea is to compare: keep the average picture of each digit and call a new recording by the average it is closest to. For speech this reads only 31% of our 1,000 test recordings correctly, against 77% for handwritten digits. The same word starts at different moments and lasts different lengths, so the averages blur into smudges.

Why neural networks do so well

• A spectrogram turns sound into a picture. Once sound is a picture, the tools of chapter 5 apply: this chapter's network is chapter 5's code, unchanged except for the size of its last layer.
• Convolution looks everywhere. A sound pattern means the same thing whenever it happens, so the same stencil looks for it at every moment, and a word that starts a little later is still recognized. (The stencils slide along pitch too; Secret 3 shows why that helps less.)
• They learn from their mistakes. The network is trained to tell the ten words apart, by gradient descent, on 8,000 recordings by 1,296 different people.

What this is

This film follows one spoken "seven" from the microphone's numbers to the network's answer, then shows where it listens and one thing it cannot shrug off.

The question

In chapter 5, the convolutional network coped far better than a plain one when a digit was moved. A spectrogram can be moved in two directions: to the right, which means the word starts later, or up, which means every pitch in it is higher. Move this recording either way, and watch both networks.

Moved two columns later (63 ms), the convolutional network still recognizes 89% of the test recordings. Moved two bands up, it drops to 74%, and three bands up to 49%; two bands up makes most pitches 20 to 40% higher, and the lowest ones, where a voice's own pitch lies, far more. The plain network shows the same pattern from a lower start: 61% two columns later, 40% two bands up. Before explaining why, here is what a spectrogram is.

Tool 1: sound is a wave

A microphone measures how the air pressure rises and falls, thousands of times a second; a loudspeaker pushes the air back the same way. A pure tone is a smooth wave that repeats: the faster it repeats, the higher the pitch. Real sounds are many waves added together. A spectrogram asks, moment by moment, which pitches are in the sound and how strong each one is. Pick a sound, listen, and compare its wave with its spectrogram.

Tool 2: from a wave to the network's picture

The network's spectrogram is made in four steps, repeated for 32 moments of the second. Each moment takes a window of 512 numbers (32 ms) and fades its edges. The Fourier transform finds how strong each of 257 frequencies is in that window. Those are added up into 32 mel bands, narrow at low pitches and wide at high ones, because that is roughly how we hear. Finally each band's strength is put on a logarithmic scale, in decibels, which is also how we hear loudness. Drag the slider to choose the moment.

Tool 3: the same network as chapter 5

The network is chapter 5's convolutional network, with the same code: 8 stencils of 5 × 5, pooling, 16 stencils, pooling, and a dense layer that weighs everything into 10 probabilities. The only change is the size of the last layer, because a 32 × 32 picture leaves 400 numbers at the end instead of 256. It reads each spectrogram value minus 0.5, so quiet and loud parts push in opposite directions; that helps it learn a little faster early on.

Secret 1: what it listens for

As in chapter 5, nobody chose the stencils: training started them as tiny random numbers. Each row below is one moment of training; the first five rows are a fresh copy of the network training in your browser right now, and the last is the finished network.

On a spectrogram a stencil's left-to-right is time and its bottom-to-top is pitch, so its patterns mean something about sound. A stencil that is orange on the left and blue on the right fires where quiet turns into loud: the moment a sound starts. One that is blue above and orange below fires where the bands just above are loud and those just below quiet: the lower edge of a band of sound, such as the bottom of a voice. Another fires where loud turns into quiet: where a sound ends. As in chapter 5, layer 1's starting weights are 50 times smaller than usual, which keeps these patterns visible; with a normal start the network learns as well, but its stencils keep much of their starting noise.

Secret 2: where it listens

The backward pass can be carried down to the spectrogram itself, as in chapter 5: for every spot of the picture it says how much a little more sound there would raise or lower the network's confidence in its answer. That shows where it listens.

It listens mostly to the word and pays little attention to the silence around it, and within the word it attends to particular bands and moments: the vowel's strongest bands, the hiss of an "s" or the burst of a "t".

Secret 3: it only shrugs off what it was shown

Here is the answer to the question. The networks were measured on all 1,000 test recordings, moved later or earlier by up to 6 columns (188 ms) and up or down by up to 3 bands.

Moving a word in time barely matters; moving it in pitch matters a lot. Yet the network is convolutional in both directions: its stencils slide up and down the bands just as they slide along in time. What differs is what training showed it. Its 8,000 recordings start their words at many different moments, so it learned that when a word starts hardly matters. They rarely show a whole word moved two or three bands up, so it never learned to ignore that. The dashed orange line is the same network trained on copies of the recordings moved up and down by up to three bands: it keeps 87% at two bands up and 82% at three, and 89% unmoved. (A network should not ignore pitch entirely: which bands light up is part of what makes an "ee" different from an "oo".) A network learns to shrug off only the differences its training showed it. Speech systems handle different voices by training on many speakers, and on recordings with their pitch changed; you can try it in the playground.

Secret 4: noise

Real rooms are noisy. Add hiss to a recording, at a level measured in decibels below the word, and listen to what the networks still recognize.

On all 1,000 test recordings, with hiss 20 dB below the word the convolutional network still recognizes 86%, at 10 dB below 74%, at 5 dB below 64%, and with the hiss as loud as the word 47%. The plain network falls much faster: 44%, 29%, 21% and 16%. (These numbers were measured in advance on the original recordings, which this page does not carry.) As Secret 3 suggests, training on recordings with noise added, as real systems do, makes networks far more robust to it.

Pulling sounds apart

The spectrogram is also where sounds can be separated. In a mix, each spot of the spectrogram mostly belongs to one sound or the other. A mask says which spots to keep: multiply the mix's spectrogram by it, turn the result back into a wave, and you hear only what you kept. Here a spoken "seven" is mixed, equally loud, with a hum or with another person saying "three". Paint on the spectrogram to remove or keep parts, then listen.

The ideal mask is made with knowledge you never have in practice: the two sounds before they were mixed. That is the kind of mask a separation network learns to predict, usually a soft version that keeps part of each spot. It is trained on mixes made from known sounds: it looks at a mix's spectrogram, predicts a mask, and gets better by gradient descent on how far the result is from the known sound. Many networks that pull voices out of songs, or one speaker out of a crowded room, work in this space; others work on the wave itself.

The algorithm: making the picture

The network's code is chapter 5's, unchanged. What is new is the code that turns a recording into its 32 × 32 picture, here in four parts. It is the JavaScript this page actually runs, on the recordings and on your microphone, or the same program in Python.

The playground

Say a digit into your microphone and see what both networks hear. Or pick a recording. The sound never leaves your browser.