Part 3 · Deep Learning & AI

3.3 CNNs and image recognition

Picking up features from images

10-minute readPrerequisite: 3.1 Neural networks▶ Video (about 5 min)

What you will learn

Explain that an image is an array of numbers (pixels), and why the fully connected layers of 3.1 are inefficient for it
Carry out a convolution (sliding a small filter across the whole image) and explain the roles of feature maps and pooling
Explain how later layers move from simple features to whole objects, and the weaknesses of CNNs (small distant objects, degraded images, misleading cues)

Build intuition

To a computer, an image is an array of numbers. Each point — each pixel — holds a number for its brightness, and a color image has three such layers, for red, green and blue. A 1920 × 1080-pixel image from a vehicle camera contains about 6.2 million numbers.

Feeding that straight into a fully connected layer from 3.1 would take about 6.2 billion weights for a layer of just 1,000 neurons. Worse, a pedestrian on the left of the image and one on the right are the same kind of thing, yet the network would have to learn each separately.

So instead of looking at the whole image at once, we slide a small window — a filter, say 3 × 3 pixels — across the image a little at a time. Because the same filter (the same weights) is used at every position, very few weights are needed, and the same feature is found wherever it appears — like examining the image from end to end with a magnifying glass. This calculation is called convolution, and a neural network that uses it is a CNN (convolutional neural network).

Convolution with a filter that finds vertical edges. Only at positions straddling the boundary between the dark left and the bright right does the value become large (27). Sliding the filter one pixel at a time across the whole image produces the feature map on the right.

A CNN has dozens to hundreds of such filters in each layer and stacks many layers. People do not design what the filters look for; they are learned from data by the backpropagation of 3.2. As training proceeds, the early layers are known to come to represent simple features such as edges and patterns, and later layers parts such as wheels and legs — and finally whole objects such as cars and pedestrians.

The further along the layers, the larger and more complex the patterns they represent. This emerges naturally from training; nobody decides it.

Check with math

① The convolution calculation

With filter weights $w_{m,n}$ and image pixels $x$, the value at position (i, j) of the feature map is “multiply and add” over the region the filter covers — the same form as the weighted sum of a neuron in 3.1:

$$y_{i,j} = \sum_{m}\sum_{n} w_{m,n}\, x_{i+m,\, j+n} + b$$

In the red window of the figure, each of the three rows has pixels “left 0, middle 0, right 9” multiplied by the filter “−1, 0, 1”, giving $-1 \times 0 + 0 \times 0 + 1 \times 9 = 9$ per row and 27 over three rows. At the leftmost position, by contrast, everything in the window is 0, so the value is 0. Where the brightness is uniform, left and right cancel out to 0; only at an edge that is dark on the left and bright on the right does the value become large. The result then passes through ReLU (3.1) to indicate whether the feature is present.

② Weight sharing: remarkably few parameters

A single 3 × 3 filter has just 9 weights and 1 bias, however large the image. Even a layer applying 64 kinds of 3 × 3 filter to a color image (3 layers) has only 3 × 3 × 3 × 64 + 64 = 1,792 parameters — orders of magnitude fewer than the 6.2 billion of the fully connected layer.

③ Pooling: not minding small shifts

Convolution is often followed by pooling, which, for example, keeps only the largest value in each 2 × 2 block of pixels. This halves the height and width of the feature map, reducing computation and making the result less sensitive to an object shifting by a pixel or two. After stacking convolution and pooling to summarize the features, a final fully connected layer and the softmax function (2.4) produce probabilities such as “pedestrian 0.9, cyclist 0.07, …”.

④ Distant objects have few pixels

With a camera 1920 pixels wide and a 60° horizontal field of view, a pedestrian 1.7 m tall is about 113 pixels tall at 25 m, but only about 57 pixels at 50 m and about 28 pixels at 100 m. The fewer the pixels, the fewer features the filters can pick up, and the harder the pedestrian is to recognize.

Summary

An image is an array of pixel values. Fully connected layers would need far too many weights and would have to relearn things for every position
A CNN slides small filters across the whole image (convolution). Reusing the same weights keeps parameters few and finds the same feature wherever it appears; pooling adds tolerance to small shifts
Moving through the layers, what is represented goes from edges to patterns to parts to objects. Watch out for small distant objects, degraded images and misleading cues, and evaluate by condition

Quick quiz

Q1. Applying the same filter as in the figure (each row −1, 0, 1) to a region whose three rows are all “2, 2, 8”, what value do you get?

$-1 \times 2 + 0 \times 2 + 1 \times 8 = 6$ per row, so 18 over three rows.

Q2. What value does the same filter give on a uniform region where all nine pixels are 5? What does that mean?

−5 + 0 + 5 = 0 per row, so the value is 0. The filter does not respond where there is no edge in brightness.

Q3. How many parameters does a convolution layer have that applies 16 kinds of 3 × 3 filter to a color image (3 layers)?

3 × 3 × 3 × 16 = 432 weights plus 16 biases: 448 in total.

Q4. With the same detector, pedestrians 100 m away are missed more often than those 25 m away. Why? How should evaluation handle this?

A pedestrian 100 m away is only about 28 pixels tall (about 113 at 25 m), so the filters can pick up few features. Evaluate separately by distance, and check that pedestrians are found reliably at the distance needed to stop.