Part 3 · Deep Learning & AI

3.1 Neural networks

Stacking simple calculations

10-minute readPrerequisites: 2.4 Logistic regression, 0.2 Reading math notation (dot products)▶ Video (about 5 min)

What you will learn

Carry out the calculation inside one neuron (a weighted sum → an activation function)
Explain why stacking layers lets a network separate things that no single straight line can
Explain parameter counts, and the strengths and weaknesses of neural networks (hard to read inside, hungry for data)

Build intuition

Logistic regression in 2.4 multiplied the inputs by weights, added them up and squeezed the result into 0–1 with the sigmoid function. That is exactly the calculation of one neuron, the building block of a neural network. A neural network is nothing more than many of these simple parts placed side by side and stacked in layers.

So why stack them? Consider deciding, from a pedestrian’s lateral position x, whether they are within the width of our vehicle’s path (1.5 m either side of its center). The danger is in the middle; straying to either side is safe. A single neuron can draw only one boundary, so it can express either “right of −1.5” or “left of 1.5”, but not both.

So we put two neurons side by side. The first judges “is the pedestrian right of the left edge?”, the second “left of the right edge?”, and the final neuron answers “danger” only when both say yes. Now the network can express “dangerous only in the middle”.

Left: a small network with one input, two neurons in the hidden layer and one output (the numbers on the lines are weights; biases are omitted). Right: its output. Only within the width of the vehicle’s path (the shaded band, ±1.5 m from the center) is the probability of “danger” close to 1. A single neuron cannot produce this hump.

Real neural networks use millions of these parts. In a model that handles images, the first layers pick up simple features such as changes in color, and later layers represent more complex things — shapes, parts, whole objects (3.3 CNNs and image recognition). Networks with many stacked layers are called deep learning.

Check with math

① One neuron

$$z = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b = \mathbf{w} \cdot \mathbf{x} + b, \qquad a = f(z)$$

Multiply the inputs x by the weights w and add them up (the dot product from 0.2), then add the bias b to get z. Passing z through the activation function f gives a, the neuron’s output. With the sigmoid function as f, this is exactly the logistic regression of 2.4.

② Activation functions

Besides the sigmoid function (2.4), deep networks commonly use ReLU, because it is cheap to compute and lets learning reach deep layers more easily:

$$\mathrm{ReLU}(z) = \max(0,\ z)$$

It outputs 0 when z is negative and z itself when it is positive — nothing more. Activation functions are needed because stacking straight lines (linear expressions) only ever gives another straight line. For example, $y = w_2(w_1 x + b_1) + b_2$ simplifies to $(w_2 w_1)x + (w_2 b_1 + b_2)$, a single straight line. Inserting a function that “bends” in between is what gives stacking its point.

③ Stacking layers

A row of neurons is called a layer, and the layers between input and output are hidden layers. A layer’s calculation can be written compactly with a matrix W, a table of numbers arranged in rows and columns (see Notation):

$$\mathbf{h} = f(W_1 \mathbf{x} + \mathbf{b}_1), \qquad \hat{y} = g(W_2 \mathbf{h} + \mathbf{b}_2)$$

In the figure’s example, with σ the sigmoid function:

$$h_1 = \sigma(4x + 6), \quad h_2 = \sigma(-4x + 6), \quad \hat{y} = \sigma(10 h_1 + 10 h_2 - 15)$$

With the pedestrian in the middle (x = 0), $h_1 = h_2 = \sigma(6) \approx 1.00$, so $\hat{y} = \sigma(10 + 10 - 15) = \sigma(5) \approx 0.99$ (danger). At x = 3 m, $h_1 \approx 1$ and $h_2 = \sigma(-6) \approx 0$, so $\hat{y} \approx \sigma(-5) \approx 0.01$ (safe). Right at the edge, x = 1.5 m, $\hat{y} = 0.5$.

In this example the weights were chosen by hand to keep things clear. In practice, they are learned from data with the gradient descent of 2.3. The method for computing the slopes of all those weights at once is the subject of the next lesson, 3.2 Backpropagation.

④ Parameter counts

A layer with n inputs and m neurons, where every input connects to every neuron (a fully connected layer), has n × m weights and m biases. Feeding a 32 × 32-pixel color image (32 × 32 × 3 = 3,072 inputs) into a layer of 100 neurons already takes 3,072 × 100 + 100 = 307,300 parameters. Real image-recognition models have millions to billions of parameters.

In theory, a network with enough neurons in its hidden layer can approximate any function as closely as you like (the universal approximation theorem). But being able to represent something and being able to learn it well from limited data are separate questions (2.6).

Summary

A neuron is a simple part that passes “weighted sum + bias” through an activation function. Placing neurons side by side in layers and stacking many layers gives a neural network
Activation functions (sigmoid, ReLU) “bend” the calculation in between, so stacked layers can represent complex boundaries that no single straight line can
Parameters can number in the millions or billions: highly expressive, but hard to read inside, and demanding lots of data and careful checking

Quick quiz

Q1. A neuron has inputs (x₁, x₂) = (2, 1), weights (w₁, w₂) = (0.5, −1) and bias b = 0.5. With ReLU as the activation function, what is its output?

z = 0.5 × 2 + (−1) × 1 + 0.5 = 0.5, and ReLU(0.5) = 0.5.

Q2. When z = −2, what do ReLU and the sigmoid function output?

ReLU gives max(0, −2) = 0; the sigmoid function gives $1/(1 + e^{2}) \approx 0.12$.

Q3. A network stacks ten layers without any activation function. What is it equivalent to?

A single linear model (the weighted sum of 2.2). Stacking linear expressions only gives another linear expression, so the extra layers achieve nothing.

Q4. How many parameters does a fully connected layer with 100 inputs and 50 neurons have?

100 × 50 = 5,000 weights plus 50 biases: 5,050 in total.