Artificial neuron

A neuron weighs its inputs, adds them, and bends the sum

x1 = 0.5× 0.9x2 = 0.8× -0.4x3 = 0.2× 0.6Σ + bb = 0.1z = 0.35activationReLU: max(0, z)output 0.35passed to the next layerz = 0.5×0.9 + 0.8×(−0.4) + 0.2×0.6 + 0.1 = 0.35A neuron multiplies each input by a weight, adds them up, and bends the result.The weights and the bias are the numbers training learns. A negative weight means that input argues against firing. The activationfunction then bends the sum. Without that bend a network of any depth would collapse into one weighted sum, a linear model.

Weights and a bias are learned; the activation bends the result.

Activation functions

Without a bend, any stack of layers is one straight line

-303-0.53ReLUhidden layers, by default: cheap, fast to train-60601Sigmoida probability for yes or no outputs-303-11Tanhcentred on 0; older sequence modelsThe bend is what lets stacked layers learn curves, edges and meaning.ReLU passes positive values and zeroes the rest: simple, and the default inside networks. Sigmoid squeezes into 0 to 1 for aprobability; softmax does the same across many classes so they add up to 1. Sigmoid and tanh flatten at the ends, where gradientsvanish, which is why deep networks moved to ReLU and its relatives (GELU in transformers).

ReLU inside the network, sigmoid or softmax at the output.

Softmax and cross-entropy · Going deeper

Softmax makes scores into probabilities; cross-entropy grades them

ClassLogit ze^zProbability pTarget yGradient p − ycat2.07.3890.6590+0.659dog1.02.7180.2421−0.758bird0.11.1050.0990+0.099sum11.2131.00010.000p = e^z ÷ sum of e^zThe picture is a dog: loss = −ln(0.242) = 1.417The gradient sent back to each logit is just p − y: push the dog score up, the others down.Which output layer?Yes or noone sigmoid, binary cross-entropyExactly one of manysoftmax, cross-entropySeveral can be true (tags)a sigmoid per classA numberno activation, squared errorSoftmax turns the last layer's scores into probabilities that add up to 1; cross-entropy scores them.Raw outputs (logits) can be any size or sign. Softmax exponentiates and normalises them, so the largest logit takes the largestshare: a lead of 1.0 over the next class is a factor of e, 2.72, in probability. Paired with cross-entropy, the gradient is the plain differencebetween prediction and target, which is why the pair is used everywhere, including for the next token of a language model.

Softmax for exactly one class of many; a sigmoid per class when several can be true.

Layers

Layers turn raw input into features, step by step

Input4 featuresHidden 1simple patternsHidden 2combinationsOutput3 classes4×6+6 + 6×6+6 + 6×3+3= 93 parametersGPT-3 (2020):175,000,000,000Layers turn raw input into useful features, one step of abstraction at a time.Every line is a weight and every node has a bias: this small network has 93 parameters. Deep learning means many hidden layers.In an image network the first layer finds edges, the next shapes, the last whole objects, and nobody programmed those features.

Every line is a weight. Deep learning means many hidden layers.

Forward pass

One forward pass turns an input into a prediction, in numbers

x1 = 1x2 = 0.5h1 = 0.599z = 0.4h2 = 0.646z = 0.6ŷ = 0.574z = 0.2960.40.3-0.20.8w = 0.7w = -0.5target 1, loss = ½(ŷ − 1)² = 0.091h1 = σ(0.4×1 − 0.2×0.5 + 0.1) = σ(0.4) = 0.599ŷ = σ(0.7×h1 − 0.5×h2 + 0.2) = 0.574A forward pass pushes the input through every layer to a prediction.Each node is one neuron from two pages ago: weighted sum, plus bias, through the sigmoid. The network predicts 0.57 for an examplewhose answer is 1, so the loss is 0.091. Inference is just this forward pass. Training adds the backward pass on the next page.

Weighted sum, bias, sigmoid, layer by layer. Inference is only this.

Backpropagation

The error flows backward; each weight gets its share of blame

x1 = 1x2 = 0.5h1 = 0.599δ = -0.018h2 = 0.646δ = 0.012ŷ = 0.574δ = -0.1040.4 → 0.4090.3 → 0.294-0.2 → -0.1960.8 → 0.7970.7 → 0.731-0.5 → -0.466after one step: ŷ = 0.597, loss 0.081output δ = (ŷ − t)·ŷ(1 − ŷ) = -0.104each weight: w ← w − 0.5 × δ × its inputBackpropagation sends the error backwards and gives every weight its share of the blame.The chain rule works out, layer by layer from the output, how much each weight contributed to the error (its gradient). Every weightthen takes one gradient descent step. After a single update the prediction moves from 0.574 to 0.597. Training repeats this millions of times.

Backpropagation computes every gradient; gradient descent updates every weight.

Vanishing gradients · Going deeper

Ten sigmoid layers pass back less than a millionth of the error

05101520110⁻³10⁻⁶10⁻⁹10⁻¹²layers the error has passed back throughgradient left, at best (log scale)ReLU on an active path: × 15 layers: 9.8 × 10⁻⁴10 layers: 9.5 × 10⁻⁷sigmoid: × 0.25 a layer at most20 layers: 9.1 × 10⁻¹³What keeps gradients alive in deep networksReLU and its relativesslope 1 wherever the neuron is activeResidual (skip) connectionsoutput = x + f(x): a path where the gradient is × 1Normalisation layersbatch norm, layer norm keep signals in a usable rangeCareful initialisationHe for ReLU, Xavier for tanh: variance kept per layerGradient clippingfor the opposite fault, gradients that explodeBackpropagation multiplies one slope per layer, and a sigmoid's slope is never more than 0.25.The sigmoid's slope is σ(z)(1 − σ(z)): 0.25 at z = 0 and far less at the flat ends. Ten such layers leave at most 9.5 × 10⁻⁷ of the error signal,so the early layers of a deep sigmoid network barely learn. That is the vanishing gradient. Deep learning became practical whenReLU, normalisation and skip connections gave the gradient a path that does not shrink. Weights are left out of this best case.

Vanishing gradients: fixed by ReLU, normalisation and skip connections.

Convolutional networks

A convolution slides one small filter across the whole image

Image: dark left, bright right000999000999000999000999000999000999Filter (learned)-101-101-101slideFeature map027270027270027270027270Highlighted window × filter:(0·−1 + 0·0 + 9·1) × 3 rows = 27Strong response only where dark meets bright:the filter is a vertical edge detector.Real networks learn dozens of filters per layer,then pool and repeat: edges → shapes → objects.A convolution slides one small filter over the whole image, so the same pattern is found anywhere.The filter’s nine weights are learned. Because one filter is reused at every position, a CNN needs far fewer weights than a fullyconnected network and finds a cat in any corner of the picture. Pooling then shrinks the map, keeping the strongest responses.CNNs power image classification, object detection and OCR; vision transformers now compete with them.

One learned filter, reused everywhere, finds a pattern wherever it is.

Convolutional networks · Going deeper · 1 of 2

The main architectures and what each is for

ArchitectureBuilt forWhere you meet itFully connectedtabular inputs, final layersthe last layers of most networksCNNgrids: images, spectrogramsvision, OCR, medical scansRNN, LSTMsequences, one step at a timeolder speech and text modelsTransformersequences, all positions at onceLLMs, translation, vision transformersDiffusionremoving noise step by stepimage and video generationGAN, VAEgenerating by competing or encodingolder image generation, synthetic dataEach architecture builds an assumption about the data into the network.CNNs assume nearby pixels belong together. RNNs read a sequence in order, carrying a memory, which makes them slow and forgetfulover long texts. Transformers look at every position at once with attention (lesson 8), train in parallel on GPUs, and replacedRNNs for language. Diffusion models generate images by learning to remove noise.

CNNs for grids, transformers for sequences, diffusion for generating images.

Stride and padding · Going deeper · 2 of 2

One formula gives every feature map its size

LayerInput WKernel KStride SPadding POutput = ⌊(W − K + 2P) ÷ S⌋ + 1The main page's filter6 × 63 × 310⌊(6 − 3 + 0) ÷ 1⌋ + 1 = 4Padding 1: size kept6 × 63 × 311⌊(6 − 3 + 2) ÷ 1⌋ + 1 = 6Stride 2: size halved6 × 63 × 321⌊(6 − 3 + 2) ÷ 2⌋ + 1 = 3First layer of a ResNet-style network224 × 2247 × 723⌊(224 − 7 + 6) ÷ 2⌋ + 1 = 112Pooling after it112 × 1123 × 321⌊(112 − 3 + 2) ÷ 2⌋ + 1 = 56Parameters, from a 224 × 224 colour image (3 channels) to 64 outputsConvolution, 64 filters of 7 × 7 × 3: (7 × 7 × 3 + 1) × 64 = 9,472Fully connected, 64 neurons: 224 × 224 × 3 × 64 + 64 = 9,633,856 (1,017 times more)Kernel, stride and padding fix the size of every feature map; the filter count fixes the parameters.Padding adds a border of zeros so the filter can sit on the edge pixels; "same" padding keeps the size. Stride is the step betweenpositions, so stride 2 halves each side and quarters the work. A filter spans every input channel and has one bias, and the countdoes not depend on the image size at all: that reuse is why vision models are convolutional. Pooling has no parameters.

Output = ⌊(W − K + 2P) ÷ S⌋ + 1. Stride shrinks the map; padding keeps it.

GPUs and accelerators

Training is matrix maths, which GPUs do in parallel

Cores that can work at the same timeServer CPU64 cores64NVIDIA H100 GPU16,896 CUDA coresOne layer of a large model: a 4,096 × 4,096 matrix multiplication = 2 × 4,096³ = 137 billion operationsAt the H100’s ~989 trillion BF16 operations a second: 0.14 ms (at full use, in theory)Also built for this: Google TPUs, AWS Trainium (training) and Inferentia (inference), with fast memory: 80 GB at 3.35 TB/s on the H100.Training is mostly matrix multiplication, which GPUs do thousands of operations at a time.A CPU has a few powerful cores built for varied, branching work. A GPU has thousands of simple ones that all run the samemultiply-and-add on different numbers, exactly what a neural network layer needs. Large models train on thousands of GPUs at once;memory size and bandwidth, not just speed, decide which model fits on which chip.

GPUs, TPUs, Trainium and Inferentia run thousands of multiplies at once.

Lesson complete

You finished “Neural networks”

Nicely done.

Open in a new tab