Artificial neuron
A neuron weighs its inputs, adds them, and bends the sum
Weights and a bias are learned; the activation bends the result.
A neural network is built from one simple part. Each input is multiplied by a weight. Here the second weight is negative, so that input argues against. The products are added, with a bias. The sum goes through an activation function, here ReLU, which keeps positive values and turns negative ones to zero. The result is passed on to the next layer.
Activation functions
Without a bend, any stack of layers is one straight line
ReLU inside the network, sigmoid or softmax at the output.
Why bend at all? Because a weighted sum of weighted sums is still just a weighted sum. Without a bend, a hundred layers can only draw a straight line. ReLU is the usual choice inside a network: it is cheap, and it trains well. Sigmoid squeezes any number into zero to one, for a probability at the output. Tanh is similar, centred on zero.
Softmax and cross-entropy · Going deeper
Softmax makes scores into probabilities; cross-entropy grades them
Softmax for exactly one class of many; a sigmoid per class when several can be true.
Layers
Layers turn raw input into features, step by step
Every line is a weight. Deep learning means many hidden layers.
Neurons are arranged in layers. The input layer holds the features. The first hidden layer combines them into simple patterns. The next combines those patterns. The output layer gives the answer, here a score for each of three classes. Every neuron connects to every neuron in the next layer, and every connection is a weight: ninety three parameters in this small network. GPT-3 had one hundred and seventy five billion.
Forward pass
One forward pass turns an input into a prediction, in numbers
Weighted sum, bias, sigmoid, layer by layer. Inference is only this.
Here is a real calculation. Two inputs, two hidden neurons, one output. Each hidden neuron takes the inputs, multiplies them by its weights, adds its bias, and squashes the sum. The output neuron does the same with the hidden values. The network predicts about zero point five seven. The right answer is one, so the loss is about zero point zero nine. Running a trained model, inference, is just this pass.
Backpropagation
The error flows backward; each weight gets its share of blame
Backpropagation computes every gradient; gradient descent updates every weight.
Now training. Start at the output: the prediction was too low by zero point four. The chain rule turns that into a share of blame for each weight, working backwards one layer at a time. That is backpropagation. Each weight then takes one small step that would have reduced the error. After a single update, the same input gives a higher prediction and a lower loss. Repeat across millions of examples, and the network learns.
Vanishing gradients · Going deeper
Ten sigmoid layers pass back less than a millionth of the error
Vanishing gradients: fixed by ReLU, normalisation and skip connections.
Convolutional networks
A convolution slides one small filter across the whole image
One learned filter, reused everywhere, finds a pattern wherever it is.
Images are too big to connect every pixel to every neuron. A convolutional network slides a small filter across the image instead. This filter has nine weights. Over the dark half it gives zero. Where dark meets bright it fires strongly. It has become a vertical edge detector. A network learns dozens of such filters, and layers of them build edges into shapes, and shapes into objects.
Convolutional networks · Going deeper · 1 of 2
The main architectures and what each is for
CNNs for grids, transformers for sequences, diffusion for generating images.
Stride and padding · Going deeper · 2 of 2
One formula gives every feature map its size
Output = ⌊(W − K + 2P) ÷ S⌋ + 1. Stride shrinks the map; padding keeps it.
GPUs and accelerators
Training is matrix maths, which GPUs do in parallel
GPUs, TPUs, Trainium and Inferentia run thousands of multiplies at once.
Every layer is a big matrix multiplication: the same multiply and add, done billions of times on different numbers. A server CPU has a few dozen cores built for varied work. A modern GPU has close to seventeen thousand simple ones, built to do exactly the same operation at the same time. One layer of a large model, about a hundred and forty billion operations, takes a fraction of a millisecond. That is why AI runs on GPUs and on chips designed for it.
Lesson complete
You finished “Neural networks”
Nicely done.