The task
Eight houses have sold. What will the ninth sell for?
A model learns from training data: each example has a feature (size) and a label (price). The task is to predict the label of a new example.
Every model in this course is an answer to a task, so start with the task. Eight houses have sold. For each you know the size and the price. Put them on a chart and a pattern shows: bigger houses cost more, and the dots nearly form a line. Now a ninth house comes up for sale, ninety-five square metres. What will it sell for? You can read an answer off the chart, because your eye drew a line through the dots. Making a machine draw that line, and draw it well, is what the rest of this lesson is about.
Model parameters
A model is a function with knobs; training sets them
The learned numbers are parameters, or weights. Training finds their values.
Strip away the mystery and a model is a function with adjustable numbers. This one predicts a house price from its size: price equals w times size plus b. Two numbers, w and b. Training looked at eight houses and found the values that fit them best: about three point six thousand dollars for every square metre, plus a starting value. A large language model works on exactly this idea, only with billions of numbers instead of two.
Loss
Loss turns "how wrong" into one number to push down
Mean squared error for numbers, cross-entropy for classes. Lower is better.
To improve a model, you need to measure how wrong it is. On the left, a guessed line. Each red line is one error, the gap between the prediction and the real price. Square each error and take the average: that is the mean squared error, here about eleven hundred. On the right, the trained line. Its errors are small, and so is its loss. Training is nothing more than the search for the parameters with the lowest loss.
Choosing a loss · Going deeper
Squared error chases an outlier; absolute error holds the line
MSE punishes large misses; MAE resists outliers; Huber loss does both.
Gradient descent
Each step measures the slope and moves the knobs downhill
New weight = old weight − learning rate × gradient. Repeat.
How does training find the lowest loss? Here is the loss for every value of w. It is a valley. We start at two point five, high on the wall. The gradient is the slope at that point. We step against it, downhill. Again. And again. Each step is smaller, because the slope flattens near the bottom. After six steps we are at the floor of the valley. That is gradient descent, and every neural network, however large, is trained this way.
The gradient · Going deeper
One gradient step, worked from the eight houses
The gradient for a weight is the average of error × input, times two.
Learning rate
Too small crawls; too large overshoots and diverges
A loss that jumps or climbs means the learning rate is too high.
The size of each step is the learning rate, and it matters more than almost anything else. Too small, and after eight steps you have barely moved: training would take forever. About right, and you settle at the bottom in a few steps. Too large, and each step jumps past the valley floor to the other wall, higher than before, and the loss grows without limit.
Epochs and batches
An epoch is one pass over all the data, a batch at a time
Batch size examples make one step; all of them once make one epoch.
Training data is fed in batches. Here ten thousand examples are cut into batches of one hundred. Each batch gives one step: the gradient is averaged over the hundred examples and every weight is updated once. A hundred steps later, the model has seen every example once. That is one epoch. Twenty epochs is two thousand steps.
Batch size · Going deeper
Smaller batches buy more updates from the same pass
Batch, mini-batch and stochastic gradient descent differ only in examples per step.
Linear regression
A weighted sum of the features predicts a number
Each weight says how much one feature moves the answer.
The oldest model is still one of the most useful. Linear regression predicts a number as a sum: a starting value, plus each feature times its own weight. Each extra square metre adds three point four thousand dollars. Each bedroom adds twelve. Each kilometre from the centre takes off six. Each year of age takes off a little. Training finds these weights. And because you can read them, you can explain every prediction.
Lesson complete
You finished “A first model: linear regression”
Nicely done.