The task

Eight houses have sold. What will the ninth sell for?

Eight houses that soldsize, m²price, $k5021060260702708033090350100400110420120470507090110130200300400500size, m²price, $k95?a ninth, for sale95 m²: what price?Machine learning starts from a task like this one: known examples in, a prediction for a new case out.Nobody wrote a rule for house prices here. The eight sold houses are the training data: size is the feature, the knownprice is the label. By eye you would say about 370, because you drew a line through the dots in your head. A modeldoes the same thing with numbers. The next pages build that line: what it is, how wrong it is, and how training improves it.

A model learns from training data: each example has a feature (size) and a label (price). The task is to predict the label of a new example.

Model parameters

A model is a function with knobs; training sets them

507090110130200300400500size, m²price, $kThe modelprice = w × size + bTraining foundw = 3.61 ($k per m²)b = 32.1 ($k at size 0)so a 85 m² house: 3.61 × 85 + 32.1 = 339kA model is a function with adjustable numbers; training sets them so it fits the data.These numbers are the parameters, or weights. This model has two. A large language model has the same idea with billions,and the same question drives every one: which values make the predictions closest to the known answers?

The learned numbers are parameters, or weights. Training finds their values.

Loss

Loss turns "how wrong" into one number to push down

5090130200300400500size, m²a guess: w = 2.2, b = 160loss (MSE) = 1,1885090130200300400500size, m²trained: w = 3.61, b = 32.1loss (MSE) = 80Loss turns "how wrong is the model" into one number to push down.Each vertical line is one error: the prediction minus the true price. Mean squared error squares each error and averages them,so big misses cost far more than small ones. Training is a search for the parameters with the lowest loss. Classifiers usecross-entropy (log loss) instead, which punishes confident wrong answers hardest.

Mean squared error for numbers, cross-entropy for classes. Lower is better.

Choosing a loss · Going deeper

Squared error chases an outlier; absolute error holds the line

507090110130200300400500size, m²price, $kone odd salesquaredabsoluteLoss (cost of an error e)Fitted lineAt 95 m²Squared (MSE): e²2.56 × size + 103.1346kAbsolute (MAE): |e|3.50 × size + 35.0368kHuber, δ = 303.39 × size + 46.6369kSquared, odd sale removed3.61 × size + 32.1375kThe odd sale is 197k off the trend. Squared, that is 38,795:more than the other eight errors put together (639).Squared error bends the whole line towards one outlier; absolute error barely moves.The loss you choose decides what "best fit" means. Squaring makes one large miss outweigh many small ones, so the line tilts toreduce it: right when big misses really are that costly, wrong when the point is a data error. Absolute error fits the median andshrugs off outliers. Huber is squared for small errors and linear beyond δ: smooth to train, hard to drag. Invented example.

MSE punishes large misses; MAE resists outliers; Huber loss does both.

Gradient descent

Each step measures the slope and moves the knobs downhill

2.533.544.5505,00010,000w (b held at its best)lossStepwloss02.59,58012.953,46623.211,28733.3751043.4723353.5213563.5699Gradient descent: measure the slope of the loss, step downhill, repeat.The gradient says which way each parameter should move to cut the loss, and how steeply. Each step moves w by the learning rate,here 0.000026, times the gradient. Big steps while the slope is steep, small ones near the bottom. With millions of parameters thepicture has millions of dimensions, but each step works exactly the same way.

New weight = old weight − learning rate × gradient. Repeat.

The gradient · Going deeper

One gradient step, worked from the eight houses

size xprice ypredictionerror ee × x50210157.1-52.9-2,64360260182.1-77.9-4,67170270207.1-62.9-4,40080330232.1-97.9-7,82990350257.1-92.9-8,357100400282.1-117.9-11,786110420307.1-112.9-12,414120470332.1-137.9-16,543sum-752.9-68,643prediction = 2.5 × x + 32.1 (b already at its best); error = prediction − priceSlope of the loss for w∂L/∂w = 2 × Σ e·x ÷ n = 2 × −68,643 ÷ 8 = −17,161One stepw ← 2.5 − 0.000026 × (−17,161) = 2.95Slope for b∂L/∂b = 2 × Σ e ÷ n = 2 × −752.9 ÷ 8 = −188.2No steps needed for a straight linew = cov(x, y) ÷ var(x) = 3.61, b = 32.1The gradient is two sums over the data: the errors, and the errors times the inputs.The slope for w is 91 times the slope for b, because every error is multiplied by a size of 50 to 120. That is why the learningrate on the main page had to be as small as 0.000026. Standardise the size first and the same slope is −50.7, so a rate near 0.1 works forboth knobs. A straight line also has an exact answer (the normal equation); gradient descent is for models that have none.

The gradient for a weight is the average of error × input, times two.

Learning rate

Too small crawls; too large overshoots and diverges

learning rate too small: 0.0000039crawls: barely moves in 8 stepslearning rate about right: 0.000026settles in a few stepslearning rate too large: 0.00013overshoots, and the loss growsThe learning rate sets the step size: too small crawls, too large overshoots and diverges.All three start at the same point and take eight steps on the same loss curve. Only the step size differs. A loss that jumpsaround or climbs during training is the classic sign of a learning rate set too high. Many optimisers (Adam is the common one)adapt the step for each parameter, and schedules shrink it as training goes on.

A loss that jumps or climbs means the learning rate is too high.

Epochs and batches

An epoch is one pass over all the data, a batch at a time

A training set of 10,000 examples, cut into batches of 100one batch = 100 examples = one step (one update of every weight)TermHereMeaningBatch size100examples per updateSteps per epoch10010,000 ÷ 100Epoch1 passevery example seen onceSteps in 20 epochs2,000100 × 20An epoch is one pass over all the training data, taken a batch at a time.The gradient is averaged over a batch, then every weight is updated once: one step. Small batches give noisy but frequent updates;large batches give smooth updates and use the GPU well but need more memory. Training runs for many epochs, until thevalidation loss stops improving.

Batch size examples make one step; all of them once make one epoch.

Batch size · Going deeper

Smaller batches buy more updates from the same pass

01234561001,00010,000100,000epochs (passes over the 8 houses)loss (MSE), log scalebest possible: 80Batch sizeStepsLoss after 6 epochsBatch: all 8 at once68,435Mini-batch: 2 at a time2483Stochastic: 1 at a time4884dashed: batch · purple: mini-batch · thin dots: stochasticThe batch size sets how many updates one pass over the data buys.All three runs see each house 6 times at a learning rate of 0.1 on standardised sizes. Full-batch descent takes one smooth step an epoch.Stochastic descent takes eight noisy ones and never quite settles. Mini-batches sit between, and are the default because a GPUprocesses a batch in one go: 32 to 512 examples is usual. A larger batch gives a steadier gradient and usually wants a larger rate.

Batch, mini-batch and stochastic gradient descent differ only in examples per step.

Linear regression

A weighted sum of the features predicts a number

price =45 + 3.4 × size + 12 × beds − 6.1 × km − 0.8 × agesize, per m²+3.4keach bedroom+12kkm from centre, each−6.1kage of house, per year−0.8kintercept45kpushes price down ← → pushes price upEach weight is learned from data.Read it as: one more unit of this,all else equal, changes price by that.Fast, explainable, a strong baseline.Assumes effects simply add up.Linear regression predicts a number as a weighted sum of the features.Each feature gets a weight, and the sign and size of the weight say how that feature moves the answer. That makes the model easyto explain, which regulators and business owners value. It cannot see curves or interactions unless you add them as features.Illustrative weights; note that weights are only comparable when the features are standardised.

Each weight says how much one feature moves the answer.

Lesson complete

You finished “A first model: linear regression”

Nicely done.

Open in a new tab