Data cleaning
Most of the work is cleaning: gaps, outliers, duplicates
Impute gaps, remove impossible values, make codes consistent, deduplicate.
Real data is messy. In seven rows here, one age is missing, so we fill it with the median of the others. One country is written in lower case, so it would count as a different country. One age is two hundred and thirty, which cannot be true. And one customer appears twice. Each problem would teach the model something false. Most of the time in a real project goes here.
Data cleaning · Going deeper
Columnar files read only the columns a job needs
Parquet and ORC store by column. CSV, JSON and Avro store by row.
Training data is usually stored as files, and the layout matters. A row format such as CSV, JSON or Avro writes each record whole. To read two columns, you still read everything. A columnar format such as Parquet or ORC stores each column together, so a job that needs two of ten columns reads a fifth of the data, and similar values compress far better.
Feature engineering
Every input becomes a number on a comparable scale
One-hot for categories, standardise numbers, bin and derive where it helps.
A model only reads numbers. A plan name becomes a one-hot column: one for Pro, zero otherwise. Countries have no order, so each gets its own column rather than a rank, which would tell the model that France is somehow more than Britain. Income is standardised, so its large numbers do not drown the small ones. Age is grouped into bands. And a signup date becomes tenure in days, which says more than the date itself.
Feature engineering · Going deeper
Scaling: standardise, normalise or log
Fit the scaler on training data only, then reuse it everywhere.
Data splits
Keep a test set the model never sees until the end
Train to learn, validate to choose and stop, test once for the final score.
Before training, split the data three ways. Most of it, seventy percent here, is for training. A validation set is used to compare settings and decide when to stop. A test set is kept locked away for one final, honest score. On the right is why the validation set matters. Training loss keeps falling. Validation loss falls too, then turns up: from here the model is memorising the training examples instead of learning the pattern. Stop at the bottom of the validation curve.
Data splits · Going deeper · 1 of 2
k-fold cross-validation: every example validates once
Five folds, five runs, one averaged score.
Data leakage · Going deeper · 2 of 2
Leakage: pure noise scores well once the test rows have been seen
Leakage lets test information into training. Fit every step on the training fold only.
Overfitting
A model that memorises its data fails on anything new
Underfit: too simple. Overfit: memorised the noise. Aim between.
Three models, the same eight points. A straight line misses the obvious bend: it underfits, too simple for the data. A gentle curve follows the trend and ignores the noise. A curve of degree seven passes exactly through every point. Its training error is zero. But look between the points: it swings wildly. On a new house it would be badly wrong. That is overfitting, and it is the most common way a model fails.
Overfitting · Going deeper · 1 of 2
Fixing it: regularisation, dropout, more data
L1, L2, dropout and early stopping fight overfitting.
Learning curves · Going deeper · 2 of 2
A learning curve says whether more data would help
A gap between the curves means variance: add data. Two high curves mean bias: change the model.
Hyperparameters
Settings chosen before training; tuning searches for the best
Grid, random or Bayesian search. Each try is a full training run.
Some settings are not learned: the learning rate, the batch size, how many trees or layers. These are hyperparameters, and finding good ones is a search where every try is a whole training run. A grid tries every combination of a few values, so it only ever tests three values of each. Random search, with the same nine runs, tests nine. Bayesian optimisation looks at the results so far and aims each new run where the best model is likely to be.
Regression metrics
MAE and RMSE measure the miss in the target’s units
MAE: the typical miss. RMSE: punishes big misses. R²: variation explained.
For predictions of a number, judge the size of the errors. Mean absolute error averages the misses: a typical prediction is off by about twenty thousand dollars. Root mean squared error squares them first, so the one big miss of fifty five counts heavily, and it comes out higher. R squared says what share of the variation in prices the model explains.
Baselines
A model is only good compared with a simple baseline
Beat the majority class, the current rule and a simple model first.
A score means nothing on its own. The majority class, always saying nobody cancels, scores zero here. The rule the business uses today scores thirty one percent. A simple logistic regression scores fifty two. The complex model scores fifty eight. It is better, but only six points better than the simple model. Whether that is worth the cost and the lost clarity is a real decision.
Lesson complete
You finished “Data and generalisation”
Nicely done.