Data cleaning

Most of the work is cleaning: gaps, outliers, duplicates

CustomerAgeCountryIncomeProblem and fixc-10134UK52,000c-102—UK61,000missing: fill with the median, 38c-10329uk48,000inconsistent: normalise to UKc-10441FR75,000c-105230DE58,000outlier: impossible age: drop or capc-10638FR66,000c-10441FR75,000duplicate: remove the second copyMost of the work in machine learning is cleaning the data.Missing values are filled (imputed) with a typical value; the median, 38, resists the outlier that would drag the mean. Values thatcannot be true are removed or capped, codes are made consistent, and duplicates are dropped so no customer counts twice.A model trained on dirty data learns the dirt. Tools: SageMaker Data Wrangler, AWS Glue DataBrew, pandas, Spark.

Impute gaps, remove impossible values, make codes consistent, deduplicate.

Data cleaning · Going deeper

Columnar files read only the columns a job needs

Row format: CSV, JSON, Avroa query that needs 2 of 10 columns still reads every row in fullreads 60 of 60 cells: 100%Column format: Parquet, ORCeach column is stored together, so only the 2 needed are readreads 12 of 60 cells: 20%Also: columns of one type compress well; Parquet stores the schema; Avro suits streaming records; RecordIO is SageMaker’s training format.Columnar files read only the columns a job needs.Training and analytics read a few columns of many rows, so Parquet or ORC is usually the right store for them. Row formats suitwriting one whole record at a time: an event stream or an API. CSV and JSON are readable but slow and untyped at scale.

Parquet and ORC store by column. CSV, JSON and Avro store by row.

Feature engineering

Every input becomes a number on a comparable scale

One raw rowplan: "Pro"country: "FR"income: 75,000age: 41signup: 2024-03-18What the model sees: numbersFeatureValueHowplan_pro1one-hot: a 0/1 column per plancountry_FR / UK / DE1, 0, 0one-hot: countries have no orderincome_z1.51standardised: (x − mean) / sdage_band40–49 → 3binned, then ordinaltenure_days609derived from signup, at 2025-11-17A model reads only numbers, so every input is turned into one, on a comparable scale.Categories with no order become one-hot columns; ordered ones become ranks (label encoding). Numbers on very different scales arestandardised or normalised so that income in thousands does not drown age in tens; skewed values get a log transform.Derived features, like tenure from a date, often carry more signal than the raw column.

One-hot for categories, standardise numbers, bin and derive where it helps.

Feature engineering · Going deeper

Scaling: standardise, normalise or log

IncomeStandardised (z)Normalised (min-max)Log1052,000-0.870.154.7261,0000.060.484.7948,000-1.2804.6875,0001.5114.8866,0000.580.674.82mean 60,400, standard deviation 9,687, min 48,000, max 75,000Standardising centres a column on 0; normalising squeezes it into 0 to 1.Standardisation, z = (x − mean) / sd, suits most models and keeps outliers visible. Min-max normalisation suits bounded inputs such aspixel values, but one outlier squashes everything else. Compute the mean and sd on the training data only, then apply the samenumbers to validation, test and live traffic; otherwise the test set leaks into training.

Fit the scaler on training data only, then reuse it everywhere.

Data splits

Keep a test set the model never sees until the end

Training 70%the model learns from itValidation 15%choose settings, stop earlyTest 15%one final, honest scoreShuffle before splitting. For time series, split by time:train on the past, test on the future.010203040epochslosstraining lossvalidation lossstop at epoch 18Keep a test set the model never sees until the very end.The model learns from the training set. The validation set picks settings and says when to stop: when its loss turns up while thetraining loss still falls, the model has begun to memorise (early stopping). The test set is touched once, for the final score; use itto choose anything, and its score is no longer honest.

Train to learn, validate to choose and stop, test once for the final score.

Data splits · Going deeper · 1 of 2

k-fold cross-validation: every example validates once

Run 1validatetraintraintraintrainRun 2trainvalidatetraintraintrainRun 3traintrainvalidatetraintrainRun 4traintraintrainvalidatetrainRun 5traintraintraintrainvalidatescore0.810.840.790.830.82mean 0.818k-fold cross-validation uses every example for validation exactly once.The data is cut into k folds (here 5). Each run trains on four and validates on the fifth, and the k scores are averaged. It gives asteadier estimate than one split when data is scarce, at k times the training cost. The test set still stays outside, untouched.

Five folds, five runs, one averaged score.

Data leakage · Going deeper · 2 of 2

Leakage: pure noise scores well once the test rows have been seen

Pick the 10 best columns on all rows, then cross-validate91%the choice of columns has already seen every test rowPick the 10 best columns inside each training fold53%the test fold stays unseen until it is scoredData: 40 rows, 1,000 columns of random noise, coin-toss labels.Honest answer: 50%. Nearest-centroid classifier, 5 folds, mean of 20 datasets.Three leaks to look for1 Preparation fitted on all the datascaler, imputer, feature selection or oversamplingdone before the split2 A feature from the futurea column filled in after the outcome, such as"refund issued" in a fraud model3 The same thing on both sidesone customer's rows, or near-duplicates, in train andtest; a random split of a time seriesLeakage is information in training that will not exist at prediction time. It makes a score too good to be true.Pure noise "scores" 91% because the ten columns were chosen with the test rows in view: among 1,000 columns, some matchthe labels by chance, and cross-validation then confirms the accident. Fit every preparation step on the training fold only (apipeline object does this for you), split by time or by customer when rows are related, and distrust any score that looks too good.

Leakage lets test information into training. Fit every step on the training fold only.

Overfitting

A model that memorises its data fails on anything new

underfit: a straight linetoo simple: misses the bendgood fit: a gentle curvefollows the trend, not the noiseoverfit: passes every pointmemorised the noise; wild between pointsAn overfit model memorises its training data and fails on anything new.All three are fitted to the same eight points. The straight line is too simple (high bias). The degree-7 curve hits every point, so itstraining error is zero, yet it swings wildly between them (high variance). Overfitting shows as a high training score and a muchlower validation score; underfitting as both low.

Underfit: too simple. Overfit: memorised the noise. Aim between.

Overfitting · Going deeper · 1 of 2

Fixing it: regularisation, dropout, more data

RemedyFixesMore data, or augmented dataoverfittingRegularisation: L2 (weight decay) shrinks weights; L1 drives some to 0overfittingDropout: switch off random neurons while trainingoverfittingEarly stopping on the validation lossoverfittingFewer features, or a simpler modeloverfittingMore features, or a bigger modelunderfittingTrain longer; less regularisationunderfittingRegularisation trades a little training accuracy for much better accuracy on new data.L2 adds the sum of squared weights to the loss, so large weights must earn their place; L1 adds their absolute values and tends toswitch useless features off entirely. Fine-tuning a large model brings one more risk: catastrophic forgetting, where new trainingoverwrites what the model knew. A low learning rate and mixing in general data reduce it.

L1, L2, dropout and early stopping fight overfitting.

Learning curves · Going deeper · 2 of 2

A learning curve says whether more data would help

A straight lineboth errors high and together: underfitting16326412825600.20.40.60.8training exampleserror (RMSE)validation 0.46training 0.46A degree-9 polynomiala gap that closes as data arrives: overfitting16326412825600.20.40.60.8training exampleserror (RMSE)validation 0.22training 0.21A learning curve tells you whether more data will help, before you pay for it.Train on growing slices of the data and plot both errors. The line stalls at 0.46 however much data it gets: more rows are wasted, itneeds a richer model or better features. The polynomial starts wild (1.78 at 16 rows) and falls to 0.22, near the noise floor of 0.21(dashed): more data was the cure. A gap means variance; two high, close curves mean bias. Generated data, median of 40 runs.

A gap between the curves means variance: add data. Two high curves mean bias: change the model.

Hyperparameters

Settings chosen before training; tuning searches for the best

Grid: 9 runsonly 3 values of each settingRandom: 9 runs9 different values of eachBayesian: 9 runseach run aimed by the lastaxes: two hyperparameters, such as learning rate and tree depth; shaded: where the best model isHyperparameters are chosen before training; tuning is a search for the best ones.Parameters are learned. Hyperparameters are set by you: learning rate, batch size, number of trees, number of layers. Each try is afull training run, so tries are expensive. Random search beats a grid for the same budget; Bayesian optimisation uses earlier resultsto pick the next try, and usually wins. SageMaker automatic model tuning, Azure ML sweeps and Vertex AI Vizier all offer these.

Grid, random or Bayesian search. Each try is a full training run.

Regression metrics

MAE and RMSE measure the miss in the target’s units

Actual $kPredictedError|Error|310296-1414395410+1515468442-2626265271+66520575+5555350344-66MAE = 20.33kmean of |error|: the typical missRMSE = 26.44ksquare, average, root: punishes the one big miss (+55)R² = 0.91share of the variation in price the model explainsRegression is judged by the size of its errors, in the target’s own units.MAE says a typical prediction is off by this many thousand dollars. RMSE is always at least as large and grows when a few errors arebig, so it suits problems where a large miss is much worse than several small ones. R² near 1 means the model explains almost all ofthe variation; 0 means it does no better than always predicting the average.

MAE: the typical miss. RMSE: punishes big misses. R²: variation explained.

Baselines

A model is only good compared with a simple baseline

Churn model: share of next month’s leavers among the 5% flagged (recall at 5%)Always "will not cancel"majority class0%Today’s rule: no login for 14 daysthe current process31%Logistic regressionsimple model52%Gradient-boosted treesthe candidate58%A model is only good compared with a baseline.Always start with the simplest thing that could work: the majority class, the average, the rule in use today. Here the complex modelbeats the rule by 27 points, but beats a simple logistic regression by only 6. Whether that gap is worth the extra cost and thelost interpretability is a business decision. Illustrative figures.

Beat the majority class, the current rule and a simple model first.

Lesson complete

You finished “Data and generalisation”

Nicely done.

Open in a new tab