Decision trees
A tree asks one question at a time, narrowing the data
Each split makes the groups below it purer. Read the path, and you have the reason.
A decision tree learns a flowchart. At the top are all sixty loans, mixed. It looks for the single question that best separates repaid from defaulted: is income under forty five thousand? If yes, most defaulted. If no, the group is still mixed, so it asks again: is the debt ratio under zero point four? Now each group is nearly pure. On the right are the same two questions, drawn as boxes on the data.
Decision trees · Going deeper
How a tree chooses a split: Gini impurity
Pick the split with the lowest weighted impurity.
Random forests
Many different trees vote, and their errors cancel
Bagging: each tree sees different rows and features. The majority wins.
One tree is easy to read but easily fooled. So train many. Each tree sees a different random sample of the loans, and a random handful of features at each split, so each makes different mistakes. Ask them all about a new application. Some say repay, some say default. The majority wins. Because the trees’ errors are different, they mostly cancel. That is a random forest.
Why forests work · Going deeper
Voting cancels independent errors; shared errors set the floor
Variance of a forest falls to ρσ², so the trees are made to disagree.
Gradient boosting
Each new tree fixes the errors the earlier ones left
Boosting adds trees in sequence, each fitted to the remaining residuals.
Boosting builds its trees in a line rather than side by side. The first small tree makes a rough fit. The second is trained only on what the first got wrong, and so on, each one adding a little correction. After eight rounds the fit follows the curve closely. The bars show the error left after each round, shrinking every time. XGBoost and its cousins do this, and on tables of data they are very hard to beat.
Boosting settings · Going deeper
Small learning rate, many rounds, and early stopping to choose
Lower eta with more rounds, and stop on the validation error.
k-means clustering
k centres move until each sits in the middle of its group
Assign each point to the nearest centre, move centres to the mean, repeat.
k-means is the classic way to find groups. You choose how many, here three, and drop three centres anywhere, even badly. Each point joins its nearest centre. Each centre then moves to the average of its points. Repeat. The dashed lines show each centre’s journey, from a poor start to the middle of a real group, in a few steps.
Choosing k · Going deeper
The elbow shows where extra clusters stop paying
Inertia always falls with k; the silhouette score peaks at a good k.
Choosing an algorithm
Start from the task; for tables, try boosted trees first
Match the task to a family, then compare against a simple baseline.
Choose an algorithm from the task. A number or a class from a table of data: start with gradient boosted trees and a linear model, and compare. No labels: clustering or anomaly detection. Too many columns: PCA. Recommendations, forecasts, images and text each have their own families. The right column lists the built-in algorithm Amazon SageMaker AI offers for each, which the machine learning engineer exam expects you to know.
Lesson complete
You finished “Trees, forests and clustering”
Nicely done.