Logistic regression

A squashed weighted sum becomes a probability

024681000.51hours studiedP(pass)decision boundary: 4.5 hdots: 30 students, failed (0) or passed (1)P(pass) = 1 / (1 + e^−(1.26·h − 5.72))5 hours → 64% chance of passing8 hours → 99%above the 0.5 line: predict passLogistic regression squashes a weighted sum into a probability between 0 and 1.The same weighted sum as linear regression goes through the S-shaped sigmoid curve, so the output can be read as a probability.A threshold, 0.5 by default, turns it into a class. Despite the name, it is a classifier, and the usual first model for yes-or-noproblems: fast, calibrated and explainable.

The sigmoid turns a score into 0 to 1; a threshold makes it a class.

Log loss · Going deeper

A confident wrong answer costs far more than a hesitant one

00.250.50.751012345probability the model gave to the true classloss = −ln(p)12345#ActualModel: P(yes)P for the truthLoss1yes0.900.900.1052yes0.600.600.5113no0.200.800.2234yes0.100.102.3035no0.990.014.605mean (log loss)1.549Accuracy sees 3 right, 2 wrong. Cases 4 and 5 are 89% of the total loss.Log loss charges little for a hesitant miss and a great deal for a confident one.Cross-entropy, or log loss, is −ln of the probability the model gave to what actually happened. Sure and right costs almostnothing; 0.99 on the wrong class costs 4.6, and a flat 1.0 would cost infinity, so libraries clip. Logistic regression and neuralclassifiers are trained by pushing this number down, which is what makes their scores usable as probabilities. Invented batch.

Log loss is −ln of the probability given to the true class, averaged.

Confusion matrix

Every prediction lands in one of four cells

Predicted: fraudPredicted: fineActually fraud100 paymentsActually fine9,900 payments80true positivefraud caught20false negativefraud missed200false positivegood customer blocked9,700true negativecorrectly let through10,000 card payments100 of them fraud (1.0%)the model flags 280every metric on the next pagesis computed from these four cellsEvery prediction of a classifier lands in one of four cells.Right answers sit on the diagonal: fraud caught and good payments let through. The two kinds of mistake are off it, and they rarelycost the same: a missed fraud loses money, a blocked customer loses goodwill. Choosing a metric means choosing which mistakematters more.

True positive, false negative, false positive, true negative.

Multi-class metrics · Going deeper

Three classes: one matrix, three scores each, and a choice of average

PredictedcardpaymentloginActualcard100 tickets85105payment200 tickets1016030login50 tickets31235350 support tickets; the diagonal is rightClassPrecisionRecallF1Ticketscard86.7%85.0%85.9%100payment87.9%80.0%83.8%200login50.0%70.0%58.3%50Macro average74.9%78.3%76.0%Weighted average82.2%80.0%80.7%Micro average = accuracy80.0%350login precision: 35 right of 70 predicted login. 30 of them were payment tickets.With more than two classes, score each class one against the rest, then choose how to average.Each class has its own precision and recall, read down its column and along its row. Macro averaging counts every class equally,so the weak small class drags it to 76.0%. Weighted averaging counts every ticket equally (80.7%) and can hide that class. Microaveraging pools every decision and equals accuracy. Report macro F1 when the rare classes matter. Invented counts.

Macro averages classes equally; weighted by size; micro equals accuracy.

Accuracy

Accuracy hides the rare class: doing nothing scores higher

Our model97.8% accuracy(80 + 9,700) ÷ 10,000: catches 80 of 100 fraudsA model that always says "fine"99.0% accuracy9,900 ÷ 10,000: catches nonebars start at 90%, so the difference is visibleAccuracy hides a rare class: doing nothing scores higher than the real model.Accuracy is the share of all predictions that were right. With fraud at 1.0% of payments, a model that never flags anything is99.0% accurate and useless. Whenever one class is rare (fraud, disease, defects, churn), accuracy is the wrong metric.Use precision, recall and F1, and fix the imbalance in training by resampling, class weights or synthetic examples (SMOTE).

On rare classes, accuracy rewards the model that never says yes.

Precision and recall

Precision: were the alarms real? Recall: were the cases caught?

Precision: of the 280 alarms, how many were fraud?TP ÷ (TP + FP) = 80 ÷ 28028.6%Recall: of the 100 frauds, how many did we catch?TP ÷ (TP + FN) = 80 ÷ 10080.0%F1: one number for both2 × P × R ÷ (P + R)42.1%the harmonic mean: low if either is lowAlso: specificity = TN ÷ (TN + FP) = 98.0%Precision asks how many alarms were real; recall asks how many real cases were caught.High recall matters when a miss is expensive: cancer screening, fraud, safety faults. High precision matters when a false alarm isexpensive: blocking a payment, a spam filter that hides real mail, an arrest. Raising one usually lowers the other, so F1 is usedwhen both matter and the classes are imbalanced.

Precision = TP ÷ all flagged. Recall = TP ÷ all real cases. F1 balances them.

Decision threshold

Moving the threshold trades precision against recall

Model scores for 1,000 cases■ 800 negatives■ 200 positives01scorethreshold 0.50 · flagged →Drag to move the thresholdFlaggedNot flaggedPositiveNegative172caught28missed88false alarm712left alonePrecision 66%Recall 86%Moving the decision threshold trades precision against recall.A classifier outputs a score, and a threshold turns it into a yes or no. Lower the threshold and more cases are flagged: recall rises,precision falls. Raise it and the opposite happens. The model is the same; only the business choice changes. Pick the thresholdfrom the cost of each kind of mistake, or from capacity, such as how many cases a team can review.

Lower the threshold: more caught, more false alarms. Raise it: the reverse.

Decision threshold · Going deeper · 1 of 2

The ROC curve shows every threshold; AUC sums it up

00.5100.51false positive ratetrue positive rate (recall)random guessing, AUC 0.5AUC 0.9520.20.350.50.650.8Each point is one threshold: high thresholds bottom left,low thresholds top right.AUC = 0.952: pick a random positive and a random negative;the model scores the positive higher 95% of the time.1.0 is perfect; 0.5 is a coin toss.The ROC curve shows every threshold at once; AUC sums it up as one number.It plots recall against the false positive rate as the threshold sweeps from 1 to 0. AUC, the area under it, compares modelswithout choosing a threshold first. On very rare classes the precision-recall curve is more telling, because the false positive ratestays tiny even when most alarms are false.

AUC 1.0 is perfect, 0.5 is guessing. It compares models before a threshold is chosen.

Choosing the threshold · Going deeper · 2 of 2

Price each mistake, and one threshold is cheapest

00.250.50.751$0k$20k$40k$60k$80kdecision thresholdcost of the mistakes on 1,000 cases0.36: $7,9250.50: $13,400ThresholdMissedFalse alarmsCost0.50, the default2888$13,4000.36, cheapest7205$7,9250.65, capacity7422$30,150Example costs: a miss $400, a false alarm $25Capacity: a team that can review 150 of 1,000 casesneeds 0.65, the lowest threshold flagging at most 150.Price the two mistakes, and the threshold stops being a matter of taste.Total cost = misses × $400 + false alarms × $25, computed at every threshold for the 1,000 scored cases of the main page. A miss costs16 times a false alarm, so the cheapest threshold is far below 0.5: 0.36 cuts the bill by 41%. The curve is shallow on the low side andsteep on the high side: when misses are dear, erring low is the cheaper mistake. Choose it on validation data, never on the test set.

Pick the threshold that minimises misses × their cost + false alarms × theirs.

Lesson complete

You finished “Classification”

Nicely done.

Open in a new tab