ML lifecycle

A model is never finished: the lifecycle is a loop

Frame the problemCollect dataPrepare dataEngineer featuresTrainEvaluateDeployMonitormonitoring sends you backto collect new data and retrainWhat goes wrong at each stageFrame the problem✗wrong target, no baselineCollect data✗too little, biased, unlabelledPrepare data✗missing values, leakageEngineer features✗scales and codes that misleadTrain✗overfits, never convergesEvaluate✗wrong metric, leaked test dataDeploy✗too slow, too costlyMonitor✗drift goes unnoticedA model is never finished: the machine learning lifecycle is a loop.The world the model learned from keeps changing, so a deployed model slowly gets worse. Monitoring notices, and the loop runsagain with fresh data. Each stage has its own typical failure, and most projects fail at the first two, not at training.MLOps, later in this lesson, is the practice of automating this loop.

Frame, collect, prepare, train, evaluate, deploy, monitor, and around again.

Problem framing

A business goal becomes one prediction you can measure

Business goalKeep more subscribersPredictionWill this customer cancel in the next 30 days?ML taskBinary classificationLabelcancelled within 30 days of the snapshot: yes or noFeaturestenure, logins last week, plan, support tickets, price changeMetricrecall among the 5% flagged; retention team can call 5%Baselinetoday’s rule: no login for 14 daysFraming turns a business goal into one prediction, with a label, a metric and a baseline.Without a measurable target nobody can say whether the model works. The metric must match the action the business will take,and the baseline is what the model has to beat to be worth building. A vague goal, like using AI on churn, is not a problem yet.

Name the prediction, the label, the metric and the baseline before any data work.

Inference options

Serve by who is waiting and how often requests come

Inference optionLatencyFitsCost shapeReal-time endpointmillisecondssteady traffic, a user is waitingpay while it runsServerless endpointms, plus cold startsspiky or rare trafficpay per requestAsynchronousseconds to an hourbig payloads (up to 1 GB), long jobsscales to zero when idleBatchminutes to hoursa whole dataset, no one waitingpay for the job onlyLimits are Amazon SageMaker AI’s. Azure ML (online, batch endpoints) and Vertex AI (online, batch prediction) offer the same shapes.How a model is served depends on who is waiting and how often requests come.A fraud check at checkout needs a real-time endpoint. Nightly scores for every customer are a batch job. A one-hour video analysis isasynchronous: the caller gets a ticket and is told when it is done. Serverless fits traffic that comes in bursts with idle gaps.

Real-time, serverless, asynchronous or batch.

Sizing an endpoint · Going deeper

Size an endpoint from its tail latency and its throughput

Latency of 2,000 requests to one model (generated)0100200300400millisecondsp50 80 msp95 192 msp99 278 msSizing for a peak of 150 requests a secondmean latency 92 msone instance, 4 workers 43.5 req/srun at 70% of that 30.5 req/sinstances ⌈150 ÷ 30.5⌉ 5scaling target 1,829 a minuteThe scaling target is per instance per minute, the unit ofSageMakerVariantInvocationsPerInstance in target tracking.Example workload. Keep at least 2 instances, in different zones.Promise a percentile, not the average: the slow tail is what users and time-outs meet.Half of these requests finish in 80 ms, but one in twenty takes over 192 ms and one in a hundred over 278 ms. A latency target iswritten as p95 or p99 for that reason. Capacity comes from throughput: workers ÷ mean latency, held below full load so a burst queuesbriefly, not for ever. Load-test for the real numbers, then let auto scaling hold invocations per instance at the target.

Set targets at p95 or p99; instances = peak load ÷ safe throughput per instance.

MLOps

MLOps turns training and release into a repeatable pipeline

Dataversioned, validatedTraintracked runsEvaluateagainst a baselineRegisterversion, approveDeploysafely, see nextMonitorquality, drift, costdrift or a schedule triggers retrainingModel registry: every version with its data, code, metrics and approval status; deploy only approved versions; roll back to the last.MLOps automates the path from data to a monitored model, so retraining is routine, not a project.A pipeline runs the steps as code (SageMaker Pipelines, Azure Machine Learning pipelines, Vertex AI Pipelines). Each run istracked, each model versioned in a registry, and a gate compares the new model with the current one before it is approved.For LLM apps the same discipline applies to prompts, retrieval settings and model versions: version, evaluate, then ship.

Data, train, evaluate, register, deploy, monitor, and around again.

Feature stores · Going deeper

A feature must be read as it was on the day of the prediction

One customer's logins a week, and a model that predicts cancelling within 30 days5455210000snapshot: 1 Marchcancels 20 Marchtraining run: 12 April✓logins_7d as of the snapshot = 5: what the model would have known✗logins_7d read today = 0: the customer has already left, so the label leaksone feature definition, computed onceOffline storeevery value, with its timepoint-in-time joinsbuilds training setsOnline storelatest value per keyread in millisecondsfeeds live predictionsSame code for both: no training-serving skew.Training rows must hold each feature as it was at prediction time; a feature store keeps the history to do it.Joining labels to the latest feature values is the commonest leak in production ML: the features already reflect the outcome. Apoint-in-time join takes, for each label, the newest value at or before its timestamp. The second fault is skew: a feature coded oncein SQL for training, again in the service. The feature stores of SageMaker AI, Vertex AI and Azure Machine Learning fix both.

A feature store gives point-in-time training data and the same features online.

Deployment strategies

Shift traffic to a new model in steps you can undo

All at once (blue/green)0%50%100%all at once; old fleet on standbyCanary10% first, then the restLinearequal steps on a timerShadowmirroredcopy of traffic, replies not servedLines show the share of live traffic answered by the new model over time. A CloudWatch or Azure Monitor alarm during thebake time rolls traffic back automatically (SageMaker deployment guardrails). A/B testing splits traffic to compare results.Shift traffic to a new model in steps you can watch and undo.Never swap a model for everyone at once without a way back. Blue/green keeps the old fleet until the new one proves itself;canary and linear expose a slice first; shadow tests a model on real traffic with no user seeing its answers. Each step waitsa bake time while alarms watch errors and latency, and rolls back on its own if one fires.

Blue/green, canary, linear, shadow: each with automatic rollback.

Deployment strategies · Going deeper · 1 of 2

Real-time, serverless, asynchronous or batch

SageMaker AI inferenceHowFitsReal-timepersistent instances, low latencyinteractive apps, steady trafficServerlessscales to zero; cold startsintermittent traffic, small modelsAsynchronousqueued; payloads up to 1 GB, up to 1 hourlarge inputs, long processingBatch transforma whole dataset, no endpointscoring files overnightChoose the endpoint type by latency need, traffic shape and payload size.Real-time endpoints are always on and auto-scale on a metric such as invocations per instance. Serverless pays per requestbut the first call after idle is slow. Asynchronous queues requests and can scale to zero. Batch transform runs offline overfiles in S3. Azure Machine Learning and Vertex AI have online and batch endpoints that map to the same choices.

Pick the endpoint by latency, traffic shape and payload.

A/B testing · Going deeper · 2 of 2

Before calling the new model better, check the gap is not chance

Share of users who buy after a recommendation, with 95% confidence intervalsModel A, current500 of 10,0005.0%± 0.43%Model B, new560 of 10,0005.6%± 0.45%4%5%6%The intervals overlap: the gap could be chance.lift 0.6% points (12% relative)z 1.89p-value 0.058 (not below 0.05)needed 21,900 users per arm for 80% power at this liftz = (pB − pA) ÷ √(p(1 − p) × 2 ÷ n), with p the pooled rateCanary and linear shifts test safety: errors, latency.An A/B test asks a different question: is it better?A 12% lift on 10,000 users each is still not proof: size the test before you start it.B converts 560 users against 500. A two-proportion z-test gives p = 0.058: equal models would show a gap this large about 1 time in 17.Detecting this lift reliably takes about 21,900 users per arm. Fix the metric, the sample size and the stopping rule in advance: checkingdaily and stopping on the first good day manufactures winners. On SageMaker the two models are production variants with weights.

An A/B test needs a metric, a sample size and a stopping rule fixed in advance.

Model drift

Accuracy decays as the world drifts from the training data

03691260%80%100%months after deploymentaccuracyretrain below 85%new competitor launchesbasket size, $training datalive traffic nowAccuracy decays as the world drifts away from the data the model learned from.Data drift: the inputs change, as when basket sizes grow. Concept drift: the link between inputs and answer changes, as when acompetitor makes loyal customers leave. Monitor input distributions against the training baseline, and model quality as true labelsarrive; set a threshold that triggers retraining. SageMaker Model Monitor, Azure ML data drift and Vertex AI Model Monitoring do this.

Data drift: the inputs change. Concept drift: what they mean changes.

Model drift · Going deeper

Watch live data drift away from the training data

0%10%20%30%40%share of customers, by age< 2525–3435–4445–5455 +training datalast 30 daysPSI = Σ (actual − expected) × ln(actual ÷ expected)< 25 (0.05 − 0.10) × ln(0.50) = 0.034725–34 (0.12 − 0.20) × ln(0.60) = 0.040935–44 (0.28 − 0.30) × ln(0.93) = 0.001445–54 (0.30 − 0.25) × ln(1.20) = 0.009155 + (0.25 − 0.15) × ln(1.67) = 0.0511PSI = 0.137: moderate shift: investigateunder 0.1 stable · 0.1 to 0.25 moderate · over 0.25 significantMonitoring compares live data with training data; a drift score tells you when to look.Data drift: the inputs change, as here, where customers got older. Concept drift: the link between inputs and the right answerchanges, so accuracy falls even when inputs look the same; only ground-truth labels reveal it. SageMaker Model Monitor checksdata quality, model quality, bias drift and feature-attribution drift against a baseline and raises alarms.

Data drift scored by PSI = 0.137; concept drift needs labels.

Lesson complete

You finished “From model to production”

Nicely done.

Open in a new tab