ML lifecycle
A model is never finished: the lifecycle is a loop
Frame, collect, prepare, train, evaluate, deploy, monitor, and around again.
A machine learning project runs as a loop, not a line. It starts by framing the business problem. Then data is collected, cleaned and prepared, and turned into features. A model is trained and evaluated. If it is good enough it is deployed. And then it is monitored, because the world changes, the model gets worse, and the loop starts again with new data. Every stage has its own way to fail, listed on the right.
Problem framing
A business goal becomes one prediction you can measure
Name the prediction, the label, the metric and the baseline before any data work.
Before any data, frame the problem. The business wants to keep more subscribers. As a prediction: will this customer cancel in the next thirty days? That is binary classification. The label comes from history: did they cancel within thirty days? The features are what we know before that moment. The metric follows the action: the retention team can call five percent of customers, so measure how many leavers are in that five percent. And the baseline is today’s simple rule, which the model has to beat.
Inference options
Serve by who is waiting and how often requests come
Real-time, serverless, asynchronous or batch.
Once trained, a model has to be served. If a person is waiting, and traffic is steady, use a real-time endpoint that is always on. If requests come in bursts with long quiet gaps, a serverless endpoint costs nothing while idle, at the price of a slow first request. If each request is large or slow, like an hour of video, use asynchronous inference: the caller is told when it is done. And if you need scores for a whole dataset, with nobody waiting, run a batch job.
Sizing an endpoint · Going deeper
Size an endpoint from its tail latency and its throughput
Set targets at p95 or p99; instances = peak load ÷ safe throughput per instance.
MLOps
MLOps turns training and release into a repeatable pipeline
Data, train, evaluate, register, deploy, monitor, and around again.
MLOps applies software discipline to models. Data is versioned and checked. Training runs are tracked. Each model is evaluated against the one in production. The winner is registered, with its data, code and metrics, and approved. It is deployed safely. And monitored. When monitoring sees drift, or on a schedule, the pipeline runs again.
Feature stores · Going deeper
A feature must be read as it was on the day of the prediction
A feature store gives point-in-time training data and the same features online.
Deployment strategies
Shift traffic to a new model in steps you can undo
Blue/green, canary, linear, shadow: each with automatic rollback.
Four ways to release a model. All at once: traffic moves together, and the old fleet stays ready to switch back. Canary: ten percent first, the rest after a bake time. Linear: equal steps on a timer. Shadow: the new model gets a copy of real traffic, but its answers are only logged, never served. Alarms watch each step and roll back automatically.
Deployment strategies · Going deeper · 1 of 2
Real-time, serverless, asynchronous or batch
Pick the endpoint by latency, traffic shape and payload.
A/B testing · Going deeper · 2 of 2
Before calling the new model better, check the gap is not chance
An A/B test needs a metric, a sample size and a stopping rule fixed in advance.
Model drift
Accuracy decays as the world drifts from the training data
Data drift: the inputs change. Concept drift: what they mean changes.
A model is a snapshot of the past. Here its accuracy holds for five months, then a competitor launches and it falls quickly. On the right is why one kind of drift happens: the basket sizes the model sees today are larger than the ones it learned from. That is data drift. When the inputs look the same but the right answer has changed, that is concept drift. Either way, monitoring compares live data and results with the training baseline, and a threshold triggers retraining.
Model drift · Going deeper
Watch live data drift away from the training data
Data drift scored by PSI = 0.137; concept drift needs labels.
A model learns the world as it was in its training data. Monitoring checks whether today still looks like that. Compare the age of customers then and now, band by band. Fewer under twenty fives, and fewer in their late twenties, about the same in the middle, and more older customers. The population stability index adds these differences up: 0.14, a moderate shift, worth investigating before accuracy falls.
Lesson complete
You finished “From model to production”
Nicely done.