MLOps Basics: Monitoring Models and Detecting Data Drift

Deploying a model is not the end of the work. The world changes: customers behave differently, prices rise, new products appear, sensors age. A model trained on last year's data slowly becomes wrong. MLOps (machine learning operations) is the set of practices that keeps models reliable in production: tracking experiments, reproducible training, automated testing and deployment, monitoring, and retraining.

The lifecycle#

1. Data: collect, clean and version the training data.
2. Train and track: record every experiment's code version, data, settings and scores.
3. Validate: test the model on held-out data and check it beats the current model.
4. Deploy: release it behind an API or batch job, ideally gradually.
5. Monitor: watch inputs, predictions and (when labels arrive) accuracy.
6. Retrain: when performance drops, retrain on fresh data and repeat.

Reproducibility#

Anyone (including you in six months) should be able to recreate a model exactly:

• the code in Git, with the exact commit recorded for each model;
• pinned library versions (requirements.txt or a lock file);
• a fixed random_state or seed;
• a record of exactly which data was used (a snapshot, a query with a date, or a data versioning tool).

Experiment tracking#

Trying many models and settings quickly gets confusing. Experiment trackers such as MLflow or Weights & Biases record parameters, metrics and model files for every run. The idea, in plain Python:

import json

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

X, y = load_breast_cancer(return_X_y=True)
runs = []
for depth in [2, 4, None]:
    params = {"model": "RandomForestClassifier", "max_depth": depth, "n_estimators": 100, "random_state": 0}
    model = RandomForestClassifier(max_depth=depth, n_estimators=100, random_state=0)
    runs.append({"params": params, "cv_accuracy": round(cross_val_score(model, X, y, cv=5).mean(), 4)})

with open("experiments.json", "w") as f:
    json.dump(runs, f, indent=2)
best = max(runs, key=lambda r: r["cv_accuracy"])
print("runs logged:", len(runs))
print("best:", best["params"]["max_depth"], best["cv_accuracy"])

runs logged: 3
best: None 0.9631

With MLflow the same loop would call mlflow.log_params(params) and mlflow.log_metric("cv_accuracy", score) inside with mlflow.start_run():, and offer a web interface for comparing runs.

Data drift#

Data drift means the inputs in production no longer look like the training data. A quick check compares each feature's distribution in training and in recent production data. The Kolmogorov–Smirnov (KS) test measures how different two samples' distributions are:

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

rng = np.random.default_rng(0)
train = pd.DataFrame({
    "age": rng.normal(35, 8, 5000),
    "monthly_spend": rng.normal(2000, 400, 5000),
    "visits": rng.poisson(6, 5000),
})
recent = pd.DataFrame({
    "age": rng.normal(35, 8, 1000),                 # unchanged
    "monthly_spend": rng.normal(2400, 450, 1000),   # prices went up
    "visits": rng.poisson(6, 1000),                 # unchanged
})

for column in train.columns:
    result = ks_2samp(train[column], recent[column])
    flag = "DRIFT" if result.pvalue < 0.01 else "ok"
    print(f"{column:>14}: KS statistic {result.statistic:.3f}, p-value {result.pvalue:.3g}  {flag}")

           age: KS statistic 0.051, p-value 0.0265  ok
 monthly_spend: KS statistic 0.367, p-value 1.21e-100  DRIFT
        visits: KS statistic 0.010, p-value 1  ok

Only the spending feature is flagged. Note that age got a p-value of 0.0265 even though its distribution did not change at all: with a common 0.05 cut-off it would have been a false alarm. Checking many features every day produces such false alarms, which is why this check uses a stricter threshold. The spending feature has shifted, so the model is now making predictions on inputs unlike anything it was trained on. That is a signal to investigate and probably to retrain on recent data. With very large samples, even tiny, harmless shifts become "significant", so also look at the size of the change (the KS statistic, or the difference in means), not just the p-value.

Concept drift is different: the inputs look the same but the relationship to the target changes (for example, the same browsing behaviour no longer predicts a purchase after a competitor launches). It only shows up when you compare predictions with real outcomes.

Monitoring#

Track, on a dashboard with alerts:

• service health: request rate, errors, response time;
• inputs: missing values, out-of-range values, drift per feature;
• predictions: the share of each predicted class or the average prediction, which shifts when something changes;
• performance: accuracy or error once true labels arrive (often days or weeks later).

Deploying safely#

• Shadow mode: run the new model alongside the old one, logging its predictions without using them, and compare.
• Canary release: send a small share of traffic (say 5%) to the new model first.
• A/B test: compare business outcomes between models on random user groups.
• Keep the previous model ready to roll back to.

Common mistakes#

Practice#

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(1)
train = rng.exponential(5, 3000)
recent = rng.exponential(8, 800)
result = ks_2samp(train, recent)
print(f"KS statistic {result.statistic:.3f}, p-value {result.pvalue:.3g}")
print("recent mean:", round(recent.mean(), 2), " training mean:", round(train.mean(), 2))

KS statistic 0.172, p-value 1.08e-16
recent mean: 7.69  training mean: 4.94

Key takeaway#

Next, put it all together: Project: House Price Prediction.