Machine Learning course menu
Cross-Validation in scikit-learn: K-Fold, Stratified and Time Series
Learn cross-validation: why one train/test split is unreliable, k-fold and stratified k-fold, cross_val_score and cross_validate with several metrics, group k-fold for related rows, TimeSeriesSplit, and comparing models fairly.
On this page
A single train/test split gives one score, and that score depends on which rows happened to land in the test set. Cross-validation repeats the split several times and averages the results, giving a more reliable estimate and a sense of how much the score varies.
One split is noisy#
The same model, scored on five different random splits:
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
for seed in range(5):
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=seed)
score = DecisionTreeClassifier(random_state=0).fit(X_train, y_train).score(X_test, y_test)
print(f"split {seed}: {score:.3f}")split 0: 0.933
split 1: 0.956
split 2: 0.956
split 3: 0.867
split 4: 0.911Depending on the split, you could report a very different accuracy for exactly the same model.
K-fold cross-validation#
K-fold splits the data into k equal folds. Each fold takes one turn as the validation set while the model trains on the other k − 1 folds. Every row is used for validation exactly once:
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
scores = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5)
print(scores.round(3))
print(f"mean {scores.mean():.3f}, standard deviation {scores.std():.3f}")[0.944 0.833 0.889 0.914 0.857]
mean 0.888, standard deviation 0.040Report the mean and the spread. 5 or 10 folds are the usual choices. For classifiers, cv=5 automatically uses stratified folds, which keep the class proportions the same in every fold.
Controlling the folds#
import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold
y = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
X = np.arange(10).reshape(-1, 1)
for name, cv in [("KFold", KFold(n_splits=2)), # no shuffling
("StratifiedKFold", StratifiedKFold(n_splits=2, shuffle=True, random_state=0))]:
print(name)
for train_idx, val_idx in cv.split(X, y):
print(" validation labels:", y[val_idx].tolist())KFold
validation labels: [0, 0, 0, 0, 0]
validation labels: [0, 1, 1, 1, 1]
StratifiedKFold
validation labels: [0, 0, 0, 1, 1]
validation labels: [0, 0, 0, 1, 1]The labels here are sorted (all the 0s first), as data often is when exported from a database. Plain KFold without shuffling cuts the data in order, so the first validation fold has no 1s at all. StratifiedKFold keeps every fold representative. When you build folds yourself, shuffle them (shuffle=True) or stratify.
Several metrics at once#
cross_validate returns several metrics and the training scores, which helps spot overfitting:
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate
X, y = load_breast_cancer(return_X_y=True)
results = cross_validate(RandomForestClassifier(n_estimators=200, random_state=0), X, y, cv=5,
scoring=["accuracy", "recall", "roc_auc"], return_train_score=True)
for key in ["train_accuracy", "test_accuracy", "test_recall", "test_roc_auc"]:
print(f"{key:>15}: {results[key].mean():.3f}") train_accuracy: 1.000
test_accuracy: 0.960
test_recall: 0.975
test_roc_auc: 0.990A training accuracy of 1.0 against a validation accuracy around 0.96 shows the forest memorises its training data, yet still generalises well; the validation numbers are the ones that matter.
Groups: keep related rows together#
If several rows belong to the same patient, customer or device, they must not be split between training and validation: the model would recognise the individual instead of learning the pattern. GroupKFold keeps each group in one fold:
import numpy as np
from sklearn.model_selection import GroupKFold
patients = np.array(["A", "A", "A", "B", "B", "C", "C", "C", "D", "D"])
X = np.arange(10).reshape(-1, 1)
for train_idx, val_idx in GroupKFold(n_splits=2).split(X, groups=patients):
print("validation patients:", sorted(str(p) for p in set(patients[val_idx])))validation patients: ['C', 'D']
validation patients: ['A', 'B']Time series: never train on the future#
For data over time, TimeSeriesSplit always validates on data that comes after the training data:
import numpy as np
from sklearn.model_selection import TimeSeriesSplit
X = np.arange(12).reshape(-1, 1)
for train_idx, val_idx in TimeSeriesSplit(n_splits=3).split(X):
print("train", train_idx.tolist(), "-> validate", val_idx.tolist())train [0, 1, 2] -> validate [3, 4, 5]
train [0, 1, 2, 3, 4, 5] -> validate [6, 7, 8]
train [0, 1, 2, 3, 4, 5, 6, 7, 8] -> validate [9, 10, 11]Comparing models#
Use the same folds for every model so the comparison is fair, and look at the spread before declaring a winner:
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
folds = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
models = {
"logistic": make_pipeline(StandardScaler(), LogisticRegression()),
"forest": RandomForestClassifier(n_estimators=200, random_state=0),
}
for name, model in models.items():
s = cross_val_score(model, X, y, cv=folds)
print(f"{name:>8}: {s.mean():.3f} ± {s.std():.3f}")logistic: 0.977 ± 0.019
forest: 0.963 ± 0.018When the difference in means is smaller than the spread, the models are effectively tied on this data, and simplicity, speed or interpretability should decide.
Common mistakes#
Practice#
Practice exercise
Compare a scaled LogisticRegression and a DecisionTreeClassifier(max_depth=3, random_state=0) on the wine dataset with 10-fold stratified cross-validation (shuffle=True, random_state=1). Print each model's mean accuracy and standard deviation (3 decimals).
Show hint
Create one StratifiedKFold(n_splits=10, shuffle=True, random_state=1) and pass it as cv for both models.
Show solution
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
folds = StratifiedKFold(n_splits=10, shuffle=True, random_state=1)
for name, model in [("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
("tree", DecisionTreeClassifier(max_depth=3, random_state=0))]:
s = cross_val_score(model, X, y, cv=folds)
print(name, round(s.mean(), 3), round(s.std(), 3))logistic 0.983 0.025
tree 0.905 0.043Key takeaway#
Next: Hyperparameter Tuning.
Finished this lesson?