Skip to content
Laviyx Learn

Try

Machine Learning course menu

Hyperparameter Tuning: GridSearchCV and RandomizedSearchCV

Tune machine learning models in scikit-learn: parameters vs hyperparameters, grid search and randomized search with cross-validation, reading cv_results_, searching on a log scale, nested cross-validation, and avoiding overfitting the validation data.

Intermediate8 min readUpdated
On this page

Parameters are what a model learns from data (the coefficients of a linear regression, the splits of a tree). Hyperparameters are settings you choose before training: the depth of a tree, the number of neighbours in KNN, the regularisation strength C. Tuning means finding hyperparameters that give the best performance on unseen data.

GridSearchCV tries every combination in a grid and scores each with cross-validation:

Python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

grid = {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5]}
search = GridSearchCV(RandomForestClassifier(n_estimators=150, random_state=0), grid, cv=5, n_jobs=-1)
search.fit(X_train, y_train)

print("best settings:", search.best_params_)
print("best cross-validated accuracy:", round(search.best_score_, 3))
print("test accuracy of the refitted best model:", round(search.score(X_test, y_test), 3))
Output
best settings: {'max_depth': 6, 'min_samples_leaf': 1}
best cross-validated accuracy: 0.96
test accuracy of the refitted best model: 0.944

That grid has 3 × 2 = 6 combinations, each trained 5 times: 30 fits. After the search, GridSearchCV refits the best combination on all the training data, so search can be used directly as a model.

Reading the results#

cv_results_ holds the score of every combination, which shows how sensitive the model is to each setting:

Python
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

X, y = load_breast_cancer(return_X_y=True)
grid = {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5]}
search = GridSearchCV(RandomForestClassifier(n_estimators=150, random_state=0), grid, cv=5, n_jobs=-1).fit(X, y)

results = pd.DataFrame(search.cv_results_)[["param_max_depth", "param_min_samples_leaf", "mean_test_score", "std_test_score"]]
print(results.sort_values("mean_test_score", ascending=False).round(3).to_string(index=False))
Output
param_max_depth  param_min_samples_leaf  mean_test_score  std_test_score
              6                       1            0.963           0.021
           None                       1            0.960           0.024
              6                       5            0.958           0.024
           None                       5            0.958           0.024
              3                       1            0.953           0.025
              3                       5            0.953           0.025

When the top scores are all within one standard deviation of each other, the setting barely matters; prefer the simpler or faster option.

Searching on a log scale#

Strength-like hyperparameters (C, alpha, gamma, learning rates) matter by order of magnitude, so search them on a log scale:

Python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": np.logspace(-2, 2, 5), "svc__gamma": np.logspace(-3, 0, 4)}
print("C values:", grid["svc__C"])
search = GridSearchCV(pipe, grid, cv=5, n_jobs=-1).fit(X, y)
print({k: float(v) for k, v in search.best_params_.items()}, round(search.best_score_, 3))
Output
C values: [1.e-02 1.e-01 1.e+00 1.e+01 1.e+02]
{'svc__C': 10.0, 'svc__gamma': 0.01} 0.979

np.logspace(-2, 2, 5) gives 0.01, 0.1, 1, 10, 100. Pipeline step parameters use step__name (here make_pipeline named the SVM step svc).

A grid grows very quickly: 5 values for each of 4 hyperparameters is 625 combinations. RandomizedSearchCV samples a fixed number of random combinations instead, usually finding nearly as good settings much faster, and it can sample from continuous distributions:

Python
from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import RandomizedSearchCV

X, y = load_breast_cancer(return_X_y=True)
distributions = {
    "learning_rate": loguniform(0.01, 0.3),
    "max_leaf_nodes": randint(8, 64),
    "l2_regularization": loguniform(1e-4, 1.0),
}
search = RandomizedSearchCV(HistGradientBoostingClassifier(random_state=0), distributions,
                            n_iter=15, cv=5, random_state=0, n_jobs=-1)
search.fit(X, y)
print({k: v if isinstance(v, int) else round(float(v), 4) for k, v in search.best_params_.items()})
print(round(search.best_score_, 3))
Output
{'l2_regularization': 0.0006, 'learning_rate': 0.1226, 'max_leaf_nodes': 36}
0.968

15 combinations × 5 folds = 75 fits, no matter how many hyperparameters there are.

Don't overfit the validation data#

Trying many combinations and picking the best cross-validation score slightly overestimates how well the winner will do, because some of its advantage is luck. Two defences:

  1. Keep a test set outside the search and score the final model on it once (as in the first example).
  2. For a fully honest estimate on small data, use nested cross-validation: an outer cross-validation loop around the whole search.
Python
from sklearn.datasets import load_wine
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
inner = GridSearchCV(make_pipeline(StandardScaler(), KNeighborsClassifier()),
                     {"kneighborsclassifier__n_neighbors": [1, 3, 5, 9, 15]}, cv=5)
outer_scores = cross_val_score(inner, X, y, cv=5)
print(outer_scores.round(3), round(outer_scores.mean(), 3))
Output
[0.944 0.944 0.972 1.    0.943] 0.961

Common mistakes#

Practice#

Practice exercise

Tune a DecisionTreeClassifier(random_state=0) on the wine dataset with GridSearchCV (5 folds) over max_depth in [2, 3, 4, 6, None] and min_samples_leaf in [1, 3, 5]. Print the best parameters, the best score (3 decimals) and how many fits were run.

Show hint

The number of fits is the number of combinations times the number of folds; len(search.cv_results_["params"]) gives the combinations.

Show solution
Python
from sklearn.datasets import load_wine
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
grid = {"max_depth": [2, 3, 4, 6, None], "min_samples_leaf": [1, 3, 5]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5).fit(X, y)
print(search.best_params_, round(search.best_score_, 3))
print(len(search.cv_results_["params"]) * 5, "fits")
Output
{'max_depth': 4, 'min_samples_leaf': 1} 0.916
75 fits

Key takeaway#

Next: Overfitting and the Bias-Variance Trade-off.

Finished this lesson?

Related lessons

Machine LearningCross-Validation in scikit-learn: K-Fold, Stratified and Time SeriesLearn cross-validation: why one train/test split is unreliable, k-fold and stratified k-fold, cross_val_score and cross_validate with several metrics, group k-fold for related rows, TimeSeriesSplit, and comparing models fairly.Intermediate9 min readMachine Learningscikit-learn Pipelines and ColumnTransformer ExplainedCombine preprocessing and models into one object with scikit-learn Pipeline and ColumnTransformer: different steps for numeric and categorical columns, preventing data leakage, cross-validating and tuning whole pipelines, and predicting on raw data.Intermediate9 min readMachine LearningOverfitting, Underfitting and the Bias-Variance Trade-offUnderstand overfitting and underfitting, bias and variance, how to diagnose them with training vs validation scores, validation curves and learning curves in scikit-learn, and the practical fixes for each.Intermediate8 min read