Gradient Boosting in Python: How It Works and HistGradientBoosting

Gradient boosting also combines many trees, but differently from a random forest. Instead of training independent trees and averaging, it trains small trees one after another, each one focusing on the mistakes the previous ones still make. Boosted trees are among the most accurate models for tabular data and win many competitions.

Boosting from scratch#

For regression the idea is easy to see. Start with a simple prediction (the mean). Then repeatedly fit a small tree to the residuals (what is still wrong) and add a fraction of its predictions:

import numpy as np
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 6, 80)).reshape(-1, 1)
y = np.sin(x).ravel() * 10 + rng.normal(0, 1, 80)

learning_rate = 0.3
prediction = np.full_like(y, y.mean())
for step in range(1, 51):
    residuals = y - prediction
    tree = DecisionTreeRegressor(max_depth=2).fit(x, residuals)
    prediction += learning_rate * tree.predict(x)
    if step in (1, 5, 20, 50):
        print(f"after {step:>2} trees: mean absolute error {np.abs(y - prediction).mean():.2f}")

after  1 trees: mean absolute error 4.78
after  5 trees: mean absolute error 1.56
after 20 trees: mean absolute error 0.50
after 50 trees: mean absolute error 0.32

Each tree is weak on its own (depth 2), but together they fit the curve closely. "Gradient" refers to the fact that fitting residuals is gradient descent on the squared error; other losses (like log loss for classification) use the same machinery.

Learning rate and number of trees#

A smaller learning rate adds each tree more cautiously, so it needs more trees but usually generalises better. Too many trees eventually fit noise. Early stopping solves the "how many?" question: it holds out part of the training data and stops adding trees once the validation score stops improving.

HistGradientBoosting in scikit-learn#

HistGradientBoostingClassifier and HistGradientBoostingRegressor are scikit-learn's fast boosting implementations. They bin feature values into histograms (hence the name), handle missing values natively, and use early stopping automatically on larger datasets:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score

X, y = load_breast_cancer(return_X_y=True)
models = {
    "random forest": RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1),
    "gradient boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, model in models.items():
    print(f"{name:>17}: {cross_val_score(model, X, y, cv=5).mean():.3f}")

    random forest: 0.963
gradient boosting: 0.967

On this small, clean dataset both ensembles do very well; differences between strong models are often small, and on larger, messier tabular datasets boosting frequently has the edge.

Missing values without imputation#

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 3))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
X[rng.random((500, 3)) < 0.1] = np.nan          # 10% missing everywhere

model = HistGradientBoostingClassifier(random_state=0).fit(X[:400], y[:400])
print("missing values in training data:", int(np.isnan(X[:400]).sum()))
print("test accuracy:", round(model.score(X[400:], y[400:]), 3))

missing values in training data: 138
test accuracy: 0.95

The model learns which branch missing values should follow at each split, so no filling is required.

Tuning boosting#

The parameters that matter most:

• learning_rate (often 0.03 to 0.2) together with max_iter (number of trees), or early stopping;
• max_depth / max_leaf_nodes (tree size; small trees are typical);
• l2_regularization and min_samples_leaf to reduce overfitting.

Tune them with cross-validated search.

XGBoost, LightGBM and CatBoost#

Separate libraries implement gradient boosting with extra speed and features: XGBoost, LightGBM and CatBoost (which handles categorical features especially well). They follow the same scikit-learn-style fit/predict interface and the same ideas, so everything in this lesson transfers. scikit-learn's HistGradientBoosting is a good default that needs no extra installation.

Common mistakes#

Practice#

import numpy as np
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 6, 80)).reshape(-1, 1)
y = np.sin(x).ravel() * 10 + rng.normal(0, 1, 80)

def boost(learning_rate, trees=50):
    prediction = np.full_like(y, y.mean())
    for _ in range(trees):
        tree = DecisionTreeRegressor(max_depth=2).fit(x, y - prediction)
        prediction += learning_rate * tree.predict(x)
    return np.abs(y - prediction).mean()

for lr in [0.05, 0.5]:
    print(lr, round(boost(lr), 2))

0.05 0.95
0.5 0.18

Key takeaway#

Next: Support Vector Machines.