Random Forests in Python: Bagging, Feature Importance and Tuning

A single decision tree is unstable and overfits. A random forest trains many trees, each on a slightly different version of the data, and averages their votes. The individual trees make different mistakes, and averaging cancels much of that noise. Random forests are accurate, robust and hard to break, which is why they are a favourite first serious model for tabular data.

Two sources of randomness#

1. Bootstrap samples (bagging): each tree trains on a random sample of the rows, drawn with replacement, so every tree sees a slightly different dataset.
2. Random feature subsets: at each split, a tree only considers a random subset of the features. This stops every tree from relying on the same strongest feature, making the trees more different from each other.

One tree versus a forest#

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
tree = DecisionTreeClassifier(random_state=0)
forest = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1)

for name, model in [("single tree", tree), ("random forest", forest)]:
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:>13}: mean accuracy {scores.mean():.3f} (spread {scores.std():.3f})")

  single tree: mean accuracy 0.917 (spread 0.016)
random forest: mean accuracy 0.960 (spread 0.019)

The forest is clearly more accurate (0.960 against 0.917 on average). Both are about equally consistent across the five folds here; with noisier data, the forest's averaging usually makes it more stable too. n_jobs=-1 trains trees in parallel on all CPU cores.

Out-of-bag score#

Each tree's bootstrap sample leaves out roughly a third of the rows. Those out-of-bag rows act as a free validation set for that tree, so a forest can estimate its own accuracy without a separate split:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
forest = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0, n_jobs=-1).fit(X_train, y_train)
print("out-of-bag accuracy:", round(forest.oob_score_, 3))
print("test accuracy:      ", round(forest.score(X_test, y_test), 3))

out-of-bag accuracy: 0.962
test accuracy:       0.944

The two estimates are close, which is what you would expect if the out-of-bag score is a fair estimate.

Feature importance#

Forests report impurity-based importances like single trees, but these can favour features with many distinct values. Permutation importance is more reliable: shuffle one feature in the test set and measure how much the score drops. If the score barely changes, the model did not need that feature:

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.25, random_state=0, stratify=data.target)
forest = RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1).fit(X_train, y_train)

result = permutation_importance(forest, X_test, y_test, n_repeats=10, random_state=0, n_jobs=-1)
importance = pd.Series(result.importances_mean, index=data.feature_names)
print(importance.sort_values(ascending=False).head(5).round(4))

worst area              0.0084
worst perimeter         0.0084
worst concave points    0.0070
mean concavity          0.0056
mean concave points     0.0056
dtype: float64

With many correlated features (as here, where several measure tumour size in different ways), shuffling one barely hurts because others carry the same information, so individual importances look small. Interpret importances for groups of related features, not one feature in isolation.

Main hyperparameters#

Random forests work well with default settings, which is part of their appeal. RandomForestRegressor does the same for numbers, averaging the trees' predictions.

Common mistakes#

Practice#

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

X, y = load_wine(return_X_y=True)
for n in [10, 300]:
    score = cross_val_score(RandomForestClassifier(n_estimators=n, random_state=0), X, y, cv=5).mean()
    print(n, round(score, 3))

10 0.944
300 0.967

Key takeaway#

Next: Gradient Boosting.