Decision Trees in Python: How They Split, Overfit and Explain

A decision tree makes predictions by asking a series of yes/no questions about the features, like a flowchart: "Is income above ₹50,000? If yes, is the loan amount above ₹5 lakh? …" Each path ends in a leaf with a prediction. Trees are easy to read, need no feature scaling, and are the building block of the most successful models for tabular data: random forests and gradient boosting.

A tree you can read#

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(iris.data, iris.target)
print(export_text(tree, feature_names=list(iris.feature_names)))

|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Read it as rules: if petal width is at most 0.8 cm, the flower is class 0 (setosa); otherwise, a second question on petal width (at most 1.75 cm or not) decides between the other two species. export_text makes the model fully explainable, which matters when decisions must be justified.

How a tree chooses splits#

At each node, the tree tries every feature and threshold and picks the split that makes the resulting groups purest. For classification the usual measure is Gini impurity: 0 when a group contains a single class, higher when classes are mixed:

import numpy as np

def gini(labels):
    _, counts = np.unique(labels, return_counts=True)
    p = counts / counts.sum()
    return 1 - (p ** 2).sum()

print(round(gini([0, 0, 0, 0]), 3))         # pure
print(round(gini([0, 0, 1, 1]), 3))         # 50/50 mix
print(round(gini([0, 1, 2, 0, 1, 2]), 3))   # three classes evenly mixed

0.0
0.5
0.667

The tree keeps splitting until the leaves are pure or a stopping rule (like max_depth) is reached. For regression trees, the split minimises the squared error instead, and each leaf predicts the average of its training values.

Trees overfit easily#

With no limits, a tree keeps splitting until it has memorised the training data:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=4, stratify=y)

for depth in [1, 2, 3, 5, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={str(depth):>4}: leaves {tree.get_n_leaves():>2}, "
          f"train {tree.score(X_train, y_train):.3f}, test {tree.score(X_test, y_test):.3f}")

max_depth=   1: leaves  2, train 0.927, test 0.901
max_depth=   2: leaves  4, train 0.950, test 0.930
max_depth=   3: leaves  8, train 0.975, test 0.930
max_depth=   5: leaves 16, train 0.995, test 0.942
max_depth=None: leaves 19, train 1.000, test 0.942

The unlimited tree reaches perfect training accuracy, but its test accuracy is no better than the depth-5 tree's and only slightly better than a 4-leaf tree's. The extra splits mostly memorise the training data. Limit the tree with:

• max_depth: the maximum number of questions on any path;
• min_samples_leaf: the minimum number of training examples in a leaf;
• max_leaf_nodes or ccp_alpha (cost-complexity pruning).

Choose these with cross-validation.

Feature importance#

A tree records how much each feature reduced impurity across all its splits:

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier

data = load_breast_cancer()
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(data.data, data.target)
importance = pd.Series(tree.feature_importances_, index=data.feature_names)
print(importance.sort_values(ascending=False).head(4).round(3))
print("features used:", int((importance > 0).sum()), "of", len(importance))

worst radius            0.757
worst concave points    0.117
worst texture           0.042
compactness error       0.035
dtype: float64
features used: 7 of 30

Importances sum to 1. A depth-3 tree uses only a few features; the rest get 0. Importance shows what the model relies on, not what causes the outcome.

Strengths and weaknesses#

The instability and modest accuracy of single trees are exactly what ensembles fix.

Common mistakes#

Practice#

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
for depth in [2, 4, None]:
    scores = cross_val_score(DecisionTreeClassifier(max_depth=depth, random_state=0), X, y, cv=5)
    print(depth, round(scores.mean(), 3))

2 0.821
4 0.916
None 0.888

Key takeaway#

Next: many trees together, in Random Forests.