Skip to content
Laviyx Learn

Try

Machine Learning course menu

Handling Imbalanced Data in Machine Learning (Python)

Train and evaluate classifiers when one class is rare: why accuracy fails, stratified splits, class_weight='balanced', threshold tuning with precision-recall curves, average precision, and resampling with under- and over-sampling.

Intermediate8 min readUpdated
On this page

In many important problems the interesting class is rare: fraud is a small fraction of transactions, defects a small fraction of products, churners a minority of customers. A model trained naively on such data learns that always predicting the common class is "accurate", and never finds the rare cases you care about.

The problem#

Simulated transactions where 3% are fraud:

Python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, weights=[0.97, 0.03],
                           class_sep=0.8, random_state=0)
print("fraud share:", round(y.mean(), 3))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), digits=3))
Output
fraud share: 0.034
              precision    recall  f1-score   support

           0      0.968     0.997     0.982      1738
           1      0.400     0.065     0.111        62

    accuracy                          0.964      1800
   macro avg      0.684     0.531     0.546      1800
weighted avg      0.948     0.964     0.952      1800

Overall accuracy looks excellent, but the fraud row (class 1) tells the real story: the model catches only a small share of fraud cases (its recall). Always use stratified splits (as here) so the rare class appears in every split, and judge the model on the rare class's precision and recall.

Fix 1: class weights#

class_weight="balanced" makes mistakes on the rare class cost more during training, in proportion to how rare it is. Most scikit-learn classifiers support it:

Python
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, weights=[0.97, 0.03],
                           class_sep=0.8, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

for weight in [None, "balanced"]:
    model = LogisticRegression(max_iter=1000, class_weight=weight).fit(X_train, y_train)
    pred = model.predict(X_test)
    print(f"class_weight={str(weight):>8}: precision {precision_score(y_test, pred):.3f}, recall {recall_score(y_test, pred):.3f}")
Output
class_weight=    None: precision 0.400, recall 0.065
class_weight=balanced: precision 0.112, recall 0.758

Balanced weights catch far more fraud (higher recall) at the cost of more false alarms (lower precision). Whether that trade is worth it depends on what a missed fraud costs compared with a false alarm.

Fix 2: choose the threshold#

The model's probabilities already rank cases; the default 0.5 cut-off is just one choice. The precision-recall curve shows every option, and average precision (the area under it) summarises ranking quality on imbalanced data better than ROC AUC:

Python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, weights=[0.97, 0.03],
                           class_sep=0.8, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
proba = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]

print("ROC AUC:", round(roc_auc_score(y_test, proba), 3), " average precision:", round(average_precision_score(y_test, proba), 3))
precision, recall, thresholds = precision_recall_curve(y_test, proba)
target = np.argmax(recall[:-1] <= 0.6)          # first threshold where recall drops to 60%
print(f"threshold {thresholds[target]:.3f}: precision {precision[target]:.3f}, recall {recall[target]:.3f}")
Output
ROC AUC: 0.816  average precision: 0.271
threshold 0.064: precision 0.157, recall 0.597

A business can pick the threshold that meets a requirement such as "catch at least 60% of fraud" and see what precision that implies. ROC AUC can look reassuringly high on imbalanced data even when precision is poor; average precision exposes that.

Fix 3: resampling#

Resampling changes the training data so the classes are more balanced:

  • undersampling removes some majority-class rows;
  • oversampling repeats (or synthesises) minority-class rows. SMOTE, from the imbalanced-learn library, creates synthetic examples between existing minority points.

Simple random oversampling with plain NumPy:

Python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, weights=[0.97, 0.03],
                           class_sep=0.8, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

rng = np.random.default_rng(0)
minority = np.flatnonzero(y_train == 1)
extra = rng.choice(minority, size=(y_train == 0).sum() - minority.size, replace=True)
X_bal = np.vstack([X_train, X_train[extra]])
y_bal = np.concatenate([y_train, y_train[extra]])
print("training class counts after oversampling:", np.bincount(y_bal))

pred = LogisticRegression(max_iter=1000).fit(X_bal, y_bal).predict(X_test)
print(f"precision {precision_score(y_test, pred):.3f}, recall {recall_score(y_test, pred):.3f}")
Output
training class counts after oversampling: [4056 4056]
precision 0.110, recall 0.758

Resample only the training data, after splitting (and inside each cross-validation fold). Resampling before the split puts copies of the same row in both training and test sets, which inflates the scores.

Which approach?#

  1. Start with stratified splits and the right metrics (precision, recall, average precision).
  2. Try class_weight="balanced": no data changes, one line of code.
  3. Tune the threshold on validation data to match the real costs.
  4. Consider resampling if the above is not enough.

Common mistakes#

Practice#

Practice exercise

Using the same simulated data and split as above, train RandomForestClassifier(n_estimators=200, random_state=0) with and without class_weight="balanced", and print precision, recall and average precision (from predict_proba) for each (3 decimals).

Show hint

Loop over the two class_weight values; average_precision_score(y_test, model.predict_proba(X_test)[:, 1]).

Show solution
Python
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import average_precision_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, weights=[0.97, 0.03],
                           class_sep=0.8, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
for weight in [None, "balanced"]:
    model = RandomForestClassifier(n_estimators=200, random_state=0, class_weight=weight, n_jobs=-1).fit(X_train, y_train)
    pred = model.predict(X_test)
    ap = average_precision_score(y_test, model.predict_proba(X_test)[:, 1])
    print(weight, round(precision_score(y_test, pred), 3), round(recall_score(y_test, pred), 3), round(ap, 3))
Output
None 0.923 0.194 0.467
balanced 1.0 0.016 0.33

Surprisingly, balanced weights made the forest catch fewer fraud cases here. A forest's fully grown trees end in pure leaves, so reweighting barely changes which leaf a case lands in but does change the votes in ways that can backfire. class_weight is not a reliable fix for tree ensembles; tuning the threshold on predict_proba (whose average precision shows the ranking is decent) is usually the better lever. Always measure rather than assume.

Key takeaway#

Next, learning without labels: k-Means Clustering.

Finished this lesson?

Related lessons

Machine LearningClassification Metrics: Confusion Matrix, Precision, Recall, F1 and ROC AUCEvaluate classifiers properly: why accuracy misleads on imbalanced data, the confusion matrix, precision, recall, F1 score, classification_report, the precision-recall trade-off, ROC curves and AUC, with scikit-learn.Intermediate9 min readMachine LearningLogistic Regression in Python: Classification with ProbabilitiesLearn logistic regression for classification: the sigmoid function, predicting probabilities with predict_proba, the decision threshold, interpreting coefficients as odds, multi-class classification, and scaling with a pipeline.Intermediate9 min readMachine LearningCross-Validation in scikit-learn: K-Fold, Stratified and Time SeriesLearn cross-validation: why one train/test split is unreliable, k-fold and stratified k-fold, cross_val_score and cross_validate with several metrics, group k-fold for related rows, TimeSeriesSplit, and comparing models fairly.Intermediate9 min read