Regression Metrics: MAE, MSE, RMSE, R² and MAPE Explained

A regression model's predictions are never exactly right. Metrics summarise how wrong they are, in a way you can compare between models and explain to others. Each metric answers a slightly different question.

import numpy as np

actual = np.array([200, 150, 300, 250, 180])
predicted = np.array([210, 140, 280, 255, 220])
errors = actual - predicted
print(errors)

[-10  10  20  -5 -40]

MAE: mean absolute error#

The average size of the errors, ignoring their sign. It is in the same units as the target, so it is easy to explain: "on average we are off by about 17 units".

MSE and RMSE#

MSE (mean squared error) averages the squared errors. Squaring punishes big errors much more than small ones. Its units are squared (rupees²), so we usually report RMSE (its square root), which is back in the original units.

R²: how much better than the average?#

R² compares the model with simply predicting the mean every time: 1 is perfect, 0 means no better than the mean, and negative means worse than the mean.

MAPE#

MAPE (mean absolute percentage error) averages the errors as a percentage of the actual value: "off by 8% on average". It is easy to communicate but breaks down when actual values are near zero.

Computing them#

import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_absolute_percentage_error,
                             mean_squared_error, r2_score, root_mean_squared_error)

actual = np.array([200, 150, 300, 250, 180])
predicted = np.array([210, 140, 280, 255, 220])

print("MAE ", mean_absolute_error(actual, predicted))
print("MSE ", mean_squared_error(actual, predicted))
print("RMSE", round(root_mean_squared_error(actual, predicted), 2))
print("R²  ", round(r2_score(actual, predicted), 3))
print("MAPE", round(mean_absolute_percentage_error(actual, predicted) * 100, 1), "%")

MAE  17.0
MSE  445.0
RMSE 21.1
R²   0.842
MAPE 8.5 %

root_mean_squared_error is available in scikit-learn 1.4 and later; with older versions use np.sqrt(mean_squared_error(...)).

Outliers: MAE vs RMSE#

One very bad prediction moves RMSE much more than MAE:

import numpy as np
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

actual = np.array([100, 100, 100, 100, 100])
steady = np.array([110, 90, 110, 90, 110])      # every prediction off by 10
one_miss = np.array([100, 100, 100, 100, 150])  # perfect except one big miss

for name, pred in [("steady", steady), ("one big miss", one_miss)]:
    print(f"{name:>12}: MAE {mean_absolute_error(actual, pred):.1f}, RMSE {root_mean_squared_error(actual, pred):.1f}")

      steady: MAE 10.0, RMSE 10.0
one big miss: MAE 10.0, RMSE 22.4

Both have the same MAE, but RMSE is more than twice as high for the single big miss. Use RMSE when large errors are especially costly (an under-stocked festival weekend); use MAE when every unit of error costs about the same.

Always compare with a baseline#

A metric alone is hard to judge. Is an MAE of 40 good? Compare with predicting the training average:

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=2)

for name, model in [("predict the mean", DummyRegressor()), ("linear regression", LinearRegression())]:
    model.fit(X_train, y_train)
    print(f"{name:>17}: MAE {mean_absolute_error(y_test, model.predict(X_test)):.1f}")

 predict the mean: MAE 61.9
linear regression: MAE 44.1

The model's errors are clearly smaller than the baseline's. That comparison is what makes a metric meaningful.

Choosing a metric#

Report the metric on the test set (or with cross-validation), never only on the training data.

Common mistakes#

Practice#

import numpy as np
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

actual = np.array([30, 45, 25, 60, 40])
models = {"A": np.array([32, 44, 27, 55, 41]), "B": np.array([30, 45, 25, 60, 70])}
for name, pred in models.items():
    print(name, round(mean_absolute_error(actual, pred), 2), round(root_mean_squared_error(actual, pred), 2))

A 2.2 2.65
B 6.0 13.42

Key takeaway#

Next, predicting categories: Logistic Regression.