Ridge and Lasso Regression: Regularisation Explained

A linear model with many features, especially compared with the number of rows, can fit noise by giving some features huge coefficients. Regularisation adds a penalty for large coefficients to the training objective, so the model prefers simpler explanations. It usually improves performance on new data.

The two penalties#

• Ridge (L2) adds alpha × (sum of squared coefficients). It shrinks all coefficients towards zero, but rarely to exactly zero.
• Lasso (L1) adds alpha × (sum of absolute coefficients). It can shrink coefficients to exactly zero, removing features: built-in feature selection.

alpha controls the strength: 0 means ordinary linear regression; larger values mean a simpler model.

Seeing the effect#

The data below has only 50 training rows but 30 features, and only 5 features truly matter; the rest are noise. That is exactly the situation where plain linear regression overfits:

import numpy as np
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
X = rng.normal(size=(80, 30))
true_coef = np.zeros(30)
true_coef[:5] = [4, -3, 2.5, 2, -1.5]
y = X @ true_coef + rng.normal(0, 2, 80)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=30, random_state=0)

for name, model in [("linear", LinearRegression()), ("ridge", Ridge(alpha=1)), ("lasso", Lasso(alpha=0.3))]:
    model.fit(X_train, y_train)
    nonzero = int((np.abs(model.coef_) > 1e-6).sum())
    print(f"{name:>6}: train R² {model.score(X_train, y_train):.3f}, test R² {model.score(X_test, y_test):.3f}, non-zero coefficients {nonzero}")

linear: train R² 0.956, test R² 0.748, non-zero coefficients 30
 ridge: train R² 0.955, test R² 0.772, non-zero coefficients 30
 lasso: train R² 0.921, test R² 0.855, non-zero coefficients 9

Plain linear regression fits the training data best but generalises worst. Both regularised models do better on the test set, and Lasso, which matches this data (only a few features matter), does best while keeping only a handful of features. As the next sections show, the right alpha matters: too much regularisation makes things worse.

What Lasso keeps#

import numpy as np
from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
X = rng.normal(size=(80, 30))
true_coef = np.zeros(30)
true_coef[:5] = [4, -3, 2.5, 2, -1.5]
y = X @ true_coef + rng.normal(0, 2, 80)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=30, random_state=0)

lasso = Lasso(alpha=0.3).fit(X_train, y_train)
kept = np.flatnonzero(np.abs(lasso.coef_) > 1e-6)
print("features kept:", kept.tolist())
print("their coefficients:", lasso.coef_[kept].round(2).tolist())

features kept: [0, 1, 2, 3, 4, 5, 7, 23, 26]
their coefficients: [3.16, -2.54, 2.46, 1.69, -0.92, 0.01, -0.05, -0.02, 0.09]

The first five features, the ones that really matter, are kept with coefficients somewhat shrunk towards zero (that shrinking is the price of regularisation). Any other kept feature has a small coefficient.

Choosing alpha#

Too small an alpha barely regularises; too large shrinks everything and underfits. Try a range and compare on validation data, or let RidgeCV / LassoCV choose with cross-validation:

import numpy as np
from sklearn.linear_model import LassoCV, Ridge
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
X = rng.normal(size=(80, 30))
true_coef = np.zeros(30)
true_coef[:5] = [4, -3, 2.5, 2, -1.5]
y = X @ true_coef + rng.normal(0, 2, 80)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=30, random_state=0)

for alpha in [0.1, 1, 10, 100, 1000]:
    score = Ridge(alpha=alpha).fit(X_train, y_train).score(X_test, y_test)
    print(f"ridge alpha={alpha:>6}: test R² {score:.3f}")

cv = LassoCV(cv=5, random_state=0).fit(X_train, y_train)
print("LassoCV chose alpha =", round(cv.alpha_, 3), "test R²", round(cv.score(X_test, y_test), 3))

ridge alpha=   0.1: test R² 0.752
ridge alpha=     1: test R² 0.772
ridge alpha=    10: test R² 0.686
ridge alpha=   100: test R² 0.349
ridge alpha=  1000: test R² 0.035
LassoCV chose alpha = 0.39 test R² 0.851

Scale your features first#

The penalty treats every coefficient equally, but coefficients depend on feature units. Without scaling, a feature measured in rupees and one measured in lakhs would be penalised very differently. Always standardise before Ridge or Lasso, ideally inside a pipeline: make_pipeline(StandardScaler(), Ridge(alpha=1.0)).

ElasticNet#

ElasticNet mixes both penalties (l1_ratio sets the mix). It keeps Lasso's feature selection while behaving better when several features are strongly correlated, where Lasso tends to pick one at random.

Common mistakes#

Practice#

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
models = {"linear": LinearRegression(), "ridge": Ridge(alpha=1), "lasso": Lasso(alpha=0.5)}
for name, m in models.items():
    print(name, round(m.fit(X_train, y_train).score(X_test, y_test), 3))
print("lasso zero coefficients:", int(np.sum(models["lasso"].coef_ == 0)))

linear 0.444
ridge 0.384
lasso 0.383
lasso zero coefficients: 6

Key takeaway#

Next: Regression Metrics.