Intermediate Machine Learning Interview Questions and Answers

A model's error on new data comes from bias (systematic error from being too simple: it misses real patterns, so it underfits) and variance (sensitivity to the particular training sample: it learns noise, so it overfits). Making a model more flexible lowers bias but raises variance. The goal is the complexity with the lowest total error, found with validation data. More training data reduces variance without raising bias.

Both add a penalty on coefficient size to the loss. L2 (Ridge) penalises squared coefficients and shrinks them all smoothly towards zero. L1 (Lasso) penalises absolute values and drives some coefficients exactly to zero, performing feature selection.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso, Ridge
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X = StandardScaler().fit_transform(X)
print("Ridge zero coefficients:", int(np.sum(Ridge(alpha=10).fit(X, y).coef_ == 0)))
print("Lasso zero coefficients:", int(np.sum(Lasso(alpha=5).fit(X, y).coef_ == 0)), "of", X.shape[1])

Ridge zero coefficients: 0
Lasso zero coefficients: 5 of 10

Elastic Net combines both.

Both are ensembles of decision trees. A random forest trains many deep trees independently on bootstrap samples with random feature subsets and averages them, which reduces variance; it is robust and needs little tuning. Gradient boosting trains shallow trees sequentially, each correcting the errors of the ones before, which reduces bias; it is usually more accurate on tabular data but needs more careful tuning (learning rate, number of trees, depth) and can overfit.

At each node it tries candidate thresholds on each feature and picks the split that makes the child nodes purest: for classification it minimises Gini impurity or entropy; for regression, the variance (squared error) within the children. It repeats recursively until a stopping rule such as max_depth or min_samples_leaf.

When interpretability matters (coefficients show each feature's effect), data is limited, the relationship is roughly linear in the features, you need well-calibrated probabilities and fast predictions, or as a baseline. If a complex model is only slightly better, the simpler one is often the better business choice.

A single split's score depends on which rows happened to land in it. k-fold cross-validation trains k times, each time validating on a different fold, and averages the scores, so the estimate is more reliable and its spread shows how stable it is. Use stratified folds for classification, GroupKFold when rows from the same group (like one customer) must stay together, and time-ordered splits for time series.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
scores = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression()), X, y, cv=5)
print(scores.round(3), "mean", round(scores.mean(), 3))

[0.982 0.982 0.974 0.974 0.991] mean 0.981

The ROC curve plots the true positive rate against the false positive rate at every threshold. AUC (area under it) is the probability that the model ranks a random positive example above a random negative one: 0.5 is random guessing, 1.0 is perfect ranking. It is threshold-independent. For heavily imbalanced data, average precision (area under the precision-recall curve) is often more informative.

Grid search tries every combination of listed values; the cost multiplies with each parameter. Random search samples a fixed number of combinations from distributions, which usually finds equally good settings much faster when only a few parameters really matter. Either way, tune with cross-validation on training data and keep the test set for the final check.

So every preprocessing step (imputation, scaling, encoding) is fitted only on the training folds during cross-validation, preventing leakage, and so the exact same transformations are applied at prediction time. It also saves and loads as a single object.

Look at the inertia for several k and find the "elbow" where improvement slows; compare silhouette scores (higher means better-separated clusters); and, most importantly, check whether the clusters are stable and useful for the business. The methods often disagree, so judgement is needed.

PCA finds new axes (principal components), ordered by how much variance they capture, as linear combinations of the original features. Use it to visualise high-dimensional data in 2D, to compress many correlated features before a model, or to remove redundancy. Scale features first; the downside is that components are hard to interpret.