Feature Engineering for Machine Learning in Python

Models only see numbers. Feature engineering turns raw data (text categories, dates, skewed amounts, missing values) into numeric features that make patterns easy to learn. Good features often improve a model more than switching algorithms.

Scaling numeric features#

Distance-based and regularised models (KNN, SVM, logistic regression, neural networks) need features on similar scales. Tree models do not.

import numpy as np
from sklearn.preprocessing import MinMaxScaler, StandardScaler

X = np.array([[25, 30000], [32, 52000], [47, 120000], [51, 61000]], dtype=float)   # age, income
print(StandardScaler().fit_transform(X).round(2))
print(MinMaxScaler().fit_transform(X).round(2))

[[-1.29 -1.07]
 [-0.63 -0.41]
 [ 0.78  1.63]
 [ 1.15 -0.14]]
[[0.   0.  ]
 [0.27 0.24]
 [0.85 1.  ]
 [1.   0.34]]

• StandardScaler: each column gets mean 0 and standard deviation 1. The usual default.
• MinMaxScaler: each column is squeezed into 0–1. Useful when a bounded range is needed.

Fit the scaler on the training data only, then apply it to the test data (pipelines do this for you).

Encoding categories#

Categories must become numbers. One-hot encoding creates a 0/1 column per category, so the model does not assume any order:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

cities = pd.DataFrame({"city": ["Mumbai", "Delhi", "Pune", "Delhi"]})
encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
encoded = encoder.fit_transform(cities)
print(encoder.get_feature_names_out().tolist())
print(encoded)
print(encoder.transform(pd.DataFrame({"city": ["Chennai"]})))   # unseen city: all zeros

['city_Delhi', 'city_Mumbai', 'city_Pune']
[[0. 1. 0.]
 [1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]]
[[0. 0. 0.]]

handle_unknown="ignore" matters in production: a category never seen during training becomes all zeros instead of crashing the model.

For categories with a real order (small < medium < large, or education levels), OrdinalEncoder with an explicit order keeps that information in one column:

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder

sizes = pd.DataFrame({"size": ["M", "S", "L", "M"]})
encoder = OrdinalEncoder(categories=[["S", "M", "L"]])
print(encoder.fit_transform(sizes).ravel())

[1. 0. 2. 1.]

Features from dates#

A raw timestamp means little to a model; its parts often mean a lot:

import pandas as pd

orders = pd.DataFrame({"ordered_at": pd.to_datetime(["2026-03-07 21:15", "2026-03-09 09:30", "2026-12-24 18:00"])})
orders["hour"] = orders["ordered_at"].dt.hour
orders["weekday"] = orders["ordered_at"].dt.dayofweek          # Monday = 0
orders["is_weekend"] = orders["weekday"] >= 5
orders["month"] = orders["ordered_at"].dt.month
orders["days_to_new_year"] = (pd.Timestamp("2027-01-01") - orders["ordered_at"]).dt.days
print(orders)

           ordered_at  hour  weekday  is_weekend  month  days_to_new_year
0 2026-03-07 21:15:00    21        5        True      3               299
1 2026-03-09 09:30:00     9        0       False      3               297
2 2026-12-24 18:00:00    18        3       False     12                 7

Ratios, differences and interactions#

Domain knowledge creates the most powerful features. A loan applicant's debt-to-income ratio says more than debt and income separately; price per square foot compares houses of different sizes:

import pandas as pd

applicants = pd.DataFrame({"income": [50000, 120000, 80000], "monthly_debt": [20000, 30000, 8000]})
applicants["debt_to_income"] = (applicants["monthly_debt"] / applicants["income"]).round(2)
print(applicants)

   income  monthly_debt  debt_to_income
0   50000         20000            0.40
1  120000         30000            0.25
2   80000          8000            0.10

Linear models in particular benefit, because they cannot invent ratios or interactions on their own; tree models can approximate them but still learn faster from good features.

Skewed values: log transforms#

Amounts like income or order value are often right-skewed: most values are small and a few are huge. A log transform compresses the long tail, which helps linear models and distance-based ones:

import numpy as np
import pandas as pd

amounts = pd.Series([120, 250, 300, 450, 800, 1500, 25000])
print(amounts.skew().round(2), np.log1p(amounts).skew().round(2))
print(np.log1p(amounts).round(2).tolist())

2.63 1.55
[4.8, 5.53, 5.71, 6.11, 6.69, 7.31, 10.13]

np.log1p(x) computes log(1 + x), which also works for zeros.

Missing values#

Most scikit-learn models cannot handle NaN (gradient boosting is an exception). SimpleImputer fills them, learning the fill values from the training data, and can add a flag column marking where values were missing:

import numpy as np
from sklearn.impute import SimpleImputer

np.set_printoptions(suppress=True)          # plain numbers instead of 5.0e+04
X = np.array([[25, 50000], [np.nan, 62000], [41, np.nan], [33, 58000]])
imputer = SimpleImputer(strategy="median", add_indicator=True)
print(imputer.fit_transform(X))
print(imputer.statistics_)

[[   25. 50000.     0.     0.]
 [   33. 62000.     1.     0.]
 [   41. 58000.     0.     1.]
 [   33. 58000.     0.     0.]]
[   33. 58000.]

The last two columns flag which values were filled; sometimes "this was missing" is itself predictive.

Common mistakes#

Practice#

import pandas as pd

d = pd.DataFrame({
    "distance_km": [2.5, 8.0, 15.0, 4.0],
    "minutes": [12, 30, 55, 18],
    "ordered_at": pd.to_datetime(["2026-05-01 12:30", "2026-05-02 20:15", "2026-05-03 09:00", "2026-05-03 19:45"]),
})
d["speed_kmh"] = (d["distance_km"] / (d["minutes"] / 60)).round(1)
d["hour"] = d["ordered_at"].dt.hour
d["is_evening"] = d["hour"] >= 18
d["is_weekend"] = d["ordered_at"].dt.dayofweek >= 5
print(d.drop(columns="ordered_at"))

   distance_km  minutes  speed_kmh  hour  is_evening  is_weekend
0          2.5       12       12.5    12       False       False
1          8.0       30       16.0    20        True        True
2         15.0       55       16.4     9       False        True
3          4.0       18       13.3    19        True        True

Key takeaway#

Next: put the steps together safely with Pipelines and Column Transformers.