Skip to content
Laviyx Learn

Try

Machine Learning course menu

k-Nearest Neighbours (KNN) Classifier in Python

Learn the k-nearest neighbours algorithm: predicting from the closest training examples, distance metrics, choosing k, why feature scaling is essential, KNN for regression, and its strengths and limits.

Intermediate6 min readUpdated
On this page

k-nearest neighbours (KNN) is the simplest classifier to understand: to classify a new example, find the k training examples closest to it and take a majority vote of their labels. There is no real "training" step; the model just stores the data.

How it works#

Python
import numpy as np

# fruits described by (weight in grams, colour score from 0 = green to 10 = red)
X = np.array([[150, 8], [170, 9], [140, 7], [120, 2], [130, 3], [110, 1]])
labels = np.array(["apple", "apple", "apple", "lime", "lime", "lime"])
new = np.array([135, 6])

distances = np.sqrt(((X - new) ** 2).sum(axis=1))      # Euclidean distance
nearest = np.argsort(distances)[:3]                     # k = 3
print(distances.round(1))
print(labels[nearest], "->", max(set(labels[nearest]), key=list(labels[nearest]).count))
Output
[15.1 35.1  5.1 15.5  5.8 25.5]
['apple' 'lime' 'apple'] -> apple

The three closest fruits vote, and the majority label wins. scikit-learn does the same with efficient search structures.

KNN in scikit-learn#

Python
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

unscaled = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)).fit(X_train, y_train)
print("without scaling:", round(unscaled.score(X_test, y_test), 3))
print("with scaling:   ", round(scaled.score(X_test, y_test), 3))
Output
without scaling: 0.722
with scaling:    0.963

Why scaling matters so much#

Distances add up differences across features. In the wine data, one feature (proline) has values in the hundreds or thousands while others are below 1, so without scaling the distance is almost entirely proline and the other 12 features are ignored. StandardScaler puts every feature on the same scale, which is why accuracy jumps. Always scale features for KNN (and for any distance-based method).

Choosing k#

A small k follows the training data closely (and its noise); a large k smooths decisions but can blur real boundaries:

Python
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
for k in [1, 3, 5, 11, 25, 51]:
    model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))
    scores = cross_val_score(model, X, y, cv=5)
    print(f"k={k:>2}: mean accuracy {scores.mean():.3f}")
Output
k= 1: mean accuracy 0.950
k= 3: mean accuracy 0.944
k= 5: mean accuracy 0.949
k=11: mean accuracy 0.955
k=25: mean accuracy 0.961
k=51: mean accuracy 0.961

Cross-validation averages the score over five different splits, a more reliable way to pick k than a single test set. Odd values of k avoid ties in two-class problems.

KNN for regression#

KNeighborsRegressor predicts the average of the neighbours' values:

Python
import numpy as np
from sklearn.neighbors import KNeighborsRegressor

sizes = np.array([[600], [750], [900], [1100], [1300], [1500]])
prices = np.array([40, 52, 63, 80, 95, 110])
model = KNeighborsRegressor(n_neighbors=2).fit(sizes, prices)
print(model.predict([[1000], [1600]]))
Output
[ 71.5 102.5]

For 1,600 sq ft the prediction is the average of the two largest houses: KNN can never predict beyond the range of values it has seen.

Strengths and limits#

StrengthsLimits
simple and easy to explain ("these similar customers did X")slow to predict on large datasets (it compares with all stored examples)
no training timeneeds scaled features and a sensible distance
works for oddly shaped class boundariessuffers in high dimensions, where all points become nearly equally far apart

The same "find the most similar items" idea powers recommendation and semantic search, using embeddings as the features.

Common mistakes#

Practice#

Practice exercise

On the iris dataset, use 5-fold cross-validation to compare scaled KNN with k = 1, 7 and 15, and print the mean accuracy for each (3 decimals).

Show hint

Loop over k, build make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k)) and call cross_val_score(model, X, y, cv=5).mean().

Show solution
Python
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
for k in [1, 7, 15]:
    model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))
    print(k, round(cross_val_score(model, X, y, cv=5).mean(), 3))
Output
1 0.947
7 0.96
15 0.94

Key takeaway#

Next: Decision Trees.

Finished this lesson?

Related lessons

Machine LearningFeature Engineering for Machine Learning in PythonTurn raw data into useful model features: scaling numbers with StandardScaler and MinMaxScaler, one-hot and ordinal encoding of categories, date features, ratios and interactions, log transforms for skewed values, and handling missing values with SimpleImputer.Intermediate9 min readMachine LearningDecision Trees in Python: How They Split, Overfit and ExplainLearn decision tree classifiers and regressors: how trees choose splits with Gini impurity, reading a tree as if-else rules, controlling depth to prevent overfitting, feature importances, and when trees work well.Intermediate7 min readMachine LearningEmbeddings and Semantic Search Explained (with Python)Understand embeddings: representing words and documents as vectors, cosine similarity, building a small semantic search and recommendation engine, sentence embedding models, vector databases, and retrieval-augmented generation (RAG).Advanced8 min read