Skip to content
Laviyx Learn

Try

Python for Data course menu

Find and Remove Duplicates and Outliers in pandas

Detect and remove duplicate rows with duplicated and drop_duplicates, keep the latest record per customer, and find outliers with the IQR rule and z-scores, then decide whether to remove, cap or keep them.

Intermediate9 min readUpdated
On this page

Two more problems turn up in almost every dataset: duplicate records (the same order loaded twice) and outliers (values far from the rest, such as an age of 250 or a ₹9,99,999 order). Both distort totals and averages if left alone.

Finding duplicates#

Python
import pandas as pd

orders = pd.DataFrame({
    "order_id": [101, 102, 102, 103, 104, 104],
    "customer": ["Asha", "Ravi", "Ravi", "Meera", "John", "John"],
    "amount":   [250, 120, 120, 560, 90, 95],
})
print(orders.duplicated())                     # entire row repeated?
print(orders.duplicated().sum(), "exact duplicate rows")
print(orders.duplicated(subset="order_id").sum(), "repeated order ids")
print(orders[orders.duplicated(subset="order_id", keep=False)])
Output
0    False
1    False
2     True
3    False
4    False
5    False
dtype: bool
1 exact duplicate rows
2 repeated order ids
   order_id customer  amount
1       102     Ravi     120
2       102     Ravi     120
4       104     John      90
5       104     John      95

Order 102 is an exact copy. Order 104 appears twice with different amounts: a conflict that needs a decision, not just deletion. keep=False marks every copy, so you can see all of them.

Removing duplicates#

Python
import pandas as pd

orders = pd.DataFrame({
    "order_id": [101, 102, 102, 103, 104, 104],
    "customer": ["Asha", "Ravi", "Ravi", "Meera", "John", "John"],
    "amount":   [250, 120, 120, 560, 90, 95],
})
print(orders.drop_duplicates())                               # exact copies only
print(orders.drop_duplicates(subset="order_id", keep="last"))  # one row per id, last wins
Output
   order_id customer  amount
0       101     Asha     250
1       102     Ravi     120
3       103    Meera     560
4       104     John      90
5       104     John      95
   order_id customer  amount
0       101     Asha     250
2       102     Ravi     120
3       103    Meera     560
5       104     John      95

Keeping the latest record#

A common task: one row per customer, keeping their most recent update. Sort by date, then drop duplicates keeping the last:

Python
import pandas as pd

updates = pd.DataFrame({
    "customer": ["Asha", "Ravi", "Asha", "Ravi", "Meera"],
    "updated":  pd.to_datetime(["2026-01-05", "2026-01-07", "2026-02-10", "2026-01-02", "2026-03-01"]),
    "plan":     ["basic", "pro", "pro", "basic", "basic"],
})
latest = updates.sort_values("updated").drop_duplicates("customer", keep="last")
print(latest.sort_values("customer"))
Output
  customer    updated   plan
2     Asha 2026-02-10    pro
4    Meera 2026-03-01  basic
1     Ravi 2026-01-07    pro

Asha moved to "pro" in February, so that row is kept.

Outliers with the IQR rule#

The interquartile range (IQR) is the spread of the middle half of the data: Q3 − Q1. A widely used rule flags values more than 1.5 × IQR below Q1 or above Q3. It does not depend on the mean, so a few extreme values do not hide themselves:

Python
import pandas as pd

amounts = pd.Series([250, 310, 180, 270, 4200, 290, 230, 260, 15, 300])
q1, q3 = amounts.quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(q1, q3, iqr)
print(low, high)
print(amounts[(amounts < low) | (amounts > high)].tolist())
Output
235.0 297.5 62.5
141.25 391.25
[4200, 15]

Outliers with z-scores#

A z-score says how many standard deviations a value is from the mean. Values beyond ±3 (sometimes ±2) are flagged:

Python
import pandas as pd

amounts = pd.Series([250, 310, 180, 270, 4200, 290, 230, 260, 15, 300])
z = (amounts - amounts.mean()) / amounts.std()
print(z.round(2).tolist())
print(amounts[z.abs() > 2].tolist())
Output
[-0.3, -0.25, -0.36, -0.29, 2.84, -0.27, -0.32, -0.29, -0.49, -0.26]
[4200]

The z-score method only caught 4200, and with a ±3 cut-off it would have caught nothing: the outlier itself inflates the standard deviation. That is why the IQR rule is often preferred for small or skewed data.

What to do with outliers#

  1. Check if it is an error. An age of 250 is a typo; fix or remove it.
  2. If it is real, keep it, but consider reporting the median instead of the mean, or analysing it separately (a bulk corporate order is real, but not typical).
  3. Cap extreme values to the limits (also called winsorising) when a model is sensitive to them:
Python
import pandas as pd

amounts = pd.Series([250, 310, 180, 270, 4200, 290, 230, 260, 15, 300])
print(amounts.clip(lower=105, upper=435).tolist())
print(round(amounts.mean(), 1), amounts.median())
Output
[250, 310, 180, 270, 435, 290, 230, 260, 105, 300]
630.5 265.0

One value pulls the mean far above the median; that gap is itself a sign of outliers.

Common mistakes#

Practice#

Practice exercise

Delivery times in minutes were [32, 28, 35, 30, 31, 29, 180, 33, 27, 34, 30]. Using the IQR rule, print the bounds and the outliers, then print the mean and median before and after removing the outliers (rounded to 1 decimal).

Show hint

Compute Q1 and Q3 with quantile, the bounds as Q1 − 1.5·IQR and Q3 + 1.5·IQR, then filter with between(low, high).

Show solution
Python
import pandas as pd

times = pd.Series([32, 28, 35, 30, 31, 29, 180, 33, 27, 34, 30])
q1, q3 = times.quantile([0.25, 0.75])
low, high = q1 - 1.5 * (q3 - q1), q3 + 1.5 * (q3 - q1)
print(low, high)
print(times[~times.between(low, high)].tolist())
kept = times[times.between(low, high)]
print(round(times.mean(), 1), times.median())
print(round(kept.mean(), 1), kept.median())
Output
23.5 39.5
[180]
44.5 31.0
30.9 30.5

The single 180-minute delivery raises the mean by more than 13 minutes but barely moves the median.

Key takeaway#

Next: apply, map and Custom Functions.

Finished this lesson?

Related lessons

Python for DataHandling Missing Values in pandas: isna, dropna and fillnaFind, count, drop and fill missing data in pandas: NaN and None, isna and notna, dropna with thresholds, fillna with constants, means and group means, forward fill and interpolation, and how missing values affect calculations.Intermediate12 min readPython for DataDescriptive Statistics in Python: Mean, Median, Mode, Spread and PercentilesSummarise data with descriptive statistics in pandas: mean, median and mode, when each is appropriate, range, variance and standard deviation, percentiles and the interquartile range, skewness, and describe().Intermediate8 min readPython for DataCleaning Text Data in pandas with .str MethodsClean messy text columns in pandas: strip spaces, change case, replace characters, split columns, extract values with regular expressions, check text with contains and startswith, and standardise inconsistent labels.Intermediate7 min read