Skip to content
Laviyx Learn

Try

Python for Data course menu

Handling Missing Values in pandas: isna, dropna and fillna

Find, count, drop and fill missing data in pandas: NaN and None, isna and notna, dropna with thresholds, fillna with constants, means and group means, forward fill and interpolation, and how missing values affect calculations.

Intermediate12 min readUpdated
On this page

Real datasets have gaps: a customer skipped a form field, a sensor went offline, a value failed to load. pandas marks missing values as NaN ("not a number"); None in your data is converted to NaN (or shown as None in text columns), and both are treated as missing.

The examples use this survey:

Python
import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "name":   ["Asha", "Ravi", "Meera", "John", "Priya", "Kiran"],
    "city":   ["Mumbai", None, "Pune", "Delhi", None, "Delhi"],
    "age":    [25, 31, np.nan, 42, 28, np.nan],
    "rating": [4.5, np.nan, 3.0, np.nan, 5.0, 4.0],
})
print(survey)
Output
    name    city   age  rating
0   Asha  Mumbai  25.0     4.5
1   Ravi    None  31.0     NaN
2  Meera    Pune   NaN     3.0
3   John   Delhi  42.0     NaN
4  Priya    None  28.0     5.0
5  Kiran   Delhi   NaN     4.0

Finding missing values#

Python
import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "name":   ["Asha", "Ravi", "Meera", "John", "Priya", "Kiran"],
    "city":   ["Mumbai", None, "Pune", "Delhi", None, "Delhi"],
    "age":    [25, 31, np.nan, 42, 28, np.nan],
    "rating": [4.5, np.nan, 3.0, np.nan, 5.0, 4.0],
})
print(survey.isna().sum())                  # missing per column
print((survey.isna().mean() * 100).round(1))   # percentage missing
print(survey[survey["age"].isna()])          # rows where age is missing
print(survey.isna().any(axis=1).sum(), "rows have at least one gap")
Output
name      0
city      2
age       2
rating    2
dtype: int64
name       0.0
city      33.3
age       33.3
rating    33.3
dtype: float64
    name   city  age  rating
2  Meera   Pune  NaN     3.0
5  Kiran  Delhi  NaN     4.0
5 rows have at least one gap

Always check this first: the right fix depends on how much is missing and why.

How missing values affect calculations#

pandas skips NaN in calculations by default, which changes what "average" means:

Python
import numpy as np
import pandas as pd

rating = pd.Series([4.5, np.nan, 3.0, np.nan, 5.0, 4.0])
print(rating.mean())             # average of the 4 known ratings
print(rating.count(), len(rating))
print(rating.sum(skipna=False))
Output
4.125
4 6
nan

count() counts non-missing values; len() counts all rows. Be explicit in reports about how many values an average is based on.

Dropping missing values#

Python
import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "name":   ["Asha", "Ravi", "Meera", "John", "Priya", "Kiran"],
    "city":   ["Mumbai", None, "Pune", "Delhi", None, "Delhi"],
    "age":    [25, 31, np.nan, 42, 28, np.nan],
    "rating": [4.5, np.nan, 3.0, np.nan, 5.0, 4.0],
})
print(survey.dropna())                       # rows with any gap removed
print(survey.dropna(subset=["rating"]))      # only care about rating
print(survey.dropna(thresh=3))               # keep rows with at least 3 values
Output
   name    city   age  rating
0  Asha  Mumbai  25.0     4.5
    name    city   age  rating
0   Asha  Mumbai  25.0     4.5
2  Meera    Pune   NaN     3.0
4  Priya    None  28.0     5.0
5  Kiran   Delhi   NaN     4.0
    name    city   age  rating
0   Asha  Mumbai  25.0     4.5
2  Meera    Pune   NaN     3.0
3   John   Delhi  42.0     NaN
4  Priya    None  28.0     5.0
5  Kiran   Delhi   NaN     4.0

dropna() with no arguments kept only one of the six rows here. Dropping is reasonable when few rows are affected and they are missing at random; otherwise it can bias the results (for example, if unhappy customers skip the rating).

Filling missing values#

Python
import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "name":   ["Asha", "Ravi", "Meera", "John", "Priya", "Kiran"],
    "city":   ["Mumbai", None, "Pune", "Delhi", None, "Delhi"],
    "age":    [25, 31, np.nan, 42, 28, np.nan],
    "rating": [4.5, np.nan, 3.0, np.nan, 5.0, 4.0],
})
filled = survey.fillna({
    "city": "Unknown",
    "age": survey["age"].median(),
    "rating": survey["rating"].mean(),
})
print(filled)
Output
    name     city   age  rating
0   Asha   Mumbai  25.0   4.500
1   Ravi  Unknown  31.0   4.125
2  Meera     Pune  29.5   3.000
3   John    Delhi  42.0   4.125
4  Priya  Unknown  28.0   5.000
5  Kiran    Delhi  29.5   4.000

A dictionary fills each column differently. Typical choices: a label like "Unknown" for categories, the median for skewed numbers, the mean for symmetric ones. Filling hides the gap, so consider also adding a flag column such as survey["age_missing"] = survey["age"].isna().

Filling within groups#

Filling with the overall mean ignores differences between groups. Fill each group with its own mean using groupby(...).transform:

Python
import numpy as np
import pandas as pd

prices = pd.DataFrame({
    "category": ["fruit", "fruit", "fruit", "dairy", "dairy", "dairy"],
    "price":    [40, np.nan, 60, 55, 65, np.nan],
})
group_mean = prices.groupby("category")["price"].transform("mean")
prices["price_filled"] = prices["price"].fillna(group_mean)
print(prices)
Output
  category  price  price_filled
0    fruit   40.0          40.0
1    fruit    NaN          50.0
2    fruit   60.0          60.0
3    dairy   55.0          55.0
4    dairy   65.0          65.0
5    dairy    NaN          60.0

The missing fruit gets the fruit average (50), not the overall average (55).

Time series: forward fill and interpolation#

For readings over time, the previous value or a straight line between neighbours is often the best guess:

Python
import numpy as np
import pandas as pd

temps = pd.Series([21.0, np.nan, np.nan, 24.0, 25.0, np.nan],
                  index=pd.date_range("2026-03-01", periods=6))
print(pd.DataFrame({
    "raw": temps,
    "ffill": temps.ffill(),
    "interpolated": temps.interpolate(),
}))
Output
             raw  ffill  interpolated
2026-03-01  21.0   21.0          21.0
2026-03-02   NaN   21.0          22.0
2026-03-03   NaN   21.0          23.0
2026-03-04  24.0   24.0          24.0
2026-03-05  25.0   25.0          25.0
2026-03-06   NaN   25.0          25.0

ffill (forward fill) repeats the last known value; interpolate draws a straight line between known values. Neither can fill a gap at the very start (ffill) or invent a trend that is not there.

Common mistakes#

Practice#

Practice exercise

A shop's daily sales are [1200, None, 1350, None, None, 1500, 1420] for 1 to 7 April 2026. Print how many days are missing, the average of the known days (rounded to 1 decimal), and a table with the raw values, forward-filled values and interpolated values.

Show hint

Build a Series with a pd.date_range index. None becomes NaN in a numeric Series. Use isna().sum(), mean(), ffill() and interpolate().

Show solution
Python
import pandas as pd

sales = pd.Series([1200, None, 1350, None, None, 1500, 1420],
                  index=pd.date_range("2026-04-01", periods=7), dtype="float")
print(sales.isna().sum())
print(round(sales.mean(), 1))
print(pd.DataFrame({"raw": sales, "ffill": sales.ffill(), "interp": sales.interpolate()}))
Output
3
1367.5
               raw   ffill  interp
2026-04-01  1200.0  1200.0  1200.0
2026-04-02     NaN  1200.0  1275.0
2026-04-03  1350.0  1350.0  1350.0
2026-04-04     NaN  1350.0  1400.0
2026-04-05     NaN  1350.0  1450.0
2026-04-06  1500.0  1500.0  1500.0
2026-04-07  1420.0  1420.0  1420.0

Key takeaway#

Next: Data Types and Conversion.

Finished this lesson?

Related lessons

Python for Datapandas Data Types: astype, to_numeric, to_datetime and categoryFix column types in pandas: check dtypes, convert with astype, clean numbers stored as text with to_numeric, parse dates with to_datetime, use the category type to save memory, and nullable integer and string types.Intermediate9 min readPython for DataFind and Remove Duplicates and Outliers in pandasDetect and remove duplicate rows with duplicated and drop_duplicates, keep the latest record per customer, and find outliers with the IQR rule and z-scores, then decide whether to remove, cap or keep them.Intermediate9 min readSQLSQL NULL Values: IS NULL, IS NOT NULL, COALESCE ExplainedUnderstand NULL in SQL: what it means, why = NULL never works, how to filter with IS NULL and IS NOT NULL, how NULL affects NOT, aggregates and calculations, and how to replace it with COALESCE.Beginner8 min read