Skip to content
Laviyx Learn

Try

Python for Data course menu

Cleaning Text Data in pandas with .str Methods

Clean messy text columns in pandas: strip spaces, change case, replace characters, split columns, extract values with regular expressions, check text with contains and startswith, and standardise inconsistent labels.

Intermediate7 min readUpdated
On this page

Text typed by people is inconsistent: extra spaces, mixed capitals, different spellings of the same thing. pandas gives every text column a .str accessor with vectorised versions of Python's string methods, applied to every row at once and skipping missing values.

Spaces and capitals#

Python
import pandas as pd

cities = pd.Series(["  Mumbai", "delhi ", "PUNE", " new delhi", "Mumbai"])
print(cities.str.strip().str.title().tolist())
print(cities.str.strip().str.lower().value_counts())
Output
['Mumbai', 'Delhi', 'Pune', 'New Delhi', 'Mumbai']
mumbai       2
delhi        1
pune         1
new delhi    1
Name: count, dtype: int64

Without cleaning, " Mumbai" and "Mumbai" would count as different cities. strip, lower, upper and title fix most of this.

Replacing characters#

Python
import pandas as pd

phones = pd.Series(["98450-12345", "(080) 2345 6789", "+91 99001 23456"])
digits = phones.str.replace(r"\D", "", regex=True)    # \D = any non-digit
print(digits.tolist())
print(digits.str.len().tolist())
print(digits.str[-10:].tolist())                      # last 10 digits
Output
['9845012345', '08023456789', '919900123456']
[10, 11, 12]
['9845012345', '8023456789', '9900123456']

With regex=True, the pattern is a regular expression. .str[...] slices every string.

Splitting into columns#

Python
import pandas as pd

people = pd.DataFrame({"full_name": ["Asha Rao", "Ravi Kumar", "Meera"],
                       "email": ["asha@shop.in", "ravi@mail.com", "meera@shop.in"]})
people[["first", "last"]] = people["full_name"].str.split(" ", n=1, expand=True)
people["domain"] = people["email"].str.split("@").str[1]
print(people)
Output
    full_name          email  first   last    domain
0    Asha Rao   asha@shop.in   Asha    Rao   shop.in
1  Ravi Kumar  ravi@mail.com   Ravi  Kumar  mail.com
2       Meera  meera@shop.in  Meera   None   shop.in

expand=True returns separate columns. "Meera" has no surname, so that cell is missing (None). n=1 splits only at the first space, keeping names like "Ravi Kumar Singh" as first name plus the rest.

Extracting with patterns#

str.extract pulls out the part of each string that matches a regular expression group (the part in parentheses):

Python
import pandas as pd

items = pd.Series(["Rice 5kg", "Oil 1L", "Sugar 2kg", "Salt"])
size = items.str.extract(r"(\d+)\s*(kg|L)")
size.columns = ["amount", "unit"]
print(pd.concat([items.rename("item"), size], axis=1))

codes = pd.Series(["ORD-2026-0045", "ORD-2025-1102", "REF-9"])
print(codes.str.extract(r"ORD-(\d{4})-\d+")[0].tolist())
Output
        item amount unit
0   Rice 5kg      5   kg
1     Oil 1L      1    L
2  Sugar 2kg      2   kg
3       Salt    NaN  NaN
['2026', '2025', nan]

Rows that do not match get NaN.

Checking text#

Python
import pandas as pd

emails = pd.Series(["asha@shop.in", "ravi(at)mail.com", "meera@shop.in", None])
valid = emails.str.contains("@", na=False)
print(valid.tolist())
print(emails.str.endswith(".in", na=False).sum(), "Indian domains")
print(emails[~valid].tolist())
Output
[True, False, True, False]
2 Indian domains
['ravi(at)mail.com', None]

na=False treats missing values as "does not match"; without it the result contains missing values and cannot be used directly as a filter.

Standardising labels#

The same category is often written several ways. Clean the text, then map the variants to one label:

Python
import pandas as pd

payment = pd.Series(["UPI", "upi ", "Card", "credit card", "CASH", "Debit Card", "G-Pay"])
cleaned = payment.str.strip().str.lower()
standard = cleaned.replace({
    "credit card": "card",
    "debit card": "card",
    "g-pay": "upi",
})
print(standard.value_counts())
Output
upi     3
card    3
cash    1
Name: count, dtype: int64

After cleaning, seven messy values reduce to three real payment methods.

Common mistakes#

Practice#

Practice exercise

Product codes look like [" ab-101 ", "CD-202", "ab-303", " EF-404"]. Clean them to uppercase without spaces, then create a prefix column (the letters) and a number column (as integers). Print the table and how many codes start with "AB".

Show hint

str.strip().str.upper(), then str.split("-", expand=True). Convert the number part with astype(int).

Show solution
Python
import pandas as pd

codes = pd.Series([" ab-101 ", "CD-202", "ab-303", " EF-404"]).str.strip().str.upper()
table = codes.str.split("-", expand=True)
table.columns = ["prefix", "number"]
table["number"] = table["number"].astype(int)
table.insert(0, "code", codes)
print(table)
print(codes.str.startswith("AB").sum())
Output
     code prefix  number
0  AB-101     AB     101
1  CD-202     CD     202
2  AB-303     AB     303
3  EF-404     EF     404
2

Key takeaway#

Next: Duplicates and Outliers.

Finished this lesson?

Related lessons

Python for Datapandas Data Types: astype, to_numeric, to_datetime and categoryFix column types in pandas: check dtypes, convert with astype, clean numbers stored as text with to_numeric, parse dates with to_datetime, use the category type to save memory, and nullable integer and string types.Intermediate9 min readPython for DataFind and Remove Duplicates and Outliers in pandasDetect and remove duplicate rows with duplicated and drop_duplicates, keep the latest record per customer, and find outliers with the IQR rule and z-scores, then decide whether to remove, cap or keep them.Intermediate9 min readPythonPython Regular Expressions (re): Search, Match, Extract and Replace TextLearn Python regular expressions with the re module: pattern syntax, findall, search, match, groups and named groups, sub and split, flags, validating emails and phone numbers, and greedy vs lazy matching.Intermediate6 min readPythonPython Strings: Indexing, Slicing, Methods and f-stringsA complete beginner guide to Python strings: creating strings, indexing and slicing, immutability, essential string methods (strip, split, join, replace, find), f-string formatting, and checking content.Beginner8 min read