Machine Learning course menu

Text Classification in Python: Bag of Words and TF-IDF

Classify text with scikit-learn: tokenisation, bag-of-words counts, TF-IDF weighting, n-grams, a spam and sentiment classifier with logistic regression and Naive Bayes, reading the most important words, and the limits of word counts.

Intermediate10 min readUpdated October 24, 2026

On this page

Bag of words
TF-IDF
A sentiment classifier
Spam detection with Naive Bayes
Which words matter?
Limits of word counts
Common mistakes
Practice
Key takeaway

Text is everywhere: reviews, support tickets, emails, chat messages. Text classification assigns a label to each piece: spam or not, positive or negative, which department a ticket belongs to. Models need numbers, so the first step is turning text into numeric features.

Bag of words#

The simplest representation counts how often each word appears, ignoring word order: a bag of words.

Python

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the food was great", "the service was slow", "great food great service"]
vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(counts.toarray())

Output

['food' 'great' 'service' 'slow' 'the' 'was']
[[1 1 0 0 1 1]
 [0 0 1 1 1 1]
 [1 2 1 0 0 0]]

Each column is a word in the vocabulary, learned from the training text; each row counts the words in one document. Real vocabularies have tens of thousands of words, so the result is stored as a sparse matrix (mostly zeros, stored efficiently).

TF-IDF#

Common words such as "the" and "was" appear everywhere and say little. TF-IDF (term frequency × inverse document frequency) gives less weight to words that appear in many documents and more to distinctive ones:

Python

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["the food was great", "the service was slow", "great food great service"]
tfidf = TfidfVectorizer()
weights = tfidf.fit_transform(docs).toarray()
for word, w in zip(tfidf.get_feature_names_out(), weights[1]):
    print(f"{word:>8}: {w:.3f}")

Output

    food: 0.000
   great: 0.000
 service: 0.460
    slow: 0.605
     the: 0.460
     was: 0.460

In the second document, "slow" (which appears only there) gets the highest weight, while "the" and "was" (shared with another document) get less.

Useful options:

ngram_range=(1, 2) adds two-word phrases, so "not good" becomes a feature of its own;
stop_words="english" drops very common English words;
min_df=2 ignores words seen in fewer than two documents;
lowercase=True (the default) treats "Great" and "great" as the same word.

A sentiment classifier#

A small set of labelled product reviews (1 = positive):

Python

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

reviews = [
    ("great phone, battery lasts all day", 1), ("terrible battery, died in an hour", 0),
    ("love the camera and the screen", 1), ("screen cracked after one week", 0),
    ("excellent value, works perfectly", 1), ("stopped working, waste of money", 0),
    ("fast delivery and great quality", 1), ("poor quality, very disappointed", 0),
    ("really happy with this purchase", 1), ("not worth the price, returned it", 0),
    ("amazing sound and great battery", 1), ("awful sound, broke quickly", 0),
]
texts, labels = zip(*reviews)

model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())
model.fit(texts, labels)

new = ["great camera and battery", "waste of money, very poor", "the battery died"]
for text, p in zip(new, model.predict_proba(new)[:, 1]):
    print(f"{p:.2f}  {text}")

Output

0.59  great camera and battery
0.40  waste of money, very poor
0.48  the battery died

The pipeline learns the vocabulary and the classifier together, and applies the same transformation to new text. With only twelve examples the probabilities are cautious; real classifiers train on thousands of labelled texts.

Spam detection with Naive Bayes#

Multinomial Naive Bayes is a fast, classic baseline for word counts: it learns how often each word appears in each class.

Python

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

spam = ["win a free prize now", "claim your free cash reward", "urgent: you won a lottery",
        "free entry, text win to claim", "cheap loans, act now", "you have won a free holiday",
        "exclusive offer, claim your prize", "limited time, free gift inside"]
ham = ["are we meeting for lunch today", "please review the attached report", "see you at the station at 6",
       "can you send me the notes", "the meeting moved to friday", "happy birthday, have a great day",
       "dinner at home tonight?", "thanks for your help yesterday"]
texts = spam + ham
labels = [1] * len(spam) + [0] * len(ham)
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.25, random_state=0, stratify=labels)

model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), target_names=["ham", "spam"]))
print(model.predict(["free prize, claim now", "lunch at the station?"]))

Output

              precision    recall  f1-score   support

         ham       0.67      1.00      0.80         2
        spam       1.00      0.50      0.67         2

    accuracy                           0.75         4
   macro avg       0.83      0.75      0.73         4
weighted avg       0.83      0.75      0.73         4

[1 0]

With only four test messages, a single missed spam message drops spam recall to 0.50, a reminder that tiny test sets give jumpy scores. The two new messages are classified correctly (1 = spam).

Which words matter?#

A linear model's coefficients show which words push towards each class, a quick way to check the model learned something sensible:

Python

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

reviews = [
    ("great phone, battery lasts all day", 1), ("terrible battery, died in an hour", 0),
    ("love the camera and the screen", 1), ("screen cracked after one week", 0),
    ("excellent value, works perfectly", 1), ("stopped working, waste of money", 0),
    ("fast delivery and great quality", 1), ("poor quality, very disappointed", 0),
    ("really happy with this purchase", 1), ("not worth the price, returned it", 0),
    ("amazing sound and great battery", 1), ("awful sound, broke quickly", 0),
]
texts, labels = zip(*reviews)
tfidf = TfidfVectorizer()
clf = LogisticRegression().fit(tfidf.fit_transform(texts), labels)

words = tfidf.get_feature_names_out()
order = np.argsort(clf.coef_[0])
print("most negative:", words[order[:4]].tolist())
print("most positive:", words[order[-4:]].tolist())

Output

most negative: ['quickly', 'awful', 'broke', 'poor']
most positive: ['perfectly', 'amazing', 'and', 'great']

Most words make sense: "awful", "broke" and "poor" push towards negative; "great", "amazing" and "perfectly" towards positive. But "and" appears among the positive words too, only because it happens to occur in several positive reviews here. With little data, models pick up such accidents; more training data (or stop_words="english") removes them.

Limits of word counts#

Bag-of-words models ignore word order and meaning:

"not good" and "good" share the word "good" (n-grams help a little);
synonyms such as "awful" and "terrible" are unrelated columns, so the model must see each in training;
they cannot handle words never seen in training.

Embeddings (next lesson) and large language models capture meaning, so "awful" and "terrible" end up close together. Even so, TF-IDF with logistic regression remains a strong, fast, cheap baseline: always try it first.

Common mistakes#

Practice#

Practice exercise

Route support tickets to a team. Train a TF-IDF + logistic regression pipeline on the tickets below and predict the team for "I was charged twice this month" and "the app crashes when I log in".

Python

tickets = [("refund my payment", "billing"), ("invoice is wrong", "billing"), ("charged twice on my card", "billing"),
           ("update my billing address", "billing"), ("app crashes on startup", "tech"), ("cannot log in to my account", "tech"),
           ("error message when saving", "tech"), ("the page will not load", "tech")]

Show hint

Unzip into texts and labels, fit make_pipeline(TfidfVectorizer(), LogisticRegression()), and call predict with the two new tickets.

Show solution

Python

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

tickets = [("refund my payment", "billing"), ("invoice is wrong", "billing"), ("charged twice on my card", "billing"),
           ("update my billing address", "billing"), ("app crashes on startup", "tech"), ("cannot log in to my account", "tech"),
           ("error message when saving", "tech"), ("the page will not load", "tech")]
texts, teams = zip(*tickets)
model = make_pipeline(TfidfVectorizer(), LogisticRegression()).fit(texts, teams)
print(model.predict(["I was charged twice this month", "the app crashes when I log in"]))

Output

['billing' 'tech']

Key takeaway#

Next: Embeddings and Semantic Search.

Finished this lesson?

Machine LearningEmbeddings and Semantic Search Explained (with Python)Understand embeddings: representing words and documents as vectors, cosine similarity, building a small semantic search and recommendation engine, sentence embedding models, vector databases, and retrieval-augmented generation (RAG).Advanced8 min read Machine LearningLogistic Regression in Python: Classification with ProbabilitiesLearn logistic regression for classification: the sigmoid function, predicting probabilities with predict_proba, the decision threshold, interpreting coefficients as odds, multi-class classification, and scaling with a pipeline.Intermediate9 min read Machine LearningUsing LLM APIs in Python: Claude API Beginner GuideCall a large language model from Python: API keys, the Anthropic SDK, messages and system prompts, structured JSON output for classification and extraction, tool use, streaming, tokens and cost, and evaluating LLM outputs like any other model.Intermediate11 min read

Bag of words#

TF-IDF#

A sentiment classifier#

Spam detection with Naive Bayes#

Which words matter?#

Limits of word counts#

Common mistakes#

Practice#

Key takeaway#

Related lessons