Python course menu

Python Web Scraping With requests and BeautifulSoup (Beginner Guide)

Learn web scraping in Python: download pages with requests, parse HTML with BeautifulSoup, find elements with find and CSS selectors, extract text and attributes, follow pagination, and scrape responsibly with robots.txt.

Intermediate8 min readUpdated October 23, 2026

On this page

Setup
A quick look at HTML
Download and parse a page
CSS selectors: select and select_one
Extracting text, attributes and numbers
Following pagination
Checking robots.txt
Limitations
Common mistakes
Practice
Key takeaway

Web scraping means extracting information from web pages automatically: prices from a shop, headlines from a news page, tables from a government site. Python's requests downloads the page, and BeautifulSoup turns the HTML into something you can search.

Setup#

Terminal

python -m pip install requests beautifulsoup4

The examples use quotes.toscrape.com, a site built specifically for practising web scraping.

A quick look at HTML#

Web pages are made of nested elements with tags, attributes and text:

HTML

<div class="quote">
  <span class="text">“The world as we have created it…”</span>
  <small class="author">Albert Einstein</small>
  <a class="tag" href="/tag/change/">change</a>
</div>

Here div, span, small and a are tags; class and href are attributes. Your browser's Inspect tool (right-click → Inspect) shows the HTML behind any part of a page, which is how you find what to target.

Download and parse a page#

Python

import requests
from bs4 import BeautifulSoup

r = requests.get("https://quotes.toscrape.com/", timeout=10)
soup = BeautifulSoup(r.text, "html.parser")

print(soup.title.string)

first = soup.find("div", class_="quote")
print(first.find("span", class_="text").get_text()[:60])
print(first.find("small", class_="author").get_text())

Output

Quotes to Scrape
“The world as we have created it is a process of our thinkin
Albert Einstein

soup.find(tag, class_=...) returns the first matching element (note the underscore: class is a Python keyword).
.get_text() returns the visible text inside an element.

CSS selectors: select and select_one#

CSS selectors are a compact, powerful way to target elements:

Python

quotes = soup.select("div.quote")
print(len(quotes))

for q in quotes[:3]:
    author = q.select_one("small.author").get_text()
    tags = [t.get_text() for t in q.select("a.tag")]
    print(author, tags)

Output

10
Albert Einstein ['change', 'deep-thoughts', 'thinking', 'world']
J.K. Rowling ['abilities', 'choices']
Albert Einstein ['inspirational', 'life', 'live', 'miracle', 'miracles']

Selector	Matches
`div`	every `div`
`.price`	elements with class `price`
`#products`	the element with id `products`
`div.quote`	`div`s with class `quote`
`ul li`	`li` elements inside a `ul`
`a[href]`	links that have an `href` attribute

Extracting text, attributes and numbers#

Practise on a small HTML string; it is fast and does not touch any website:

Python

from bs4 import BeautifulSoup

html = """
<h1>Products</h1>
<ul id="products">
  <li class="item" data-sku="P1"><span class="name">Laptop</span> <span class="price">₹55,000</span></li>
  <li class="item" data-sku="P2"><span class="name">Headphones</span> <span class="price">₹2,500</span></li>
  <li class="item sold-out" data-sku="P3"><span class="name">Monitor</span> <span class="price">₹12,000</span></li>
</ul>
<a href="/page/2">Next</a>
"""

soup = BeautifulSoup(html, "html.parser")
print(soup.h1.get_text())

for li in soup.select("li.item"):
    name = li.select_one(".name").get_text()
    price = int(li.select_one(".price").get_text().replace("₹", "").replace(",", ""))
    sold_out = "sold-out" in li["class"]
    print(li["data-sku"], name, price, sold_out)

print(soup.find("a")["href"])

Output

Products
P1 Laptop 55000 False
P2 Headphones 2500 False
P3 Monitor 12000 True
/page/2

element["attr"] reads an attribute; class is returned as a list of class names.
Text from web pages is always a string: clean and convert numbers yourself.

Following pagination#

Many sites spread content over several pages. Follow the "Next" link until there is none (here limited to 2 pages to stay polite):

Python

import time
from urllib.parse import urljoin

base = "https://quotes.toscrape.com/"
url = base
authors = set()
pages = 0

while url and pages < 2:
    soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser")
    authors.update(a.get_text() for a in soup.select("small.author"))
    next_link = soup.select_one("li.next a")
    url = urljoin(base, next_link["href"]) if next_link else None
    pages += 1
    time.sleep(1)          # be polite between requests

print(pages, len(authors), sorted(authors)[:4])

Output

2 15 ['Albert Einstein', 'Allen Saunders', 'André Gide', 'Bob Marley']

urljoin turns relative links such as /page/2/ into full URLs.

Checking robots.txt#

Python

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://quotes.toscrape.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://quotes.toscrape.com/page/2/"))

Output

True

Limitations#

Pages that build their content with JavaScript may arrive "empty" through requests. Check whether the site has an underlying API (visible in the browser's Network tab), or use a browser-automation tool such as Playwright.
Sites change their HTML, which breaks scrapers. Keep selectors simple and expect to update them.

Common mistakes#

Practice#

Practice exercise

From the first page of quotes.toscrape.com, count how often each tag appears and print the three most common.

Show hint

soup.select("div.quote a.tag") finds every tag link. Feed their text into collections.Counter and call most_common(3).

Show solution

Python

import requests
from bs4 import BeautifulSoup
from collections import Counter

soup = BeautifulSoup(requests.get("https://quotes.toscrape.com/", timeout=10).text,
                     "html.parser")
tags = Counter(t.get_text() for t in soup.select("div.quote a.tag"))
print(tags.most_common(3))

Output

[('inspirational', 3), ('life', 2), ('humor', 2)]

This was the result when the lesson was written; a live website's content can change.

Key takeaway#

Finished this lesson?

PythonWorking With APIs in Python: requests, JSON, Parameters and ErrorsLearn to use web APIs in Python with the requests library: send GET and POST requests, read JSON responses, pass query parameters, check status codes, handle timeouts and errors, and use API keys safely.Intermediate8 min read PythonPython Regular Expressions (re): Search, Match, Extract and Replace TextLearn Python regular expressions with the re module: pattern syntax, findall, search, match, groups and named groups, sub and split, flags, validating emails and phone numbers, and greedy vs lazy matching.Intermediate6 min read PythonPython File Handling: Read, Write and Append Files (with open and pathlib)Learn file handling in Python: open files with the with statement, read whole files or line by line, write and append, file modes, encodings, pathlib paths and folders, CSV files, and handling missing files.Intermediate7 min read PythonAutomate Boring Tasks With Python: Files, Folders, CSV Reports and BackupsPractical Python automation for beginners: organise a messy downloads folder by file type, batch rename files, turn CSV data into a summary report, zip backups, scan log files, and schedule scripts to run automatically.Intermediate8 min read

Setup#

A quick look at HTML#

Download and parse a page#

CSS selectors: select and select_one#

Extracting text, attributes and numbers#

Following pagination#

Checking robots.txt#

Limitations#

Common mistakes#

Practice#

Key takeaway#

Related lessons