beautifulsoup4 & lxml

BeautifulSoup parses messy HTML/XML into a navigable tree - tags, attributes, text search. Pair it with lxml as the parser backend for speed and XPath when documents grow large or malformed.

Recipe

from bs4 import BeautifulSoup
 
html = '<div class="item"><a href="/p/1">Widget</a><span class="price">$9</span></div>'
soup = BeautifulSoup(html, "lxml")
for item in soup.select(".item"):
    name = item.find("a").get_text(strip=True)
    price = item.find(class_="price").get_text(strip=True)
    print(name, price)

When to reach for this:

• Extracting data from HTML exports or legacy portals without APIs
• Cleaning CMS HTML before storage or display
• Parsing RSS/Atom or SOAP XML responses
• Test fixtures that snapshot HTML fragments

Working Example

from __future__ import annotations
 
from dataclasses import dataclass
 
import httpx
from bs4 import BeautifulSoup
 
 
@dataclass
class Product:
    sku: str
    title: str
    price: str
 
 
def parse_catalog(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products: list[Product] = []
    for card in soup.select("article.product"):
        sku = card.get("data-sku")
        title_el = card.select_one("h2.title")
        price_el = card.select_one("span.price")
        if not sku or not title_el or not price_el:
            continue
        products.append(
            Product(
                sku=sku,
                title=title_el.get_text(strip=True),
                price=price_el.get_text(strip=True),
            )
        )
    return products
 
 
def fetch_catalog(url: str) -> list[Product]:
    with httpx.Client(timeout=10.0, headers={"User-Agent": "catalog-bot/1.0"}) as client:
        response = client.get(url)
        response.raise_for_status()
        return parse_catalog(response.text)
 
 
if __name__ == "__main__":
    sample = """
    <article class="product" data-sku="W1">
      <h2 class="title">Widget</h2><span class="price">$9.00</span>
    </article>
    """
    print(parse_catalog(sample))

What this demonstrates:

• CSS selectors via select / select_one
• Defensive parsing when nodes are missing
• get_text(strip=True) for normalized text
• Separation of HTTP fetch from parse logic for testing

Deep Dive

How It Works

• Parser backends - html.parser (stdlib), lxml (fast C), html5lib (most lenient).
• Navigation - .parent, .next_sibling, .find, .find_all, .select (CSS).
• lxml XPath - tree.xpath("//div[@class='item']") when CSS is awkward.
• Encoding - BeautifulSoup detects encoding from meta tags; pass from_encoding when wrong.

Parser Choice

Python Notes

# Parse XML with lxml directly for strict schemas
from lxml import etree
 
root = etree.fromstring(xml_bytes)
for node in root.xpath("//item[@id]"):
    print(node.get("id"), node.text)

Gotchas

• Scraping without permission - Legal and ToS risk. Fix: use official APIs; throttle requests.
• Brittle selectors - CSS classes change on redesign. Fix: prefer data-* attributes; contract tests on HTML fixtures.
• Not handling pagination - Incomplete datasets. Fix: follow rel=next or API cursor.
• Executing scraped scripts - XSS if re-displayed.raw HTML. Fix: extract text only or bleach sanitize.
• Memory on huge pages - Loading 50MB HTML. Fix: stream parse with lxml iterparse.
• Timezone in scraped dates - Ambiguous strings. Fix: parse to UTC at extraction boundary.

Alternatives

FAQs

Related

• httpx & requests - fetching HTML
• re - Regular Expressions - text cleanup after parse
• Data Pipeline Skill - ETL from web sources
• Input Validation - sanitizing content

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.