Encoding & Unicode Bugs

Text bugs appear when bytes are decoded with the wrong codec, str and bytes mix without an explicit boundary, or visually identical Unicode strings compare unequal due to missing normalization.

Recipe

Quick-reference recipe card - copy-paste ready.

# Always label bytes; decode at the boundary
raw: bytes = b"\xc3\xa9"  # UTF-8 for é
text: str = raw.decode("utf-8")
 
# Writing text
path.write_text(text, encoding="utf-8")
path.write_bytes(raw)

When to reach for this:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff
Concatenating str + bytes raises TypeError
User search misses matches for accented characters
Logs show mojibake like Ã© instead of é

Working Example

import unicodedata
from pathlib import Path
 
# --- bug: implicit decoding ---
def read_config_legacy(path: str) -> str:
    with open(path) as f:  # text mode without encoding= on Windows/locale edge cases
        return f.read()
 
# --- fix: explicit UTF-8 ---
def read_config(path: str) -> str:
    return Path(path).read_text(encoding="utf-8")
 
# --- bug: mixing str and bytes ---
def broken_join(prefix: str, payload: bytes) -> bytes:
    return prefix + payload  # TypeError
 
# --- fix: encode at boundary ---
def fixed_join(prefix: str, payload: bytes) -> bytes:
    return prefix.encode("utf-8") + payload
 
# --- bug: comparison without normalization ---
def usernames_equal(a: str, b: str) -> bool:
    return a == b
 
# --- fix: NFC normalization for user identifiers ---
def usernames_equal_normalized(a: str, b: str) -> bool:
    return unicodedata.normalize("NFC", a.casefold()) == unicodedata.normalize("NFC", b.casefold())
 
# --- handling unknown bytes safely ---
def decode_best_effort(raw: bytes) -> str:
    return raw.decode("utf-8", errors="replace")
 
# demo
assert decode_best_effort(b"ok\xc3\xa9") == "oké"
assert usernames_equal_normalized("café", "café")  # visually equal forms

What this demonstrates:

Text files open with explicit encoding="utf-8"
Encode str to bytes before binary concatenation
unicodedata.normalize plus casefold for robust string equality
errors="replace" for logging dirty data without crashing

Deep Dive

How It Works

str is Unicode code points; bytes is raw octets
Encodings (UTF-8, Latin-1) map bytes ↔ str at IO boundaries
Python 3 source is UTF-8 by default (PEP 3120)
Normalization forms (NFC, NFD) align canonically equivalent sequences

Common Failure Points

Location	Mistake	Fix
`open()`	default locale encoding	`encoding="utf-8"`
HTTP	wrong `Content-Type` charset	decode per header
JSON	`json.dumps` to bytes manually	use `str` JSON, encode once
Filenames	undecoded bytes paths	`os.fsdecode` / pathlib
DB	legacy Latin-1 columns	migrate to UTF-8 or decode explicitly

Python Notes

# subprocess: text mode with explicit encoding (3.14)
import subprocess
subprocess.run(["echo", "hi"], capture_output=True, text=True, encoding="utf-8")
 
# sqlite stores TEXT as Unicode str when using text factory correctly

Gotchas

Double encoding - UTF-8 bytes decoded as Latin-1 then saved as UTF-8 produces mojibake. Fix: identify original bytes; re-decode from source.
chardet guessing in production - wrong guess corrupts data silently. Fix: fix upstream encoding; use errors="strict" on ingest.
Normalizing passwords - changes user input. Fix: normalize identifiers/display, not secrets.
Comparing hash digests of strings without encoding - hash(s) not for crypto; hashlib needs .encode("utf-8").
Emoji and legacy DB indexes - collation length limits break storage. Fix: UTF-8 MB4 in MySQL/Postgres config.
Reading binary files as text - PDF/PNG opened in text mode. Fix: binary mode + parse format.

Alternatives

Alternative	Use When	Don't Use When
UTF-8 everywhere	modern services	mandated legacy encodings
`errors="replace"`	log ingestion	financial/legal raw records
IDNA/punycode	domain names	general text storage
`bytes` end-to-end	crypto/protocol work	user-facing text without decode plan

FAQs

Should I use utf-8-sig?

Use for Excel/Windows files with BOM. Most APIs and JSON use plain UTF-8 without BOM.

What is the difference between NFC and NFD?

NFC composes characters (é as one code point); NFD decomposes (e + combining accent). Pick one form at storage and normalize on input.

How do I fix Latin-1 database data?

Identify columns, migrate schema to UTF-8, convert bytes with known source encoding in a maintenance window, verify with checksum samples.

Does requests handle encoding automatically?

response.text guesses from headers; set response.encoding or use response.content.decode("utf-8") explicitly when headers lie.

Why casefold instead of lower()?

casefold handles more Unicode special cases (e.g. German ß) for case-insensitive comparison of user-visible strings.

Are regexes Unicode-aware?

Use re.UNICODE default in Python 3; add (?u) or character classes like \w knowing they match Unicode letters.

How do pathlib paths handle Unicode?

Path uses str on Unix and Unicode APIs on Windows. Avoid passing undecoded bytes to open on Windows.

What about base64 and text?

Base64 encodes bytes. Decode base64 to bytes, then decode bytes to str with the correct text codec.

Can JSON contain non-UTF-8?

JSON is Unicode; json.dumps returns str. Encode to UTF-8 bytes only at wire/storage boundary.

How do I debug mojibake?

Inspect raw bytes with repr(raw) and try correct decode. Compare expected UTF-8 hex (e.g. c3 a9 for é).

Does Python 3.14 change encoding defaults?

UTF-8 mode (PYTHONUTF8=1) and explicit encoding= remain best practice on all supported versions including 3.14.0.

When is errors=ignore OK?

Rarely. Acceptable for lossy display of third-party logs, never for data you persist or bill on.

Migrating a Legacy Python 2/Django/Flask App - Py2 unicode migration
Floating-Point & Numeric Bugs - different class of data bugs
Data Pipeline Defects - CSV encoding in ETL
Debugging Refactor Snippets - bytes/str fixes
Debugging Tools - inspect raw bytes in pdb

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Encoding & Unicode Bugs

Text bugs appear when bytes are decoded with the wrong codec, str and bytes mix without an explicit boundary, or visually identical Unicode strings compare unequal due to missing normalization.

Recipe

Quick-reference recipe card - copy-paste ready.

# Always label bytes; decode at the boundary
raw: bytes = b"\xc3\xa9"  # UTF-8 for é
text: str = raw.decode("utf-8")
 
# Writing text
path.write_text(text, encoding="utf-8")
path.write_bytes(raw)

When to reach for this:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff
Concatenating str + bytes raises TypeError
User search misses matches for accented characters
Logs show mojibake like Ã© instead of é

Working Example

import unicodedata
from pathlib import Path
 
# --- bug: implicit decoding ---
def read_config_legacy(path: str) -> str:
    with open(path) as f:  # text mode without encoding= on Windows/locale edge cases
        return f.read()
 
# --- fix: explicit UTF-8 ---
def read_config(path: str) -> str:
    return Path(path).read_text(encoding="utf-8")
 
# --- bug: mixing str and bytes ---
def broken_join(prefix: str, payload: bytes) -> bytes:
    return prefix + payload  # TypeError
 
# --- fix: encode at boundary ---
def fixed_join(prefix: str, payload: bytes) -> bytes:
    return prefix.encode("utf-8") + payload
 
# --- bug: comparison without normalization ---
def usernames_equal(a: str, b: str) -> bool:
    return a == b
 
# --- fix: NFC normalization for user identifiers ---
def usernames_equal_normalized(a: str, b: str) -> bool:
    return unicodedata.normalize("NFC", a.casefold()) == unicodedata.normalize("NFC", b.casefold())
 
# --- handling unknown bytes safely ---
def decode_best_effort(raw: bytes) -> str:
    return raw.decode("utf-8", errors="replace")
 
# demo
assert decode_best_effort(b"ok\xc3\xa9") == "oké"
assert usernames_equal_normalized("café", "café")  # visually equal forms

What this demonstrates:

Text files open with explicit encoding="utf-8"
Encode str to bytes before binary concatenation
unicodedata.normalize plus casefold for robust string equality
errors="replace" for logging dirty data without crashing

Deep Dive

How It Works

str is Unicode code points; bytes is raw octets
Encodings (UTF-8, Latin-1) map bytes ↔ str at IO boundaries
Python 3 source is UTF-8 by default (PEP 3120)
Normalization forms (NFC, NFD) align canonically equivalent sequences

Common Failure Points

Location	Mistake	Fix
`open()`	default locale encoding	`encoding="utf-8"`
HTTP	wrong `Content-Type` charset	decode per header
JSON	`json.dumps` to bytes manually	use `str` JSON, encode once
Filenames	undecoded bytes paths	`os.fsdecode` / pathlib
DB	legacy Latin-1 columns	migrate to UTF-8 or decode explicitly

Python Notes

# subprocess: text mode with explicit encoding (3.14)
import subprocess
subprocess.run(["echo", "hi"], capture_output=True, text=True, encoding="utf-8")
 
# sqlite stores TEXT as Unicode str when using text factory correctly

Gotchas

Double encoding - UTF-8 bytes decoded as Latin-1 then saved as UTF-8 produces mojibake. Fix: identify original bytes; re-decode from source.
chardet guessing in production - wrong guess corrupts data silently. Fix: fix upstream encoding; use errors="strict" on ingest.
Normalizing passwords - changes user input. Fix: normalize identifiers/display, not secrets.
Comparing hash digests of strings without encoding - hash(s) not for crypto; hashlib needs .encode("utf-8").
Emoji and legacy DB indexes - collation length limits break storage. Fix: UTF-8 MB4 in MySQL/Postgres config.
Reading binary files as text - PDF/PNG opened in text mode. Fix: binary mode + parse format.

Alternatives

Alternative	Use When	Don't Use When
UTF-8 everywhere	modern services	mandated legacy encodings
`errors="replace"`	log ingestion	financial/legal raw records
IDNA/punycode	domain names	general text storage
`bytes` end-to-end	crypto/protocol work	user-facing text without decode plan

FAQs

Should I use utf-8-sig?

Use for Excel/Windows files with BOM. Most APIs and JSON use plain UTF-8 without BOM.

What is the difference between NFC and NFD?

NFC composes characters (é as one code point); NFD decomposes (e + combining accent). Pick one form at storage and normalize on input.

How do I fix Latin-1 database data?

Identify columns, migrate schema to UTF-8, convert bytes with known source encoding in a maintenance window, verify with checksum samples.

Does requests handle encoding automatically?

response.text guesses from headers; set response.encoding or use response.content.decode("utf-8") explicitly when headers lie.

Why casefold instead of lower()?

casefold handles more Unicode special cases (e.g. German ß) for case-insensitive comparison of user-visible strings.

Are regexes Unicode-aware?

Use re.UNICODE default in Python 3; add (?u) or character classes like \w knowing they match Unicode letters.

How do pathlib paths handle Unicode?

Path uses str on Unix and Unicode APIs on Windows. Avoid passing undecoded bytes to open on Windows.

What about base64 and text?

Base64 encodes bytes. Decode base64 to bytes, then decode bytes to str with the correct text codec.

Can JSON contain non-UTF-8?

JSON is Unicode; json.dumps returns str. Encode to UTF-8 bytes only at wire/storage boundary.

How do I debug mojibake?

Inspect raw bytes with repr(raw) and try correct decode. Compare expected UTF-8 hex (e.g. c3 a9 for é).

Does Python 3.14 change encoding defaults?

UTF-8 mode (PYTHONUTF8=1) and explicit encoding= remain best practice on all supported versions including 3.14.0.

When is errors=ignore OK?

Rarely. Acceptable for lossy display of third-party logs, never for data you persist or bill on.

Migrating a Legacy Python 2/Django/Flask App - Py2 unicode migration
Floating-Point & Numeric Bugs - different class of data bugs
Data Pipeline Defects - CSV encoding in ETL
Debugging Refactor Snippets - bytes/str fixes
Debugging Tools - inspect raw bytes in pdb

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Encoding & Unicode Bugs

Search Documentation

Encoding & Unicode Bugs