Chunking & Ingestion
Ingestion transforms raw documents into searchable chunks with embeddings. Chunk size, overlap, and metadata directly affect RAG retrieval quality.
Recipe
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(documents)When to reach for this: indexing PDFs, wikis, code repos, or any document corpus for RAG.
Working Example
"""chunking_ingestion.py - clean, chunk, embed, and index documents."""
from __future__ import annotations
import re
import chromadb
from openai import OpenAI
client = OpenAI()
RAW_DOCS = [
{"id": "doc1", "source": "handbook", "text": "# Python Testing\n\nUse pytest for unit tests. " * 20},
{"id": "doc2", "source": "handbook", "text": "# FastAPI\n\nBuild APIs with type hints. " * 20},
]
def clean_text(text: str) -> str:
text = re.sub(r"\s+", " ", text).strip()
return text
def chunk_text(text: str, size: int = 400, overlap: int = 80) -> list[str]:
chunks = []
start = 0
while start < len(text):
chunks.append(text[start:start + size])
start += size - overlap
return chunks
chroma = chromadb.Client()
collection = chroma.get_or_create_collection("handbook")
for doc in RAW_DOCS:
cleaned = clean_text(doc["text"])
for i, chunk in enumerate(chunk_text(cleaned)):
chunk_id = f"{doc['id']}_chunk_{i}"
emb = client.embeddings.create(model="text-embedding-3-small", input=[chunk])
collection.add(
ids=[chunk_id],
documents=[chunk],
embeddings=[emb.data[0].embedding],
metadatas=[{"source": doc["source"], "doc_id": doc["id"], "chunk_index": i}],
)
print(f"indexed {collection.count()} chunks")What this demonstrates: text cleaning, fixed-size chunking with overlap, metadata per chunk, and embedding at ingest time.
Deep Dive
Chunk Size Guidelines
| Content Type | Chunk Size | Overlap |
|---|---|---|
| Technical prose | 500-1000 chars | 10-20% |
| Code | Function/class level | Minimal |
| Legal/contracts | 200-500 tokens | 20% |
| Chat logs | Full turn or paragraph | 0-10% |
Gotchas
- Chunks too large - retrieval returns irrelevant sections. Fix: smaller chunks with good overlap.
- Chunks too small - lose context for generation. Fix: increase size or add parent-document retrieval.
- No metadata - cannot filter or cite sources. Fix: store source, page, title with every chunk.
- Dirty HTML - navigation and ads pollute embeddings. Fix: strip boilerplate before chunking.
- Re-embedding unchanged docs - wastes API calls. Fix: hash content; skip if unchanged.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Fixed-size chunks | General prose | Structured docs with clear sections |
| Recursive splitter | Mixed content | Code with strict boundaries |
| Semantic chunking | Variable section lengths | Simple, uniform documents |
| Parent-child chunks | Need both precision and context | Small corpora |
FAQs
What chunk size should I start with?
- 500-1000 characters for general text.
- Evaluate retrieval recall on 20 test queries.
- Adjust based on whether chunks contain enough context.
How do I chunk PDFs?
- Extract text with
pymupdforpdfplumber. - Preserve page numbers in metadata.
- Handle multi-column layouts carefully.
How do I chunk code?
- Split by function/class boundaries using AST parsing.
- Include file path and language in metadata.
Should I chunk at ingest or query time?
- Always at ingest - query time only embeds the question.
How do I handle document updates?
- Delete old chunks by doc_id; re-ingest changed sections.
- Track content hash to detect changes.
What is parent-document retrieval?
- Store small chunks for search; retrieve parent section for generation.
- Best of precision and context.
How do I ingest markdown?
- Split on headers (
#,##) for semantic sections. - Preserve heading hierarchy in metadata.
How many chunks per document?
- Depends on document length.
- Monitor average chunk count per source type.
Should I deduplicate chunks?
- Yes - near-duplicate chunks waste index space.
- Hash normalized text; skip duplicates.
How do I batch embedding at ingest?
- Embed 100-500 chunks per API call.
- OpenAI accepts up to 2048 inputs per request.
What metadata should I store?
- source, doc_id, title, page, section, created_at, content_hash.
How do I test chunking quality?
- Manual review of 20 random chunks.
- Check retrieval recall on labeled query-chunk pairs.
Related
- RAG Basics - end-to-end pipeline
- Embeddings & Similarity - embedding models
- Vector Databases - storage
- Retrieval & Re-ranking - search quality
- LlamaIndex - data framework
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.