AI Agents Best Practices

Rules for building grounded, observable, and controllable RAG systems and agents.

How to Use This List

Rules A-B cover retrieval and generation quality.
Rules C-D govern agent safety and tool use.
Rules E-F cover operations and evaluation.

A - Retrieval Quality

Chunk with appropriate size and overlap. 500-1000 chars for prose; evaluate recall@k.
Store metadata with every chunk. source, doc_id, page, section for filtering and citation.
Use hybrid search for production. Combine BM25 and dense retrieval; re-rank top candidates.
Pin embedding model per index. Never mix models or dimensions in one collection.
Return "I don't know" when retrieval is empty. Do not generate from irrelevant context.

B - Generation Quality

Instruct the model to use context only. System prompt with explicit grounding rules.
Cite sources in responses. Include chunk references for user verification.
Validate structured outputs with Pydantic. Retry on validation failure (max 3).
Separate retrieval and generation eval. Bad retrieval cannot be prompt-engineered away.
Temperature=0 for factual QA. Higher temperature only for creative tasks.

C - Agent Safety

Validate tool arguments before execution. Pydantic schemas on every tool.
Limit tool permissions to minimum required. Read-only by default; approval for writes.
Set max iterations on agent loops. 5-10 iterations prevents runaway tool calls.
Sandbox code execution tools. Isolated containers with resource limits.
Log every tool call. Name, args, result, latency for audit and debugging.

D - Observability

Trace full RAG pipeline. Query, retrieved chunks, prompt, response, latency.
Monitor retrieval recall in production. Sample queries with known relevant docs.
Track user feedback. Thumbs up/down linked to query and retrieved context.
Alert on faithfulness failures. LLM-as-judge or heuristic checks in CI.
Version index and prompts together. Log index version with each response.

E - Architecture

Start simple: raw SDK + vector DB. Add LangChain/LlamaIndex when complexity warrants.
Use LangGraph for multi-step agents. Not manual while-loops in production.
Cache frequent queries. Hash query + index version; return cached answers.
Design for index updates. Upsert/delete by doc_id without full reindex.
Separate ingest from query paths. Batch ingest offline; real-time query online.

F - Evaluation

Build eval set before launch. 20+ labeled query-answer pairs minimum.
Run eval in CI on every change. Prompt, model, chunking, or index changes.
Test adversarial inputs. Prompt injection, empty queries, off-topic questions.
Measure end-to-end latency. Retrieval + generation p50/p99 budgets.
Human review before production. Spot-check 50 responses on eval set.

FAQs

Biggest RAG mistake?

Skipping retrieval eval and only tuning prompts.

How many tools for an agent?

5-10 focused tools with clear descriptions.

When to use MCP?

When tools are shared across multiple applications or IDEs.

How do I prevent hallucinations?

Ground with retrieved context; faithfulness checks; say "I don't know."

LangChain required?

No - start with raw SDK; add frameworks when needed.

How do I handle stale documents?

Content hash at ingest; re-embed on change; TTL on chunks.

Multi-tenant RAG?

Separate collections or tenant_id metadata filter per request.

Agent vs chain?

Chain for fixed steps; agent when the LLM decides which tools to use.

Production checklist?

Eval passes, guardrails active, logging on, rate limits set, fallback messages defined.

How to review agent PRs?

Check tool validation, iteration limits, logging, and eval coverage.

Cost control?

Cache answers; use cheaper models for retrieval/reranking; limit context size.

When to reindex?

On document changes, embedding model changes, or chunk strategy changes.

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

AI Agents Best Practices

Rules for building grounded, observable, and controllable RAG systems and agents.

How to Use This List

Rules A-B cover retrieval and generation quality.
Rules C-D govern agent safety and tool use.
Rules E-F cover operations and evaluation.

A - Retrieval Quality

Chunk with appropriate size and overlap. 500-1000 chars for prose; evaluate recall@k.
Store metadata with every chunk. source, doc_id, page, section for filtering and citation.
Use hybrid search for production. Combine BM25 and dense retrieval; re-rank top candidates.
Pin embedding model per index. Never mix models or dimensions in one collection.
Return "I don't know" when retrieval is empty. Do not generate from irrelevant context.

B - Generation Quality

Instruct the model to use context only. System prompt with explicit grounding rules.
Cite sources in responses. Include chunk references for user verification.
Validate structured outputs with Pydantic. Retry on validation failure (max 3).
Separate retrieval and generation eval. Bad retrieval cannot be prompt-engineered away.
Temperature=0 for factual QA. Higher temperature only for creative tasks.

C - Agent Safety

Validate tool arguments before execution. Pydantic schemas on every tool.
Limit tool permissions to minimum required. Read-only by default; approval for writes.
Set max iterations on agent loops. 5-10 iterations prevents runaway tool calls.
Sandbox code execution tools. Isolated containers with resource limits.
Log every tool call. Name, args, result, latency for audit and debugging.

D - Observability

Trace full RAG pipeline. Query, retrieved chunks, prompt, response, latency.
Monitor retrieval recall in production. Sample queries with known relevant docs.
Track user feedback. Thumbs up/down linked to query and retrieved context.
Alert on faithfulness failures. LLM-as-judge or heuristic checks in CI.
Version index and prompts together. Log index version with each response.

E - Architecture

Start simple: raw SDK + vector DB. Add LangChain/LlamaIndex when complexity warrants.
Use LangGraph for multi-step agents. Not manual while-loops in production.
Cache frequent queries. Hash query + index version; return cached answers.
Design for index updates. Upsert/delete by doc_id without full reindex.
Separate ingest from query paths. Batch ingest offline; real-time query online.

F - Evaluation

Build eval set before launch. 20+ labeled query-answer pairs minimum.
Run eval in CI on every change. Prompt, model, chunking, or index changes.
Test adversarial inputs. Prompt injection, empty queries, off-topic questions.
Measure end-to-end latency. Retrieval + generation p50/p99 budgets.
Human review before production. Spot-check 50 responses on eval set.

FAQs

Biggest RAG mistake?

Skipping retrieval eval and only tuning prompts.

How many tools for an agent?

5-10 focused tools with clear descriptions.

When to use MCP?

When tools are shared across multiple applications or IDEs.

How do I prevent hallucinations?

Ground with retrieved context; faithfulness checks; say "I don't know."

LangChain required?

No - start with raw SDK; add frameworks when needed.

How do I handle stale documents?

Content hash at ingest; re-embed on change; TTL on chunks.

Multi-tenant RAG?

Separate collections or tenant_id metadata filter per request.

Agent vs chain?

Chain for fixed steps; agent when the LLM decides which tools to use.

Production checklist?

Eval passes, guardrails active, logging on, rate limits set, fallback messages defined.

How to review agent PRs?

Check tool validation, iteration limits, logging, and eval coverage.

Cost control?

Cache answers; use cheaper models for retrieval/reranking; limit context size.

When to reindex?

On document changes, embedding model changes, or chunk strategy changes.

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

AI Agents Best Practices

Search Documentation

AI Agents Best Practices