AI Agents Best Practices

Rules for building grounded, observable, and controllable RAG systems and agents.

How to Use This List

• Rules A-B cover retrieval and generation quality.
• Rules C-D govern agent safety and tool use.
• Rules E-F cover operations and evaluation.

A - Retrieval Quality

• Chunk with appropriate size and overlap. 500-1000 chars for prose; evaluate recall@k.
• Store metadata with every chunk. source, doc_id, page, section for filtering and citation.
• Use hybrid search for production. Combine BM25 and dense retrieval; re-rank top candidates.
• Pin embedding model per index. Never mix models or dimensions in one collection.
• Return "I don't know" when retrieval is empty. Do not generate from irrelevant context.

B - Generation Quality

• Instruct the model to use context only. System prompt with explicit grounding rules.
• Cite sources in responses. Include chunk references for user verification.
• Validate structured outputs with Pydantic. Retry on validation failure (max 3).
• Separate retrieval and generation eval. Bad retrieval cannot be prompt-engineered away.
• Temperature=0 for factual QA. Higher temperature only for creative tasks.

C - Agent Safety

• Validate tool arguments before execution. Pydantic schemas on every tool.
• Limit tool permissions to minimum required. Read-only by default; approval for writes.
• Set max iterations on agent loops. 5-10 iterations prevents runaway tool calls.
• Sandbox code execution tools. Isolated containers with resource limits.
• Log every tool call. Name, args, result, latency for audit and debugging.

D - Observability

• Trace full RAG pipeline. Query, retrieved chunks, prompt, response, latency.
• Monitor retrieval recall in production. Sample queries with known relevant docs.
• Track user feedback. Thumbs up/down linked to query and retrieved context.
• Alert on faithfulness failures. LLM-as-judge or heuristic checks in CI.
• Version index and prompts together. Log index version with each response.

E - Architecture

• Start simple: raw SDK + vector DB. Add LangChain/LlamaIndex when complexity warrants.
• Use LangGraph for multi-step agents. Not manual while-loops in production.
• Cache frequent queries. Hash query + index version; return cached answers.
• Design for index updates. Upsert/delete by doc_id without full reindex.
• Separate ingest from query paths. Batch ingest offline; real-time query online.

F - Evaluation

• Build eval set before launch. 20+ labeled query-answer pairs minimum.
• Run eval in CI on every change. Prompt, model, chunking, or index changes.
• Test adversarial inputs. Prompt injection, empty queries, off-topic questions.
• Measure end-to-end latency. Retrieval + generation p50/p99 budgets.
• Human review before production. Spot-check 50 responses on eval set.

FAQs

Related

• RAG Basics
• Evaluation & Guardrails
• Tool Use & Function Calling
• LLMs Best Practices
• Monitoring and Drift

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.