LLMs Best Practices

Rules for building reliable, safe, and cost-effective LLM applications. Walk this list during design review and before production launch.

How to Use This List

• Rules A-B cover security and API hygiene.
• Rules C-D govern prompt quality and output handling.
• Rules E-F cover cost, evaluation, and operations.
• Revisit when adding a new model, provider, or feature.

A - Security & Keys

• Load API keys from environment variables. Never commit keys to git; use secrets managers in production.
• Rotate keys if exposed. Treat accidental commits as incidents - revoke and reissue immediately.
• Sanitize user input before including in prompts. Use delimiters; never concatenate user input into system prompts.
• Validate model outputs before execution. Never eval() or exec() LLM-generated code without sandboxing.
• Log prompt hashes, not full prompts. Protect user data in logs; log enough for debugging.

B - API Usage

• Set max_tokens on every request. Prevent runaway generation and cost surprises.
• Handle rate limits with exponential backoff. Catch 429 errors; use semaphores for concurrency.
• Use the right model tier. Mini/haiku for simple tasks; full/sonnet only when needed.
• Set timeouts on API calls. Prevent hung requests from blocking your application.
• Use async clients in web servers. AsyncOpenAI in FastAPI; never block the event loop.

C - Prompt Quality

• Write specific system prompts. Role, constraints, output format, and failure behavior.
• Use temperature=0 for extraction and classification. Reserve higher temperature for creative tasks.
• Ground answers with retrieved context (RAG). Instruct "say I don't know" when context is insufficient.
• Version prompts in git. Track prompt changes like code; A/B test variants.
• Iterate prompts before fine-tuning. Prompt engineering is cheaper and faster to iterate.

D - Output Handling

• Validate structured output with Pydantic. Never trust raw JSON from the model.
• Retry on validation failure (max 3). Include the validation error in the retry prompt.
• Handle null content and tool_calls. Branch on finish_reason and response type.
• Add confidence scores or human review for high-stakes fields. Model output is probabilistic, not guaranteed.
• Stream to users for long responses. Improve perceived latency with token streaming.

E - Cost & Performance

• Log token usage on every call. Track response.usage per feature for cost attribution.
• Trim conversation history. Summarize or drop old turns to stay within context limits.
• Cache repeated queries. Hash prompt + model; return cached response when identical.
• Batch offline work. Use Batch API for bulk embedding and evaluation.
• Consider local models for high-volume simple tasks. Ollama/vLLM when API cost exceeds GPU cost.

F - Evaluation & Safety

• Build an eval set before deploying. 20-50 labeled input/expected-output pairs minimum.
• Measure accuracy, not vibes. Automated metrics on the eval set for every prompt change.
• Test edge cases. Empty input, very long input, adversarial prompts, non-English text.
• Implement content filtering. Handle refusals, policy violations, and harmful outputs gracefully.
• Monitor production quality. Track user feedback, error rates, and output validation failures.

FAQs

Related

• Prompt Engineering - prompt patterns
• Structured Output - output validation
• Tokens, Cost & Rate Limits - budgeting
• Evaluation & Guardrails - quality measurement
• AI Agents Best Practices - agent-specific rules

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.