LLMs Best Practices
Rules for building reliable, safe, and cost-effective LLM applications. Walk this list during design review and before production launch.
How to Use This List
- Rules A-B cover security and API hygiene.
- Rules C-D govern prompt quality and output handling.
- Rules E-F cover cost, evaluation, and operations.
- Revisit when adding a new model, provider, or feature.
A - Security & Keys
- Load API keys from environment variables. Never commit keys to git; use secrets managers in production.
- Rotate keys if exposed. Treat accidental commits as incidents - revoke and reissue immediately.
- Sanitize user input before including in prompts. Use delimiters; never concatenate user input into system prompts.
- Validate model outputs before execution. Never
eval()orexec()LLM-generated code without sandboxing. - Log prompt hashes, not full prompts. Protect user data in logs; log enough for debugging.
B - API Usage
- Set max_tokens on every request. Prevent runaway generation and cost surprises.
- Handle rate limits with exponential backoff. Catch 429 errors; use semaphores for concurrency.
- Use the right model tier. Mini/haiku for simple tasks; full/sonnet only when needed.
- Set timeouts on API calls. Prevent hung requests from blocking your application.
- Use async clients in web servers.
AsyncOpenAIin FastAPI; never block the event loop.
C - Prompt Quality
- Write specific system prompts. Role, constraints, output format, and failure behavior.
- Use temperature=0 for extraction and classification. Reserve higher temperature for creative tasks.
- Ground answers with retrieved context (RAG). Instruct "say I don't know" when context is insufficient.
- Version prompts in git. Track prompt changes like code; A/B test variants.
- Iterate prompts before fine-tuning. Prompt engineering is cheaper and faster to iterate.
D - Output Handling
- Validate structured output with Pydantic. Never trust raw JSON from the model.
- Retry on validation failure (max 3). Include the validation error in the retry prompt.
- Handle null content and tool_calls. Branch on
finish_reasonand response type. - Add confidence scores or human review for high-stakes fields. Model output is probabilistic, not guaranteed.
- Stream to users for long responses. Improve perceived latency with token streaming.
E - Cost & Performance
- Log token usage on every call. Track
response.usageper feature for cost attribution. - Trim conversation history. Summarize or drop old turns to stay within context limits.
- Cache repeated queries. Hash prompt + model; return cached response when identical.
- Batch offline work. Use Batch API for bulk embedding and evaluation.
- Consider local models for high-volume simple tasks. Ollama/vLLM when API cost exceeds GPU cost.
F - Evaluation & Safety
- Build an eval set before deploying. 20-50 labeled input/expected-output pairs minimum.
- Measure accuracy, not vibes. Automated metrics on the eval set for every prompt change.
- Test edge cases. Empty input, very long input, adversarial prompts, non-English text.
- Implement content filtering. Handle refusals, policy violations, and harmful outputs gracefully.
- Monitor production quality. Track user feedback, error rates, and output validation failures.
FAQs
What is the number one LLM security mistake?
- Hardcoding API keys in source code.
- Keys get committed to git and scraped by bots.
- Use environment variables and secrets managers.
How do I prevent prompt injection?
- Separate system and user content clearly.
- Use XML delimiters for user input.
- Never follow instructions embedded in user data.
When should I use RAG vs fine-tuning?
- RAG: answers need current or private data.
- Fine-tuning: consistent format/style at high volume.
- Start with RAG + prompt engineering.
How do I test LLM features?
- Eval set with expected outputs.
- Mock API in unit tests; integration tests with recorded fixtures.
- Regression test on prompt changes.
What should I log?
- Model, token counts, latency, prompt version, validation result.
- Not full prompts if they contain user PII.
How do I handle model refusals?
- Check
finish_reasonand empty content. - Show a user-friendly fallback message.
- Log for review but do not retry indefinitely.
Should I use one provider or many?
- Start with one provider for simplicity.
- Add LiteLLM abstraction when you need fallback or cost optimization.
How do I set up a dev environment without API costs?
- Ollama for local development.
- Mock responses in unit tests.
- Separate dev API keys with spending limits.
What is a minimum viable eval?
- 20 examples covering happy path and edge cases.
- Automated scoring (exact match, F1, or LLM-as-judge).
- Run on every prompt or model change.
How do I review an LLM feature PR?
- Check key handling, output validation, error handling, and logging.
- Verify eval set exists and passes.
- Test with adversarial inputs.
When is it OK to execute LLM-generated code?
- Never directly in production without sandboxing.
- Use isolated containers with resource limits.
- Human review for anything touching infrastructure.
How do I document an LLM feature?
- Model, prompt version, expected behavior, known limitations.
- Eval accuracy, cost per request, and latency p50/p99.
Related
- Prompt Engineering - prompt patterns
- Structured Output - output validation
- Tokens, Cost & Rate Limits - budgeting
- Evaluation & Guardrails - quality measurement
- AI Agents Best Practices - agent-specific rules
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.