Lessons-Learned Catalog
A catalog of recurring Python production mistakes drawn from case studies and incident themes - with prevention patterns so teams stop paying the same interest on tech debt.
Recipe
Quick-reference recipe card - copy-paste ready.
## Lesson template
**ID:** LL-014
**Pattern:** Async route calls sync ORM
**Impact:** p95 outage, low CPU
**Prevention:** AST CI check + asyncio.to_thread for CPU
**See:** Before/After Async MigrationWhen to reach for this:
- Onboarding senior hires to fleet culture
- Quarterly ops review
- Post-mortem action consolidation
- Architecture review checklist input
Working Example
### Web & API
| ID | Lesson | Prevention |
| LL-001 | Mutable default args in hot path | pytest lint; code review checklist |
| LL-002 | Missing tenant_id filter on query | repository requires tenant context |
| LL-003 | Webhook without idempotency | Idempotency-Key table |
| LL-004 | Sync sleep in async middleware | scripts/check_async_blocking.py |
### Data & Migrations
| LL-010 | Index migration without CONCURRENTLY | migration linter + row count gate |
| LL-011 | Staging 1% row count hides locks | large table seed script |
| LL-012 | Non-idempotent ETL append | MERGE on natural key |
### ML & Batch
| LL-020 | Training-serving skew | shared features package |
| LL-021 | No eval gate on promote | min AUC in train CI job |
| LL-022 | Notebook creds in git | secrets via orchestrator only |
### Ops & Delivery
| LL-030 | API rolled back, workers not paused | deploy-order.md mandatory |
| LL-031 | :latest prod tag | immutable SHA tags |
| LL-032 | Alert on ERROR log count | SLO burn alerts |"""lessons_registry.py - optional machine-readable index."""
LESSONS = {
"LL-004": {
"title": "Sync sleep in async middleware",
"guardrail": "scripts/check_async_blocking.py",
"doc": "./before-after-sync-to-async-migration/before-after-sync-to-async-migration.md",
},
}What this demonstrates:
- Lessons grouped for scanning during design review
- Each row points to concrete prevention
- IDs stable for post-mortem cross-reference
- Optional registry links docs for onboarding bots
Deep Dive
How It Works
- Sourced from incidents - Post-mortem themes become LL entries within two weeks.
- Owner - Platform rotates quarterly catalog curator.
- Onboarding - New hires acknowledge top 10 in first month.
- Review - Retire lessons when guardrail universal in scaffold template.
- Cross-link - Case studies and troubleshooting pages reference LL IDs.
Top 10 for New Hires
- LL-004 - async blocking
- LL-010 - migration locks
- LL-030 - worker/API order
- LL-003 - webhook idempotency
- LL-020 - ML skew
- LL-002 - tenant leak
- LL-031 - image tags
- LL-012 - ETL idempotency
- LL-001 - mutable defaults
- LL-032 - alert hygiene
Python Notes
## Post-mortem action → catalog
Action G2 "async CI check" closes → add LL-004 if not presentGotchas
- Catalog too long - Nobody reads. Fix: Top 10 short list + full table linked.
- Lessons without guardrails - Philosophy only. Fix: Every entry has enforceable check or runbook.
- Stale lessons - Python 2 era advice. Fix: Quarterly curator pass tied to Python 3.14 standards.
- Blameful wording - "Dev forgot." Fix: System pattern language.
- Duplicate post-mortem text - Maintenance burden. Fix: ID reference instead of copy-paste.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Wiki only | Quick start | Drift from repo |
| Incident database only | Rich search | No onboarding narrative |
| Lint rules without catalog | Tiny team | Need human context why |
| External blameless blog | Marketing | Internal operational detail |
FAQs
Who adds entries?
Post-mortem owner proposes; platform curator approves format.
How many lessons?
Grow organically; top 10 list stays stable; full catalog may reach 50+.
Link to ADRs?
Yes - ADR references LL ID when decision triggered by incident pattern.
LLM-specific lessons?
Add eval drift, prompt injection guardrails as ML section grows.
Customer-facing?
Internal only; exec summaries separate.
Automate onboarding ack?
Checkbox in onboarding ticket linking to top 10 doc.
Duplicate incidents?
Increment occurrence note; strengthen guardrail priority.
Retire when?
Scaffold template enforces prevention by default for new services.
Search?
grep LL- IDs in repo; optional JSON registry for portal.
Related case studies?
Each reference/before-after doc contributes 2-3 lessons on publish.
Related
- Turning Incidents into Guardrails - close the loop
- Before/After: Sync to Async Migration - LL-004 source
- Before/After: Django Monolith - coupling lessons
- Common Production Failure Modes - pattern detail
- Governance Best Practices - standards
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.