Lessons-Learned Catalog

A catalog of recurring Python production mistakes drawn from case studies and incident themes - with prevention patterns so teams stop paying the same interest on tech debt.

Recipe

Quick-reference recipe card - copy-paste ready.

## Lesson template
**ID:** LL-014
**Pattern:** Async route calls sync ORM
**Impact:** p95 outage, low CPU
**Prevention:** AST CI check + asyncio.to_thread for CPU
**See:** Before/After Async Migration

When to reach for this:

• Onboarding senior hires to fleet culture
• Quarterly ops review
• Post-mortem action consolidation
• Architecture review checklist input

Working Example

### Web & API
 
| ID | Lesson | Prevention |
| LL-001 | Mutable default args in hot path | pytest lint; code review checklist |
| LL-002 | Missing tenant_id filter on query | repository requires tenant context |
| LL-003 | Webhook without idempotency | Idempotency-Key table |
| LL-004 | Sync sleep in async middleware | scripts/check_async_blocking.py |
 
### Data & Migrations
 
| LL-010 | Index migration without CONCURRENTLY | migration linter + row count gate |
| LL-011 | Staging 1% row count hides locks | large table seed script |
| LL-012 | Non-idempotent ETL append | MERGE on natural key |
 
### ML & Batch
 
| LL-020 | Training-serving skew | shared features package |
| LL-021 | No eval gate on promote | min AUC in train CI job |
| LL-022 | Notebook creds in git | secrets via orchestrator only |
 
### Ops & Delivery
 
| LL-030 | API rolled back, workers not paused | deploy-order.md mandatory |
| LL-031 | :latest prod tag | immutable SHA tags |
| LL-032 | Alert on ERROR log count | SLO burn alerts |

"""lessons_registry.py - optional machine-readable index."""
LESSONS = {
    "LL-004": {
        "title": "Sync sleep in async middleware",
        "guardrail": "scripts/check_async_blocking.py",
        "doc": "./before-after-sync-to-async-migration/before-after-sync-to-async-migration.md",
    },
}

What this demonstrates:

• Lessons grouped for scanning during design review
• Each row points to concrete prevention
• IDs stable for post-mortem cross-reference
• Optional registry links docs for onboarding bots

Deep Dive

How It Works

• Sourced from incidents - Post-mortem themes become LL entries within two weeks.
• Owner - Platform rotates quarterly catalog curator.
• Onboarding - New hires acknowledge top 10 in first month.
• Review - Retire lessons when guardrail universal in scaffold template.
• Cross-link - Case studies and troubleshooting pages reference LL IDs.

Top 10 for New Hires

1. LL-004 - async blocking
2. LL-010 - migration locks
3. LL-030 - worker/API order
4. LL-003 - webhook idempotency
5. LL-020 - ML skew
6. LL-002 - tenant leak
7. LL-031 - image tags
8. LL-012 - ETL idempotency
9. LL-001 - mutable defaults
10. LL-032 - alert hygiene

Python Notes

## Post-mortem action → catalog
Action G2 "async CI check" closes → add LL-004 if not present

Gotchas

• Catalog too long - Nobody reads. Fix: Top 10 short list + full table linked.
• Lessons without guardrails - Philosophy only. Fix: Every entry has enforceable check or runbook.
• Stale lessons - Python 2 era advice. Fix: Quarterly curator pass tied to Python 3.14 standards.
• Blameful wording - "Dev forgot." Fix: System pattern language.
• Duplicate post-mortem text - Maintenance burden. Fix: ID reference instead of copy-paste.

Alternatives

FAQs

Related

• Turning Incidents into Guardrails - close the loop
• Before/After: Sync to Async Migration - LL-004 source
• Before/After: Django Monolith - coupling lessons
• Common Production Failure Modes - pattern detail
• Governance Best Practices - standards

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.