Rollback Strategies

Not every bad deploy can roll back. Delivery teams choose between reverting the previous image, forward-fixing with a patch, or killing a feature flag - based on migration compatibility and time to mitigate.

Recipe

Quick-reference recipe card - copy-paste ready.

## Rollback decision (2 minutes)
1. Customer impact active? → mitigate first
2. Deploy < 2h and image sha-prev exists? → candidate rollback
3. Migration destructive or contract? → forward-fix only
4. Bug isolated behind flag? → flag off, then plan fix
5. Record decision + commands in #incident thread

When to reach for this:

• Defining release checklist go/no-go
• Training on-call on mitigation options
• Designing migrations for reversibility
• Post-incident review of slow rollback

Working Example

"""rollback_decision.py - encode team policy as testable helpers."""
from __future__ import annotations
 
from dataclasses import dataclass
from enum import Enum
 
 
class Mitigation(str, Enum):
    FLAG_OFF = "flag_off"
    ROLLBACK_IMAGE = "rollback_image"
    FORWARD_FIX = "forward_fix"
 
 
@dataclass(frozen=True)
class DeployState:
    minutes_since_deploy: int
    migration_contract: bool  # removed column or destructive
    migration_expand: bool
    flag_isolates_bug: bool
    previous_image_available: bool
 
 
def choose_mitigation(state: DeployState) -> Mitigation:
    if state.flag_isolates_bug:
        return Mitigation.FLAG_OFF
    if state.migration_contract:
        return Mitigation.FORWARD_FIX
    if state.previous_image_available and state.minutes_since_deploy < 120:
        return Mitigation.ROLLBACK_IMAGE
    return Mitigation.FORWARD_FIX
 
 
if __name__ == "__main__":
    s = DeployState(25, migration_contract=False, migration_expand=True, flag_isolates_bug=False, previous_image_available=True)
    print(choose_mitigation(s))

What this demonstrates:

• Decision tree is explicit and teachable to on-call
• Contract migrations block image rollback automatically
• Flag kill wins when blast radius is isolated
• Policy can be unit-tested like application code

Deep Dive

How It Works

• Image rollback - Fast when schema and config are backward compatible.
• Forward-fix - Patch forward when rollback would break DB assumptions.
• Flag off - Config change without rebuild; runbook documents env key.
• Data rollback - PITR rare; separate from application rollback.
• Retention - Registry keeps last N SHAs for rollout undo.

Strategy Matrix

Python Notes

# Keep previous Gunicorn image handy
docker pull $REGISTRY/orders-api:sha-prev
kubectl set image deploy/orders-api api=$REGISTRY/orders-api:sha-prev

Gotchas

• Assuming rollback is always one command - Workers and migrations block it. Fix: Pre-flight checklist in release template.
• Forward-fix under pressure without tests - Second incident. Fix: Minimal patch branch with single integration test from prod case.
• Deleting old images - Undo impossible. Fix: Retention policy minimum 10 prod tags.
• Rollback without smoke test - Hidden partial outage. Fix: Canary tenant write+read before full promote.
• Ignoring Celery queue - Poison messages after rollback. Fix: Pause queue per runbook.

Alternatives

FAQs

Related

• Rollback & Recovery Runbooks - execution steps
• Feature Flags & Progressive Delivery - flag kill
• Release Management - train process
• Environments & Config Promotion - artifact promotion
• Zero-Downtime Deploys - deploy mechanics

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.