A/B Testing & Experimentation

A/B tests measure whether a change improves outcomes - conversion, latency, or revenue - with controlled traffic splits. Python backends assign variants server-side, log exposures, and monitor guardrails so experiments do not become outages.

Recipe

Quick-reference recipe card - copy-paste ready.

import hashlib
 
 
def variant(user_id: str, experiment: str) -> str:
    bucket = int(hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()[:8], 16) % 100
    return "B" if bucket < 50 else "A"

log.info("experiment_exposure", experiment="checkout_cta", variant=variant(uid, "checkout_cta"))

When to reach for this:

• Product asks "does new flow convert better?"
• Pricing or ranking algorithm change needs evidence
• UX copy change with measurable funnel step
• Model or heuristic swap with business KPI

Working Example

"""ab_experiment.py - assignment, exposure logging, guardrails."""
from __future__ import annotations
 
import hashlib
from dataclasses import dataclass
from enum import Enum
 
 
class Variant(str, Enum):
    A = "A"
    B = "B"
 
 
@dataclass(frozen=True)
class Experiment:
    name: str
    traffic_pct: int  # B receives this percent
    success_metric: str
    guardrail_metrics: tuple[str, ...]
 
 
def assign_variant(user_id: str, exp: Experiment) -> Variant:
    digest = hashlib.sha256(f"{exp.name}:{user_id}".encode()).hexdigest()
    bucket = int(digest[:8], 16) % 100
    return Variant.B if bucket < exp.traffic_pct else Variant.A
 
 
@dataclass
class ExposureEvent:
    experiment: str
    variant: Variant
    user_id: str
 
 
def checkout_cta_label(user_id: str, exp: Experiment) -> tuple[str, ExposureEvent]:
    v = assign_variant(user_id, exp)
    label = "Complete purchase" if v == Variant.B else "Checkout"
    return label, ExposureEvent(exp.name, v, user_id)
 
 
if __name__ == "__main__":
    exp = Experiment("checkout_cta", 50, "checkout_completed", ("p95_latency", "error_rate"))
    print(checkout_cta_label("user-42", exp))

What this demonstrates:

• Deterministic sticky assignment per user and experiment
• Exposure events logged for analysis denominators
• Guardrail metrics defined upfront (latency, errors)
• Variant logic stays server-side in Python handlers

Deep Dive

How It Works

• Hypothesis - Written before code; success metric is one primary KPI.
• Sample ratio - 50/50 common; power analysis sets duration when possible.
• Guardrails - Abort if error rate or p95 degrades beyond threshold.
• Analysis - Use proper denominators (exposed users), not all traffic.
• Ethics/compliance - Document PII in experiment logs; honor opt-out regions.

Experiment vs Feature Flag

Python Notes

# Store exposures in warehouse-friendly schema
EXPOSURE_SCHEMA = {
    "experiment": str,
    "variant": str,
    "user_id": str,
    "timestamp": str,
}

Gotchas

• Peeking at results daily and stopping early - Inflates false positives. Fix: Pre-register duration or use sequential testing methods.
• Non-sticky assignment - Users flip variants; data useless. Fix: Hash stable user ID.
• Client-side assignment - Biased and forgeable. Fix: Server assigns; client renders.
• Ignoring guardrails - Revenue up, outages up. Fix: Auto-pause experiment on SLO burn.
• Multiple experiments colliding - Interaction effects. Fix: Experiment registry; limit overlap per surface.

Alternatives

FAQs

Related

• Feature Flags & Progressive Delivery - delivery vs measurement
• DORA Metrics - guard delivery speed
• Structured Logging - exposure events
• Metrics - guardrail dashboards
• Prioritizing Platform & Tech Debt - experiment backlog

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.