Configuration Management
Configuration management separates what changes per environment from how automation applies it. Python infra projects use templating, layered config files, and secret stores so dev, staging, and prod stay structurally identical without copying entire stacks.
Recipe
Quick-reference recipe card - copy-paste ready.
from pydantic_settings import BaseSettings, SettingsConfigDict
class AppSettings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
environment: str = "dev"
database_url: str
log_level: str = "info"
settings = AppSettings() # reads env vars and optional .env locally# app.env.j2
APP_ENV={{ environment }}
DATABASE_URL={{ database_url }}
LOG_LEVEL={{ log_level }}When to reach for this:
- Same template, different values across dev/stage/prod
- Secrets outside git with references injected at deploy time
- Validating config before apply with Pydantic models
- Feature flags and limits that differ per environment
- Ansible/Pulumi/Terraform all consuming one config schema
Working Example
Layered YAML config, Jinja2 render, and secret lookup stub for a deploy script.
"""render_config.py - load env config, merge secrets, render template."""
from __future__ import annotations
import json
from pathlib import Path
import yaml
from jinja2 import Environment, FileSystemLoader, select_autoescape
ROOT = Path(__file__).parent
CONFIG_DIR = ROOT / "config"
def load_yaml(path: Path) -> dict:
with path.open() as f:
return yaml.safe_load(f) or {}
def load_config(environment: str) -> dict:
base = load_yaml(CONFIG_DIR / "base.yaml")
env = load_yaml(CONFIG_DIR / f"{environment}.yaml")
merged = {**base, **env, "environment": environment}
return merged
def fetch_secret(name: str) -> str:
# Production: boto3.client("secretsmanager").get_secret_value(...)
return f"SECRET::{name}"
def render(template_name: str, context: dict) -> str:
env = Environment(
loader=FileSystemLoader(ROOT / "templates"),
autoescape=select_autoescape(),
)
template = env.get_template(template_name)
return template.render(**context)
if __name__ == "__main__":
cfg = load_config("staging")
cfg["database_password"] = fetch_secret("db/password")
output = render("service.env.j2", cfg)
print(output)
print(json.dumps({"rendered_for": cfg["environment"]}))# config/base.yaml
log_level: info
service_port: 8080# config/staging.yaml
database_host: staging.db.internal
feature_new_ui: true# templates/service.env.j2
APP_ENV={{ environment }}
LOG_LEVEL={{ log_level }}
PORT={{ service_port }}
DATABASE_HOST={{ database_host }}
DATABASE_PASSWORD={{ database_password }}
FEATURE_NEW_UI={{ feature_new_ui }}What this demonstrates:
- Base + env YAML merge keeps structure DRY
- Secrets fetched at render time, never committed to templates
- Jinja2 renders final env files for Ansible
templateor container injection - JSON log line records which environment was rendered for audit
Deep Dive
How It Works
- Data layer: YAML/JSON/TOML per environment + schema validation
- Secret layer: SSM, Secrets Manager, Vault, or CI secret store
- Template layer: Jinja2, envsubst, or Terraform/Pulumi variables
- Apply layer: Ansible, Kubernetes, or systemd loads rendered artifacts
- Changes flow: edit config in git → validate → render → deploy → verify
Config vs Secrets
| Item | Store | Example |
|---|---|---|
| Region, instance size | Git (env YAML) | instance_type: t3.small |
| DB password, API keys | Secret manager | arn:aws:secretsmanager:... |
| Feature flags | Git or remote flag service | feature_x: false |
| Runtime overrides | Environment variables | LOG_LEVEL=debug |
Python Notes
from pydantic import Field, SecretStr
from pydantic_settings import BaseSettings
class DeploySettings(BaseSettings):
environment: str
db_password: SecretStr = Field(alias="DATABASE_PASSWORD")
def redacted(self) -> dict:
return self.model_dump(mode="json", exclude={"db_password"})Gotchas
- Secrets in
prod.yaml- credentials in git history forever. Fix: store secret ARNs or names, resolve at runtime. - Forked templates per env - drift when one env gets a fix. Fix: one template, env-specific data only.
- Missing validation - typos reach production as failed deploys. Fix: Pydantic models or JSON Schema in CI.
- World-readable rendered files -
chmod 644on.envwith passwords. Fix:0600ownership for service user only. - Templating untrusted input - SSTI if templates include user content. Fix: keep templates internal; never embed user strings in Jinja2.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Hardcoded env vars in CI only | Tiny single-env projects | Multiple envs with shared structure |
| Consul/etcd live config | Dynamic runtime tuning | Infra wants versioned, reviewed changes |
| Pulumi config alone | All settings are cloud resource params | App-level config also needed on hosts |
.env committed per developer | Local dev convenience only | Any shared or production environment |
FAQs
Should config live in git or a secret store?
Non-secret parameters belong in git with env overlays. Secrets belong in a manager with rotation; git stores references only.
Why Jinja2 for infra teams?
Ansible uses it natively, syntax is familiar, and it supports conditionals for optional blocks without duplicating whole files.
How does pydantic-settings fit?
Apps read env vars at runtime with validation. Infra renders those env files from layered YAML + secrets during deploy.
How do I prevent staging config hitting prod databases?
Separate connection strings per env file, naming conventions (staging.db.internal), and CI checks that fail if prod hostnames appear in non-prod files.
Can Terraform variables replace this?
For cloud resources, yes. Application env files on VMs/containers still benefit from templating outside Terraform outputs.
How often should I rotate secrets?
Automate rotation in Secrets Manager/SSM and re-render configs. Document rollback if rotation breaks dependent services.
What about 12-factor config?
Store config in environment variables at runtime. Your render step produces those vars for Kubernetes, Lambda, or systemd.
How do I test templates?
pytest with fixed context dicts asserts rendered strings contain expected keys and never leak placeholder tokens like {{.
Should defaults live in base.yaml or code?
Safe operational defaults in base.yaml; code defaults only for local dev fallbacks. Document overrides in README per environment.
How do I audit who changed prod config?
Git blame on env YAML plus deploy logs showing rendered SHA and operator identity from CI OIDC claims.
Related
- Infrastructure Automation Basics - environment separation
- Ansible - template module and group_vars
- Secrets Manager & SSM - AWS secret resolution
- Configuration & Secrets in Prod - runtime injection
- Infrastructure Automation Best Practices - versioned config rules
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.