Infrastructure Automation Best Practices

Idempotent, versioned, auditable infrastructure keeps production changes predictable. These rules apply whether you use Ansible, Pulumi, Terraform/CDKTF, or boto3 ensure scripts.

How to Use This List

Adopt before the first production apply - retrofits are painful after drift accumulates.
Turn unchecked items into tracked tickets with owners, not silent exceptions.
Revisit quarterly as accounts, compliance rules, and team size change.
Pair with Testing Infrastructure Code for CI enforcement.

A - Source of Truth

Store all infra definitions in git. Console hotfixes must be backported or reverted within one business day.
Pin tool versions in lockfiles. pulumi, cdktf, ansible, and provider packages install from uv/pip locks in CI.
One logical resource naming scheme. Stable logical names prevent replace churn in IaC engines.
Tag every resource with Environment, Owner, CostCenter. CI policy rejects untagged creates.
Document stack-to-account mapping in README. Prevents applying prod stack to wrong AWS account.

B - Plan and Apply Discipline

Run plan/preview on every PR touching infra. Attach output to the PR or CI artifact store.
Block unreviewed destructive changes. Deletes on stateful resources require explicit approval ticket.
Use saved plan files for apply. terraform apply tfplan - not a fresh implicit plan at apply time.
Separate state per environment. No shared Terraform/Pulumi state across dev and prod.
Fail closed on policy errors. pytest/OPA failures stop the pipeline - no manual override without incident record.

C - Secrets and Config

Never commit secrets to group_vars, tfvars, or Pulumi config. Use encrypted config or secret manager references.
Render templates from layered YAML, not forked files per env. One template, env data overlays only.
Validate config with Pydantic or schema tests before render. Typos should fail CI, not prod deploys.
Restrict rendered file permissions to service user. Mode 0600 for files containing credentials.
Rotate secrets without editing IaC resource IDs. Update secret values in manager, re-render app config.

D - Idempotency and Drift

Prefer modules over raw shell in Ansible. Preserve changed=0 reporting and convergence semantics.
Implement ensure helpers for boto3 glue code. Check existence before create; converge tags and settings.
Schedule drift detection jobs. Compare live AWS attributes to declared spec; alert on mismatch.
Second Ansible converge should report zero changes. Molecule or CI proves idempotency.
Import adopted resources into IaC state. Do not leave console-created resources untracked.

E - Operations and Audit

Emit structured JSON logs from automation. Include actor, stack, commit SHA, and action counts.
Use OIDC federation for CI AWS roles. No long-lived AWS_ACCESS_KEY_ID in GitHub/GitLab secrets.
Sandbox account for integration tests. Prod credentials never power PR pipelines.
Run ruff and pytest on infra Python packages. Same quality bar as application code.
Keep break-glass console access rare and logged. Post-incident, reconcile console changes into git.

FAQs

What is the minimum viable IaC setup?

Git repo, plan on PR, isolated state per env, mandatory tags, and secrets outside git. Add Molecule/OPA as you mature.

Can small teams skip Terraform?

Pulumi Python or disciplined boto3 ensure scripts work for small footprints - still require plan-like review and state tracking.

How do I enforce tags?

pytest policy on plan JSON, AWS Config rules, or Service Control Policies - defense in depth beats one check.

When is manual console change acceptable?

Break-glass incidents only. Open a ticket to import or revert the change in IaC before the next scheduled apply.

How often should I run drift detection?

Daily for prod-critical resources, weekly for lower tiers - tune based on incident history and change velocity.

Should developers apply to prod?

Prefer CI promotion with approval gates. Direct prod applies from laptops multiply audit and mis-account risk.

How do Ansible and Pulumi divide ownership?

Pulumi/Terraform for cloud primitives; Ansible for OS-level config on instances. Document handoff in runbooks.

What belongs in pytest vs OPA?

Simple org rules (tags, naming) in pytest for speed. Complex graph policies (no public SG rules) in OPA on full plan JSON.

How do I roll back infra?

Revert git commit and re-apply, or restore state snapshot if backend supports it - practice rollback in sandbox first.

Why structured logs?

Incidents need who changed what and when. Plain print("done") does not survive centralized log search.

Infrastructure Automation Basics - IaC fundamentals
Testing Infrastructure Code - CI gates and policy
Configuration Management - layered config and secrets
Pulumi (Python IaC) - Python-native provisioning
Provisioning Cloud Resources - repeatable cloud setup

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Infrastructure Automation Best Practices

Idempotent, versioned, auditable infrastructure keeps production changes predictable. These rules apply whether you use Ansible, Pulumi, Terraform/CDKTF, or boto3 ensure scripts.

How to Use This List

Adopt before the first production apply - retrofits are painful after drift accumulates.
Turn unchecked items into tracked tickets with owners, not silent exceptions.
Revisit quarterly as accounts, compliance rules, and team size change.
Pair with Testing Infrastructure Code for CI enforcement.

A - Source of Truth

Store all infra definitions in git. Console hotfixes must be backported or reverted within one business day.
Pin tool versions in lockfiles. pulumi, cdktf, ansible, and provider packages install from uv/pip locks in CI.
One logical resource naming scheme. Stable logical names prevent replace churn in IaC engines.
Tag every resource with Environment, Owner, CostCenter. CI policy rejects untagged creates.
Document stack-to-account mapping in README. Prevents applying prod stack to wrong AWS account.

B - Plan and Apply Discipline

Run plan/preview on every PR touching infra. Attach output to the PR or CI artifact store.
Block unreviewed destructive changes. Deletes on stateful resources require explicit approval ticket.
Use saved plan files for apply. terraform apply tfplan - not a fresh implicit plan at apply time.
Separate state per environment. No shared Terraform/Pulumi state across dev and prod.
Fail closed on policy errors. pytest/OPA failures stop the pipeline - no manual override without incident record.

C - Secrets and Config

Never commit secrets to group_vars, tfvars, or Pulumi config. Use encrypted config or secret manager references.
Render templates from layered YAML, not forked files per env. One template, env data overlays only.
Validate config with Pydantic or schema tests before render. Typos should fail CI, not prod deploys.
Restrict rendered file permissions to service user. Mode 0600 for files containing credentials.
Rotate secrets without editing IaC resource IDs. Update secret values in manager, re-render app config.

D - Idempotency and Drift

Prefer modules over raw shell in Ansible. Preserve changed=0 reporting and convergence semantics.
Implement ensure helpers for boto3 glue code. Check existence before create; converge tags and settings.
Schedule drift detection jobs. Compare live AWS attributes to declared spec; alert on mismatch.
Second Ansible converge should report zero changes. Molecule or CI proves idempotency.
Import adopted resources into IaC state. Do not leave console-created resources untracked.

E - Operations and Audit

Emit structured JSON logs from automation. Include actor, stack, commit SHA, and action counts.
Use OIDC federation for CI AWS roles. No long-lived AWS_ACCESS_KEY_ID in GitHub/GitLab secrets.
Sandbox account for integration tests. Prod credentials never power PR pipelines.
Run ruff and pytest on infra Python packages. Same quality bar as application code.
Keep break-glass console access rare and logged. Post-incident, reconcile console changes into git.

FAQs

What is the minimum viable IaC setup?

Git repo, plan on PR, isolated state per env, mandatory tags, and secrets outside git. Add Molecule/OPA as you mature.

Can small teams skip Terraform?

Pulumi Python or disciplined boto3 ensure scripts work for small footprints - still require plan-like review and state tracking.

How do I enforce tags?

pytest policy on plan JSON, AWS Config rules, or Service Control Policies - defense in depth beats one check.

When is manual console change acceptable?

Break-glass incidents only. Open a ticket to import or revert the change in IaC before the next scheduled apply.

How often should I run drift detection?

Daily for prod-critical resources, weekly for lower tiers - tune based on incident history and change velocity.

Should developers apply to prod?

Prefer CI promotion with approval gates. Direct prod applies from laptops multiply audit and mis-account risk.

How do Ansible and Pulumi divide ownership?

Pulumi/Terraform for cloud primitives; Ansible for OS-level config on instances. Document handoff in runbooks.

What belongs in pytest vs OPA?

Simple org rules (tags, naming) in pytest for speed. Complex graph policies (no public SG rules) in OPA on full plan JSON.

How do I roll back infra?

Revert git commit and re-apply, or restore state snapshot if backend supports it - practice rollback in sandbox first.

Why structured logs?

Incidents need who changed what and when. Plain print("done") does not survive centralized log search.

Infrastructure Automation Basics - IaC fundamentals
Testing Infrastructure Code - CI gates and policy
Configuration Management - layered config and secrets
Pulumi (Python IaC) - Python-native provisioning
Provisioning Cloud Resources - repeatable cloud setup

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Infrastructure Automation Best Practices

Search Documentation

Infrastructure Automation Best Practices