Provisioning Cloud Resources
Provisioning cloud resources means creating and updating AWS (or other cloud) primitives in a repeatable, reviewable way. Python teams combine IaC tools with boto3 for convergence, imports, and operational tasks that do not belong in a static template.
Recipe
Quick-reference recipe card - copy-paste ready.
import boto3
def provision_queue(sqs, name: str) -> str:
try:
resp = sqs.get_queue_url(QueueName=name)
return resp["QueueUrl"]
except sqs.exceptions.QueueDoesNotExist:
resp = sqs.create_queue(
QueueName=name,
Attributes={"VisibilityTimeout": "60"},
)
return resp["QueueUrl"]When to reach for this:
- Bootstrapping accounts before full IaC adoption
- Idempotent ensure helpers called from deploy pipelines
- Import/adopt workflows wrapping cloud APIs
- Operational runbooks codified as Python CLIs
- Pairing with Pulumi/Terraform for resources outside current modules
Working Example
A small provisioning CLI that ensures S3, SQS, and IAM policy attachments with structured logging.
"""provision.py - idempotent AWS resource bootstrap."""
from __future__ import annotations
import json
import logging
from dataclasses import dataclass
import boto3
from botocore.exceptions import ClientError
logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("provision")
@dataclass(frozen=True)
class ProvisionSpec:
bucket: str
queue: str
tags: dict[str, str]
def ensure_bucket(s3, name: str, tags: dict[str, str]) -> str:
buckets = {b["Name"] for b in s3.list_buckets().get("Buckets", [])}
if name not in buckets:
s3.create_bucket(Bucket=name)
log.info(json.dumps({"action": "create", "resource": "s3", "name": name}))
s3.put_bucket_tagging(Bucket=name, Tagging={"TagSet": [{"Key": k, "Value": v} for k, v in tags.items()]})
return name
def ensure_queue(sqs, name: str) -> str:
try:
return sqs.get_queue_url(QueueName=name)["QueueUrl"]
except ClientError as exc:
if exc.response["Error"]["Code"] != "AWS.SimpleQueueService.NonExistentQueue":
raise
url = sqs.create_queue(QueueName=name)["QueueUrl"]
log.info(json.dumps({"action": "create", "resource": "sqs", "name": name}))
return url
def main() -> None:
spec = ProvisionSpec(
bucket="company-events-dev",
queue="company-events-dev",
tags={"Environment": "dev", "Owner": "platform"},
)
session = boto3.Session()
s3 = session.client("s3")
sqs = session.client("sqs")
ensure_bucket(s3, spec.bucket, spec.tags)
ensure_queue(sqs, spec.queue)
log.info(json.dumps({"status": "ok", "bucket": spec.bucket, "queue": spec.queue}))
if __name__ == "__main__":
main()What this demonstrates:
get_queue_url+ exception handling implements idempotent queue ensure- Tagging applied on every run to converge metadata
- JSON structured logs suitable for CI artifacts
- Session per run respects
AWS_PROFILEand IAM role credentials
Deep Dive
How It Works
- Declare desired resources in code, data classes, or IaC programs
- Plan with preview tools (
pulumi preview,terraform plan) when using IaC - Apply creates/updates; state backends track what exists
- Verify with describe/get APIs and integration tests
- Document every provision path in git with PR review
Provisioning Layers
| Layer | Tooling | Best for |
|---|---|---|
| Declarative IaC | Pulumi, Terraform | VPCs, IAM baselines, data stores |
| Imperative ensure | boto3 scripts | One-off adoption, pipeline glue |
| Control plane APIs | CloudFormation | AWS-native teams |
| Config on instances | Ansible | OS packages after instance exists |
Python Notes
from botocore.config import Config
retry_config = Config(retries={"max_attempts": 10, "mode": "standard"})
s3 = boto3.client("s3", config=retry_config)Gotchas
- Create-only scripts - second run fails with
BucketAlreadyExists. Fix: check existence or use IaC with state. - No tags on resources - cost allocation and policy audits break. Fix: mandatory tag set validated in CI policy.
- Shared credentials on laptops - provisioning from dev machines with admin keys. Fix: CI OIDC roles scoped per environment.
- Hardcoded regions - resources land in wrong partition/region. Fix:
AWS_REGIONor session config from env spec. - Silent partial failure - queue created, bucket failed, script still exits 0. Fix: aggregate errors and non-zero exit codes.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Pulumi/Terraform only | Full environment lifecycle | Need quick imperative adoption |
| AWS CloudFormation | AWS-only, service control policies enforce CFN | Team standardized on Python IaC elsewhere |
| Click CLI + boto3 | Operators run vetted commands | Ad-hoc boto3 notebooks without review |
| Console clicks | Emergency break-glass only | Repeatable environments |
FAQs
When is boto3 provisioning enough?
Small idempotent ensures and glue code. Full environments with many interdependent resources belong in IaC with plan/apply.
How do I make provisioning reviewable?
Every change through git PR, CI plan/preview artifacts, and required approvals for production accounts.
What tags are required?
At minimum Environment, Owner, and CostCenter - extend per org policy and validate before apply.
How do I handle existing resources?
Import into Terraform/Pulumi state or write adoption scripts that detect and align rather than blind create.
Can I provision from Lambda?
Yes for small ensures - watch IAM permissions, timeouts, and idempotency when Lambda retries fire.
How do retries interact with provisioning?
Use botocore Config retries for throttling. Creation itself should remain idempotent to survive duplicate calls.
What about multi-account AWS?
Assume roles per account with sts:AssumeRole, separate state backends, and guardrails preventing prod applies from dev CI.
How do I test without AWS?
Use moto for unit tests of ensure functions, and LocalStack for selective integration tests - still run smoke tests in a sandbox account.
Who owns provisioning code?
Platform/SRE owns shared modules; product teams consume tagged modules rather than copying boto3 snippets.
How does this relate to deployment?
Provisioning creates shared infrastructure (queues, buckets); deployment pipelines publish app artifacts that consume those resources.
Related
- Infrastructure Automation Basics - idempotent mindset
- Pulumi (Python IaC) - declarative provisioning
- boto3 Basics - sessions and clients
- S3 - object storage operations
- SQS & SNS - messaging primitives
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.