S3

Amazon S3 is the default object store for Python services on AWS. boto3 handles uploads, downloads, streaming, and presigned URLs for browser or partner access without proxying bytes through your API.

Recipe

import boto3
 
s3 = boto3.client("s3")
s3.upload_file("report.csv", "my-bucket", "exports/report.csv")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "exports/report.csv"},
    ExpiresIn=3600,
)

When to reach for this:

• Durable file storage for exports, uploads, ML artifacts
• Presigned URLs for direct browser upload/download
• Streaming large objects without loading into memory
• Lifecycle policies managed in IaC; app code reads/writes objects

Working Example

Upload with extra args, download to path, and stream body with context manager.

import boto3
from pathlib import Path
 
BUCKET = "demo-app-data"
KEY = "incoming/data.json"
LOCAL = Path("data.json")
 
s3 = boto3.client("s3")
 
s3.upload_file(
    str(LOCAL),
    BUCKET,
    KEY,
    ExtraArgs={"ContentType": "application/json", "ServerSideEncryption": "AES256"},
)
 
s3.download_file(BUCKET, KEY, "downloaded.json")
 
resp = s3.get_object(Bucket=BUCKET, Key=KEY)
body = resp["Body"].read()
print(len(body))
 
presigned = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": BUCKET, "Key": "incoming/upload.bin"},
    ExpiresIn=900,
)
print(presigned[:80], "...")

What this demonstrates:

• ExtraArgs sets content type and SSE on upload
• get_object returns a streaming Body - use .read() or iter chunks
• Presigned put_object enables client-side uploads without sharing IAM keys

Deep Dive

How It Works

• Objects live in buckets under keys (paths are conventional, not real folders)
• upload_file/download_file use multipart transfers for large files automatically
• generate_presigned_url signs requests with caller credentials - scope IAM tightly
• Versioning and MFA delete are bucket settings - handle in IaC

Common Operations

Python Notes

from boto3.s3.transfer import TransferConfig
 
config = TransferConfig(multipart_threshold=8 * 1024 * 1024, max_concurrency=10)
s3.upload_file("big.bin", "bucket", "big.bin", Config=config)

Gotchas

• Public buckets by mistake - ACL public-read exposes data. Fix: block public access at account level; use presigned URLs.
• Loading multi-GB objects with .read() - OOM. Fix: iter_chunks() or download_file to disk.
• Presigned URLs with excessive expiry - leaked URL works for days. Fix: short TTL (15-60 min) and least-privilege key prefix.
• Assuming list_objects is complete - truncated without pagination. Fix: paginator.
• Wrong region client - permanent redirect errors. Fix: bucket region lookup or region-scoped session.

Alternatives

FAQs

Related

• boto3 Basics - sessions and clients
• Retry, Pagination & Throttling - list pagination
• Lambda - S3 event triggers
• Credential & Session Patterns - scoped IAM
• Cloud SDK Best Practices - safe S3 access

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.