S3

Q: upload_file vs put_object?

upload_file handles multipart and retries from a path. put_object sends bytes you already have in memory.

Q: How do I upload from FastAPI?

Accept upload, stream to temp file or upload_fileobj, never store on app disk in prod without size limits.

Q: Can I enforce encryption?

Yes - bucket policy denies unencrypted puts; set ServerSideEncryption in ExtraArgs.

Q: How do I delete many keys?

delete_objects with batches up to 1000 keys per call.

Q: What content type should I set?

Match actual bytes (application/json, image/png) so browsers and tools handle objects correctly.

Q: How do I copy between buckets?

copy_object with CopySource dict - same-region copies are efficient.

Q: What IAM permissions are minimal?

Scope to arn:aws:s3:::bucket/prefix/ with s3:GetObject, PutObject, DeleteObject as needed - no s3: on *.

Amazon S3 is the default object store for Python services on AWS. boto3 handles uploads, downloads, streaming, and presigned URLs for browser or partner access without proxying bytes through your API.

Recipe

import boto3
 
s3 = boto3.client("s3")
s3.upload_file("report.csv", "my-bucket", "exports/report.csv")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "exports/report.csv"},
    ExpiresIn=3600,
)

When to reach for this:

Durable file storage for exports, uploads, ML artifacts
Presigned URLs for direct browser upload/download
Streaming large objects without loading into memory
Lifecycle policies managed in IaC; app code reads/writes objects

Working Example

Upload with extra args, download to path, and stream body with context manager.

import boto3
from pathlib import Path
 
BUCKET = "demo-app-data"
KEY = "incoming/data.json"
LOCAL = Path("data.json")
 
s3 = boto3.client("s3")
 
s3.upload_file(
    str(LOCAL),
    BUCKET,
    KEY,
    ExtraArgs={"ContentType": "application/json", "ServerSideEncryption": "AES256"},
)
 
s3.download_file(BUCKET, KEY, "downloaded.json")
 
resp = s3.get_object(Bucket=BUCKET, Key=KEY)
body = resp["Body"].read()
print(len(body))
 
presigned = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": BUCKET, "Key": "incoming/upload.bin"},
    ExpiresIn=900,
)
print(presigned[:80], "...")

What this demonstrates:

ExtraArgs sets content type and SSE on upload
get_object returns a streaming Body - use .read() or iter chunks
Presigned put_object enables client-side uploads without sharing IAM keys

Deep Dive

How It Works

Objects live in buckets under keys (paths are conventional, not real folders)
upload_file/download_file use multipart transfers for large files automatically
generate_presigned_url signs requests with caller credentials - scope IAM tightly
Versioning and MFA delete are bucket settings - handle in IaC

Common Operations

Task	API
Upload file	`upload_file`, `put_object`
Download	`download_file`, `get_object`
List prefix	`list_objects_v2` + paginator
Delete	`delete_object`, `delete_objects`

Python Notes

from boto3.s3.transfer import TransferConfig
 
config = TransferConfig(multipart_threshold=8 * 1024 * 1024, max_concurrency=10)
s3.upload_file("big.bin", "bucket", "big.bin", Config=config)

Gotchas

Public buckets by mistake - ACL public-read exposes data. Fix: block public access at account level; use presigned URLs.
Loading multi-GB objects with .read() - OOM. Fix: iter_chunks() or download_file to disk.
Presigned URLs with excessive expiry - leaked URL works for days. Fix: short TTL (15-60 min) and least-privilege key prefix.
Assuming list_objects is complete - truncated without pagination. Fix: paginator.
Wrong region client - permanent redirect errors. Fix: bucket region lookup or region-scoped session.

Alternatives

Alternative	Use When	Don't Use When
EFS/EBS	POSIX filesystem semantics needed	Global durable object delivery
CloudFront + S3	Public or signed CDN delivery	Private internal-only artifacts
MinIO / on-prem	Non-AWS requirement	Already on AWS with compliance for S3

FAQs

upload_file vs put_object?

upload_file handles multipart and retries from a path. put_object sends bytes you already have in memory.

How do I upload from FastAPI?

Accept upload, stream to temp file or upload_fileobj, never store on app disk in prod without size limits.

Can I enforce encryption?

Yes - bucket policy denies unencrypted puts; set ServerSideEncryption in ExtraArgs.

How do I delete many keys?

delete_objects with batches up to 1000 keys per call.

What content type should I set?

Match actual bytes (application/json, image/png) so browsers and tools handle objects correctly.

How do presigned PUT uploads work?

Your API returns presigned URL; client HTTP PUTs file directly to S3 - saves API bandwidth.

How do I test without AWS?

moto or LocalStack for unit tests; sandbox bucket for integration smoke tests.

Does S3 guarantee read-after-write?

New object PUTs are read-after-write consistent; LIST and overwrite semantics have nuances - see AWS docs for your use case.

How do I copy between buckets?

copy_object with CopySource dict - same-region copies are efficient.

What IAM permissions are minimal?

Scope to arn:aws:s3:::bucket/prefix/* with s3:GetObject, PutObject, DeleteObject as needed - no s3:* on *.

boto3 Basics - sessions and clients
Retry, Pagination & Throttling - list pagination
Lambda - S3 event triggers
Credential & Session Patterns - scoped IAM
Cloud SDK Best Practices - safe S3 access

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

import boto3
 
s3 = boto3.client("s3")
s3.upload_file("report.csv", "my-bucket", "exports/report.csv")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "exports/report.csv"},
    ExpiresIn=3600,
)

When to reach for this:

Durable file storage for exports, uploads, ML artifacts
Presigned URLs for direct browser upload/download
Streaming large objects without loading into memory
Lifecycle policies managed in IaC; app code reads/writes objects

Working Example

Upload with extra args, download to path, and stream body with context manager.

import boto3
from pathlib import Path
 
BUCKET = "demo-app-data"
KEY = "incoming/data.json"
LOCAL = Path("data.json")
 
s3 = boto3.client("s3")
 
s3.upload_file(
    str(LOCAL),
    BUCKET,
    KEY,
    ExtraArgs={"ContentType": "application/json", "ServerSideEncryption": "AES256"},
)
 
s3.download_file(BUCKET, KEY, "downloaded.json")
 
resp = s3.get_object(Bucket=BUCKET, Key=KEY)
body = resp["Body"].read()
print(len(body))
 
presigned = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": BUCKET, "Key": "incoming/upload.bin"},
    ExpiresIn=900,
)
print(presigned[:80], "...")

What this demonstrates:

ExtraArgs sets content type and SSE on upload
get_object returns a streaming Body - use .read() or iter chunks
Presigned put_object enables client-side uploads without sharing IAM keys

Deep Dive

How It Works

Objects live in buckets under keys (paths are conventional, not real folders)
upload_file/download_file use multipart transfers for large files automatically
generate_presigned_url signs requests with caller credentials - scope IAM tightly
Versioning and MFA delete are bucket settings - handle in IaC

Common Operations

Task	API
Upload file	`upload_file`, `put_object`
Download	`download_file`, `get_object`
List prefix	`list_objects_v2` + paginator
Delete	`delete_object`, `delete_objects`

Python Notes

from boto3.s3.transfer import TransferConfig
 
config = TransferConfig(multipart_threshold=8 * 1024 * 1024, max_concurrency=10)
s3.upload_file("big.bin", "bucket", "big.bin", Config=config)

Gotchas

Public buckets by mistake - ACL public-read exposes data. Fix: block public access at account level; use presigned URLs.
Loading multi-GB objects with .read() - OOM. Fix: iter_chunks() or download_file to disk.
Presigned URLs with excessive expiry - leaked URL works for days. Fix: short TTL (15-60 min) and least-privilege key prefix.
Assuming list_objects is complete - truncated without pagination. Fix: paginator.
Wrong region client - permanent redirect errors. Fix: bucket region lookup or region-scoped session.

Alternatives

Alternative	Use When	Don't Use When
EFS/EBS	POSIX filesystem semantics needed	Global durable object delivery
CloudFront + S3	Public or signed CDN delivery	Private internal-only artifacts
MinIO / on-prem	Non-AWS requirement	Already on AWS with compliance for S3

FAQs

upload_file vs put_object?

upload_file handles multipart and retries from a path. put_object sends bytes you already have in memory.

How do I upload from FastAPI?

Accept upload, stream to temp file or upload_fileobj, never store on app disk in prod without size limits.

Can I enforce encryption?

Yes - bucket policy denies unencrypted puts; set ServerSideEncryption in ExtraArgs.

How do I delete many keys?

delete_objects with batches up to 1000 keys per call.

What content type should I set?

Match actual bytes (application/json, image/png) so browsers and tools handle objects correctly.

How do presigned PUT uploads work?

Your API returns presigned URL; client HTTP PUTs file directly to S3 - saves API bandwidth.

How do I test without AWS?

moto or LocalStack for unit tests; sandbox bucket for integration smoke tests.

Does S3 guarantee read-after-write?

New object PUTs are read-after-write consistent; LIST and overwrite semantics have nuances - see AWS docs for your use case.

How do I copy between buckets?

copy_object with CopySource dict - same-region copies are efficient.

What IAM permissions are minimal?

Scope to arn:aws:s3:::bucket/prefix/* with s3:GetObject, PutObject, DeleteObject as needed - no s3:* on *.

boto3 Basics - sessions and clients
Retry, Pagination & Throttling - list pagination
Lambda - S3 event triggers
Credential & Session Patterns - scoped IAM
Cloud SDK Best Practices - safe S3 access

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

S3

Search Documentation

S3