File & Directory Automation
pathlib and the shutil/tempfile modules replace fragile shell one-liners for copying, rotating, and generating files. Safe writes and explicit encoding prevent data loss when scripts run unattended under cron.
Recipe
from pathlib import Path
import tempfile
import os
def atomic_write(path: Path, data: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=path.parent, text=True)
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(data)
os.replace(tmp, path)
except Exception:
os.unlink(tmp)
raiseWhen to reach for this:
- Batch file transforms (rename, compress, validate)
- Report generation to known directories
- Log rotation and cleanup jobs
- Syncing artifacts between build stages on one host
Working Example
Find CSV files, validate non-empty, atomically write summary manifest.
from __future__ import annotations
import json
import os
import tempfile
from dataclasses import dataclass
from pathlib import Path
@dataclass(frozen=True)
class FileRecord:
path: Path
size: int
def discover_csv(root: Path) -> list[FileRecord]:
records: list[FileRecord] = []
for path in root.glob("**/*.csv"):
if path.is_file():
size = path.stat().st_size
if size == 0:
raise ValueError(f"empty csv: {path}")
records.append(FileRecord(path, size))
return records
def atomic_write_json(path: Path, payload: dict) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(payload, f, indent=2)
os.replace(tmp, path)
except Exception:
os.unlink(tmp)
raise
def main(root: Path, manifest: Path) -> int:
files = discover_csv(root)
atomic_write_json(
manifest,
{"count": len(files), "files": [{"path": str(r.path), "size": r.size} for r in files]},
)
return 0
if __name__ == "__main__":
raise SystemExit(main(Path("data/incoming"), Path("data/manifest.json")))What this demonstrates:
glob("**/*.csv")recursive discovery with pathlib- Empty-file guard fails fast before publishing manifest
os.replaceatomic rename on same filesystem prevents partial reads
Deep Dive
pathlib Essentials
| Operation | API |
|---|---|
| Join | base / "child" |
| Exists | path.exists(), is_file() |
| Read | read_text(encoding="utf-8") |
| Tree walk | path.rglob("*") |
Safe Write Pattern
- Write to temp in target directory (same mount)
fsyncoptional for critical durabilityos.replaceatomically swaps into place
Python Notes
import shutil
from pathlib import Path
shutil.copy2(Path("src.txt"), Path("backup/src.txt")) # preserves metadata
shutil.rmtree(Path("tmp/build"), ignore_errors=False)Gotchas
- Writing directly to final path - readers see half-written files. Fix: atomic temp + replace.
- Default encoding on Windows - mojibake on UTF-8 CSV. Fix: always
encoding="utf-8"explicitly. - Following symlinks in deletes - removes unexpected targets. Fix:
is_symlink()checks beforeunlink. - Race in mkdir - parallel scripts collide. Fix:
exist_ok=Trueor catchFileExistsError. - Huge trees in memory -
list(rglob)on millions of files. Fix: iterate generator without materializing all paths.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
os.walk | Legacy code maintenance | Greenfield - prefer pathlib |
Shell find | One-liner on ops box | Need cross-platform Python automation |
watchdog | React to FS events | Nightly batch is enough |
FAQs
pathlib vs os.path?
pathlib is the modern default in Python 3.14 code - clearer and chainable.
How do I copy trees?
shutil.copytree(src, dst) with dirs_exist_ok=True on 3.14 for merges.
How do I pick encoding?
UTF-8 for text; open(path, "rb") for binary - never assume locale encoding.
Atomic write across filesystems?
os.replace requires same mount - use unique name + app-level versioning if cross-device.
How do I exclude directories?
Filter path.parts or use fnmatch on relative path string during walk.
Safe delete patterns?
Trash to staging dir first; --dry-run lists targets before unlink.
How do permissions work?
path.chmod(0o640) after write for secret files; respect umask in shared dirs.
Concurrent writers?
File lock (fcntl/portalocker) or delegate to queue worker for single-writer guarantee.
How do I test?
tmp_path pytest fixture for isolated directory per test.
What about S3 instead of local disk?
Use boto3 for object storage; local pathlib for build agents and log processing on disk.
Related
- Scripting Basics - CLI structure
- Robust Scripts - dry-run and idempotency
- Parsing Logs & Text - read large log files
- subprocess & Shell Interop - external tar/rsync
- System Scripting Best Practices - unattended script rules
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.