File & Directory Automation

Q: How do I copy trees?

shutil.copytree(src, dst) with dirs_exist_ok=True on 3.14 for merges.

Q: How do I pick encoding?

UTF-8 for text; open(path, "rb") for binary - never assume locale encoding.

Q: Atomic write across filesystems?

os.replace requires same mount - use unique name + app-level versioning if cross-device.

Q: How do I exclude directories?

Filter path.parts or use fnmatch on relative path string during walk.

Q: Safe delete patterns?

Trash to staging dir first; --dry-run lists targets before unlink.

Q: How do permissions work?

path.chmod(0o640) after write for secret files; respect umask in shared dirs.

Q: Concurrent writers?

File lock (fcntl/portalocker) or delegate to queue worker for single-writer guarantee.

Q: How do I test?

tmp_path pytest fixture for isolated directory per test.

pathlib and the shutil/tempfile modules replace fragile shell one-liners for copying, rotating, and generating files. Safe writes and explicit encoding prevent data loss when scripts run unattended under cron.

Recipe

from pathlib import Path
import tempfile
import os
 
def atomic_write(path: Path, data: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=path.parent, text=True)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(data)
        os.replace(tmp, path)
    except Exception:
        os.unlink(tmp)
        raise

When to reach for this:

Batch file transforms (rename, compress, validate)
Report generation to known directories
Log rotation and cleanup jobs
Syncing artifacts between build stages on one host

Working Example

Find CSV files, validate non-empty, atomically write summary manifest.

from __future__ import annotations
 
import json
import os
import tempfile
from dataclasses import dataclass
from pathlib import Path
 
@dataclass(frozen=True)
class FileRecord:
    path: Path
    size: int
 
def discover_csv(root: Path) -> list[FileRecord]:
    records: list[FileRecord] = []
    for path in root.glob("**/*.csv"):
        if path.is_file():
            size = path.stat().st_size
            if size == 0:
                raise ValueError(f"empty csv: {path}")
            records.append(FileRecord(path, size))
    return records
 
def atomic_write_json(path: Path, payload: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            json.dump(payload, f, indent=2)
        os.replace(tmp, path)
    except Exception:
        os.unlink(tmp)
        raise
 
def main(root: Path, manifest: Path) -> int:
    files = discover_csv(root)
    atomic_write_json(
        manifest,
        {"count": len(files), "files": [{"path": str(r.path), "size": r.size} for r in files]},
    )
    return 0
 
if __name__ == "__main__":
    raise SystemExit(main(Path("data/incoming"), Path("data/manifest.json")))

What this demonstrates:

glob("**/*.csv") recursive discovery with pathlib
Empty-file guard fails fast before publishing manifest
os.replace atomic rename on same filesystem prevents partial reads

Deep Dive

pathlib Essentials

Operation	API
Join	`base / "child"`
Exists	`path.exists()`, `is_file()`
Read	`read_text(encoding="utf-8")`
Tree walk	`path.rglob("*")`

Safe Write Pattern

Write to temp in target directory (same mount)
fsync optional for critical durability
os.replace atomically swaps into place

Python Notes

import shutil
from pathlib import Path
 
shutil.copy2(Path("src.txt"), Path("backup/src.txt"))  # preserves metadata
shutil.rmtree(Path("tmp/build"), ignore_errors=False)

Gotchas

Writing directly to final path - readers see half-written files. Fix: atomic temp + replace.
Default encoding on Windows - mojibake on UTF-8 CSV. Fix: always encoding="utf-8" explicitly.
Following symlinks in deletes - removes unexpected targets. Fix: is_symlink() checks before unlink.
Race in mkdir - parallel scripts collide. Fix: exist_ok=True or catch FileExistsError.
Huge trees in memory - list(rglob) on millions of files. Fix: iterate generator without materializing all paths.

Alternatives

Alternative	Use When	Don't Use When
`os.walk`	Legacy code maintenance	Greenfield - prefer pathlib
Shell `find`	One-liner on ops box	Need cross-platform Python automation
`watchdog`	React to FS events	Nightly batch is enough

FAQs

pathlib vs os.path?

pathlib is the modern default in Python 3.14 code - clearer and chainable.

How do I copy trees?

shutil.copytree(src, dst) with dirs_exist_ok=True on 3.14 for merges.

How do I pick encoding?

UTF-8 for text; open(path, "rb") for binary - never assume locale encoding.

Atomic write across filesystems?

os.replace requires same mount - use unique name + app-level versioning if cross-device.

How do I exclude directories?

Filter path.parts or use fnmatch on relative path string during walk.

Safe delete patterns?

Trash to staging dir first; --dry-run lists targets before unlink.

How do permissions work?

path.chmod(0o640) after write for secret files; respect umask in shared dirs.

Concurrent writers?

File lock (fcntl/portalocker) or delegate to queue worker for single-writer guarantee.

How do I test?

tmp_path pytest fixture for isolated directory per test.

What about S3 instead of local disk?

Use boto3 for object storage; local pathlib for build agents and log processing on disk.

Scripting Basics - CLI structure
Robust Scripts - dry-run and idempotency
Parsing Logs & Text - read large log files
subprocess & Shell Interop - external tar/rsync
System Scripting Best Practices - unattended script rules

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

from pathlib import Path
import tempfile
import os
 
def atomic_write(path: Path, data: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=path.parent, text=True)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(data)
        os.replace(tmp, path)
    except Exception:
        os.unlink(tmp)
        raise

When to reach for this:

Batch file transforms (rename, compress, validate)
Report generation to known directories
Log rotation and cleanup jobs
Syncing artifacts between build stages on one host

Working Example

Find CSV files, validate non-empty, atomically write summary manifest.

from __future__ import annotations
 
import json
import os
import tempfile
from dataclasses import dataclass
from pathlib import Path
 
@dataclass(frozen=True)
class FileRecord:
    path: Path
    size: int
 
def discover_csv(root: Path) -> list[FileRecord]:
    records: list[FileRecord] = []
    for path in root.glob("**/*.csv"):
        if path.is_file():
            size = path.stat().st_size
            if size == 0:
                raise ValueError(f"empty csv: {path}")
            records.append(FileRecord(path, size))
    return records
 
def atomic_write_json(path: Path, payload: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            json.dump(payload, f, indent=2)
        os.replace(tmp, path)
    except Exception:
        os.unlink(tmp)
        raise
 
def main(root: Path, manifest: Path) -> int:
    files = discover_csv(root)
    atomic_write_json(
        manifest,
        {"count": len(files), "files": [{"path": str(r.path), "size": r.size} for r in files]},
    )
    return 0
 
if __name__ == "__main__":
    raise SystemExit(main(Path("data/incoming"), Path("data/manifest.json")))

What this demonstrates:

glob("**/*.csv") recursive discovery with pathlib
Empty-file guard fails fast before publishing manifest
os.replace atomic rename on same filesystem prevents partial reads

Deep Dive

pathlib Essentials

Operation	API
Join	`base / "child"`
Exists	`path.exists()`, `is_file()`
Read	`read_text(encoding="utf-8")`
Tree walk	`path.rglob("*")`

Safe Write Pattern

Write to temp in target directory (same mount)
fsync optional for critical durability
os.replace atomically swaps into place

Python Notes

import shutil
from pathlib import Path
 
shutil.copy2(Path("src.txt"), Path("backup/src.txt"))  # preserves metadata
shutil.rmtree(Path("tmp/build"), ignore_errors=False)

Gotchas

Writing directly to final path - readers see half-written files. Fix: atomic temp + replace.
Default encoding on Windows - mojibake on UTF-8 CSV. Fix: always encoding="utf-8" explicitly.
Following symlinks in deletes - removes unexpected targets. Fix: is_symlink() checks before unlink.
Race in mkdir - parallel scripts collide. Fix: exist_ok=True or catch FileExistsError.
Huge trees in memory - list(rglob) on millions of files. Fix: iterate generator without materializing all paths.

Alternatives

Alternative	Use When	Don't Use When
`os.walk`	Legacy code maintenance	Greenfield - prefer pathlib
Shell `find`	One-liner on ops box	Need cross-platform Python automation
`watchdog`	React to FS events	Nightly batch is enough

FAQs

pathlib vs os.path?

pathlib is the modern default in Python 3.14 code - clearer and chainable.

How do I copy trees?

shutil.copytree(src, dst) with dirs_exist_ok=True on 3.14 for merges.

How do I pick encoding?

UTF-8 for text; open(path, "rb") for binary - never assume locale encoding.

Atomic write across filesystems?

os.replace requires same mount - use unique name + app-level versioning if cross-device.

How do I exclude directories?

Filter path.parts or use fnmatch on relative path string during walk.

Safe delete patterns?

Trash to staging dir first; --dry-run lists targets before unlink.

How do permissions work?

path.chmod(0o640) after write for secret files; respect umask in shared dirs.

Concurrent writers?

File lock (fcntl/portalocker) or delegate to queue worker for single-writer guarantee.

How do I test?

tmp_path pytest fixture for isolated directory per test.

What about S3 instead of local disk?

Use boto3 for object storage; local pathlib for build agents and log processing on disk.

Scripting Basics - CLI structure
Robust Scripts - dry-run and idempotency
Parsing Logs & Text - read large log files
subprocess & Shell Interop - external tar/rsync
System Scripting Best Practices - unattended script rules

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

File & Directory Automation

Search Documentation

File & Directory Automation