System Scripting Best Practices
Scripts you can trust unattended follow the same operability rules as production services: predictable failure modes, safe defaults, and logs operators can search at 3am.
How to Use This List
- Apply before scheduling any script in cron or CI.
- Block review on missing
--dry-runfor mutating tools. - Revisit after any incident caused by automation reruns or overlapping jobs.
A - Structure and CLI
- Use
if __name__ == "__main__"and amain() -> intfunction. Keeps modules importable and testable. - Parse args with argparse or typer. Document flags in
--helpand README. - Support
--dry-runfor destructive actions. Print intended changes without side effects. - Reserve stdout for machine output; log humans to stderr. Enables piping JSON lines cleanly.
- Return documented exit codes. 0 ok, 2 usage, 1 failure - match org conventions.
B - Files and Data
- Use pathlib for all path operations. Avoid string path joins.
- Specify
encoding="utf-8"on text files. Prevent platform default surprises. - Write outputs atomically (temp + replace). Readers never see half-written files.
- Validate inputs before mutation. Fail fast on empty dirs, missing env vars, bad flags.
- Make reruns idempotent. Safe cron overlap or explicit flock/skip behavior.
C - Processes and Network
- Call subprocess with argv lists, not shell strings. Eliminate injection unless documented exception.
- Set subprocess and HTTP timeouts. Hung jobs must not block cron forever.
- Retry transient errors with backoff cap. Log final failure with context.
- Never log secrets, tokens, or presigned URLs. Redact Authorization headers.
- Verify webhook signatures before processing body. Reject unsigned inbound events.
D - Scheduling and Ops
- Use absolute paths to venv python in crontab. Include
cdto project root. - Prevent overlapping runs with file or distributed lock. Exit 0 when skipping duplicate instance if appropriate.
- Emit structured logs (JSON) for prod scripts. Include action, path, duration, correlation id.
- Monitor last-success heartbeat externally. Detect silent cron failures.
- Version scripts in git with CI running ruff/pytest. Same quality gate as application code.
E - Remote and Scale
- Prefer Ansible over ad-hoc SSH loops for fleets. SSH scripts for break-glass only.
- Reject unknown SSH host keys in automation. Managed
known_hostsor certificates. - Cap parallel remote operations. Avoid tripping rate limits and ban hammers.
- Stream large log files line-by-line. Do not read multi-GB files into memory.
- Promote stable scripts to packaged CLIs.
pyproject.tomlentry points when scope grows.
FAQs
Why is dry-run mandatory?
First prod run without preview causes irreversible deletes - dry-run is the cheapest insurance.
Can I use print in scripts?
For quick local tools yes; scheduled prod scripts should use logging with levels.
How do I choose argparse vs typer?
argparse for zero-deps cron scripts; typer when building multi-command operator tools.
What about configuration files?
YAML/TOML in git for non-secrets; env vars for secrets; CLI overrides win on conflict.
How do I document scripts for on-call?
Runbook with example commands, exit codes, expected runtime, and rollback steps.
Should scripts call AWS APIs?
Yes with reused boto3 session, retries, and least-privilege role - see Cloud SDK section.
How do I test cron scripts?
pytest main() with tmp_path; integration test in staging with dry-run then live on sample data.
When to move to Airflow/Prefect?
When dependencies, SLAs, and observability exceed one file and cron matrix becomes unmaintainable.
How do I handle timezones?
Store UTC in logs and filenames; document local-time cron only when business requires it.
What is the biggest scripting anti-pattern?
Silent success on partial failure - always aggregate errors and exit non-zero when anything failed.
Related
- Robust Scripts - full operability pattern
- Scripting Basics - entry points and CLI
- Scheduling & Cron - cron hygiene
- subprocess & Shell Interop - safe command execution
- Cloud SDK Best Practices - AWS from scripts
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.