Sysadmin Essentials

Files, permissions, processes, and services on Linux servers running Python APIs, workers, and cron jobs - the commands operators reach for before opening a Python debugger.

Recipe

ls -la /var/log/myapp/
sudo systemctl status billing-api
sudo journalctl -u billing-api -n 100 --no-pager
df -h && du -sh /var/lib/myapp/*

When to reach for this:

• SSH into staging/prod to verify deploy
• Permission denied errors on log or upload dirs
• Disk full alerts blocking writes
• Service failed to start after config change

Working Example

# New deploy directory owned by service user
sudo mkdir -p /var/lib/billing-api/uploads
sudo chown billing:billing /var/lib/billing-api/uploads
sudo chmod 750 /var/lib/billing-api/uploads
 
# Check unit and recent logs
sudo systemctl daemon-reload
sudo systemctl restart billing-api
sudo systemctl is-active billing-api
sudo journalctl -u billing-api -f

What this demonstrates:

• Least-privilege directory ownership for app user
• systemctl lifecycle after unit file change
• Follow mode -f on journal for live triage
• Disk check before large log rotation

Deep Dive

How It Works

• Users/groups - Process runs as service user; files need matching ownership.
• systemd - Unit files define ExecStart, env, restart policy.
• journald - Centralized logging for systemd services.
• Filesystem - Inodes and blocks both can exhaust independently.

Command Map

Python Notes

# Find which process holds port 8000
sudo ss -tlnp | grep 8000
# Confirm venv python path in systemd unit
systemctl cat billing-api | grep ExecStart

Gotchas

• Running app as root - Security disaster. Fix: dedicated user, capabilities only if low ports needed (prefer reverse proxy).
• chmod 777 uploads - World-writable data dir. Fix: 750 and app group.
• Editing prod without backup - Broken unit file stops service. Fix: copy unit to /etc/systemd/system/ with version comment.
• Ignoring inode exhaustion - df shows space but writes fail. Fix: df -i.
• Log disk fill - App crashes on write. Fix: logrotate + journal size limits.
• SELinux/AppArmor denials - Permission looks correct but still EPERM. Fix: ausearch / aa-status.

Alternatives

FAQs

Related

• Process & Resource Management - signals and limits
• SSH & Remote Work - access patterns
• Deploying FastAPI - production layout
• Robust Scripts - defensive automation

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.