Sysadmin Essentials
Files, permissions, processes, and services on Linux servers running Python APIs, workers, and cron jobs - the commands operators reach for before opening a Python debugger.
Recipe
ls -la /var/log/myapp/
sudo systemctl status billing-api
sudo journalctl -u billing-api -n 100 --no-pager
df -h && du -sh /var/lib/myapp/*When to reach for this:
- SSH into staging/prod to verify deploy
- Permission denied errors on log or upload dirs
- Disk full alerts blocking writes
- Service failed to start after config change
Working Example
# New deploy directory owned by service user
sudo mkdir -p /var/lib/billing-api/uploads
sudo chown billing:billing /var/lib/billing-api/uploads
sudo chmod 750 /var/lib/billing-api/uploads
# Check unit and recent logs
sudo systemctl daemon-reload
sudo systemctl restart billing-api
sudo systemctl is-active billing-api
sudo journalctl -u billing-api -fWhat this demonstrates:
- Least-privilege directory ownership for app user
systemctllifecycle after unit file change- Follow mode
-fon journal for live triage - Disk check before large log rotation
Deep Dive
How It Works
- Users/groups - Process runs as service user; files need matching ownership.
- systemd - Unit files define
ExecStart, env, restart policy. - journald - Centralized logging for systemd services.
- Filesystem - Inodes and blocks both can exhaust independently.
Command Map
| Problem | Commands |
|---|---|
| Permissions | ls -la, namei -l path, chmod, chown |
| Disk | df -h, du -xh --max-depth=1 |
| Service | systemctl status, restart, enable |
| Logs | journalctl -u, /var/log/ tail |
Python Notes
# Find which process holds port 8000
sudo ss -tlnp | grep 8000
# Confirm venv python path in systemd unit
systemctl cat billing-api | grep ExecStartGotchas
- Running app as root - Security disaster. Fix: dedicated user, capabilities only if low ports needed (prefer reverse proxy).
- chmod 777 uploads - World-writable data dir. Fix:
750and app group. - Editing prod without backup - Broken unit file stops service. Fix: copy unit to
/etc/systemd/system/with version comment. - Ignoring inode exhaustion -
dfshows space but writes fail. Fix:df -i. - Log disk fill - App crashes on write. Fix: logrotate + journal size limits.
- SELinux/AppArmor denials - Permission looks correct but still EPERM. Fix:
ausearch/aa-status.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Docker logs | Containerized Python services | Bare metal systemd VMs |
| Kubernetes kubectl | K8s deploys | Single VM staging |
| Ansible playbooks | Repeatable server setup | One-off SSH debug |
| Cloud SSH session manager | No bastion SSH keys | Air-gapped on-prem |
FAQs
systemd or supervisord?
systemd on modern Linux VMs; supervisord in legacy containers without systemd.
Where to put .env on server?
/etc/billing-api/env mode 600 owned root:billing with EnvironmentFile= in unit - not world-readable.
Restart vs reload?
reload if app supports HUP; Python APIs usually need restart after code deploy.
Check umask for uploads?
Service unit can set UMask=0027 so new files are not group/world writable.
tmpfs for /tmp uploads?
Sized tmpfs avoids disk fill but data lost on reboot - use for ephemeral scratch only.
NTP clock skew?
timedatectl status - JWT and log correlation break with bad clock.
ulimit open files?
Raise LimitNOFILE= in unit for high-connection FastAPI behind keep-alive.
find setuid binaries?
find / -perm -4000 2>/dev/null security audit - unrelated to Python but common sysadmin task.
Shared NFS uploads?
UID/GID mapping must match across nodes - prefer object storage (S3) for uploads.
Read-only root filesystem?
Mount writable paths explicitly for logs and tmp - common in hardened containers.
Related
- Process & Resource Management - signals and limits
- SSH & Remote Work - access patterns
- Deploying FastAPI - production layout
- Robust Scripts - defensive automation
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.