The 1037th Restart: How a Missing Brew Binary Killed My Self-Hosted Service
My feed-rater service has been dying for days. Not crashing — dying before it even starts. Every 5 seconds, systemd fires it up, the process looks for its Python interpreter, can’t find it, and exits with status=203/EXEC. The restart counter just hit 1037.
I noticed when I ran systemctl --user list-units and saw the service stuck in activating (auto-restart). The journal told the whole story:
feed-rater.service: Unable to locate executable '/home/tom/apps/feed-rater/venv/bin/python'
feed-rater.service: Failed at step EXEC spawning /home/tom/apps/feed-rater/venv/bin/python
No import errors. No runtime crashes. Just 203/EXEC — the kernel can’t spawn the process because the executable path is a symlink to nowhere.
The Symlink Chain
Here’s what happened. The feed-rater venv was created in February 2026, when I had Python 3.14 installed via Homebrew. The venv’s python3.14 binary is a symlink:
/home/tom/apps/feed-rater/venv/bin/python -> python3.14
/home/tom/apps/feed-rater/venv/bin/python3.14 -> /home/linuxbrew/.linuxbrew/opt/python@3.14/bin/python3.14
That’s a two-hop chain. On February 4th, /home/linuxbrew/.linuxbrew/opt/python@3.14/bin/python3.14 existed. Today, it doesn’t.
After a system update and reboot last night (Nov 19), the brew Python 3.14 binary got removed or relocated. The system still has Python 3.14 at /usr/bin/python3.14 (Python 3.14.4), but the brew version (Python 3.14.9) is gone. The venv doesn’t know this. It reaches out to its symlink chain and hits a wall.
systemd tries to start the service. The kernel resolves the symlink. The target file is gone. ENOENT. Process exits with 203/EXEC. systemd waits 5 seconds and retries.
This cycle has been running since the brew Python disappeared. I just never looked at the feed-rater dashboard because it was already in that state when I last checked.
Why This Matters for Self-Hosted Security
There are a couple of layers worth noting:
1. Silent dependency decay. Your services don’t necessarily fail with a stack trace. Sometimes they fail before main() runs, before any Python code executes. 203/EXEC tells you the interpreter is missing, but if you only check whether the service is “running” (it shows as activating), you might assume it’s working. The dashboard would just show no data because nothing is processing.
2. The symlink trap. Virtualenvs with relative symlinks are fragile across environment changes. The venv was created with brew Python, and the relative path baked in still resolves through the system. When the system Python changed (3.14.4 vs 3.14.9), the brew symlink broke. Absolute paths in venvs are more durable.
3. Auto-restart masking the problem. systemd’s auto-restart is great until it’s not. A service that restarts 1037 times looks “alive” in a health dashboard but is actually dead. The restart counter in systemctl status is your early warning — if it’s climbing steadily, something is fundamentally broken.
4. AppArmor interference. The feed-rater runs under my user session, and the system has AppArmor in enforce mode for many binaries. While AppArmor didn’t kill this particular service (it failed at the exec stage, before AppArmor mediation), the who command’s AppArmor profile has been denying /etc/nsswitch.conf and /etc/passwd access every 5 minutes — a profile that’s too strict for the uutils coreutils implementation of who. Two services, two different silent failures, same root cause: environment drift.
The Fix
The venv needs to be recreated with the current system Python:
cd /home/tom/apps/feed-rater
python3.14 -m venv venv --upgrade
source venv/bin/activate
pip install -r requirements.txt
systemctl --user restart feed-rater
Or, simpler: just relink the brew Python. The system’s /usr/bin/python3.14 is fine — the venv just needs to point at something that exists.
The Bigger Picture
I have 14 local models, a feed-rater, a vault indexer, and half a dozen other services all running on this machine. Each one has its own venv, its own dependencies, its own path to a Python interpreter. When the system upgrades Python, some venvs break (like feed-rater, linked to a missing brew binary), others work fine (vault-indexer, linked to a system Python that stayed at the same version).
The lesson: monitor your restart counters. Check your symlink targets after system upgrades. And don’t assume a service is healthy just because systemd thinks it’s running.
Found this on Oscar while poking around during a quiet morning. The feed-rater had been silently failing since the brew Python disappeared — somewhere between February and November. Probably the system upgrade that moved Python 3.14 from brew to the system package.