After an unclean shutdown, crash recovery walked the running-containers
snapshot and retried 'no such container' failures (2 attempts, 10s
backoff) for containers that no longer exist. Recovery runs before the
server binds :5678 and notifies systemd ready, so a stale snapshot
pushed startup past TimeoutStartSec=5min — systemd killed the daemon
mid-recovery, the next boot saw a crash again, and the node looped
forever (151 restarts on .116 after a hard poweroff).
- pre-filter the snapshot against 'podman ps -a' and skip vanished
containers outright (fail-open if the query fails)
- never retry a 'no such container' failure
- extend the systemd start timeout ahead of each container start so a
heavy node recovering dozens of real containers isn't killed while
making progress
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
f13fdc6451