fix(boot): signal systemd READY before heavy boot recovery + Restart=always

Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.

Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
  the unit reaches 'active' in seconds; recovery/reconcile/listener
  continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
  the daemon back. Manual  is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
  instead of a tokio-sleep child of the process being stopped, whose
  start-half was being lost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dorian
2026-07-27 00:31:54 +01:00
co-authored by Claude Fable 5
parent 56e4c30261
commit 70bcbc4acc
3 changed files with 62 additions and 18 deletions
+7 -1
View File
@@ -31,7 +31,13 @@ ExecStartPre=+/bin/bash -c 'mkdir -p /var/lib/archipelago && chown archipelago:a
# "-" so a missing/failed guard can never block the service itself.
ExecStartPre=+-/opt/archipelago/scripts/ota-crash-guard.sh
ExecStart=/usr/local/bin/archipelago
Restart=on-failure
# always (not on-failure): the OTA restart path once stopped the daemon
# cleanly and the queued start never fired (framework-pt, v1.7.114->115,
# 2026-07-26) — the node sat dead all night behind "server starting up".
# Restart=always self-heals any lost start job; an explicit
# `systemctl stop` is still honored (systemd never auto-restarts after
# a manual stop).
Restart=always
RestartSec=5
WatchdogSec=300
TimeoutStartSec=300