feat(update): auto-rollback OTA binaries that crash-loop before they can self-verify

.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.

Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-18 19:42:54 -04:00
co-authored by Claude Fable 5
parent 837cfdfd1f
commit 25162ee846
3 changed files with 101 additions and 0 deletions
+24
View File
@@ -1785,6 +1785,30 @@ pub async fn apply_update(data_dir: &Path) -> Result<()> {
.await;
}
// Install the OTA crash-loop guard as a drop-in on existing
// nodes (fresh ISOs carry it in the unit file itself). The
// guard restores the update-backup binary when a freshly
// applied binary SEGVs before it can run its own post-OTA
// verification — the .198 v1.7.103 truncated-binary loop.
// Best-effort: `+-` in the drop-in means a missing script can
// never block the service, and a failed install here must not
// abort the apply.
if Path::new("/opt/archipelago/scripts/ota-crash-guard.sh").exists() {
let dropin_dir = "/etc/systemd/system/archipelago.service.d";
let _ = host_sudo(&["mkdir", "-p", dropin_dir]).await;
let _ = host_sudo(&[
"bash",
"-c",
&format!(
"printf '%s\\n' '[Service]' \
'ExecStartPre=+-/opt/archipelago/scripts/ota-crash-guard.sh' \
> {}/ota-crash-guard.conf",
dropin_dir
),
])
.await;
}
let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
let _ =
host_sudo(&["systemctl", "enable", "--now", "archipelago-doctor.timer"]).await;