feat(update): auto-rollback OTA binaries that crash-loop before they can self-verify
.198 crash-looped 236x on a truncated OTA binary with a good backup sitting in update-backup/ — verify_pending_update() can't help when the new binary never runs. New scripts/ota-crash-guard.sh runs as root from ExecStartPre: while the post-OTA pending-verify marker exists it counts start attempts, and after 5 failures restores the backup binary (atomic rename), replaces the marker with an update-rolled-back.json tombstone, and lets the service come back on the previous version. Wiring: fresh ISOs get the ExecStartPre line in the unit file; existing nodes get a drop-in installed by apply_update's runtime component step on their next OTA. '+-' prefix so a missing or failing guard can never block the service. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
837cfdfd1f
commit
25162ee846
@@ -1785,6 +1785,30 @@ pub async fn apply_update(data_dir: &Path) -> Result<()> {
|
||||
.await;
|
||||
}
|
||||
|
||||
// Install the OTA crash-loop guard as a drop-in on existing
|
||||
// nodes (fresh ISOs carry it in the unit file itself). The
|
||||
// guard restores the update-backup binary when a freshly
|
||||
// applied binary SEGVs before it can run its own post-OTA
|
||||
// verification — the .198 v1.7.103 truncated-binary loop.
|
||||
// Best-effort: `+-` in the drop-in means a missing script can
|
||||
// never block the service, and a failed install here must not
|
||||
// abort the apply.
|
||||
if Path::new("/opt/archipelago/scripts/ota-crash-guard.sh").exists() {
|
||||
let dropin_dir = "/etc/systemd/system/archipelago.service.d";
|
||||
let _ = host_sudo(&["mkdir", "-p", dropin_dir]).await;
|
||||
let _ = host_sudo(&[
|
||||
"bash",
|
||||
"-c",
|
||||
&format!(
|
||||
"printf '%s\\n' '[Service]' \
|
||||
'ExecStartPre=+-/opt/archipelago/scripts/ota-crash-guard.sh' \
|
||||
> {}/ota-crash-guard.conf",
|
||||
dropin_dir
|
||||
),
|
||||
])
|
||||
.await;
|
||||
}
|
||||
|
||||
let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
|
||||
let _ =
|
||||
host_sudo(&["systemctl", "enable", "--now", "archipelago-doctor.timer"]).await;
|
||||
|
||||
Reference in New Issue
Block a user