fix(ota): repair a stale Restart=on-failure unit that leaves nodes dead after update

austin-sapien (100.70.96.88) sat dead for over two hours after taking
v1.7.122 — 'server starting' in the UI, service inactive, exit status
0/SUCCESS. It did not crash: the in-process updater replaces the binary and
exits cleanly for systemd to restart it, and that node's unit still carried
Restart=on-failure from an older install. systemd read the clean exit as
success and left it stopped. Every node with the old unit has this waiting
for it on the next update.

self-update.sh does refresh units, but the in-process update path never
runs it, so nothing was repairing them. The daemon now checks its own unit
at boot and rewrites only the Restart= line, so a node that starts even
once ends up with a policy that survives the next update.

Also carries the session-policy wiring: validate() now honours the
configured idle and absolute limits and the per-device class, instead of
the single hard-coded 24h constant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-05 13:50:16 -04:00
co-authored by Claude Fable 5
parent d8647f6576
commit 81033ed6f5
3 changed files with 95 additions and 4 deletions
+51
View File
@@ -1271,3 +1271,54 @@ mod tests {
assert_ne!(outcome, PodmanHealOutcome::Healthy);
}
}
/// Repair this node's own systemd restart policy.
///
/// The in-process updater replaces the binary and then asks systemd to
/// restart the service, treating `Restart=always` on the unit as its second
/// net if that request is ever lost. On austin-sapien (2026-08-05) the unit
/// was an old one carrying `Restart=on-failure`: the daemon exited cleanly
/// (status 0), systemd read that as success, and the node sat dead for over
/// two hours after a routine update — "server starting" in the UI, with
/// nothing to start it.
///
/// A node cannot be relied on to fix this via `self-update.sh` (which does
/// refresh units) because the in-process update path never runs it. So the
/// daemon checks its own unit at boot: any node that starts even once ends
/// up with a policy that survives the next update. Deliberately narrow —
/// only the `Restart=` line is touched, so local edits elsewhere in the unit
/// are preserved.
pub async fn ensure_restart_policy() {
const UNIT: &str = "/etc/systemd/system/archipelago.service";
let Ok(body) = fs::read_to_string(UNIT).await else {
return; // not a systemd install (container, dev box) — nothing to do
};
if !body.lines().any(|l| {
let l = l.trim();
l.starts_with("Restart=") && l != "Restart=always"
}) {
return; // already correct, or no Restart= line to repair
}
let patched: String = body
.lines()
.map(|l| {
if l.trim().starts_with("Restart=") && l.trim() != "Restart=always" {
"Restart=always"
} else {
l
}
})
.collect::<Vec<_>>()
.join("\n");
match write_root_if_needed(UNIT, &patched).await {
Ok(true) => {
tracing::warn!(
"repaired archipelago.service Restart= policy to always — this node would \
have stayed dead after an in-process update"
);
let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
}
Ok(false) => {}
Err(e) => tracing::warn!(error = %e, "could not repair archipelago.service restart policy"),
}
}