fix(ota): repair a stale Restart=on-failure unit that leaves nodes dead after update
austin-sapien (100.70.96.88) sat dead for over two hours after taking v1.7.122 — 'server starting' in the UI, service inactive, exit status 0/SUCCESS. It did not crash: the in-process updater replaces the binary and exits cleanly for systemd to restart it, and that node's unit still carried Restart=on-failure from an older install. systemd read the clean exit as success and left it stopped. Every node with the old unit has this waiting for it on the next update. self-update.sh does refresh units, but the in-process update path never runs it, so nothing was repairing them. The daemon now checks its own unit at boot and rewrites only the Restart= line, so a node that starts even once ends up with a policy that survives the next update. Also carries the session-policy wiring: validate() now honours the configured idle and absolute limits and the per-device class, instead of the single hard-coded 24h constant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
d8647f6576
commit
81033ed6f5
@@ -1271,3 +1271,54 @@ mod tests {
|
||||
assert_ne!(outcome, PodmanHealOutcome::Healthy);
|
||||
}
|
||||
}
|
||||
|
||||
/// Repair this node's own systemd restart policy.
|
||||
///
|
||||
/// The in-process updater replaces the binary and then asks systemd to
|
||||
/// restart the service, treating `Restart=always` on the unit as its second
|
||||
/// net if that request is ever lost. On austin-sapien (2026-08-05) the unit
|
||||
/// was an old one carrying `Restart=on-failure`: the daemon exited cleanly
|
||||
/// (status 0), systemd read that as success, and the node sat dead for over
|
||||
/// two hours after a routine update — "server starting" in the UI, with
|
||||
/// nothing to start it.
|
||||
///
|
||||
/// A node cannot be relied on to fix this via `self-update.sh` (which does
|
||||
/// refresh units) because the in-process update path never runs it. So the
|
||||
/// daemon checks its own unit at boot: any node that starts even once ends
|
||||
/// up with a policy that survives the next update. Deliberately narrow —
|
||||
/// only the `Restart=` line is touched, so local edits elsewhere in the unit
|
||||
/// are preserved.
|
||||
pub async fn ensure_restart_policy() {
|
||||
const UNIT: &str = "/etc/systemd/system/archipelago.service";
|
||||
let Ok(body) = fs::read_to_string(UNIT).await else {
|
||||
return; // not a systemd install (container, dev box) — nothing to do
|
||||
};
|
||||
if !body.lines().any(|l| {
|
||||
let l = l.trim();
|
||||
l.starts_with("Restart=") && l != "Restart=always"
|
||||
}) {
|
||||
return; // already correct, or no Restart= line to repair
|
||||
}
|
||||
let patched: String = body
|
||||
.lines()
|
||||
.map(|l| {
|
||||
if l.trim().starts_with("Restart=") && l.trim() != "Restart=always" {
|
||||
"Restart=always"
|
||||
} else {
|
||||
l
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
match write_root_if_needed(UNIT, &patched).await {
|
||||
Ok(true) => {
|
||||
tracing::warn!(
|
||||
"repaired archipelago.service Restart= policy to always — this node would \
|
||||
have stayed dead after an in-process update"
|
||||
);
|
||||
let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
|
||||
}
|
||||
Ok(false) => {}
|
||||
Err(e) => tracing::warn!(error = %e, "could not repair archipelago.service restart policy"),
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user