lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

fix(boot): signal systemd READY before heavy boot recovery + Restart=always

Browse Source 
Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.

Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
  the unit reaches 'active' in seconds; recovery/reconcile/listener
  continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
  the daemon back. Manual  is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
  instead of a tokio-sleep child of the process being stopped, whose
  start-half was being lost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dorian
2026-07-27 00:31:54 +01:00
co-authored by Claude Fable 5
parent 56e4c30261
commit 70bcbc4acc
3 changed files with 62 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/main.rs
+23 -10
View File
@@ -199,6 +199,26 @@ async fn main() -> Result<()> {
// Now mark this instance as running so the next startup can detect a crash.
crash_recovery::write_pid_marker(&config.data_dir).await?;
// Signal READY *before* the heavy synchronous boot recovery below. On a
// node with many stacks that recovery takes minutes, and the unit sat in
// `activating` the whole time — so anything that touched the service in
// that window (a superseding start/restart, a start-timeout) killed a
// half-started instance, which then exited 0 and (under the old
// Restart=on-failure) never came back: "server starting up" forever,
// reproduced on framework-pt installing apps on 2026-07-26. The daemon's
// real work (recovery, reconcile, listener) continues after READY; being
// "active" early is honest — the process is up and doing its job.
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]);
// Watchdog pings must run DURING the long recovery too, or a slow boot
// trips WatchdogSec. Spawn the keepalive here rather than after serve().
tokio::spawn(async {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(120));
loop {
interval.tick().await;
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Watchdog]);
}
});
// Run crash recovery before starting the manifest reconciler. Both paths
// mutate Podman; running them concurrently can corrupt transient runtime
// state and leave netavark/conmon unable to start containers.
@@ -479,16 +499,9 @@ async fn main() -> Result<()> {
// Notify systemd that we're ready (Type=notify)
// Note: first param `false` keeps NOTIFY_SOCKET so watchdog pings work
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]);
// Spawn systemd watchdog ping (WatchdogSec=300, ping every 120s)
tokio::spawn(async {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(120));
loop {
interval.tick().await;
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Watchdog]);
}
});
// READY + watchdog keepalive were already signalled/spawned earlier
// (before boot recovery) so the unit reaches `active` in seconds instead
// of sitting in `activating` through a minutes-long recovery.
// Graceful shutdown: wait for SIGTERM or SIGINT
let mut sigterm = signal::unix::signal(signal::unix::SignalKind::terminate())