"Bitcoin Knots disappeared again, plus other apps." Root cause is a pairing,
not a single bug: quadlet renders `podman run ... --replace --rm`, so the
container is deleted the moment it stops, and from_manifest set
Restart=on-failure, which declines to restart after a CLEAN exit. bitcoind
exits 0 on SIGTERM. So any clean stop deleted the container AND left it
deleted — the app vanished from podman and from My Apps until a later
archipelago reconcile tick noticed and recreated it. That is the
"previously-running app has no container after boot — recreating
(desired-state recovery)" line, which fired for bitcoin-knots at 18:53,
19:57 and 20:39 and for electrumx at 19:57 and 20:42 on 2026-08-07.
A crash always self-healed: on-failure restarted the unit and podman run
recreated the container. Only a clean exit stranded it, which is why this
survived so long.
The justification for on-failure was wrong on systemd's own semantics. It
read "clean exits — e.g. operator-issued systemctl stop — stay stopped", but
Restart= is never consulted for a unit stopped via systemctl stop
(systemd.service(5)), and that is exactly how archipelago stops these apps
(prod_orchestrator -> stop_service_with_timeout). Always keeps the
stopped-stays-stopped behaviour and drops the failure mode.
Always also restores the premise of the Quadlet migration — systemd owns
supervision, so an app returns without archipelago alive to notice it left.
Checked before flipping: no manifest declares a one-shot container and there
is no manifest-level restart field, so nothing gets restart-looped.
Propagation to existing nodes is via sync_quadlet_unit's drift re-render,
which rewrites the unit and daemon-reloads WITHOUT restarting the service —
running containers are undisturbed and the new policy governs the next start.
OnFailure is kept as a deliberate opt-in with a note not to wire it back to
backends. Two tests now pin the new default and assert on-failure is absent
from a rendered backend unit.
Container suite 215/215.
NOTE FOR THE OPERATOR: this changes supervision semantics for every app on
the Quadlet canary path. Wants sign-off and a lifecycle-gate run before OTA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
