Two fixes from the first full 125-test run (121 ok / 4 not ok).
1. The Quadlet unit test still asserted Restart=on-failure for backends,
which 8908fb4f deliberately changed. All 32 units on the node are
Restart=always; only archy-btcpay-db showed up because `fail` aborts at
the first offender. The old rationale — "backends need on-failure so an
operator `systemctl stop` stays stopped" — was wrong twice over:
systemd never applies Restart= to a unit stopped via `systemctl stop`,
and because quadlet renders --rm, a cleanly-exited container is deleted
and on-failure never brings it back (bitcoind exits 0 on SIGTERM, so
backends vanished). Verified on-device before that change: `podman stop
bitcoin-knots` came back in 12s, a dashboard stop stayed stopped 90s.
Now asserts Restart=always, with the corrected reasoning in-place.
2. all-apps-lifecycle cycled ten apps back-to-back and manufactured the
contention that then failed later tests: load went 10.8 at preflight to
17.6 mid-loop, and btcpay's recovery stretched from 52s on a quiet box
to 216s and then 512s, defeating waits of 180s and 300s. Widening waits
cannot win when the run sets the load they must survive, so the loop now
drains between apps — waits for load1 under 2x nproc, capped at 90s.
ARCHY_APP_SETTLE_SECS=0 disables; ARCHY_APP_SETTLE_LOAD overrides.
Verified: run.sh use-quadlet-backends-install → 6/6 (was 1 failure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop/start/restart loop carefully accumulates per-app failures into
$fails and prints them, but the three rpc_result calls were bare. Under
bats' errexit a bare call ends the test immediately, so the summary that
names the app never ran.
On 2026-08-08 that turned a single transient error into an unattributable
failure: the test died at package.stop with no indication which of the
ten targets was involved. It was mempool, and the identical call returned
{"status":"stopping"} by hand a few minutes later.
Each call now records <id>:<phase>-rpc and moves to the next app, so the
run reports what actually broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Active counterpart to the read-only all-apps-matrix.bats: drives
stop/start/restart for every installed app and, under
ARCHY_ALLOW_CASCADE_DESTRUCTIVE, a FULL teardown (uninstall →
no-ghost → reinstall) — the broad coverage F needs beyond the ~8 core
suites. App set is discovered from My Apps ∩ the node catalog; reinstall
spec comes from catalog.json {dockerImage, containerConfig}.
PROTECTED by default (never cycled or torn down): bitcoin*/electrum*
(expensive resync) AND lnd/btcpay*/fedimint* (teardown = irreversible
wallet/channel/guardian loss). The user asked to protect only
bitcoin+electrum; the wallet apps are added for safety and can be
removed via ARCHY_MATRIX_PROTECT. Heavy + destructive → a supervised
pass, not folded into run-gate. Validated on .228: discovery excludes
the 6 protected installed apps; lifecycle tier cycles a single app
(botfights) stop/start/restart green; teardown gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
