fix(orchestrator): durable uninstall marker for baseline apps + archival-bitcoin/version-report gaps

- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
  gap where installing it standalone (a legitimate direct orchestrator-install
  target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
  fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
  etc.) resurrecting itself after an explicit uninstall survives a restart or
  reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
  ("latest") as the reported running version -- a stale local :latest cache
  reported "latest" forever regardless of what latest had moved on to. Now
  falls back to asking the Bitcoin backend directly via `bitcoind --version`
  when the tag is floating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-01 06:29:11 -04:00
co-authored by Claude Sonnet 5
parent de8b2bb812
commit 5b7cd5d5d0
6 changed files with 290 additions and 4 deletions
+76
View File
@@ -1010,3 +1010,79 @@ this match.
- Definition of done: a new text sent from stock Meshtastic `3ccc` appears in `.116`
`mesh.messages` as an incoming LoRa message without a browser refresh, and `.116` -> `3ccc`
visibly arrives in the Meshtastic app.
## 11. Arch Issues (reported 2026-07-01, untriaged)
User-reported, raw, not yet root-caused. Split by owner — **do not fix the mesh items from the
non-mesh thread**; they route to the mesh/Reticulum agent (§10d owner).
- **[MESH — routes to §10d owner]** Transport-type label on mesh is delayed / requires a browser
refresh to show. Note: §10d (2026-06-30) already claims this was fixed ("Post-send message
refresh now polls briefly so FIPS/Tor/LoRa pills do not require a manual browser refresh") — this
report means it has regressed or the fix didn't fully land/deploy. Needs re-verification by the
mesh owner, not a re-fix from scratch. (The "mesh"-tag-should-read-"LoRa" report that used to be
listed alongside this was dropped 2026-07-01 — user is OK with current behavior there.)
- **[NON-MESH]** Indeedhub won't install on Arch Dev (node identity TBD — likely `.116`; confirm).
Untriaged.
- **[NON-MESH, touches bitcoin lifecycle] ROOT-CAUSED + FIX WRITTEN 2026-07-01** — Uninstalling
Bitcoin didn't stick: the container came back in My Apps and restarted IBD. Root cause:
`is_required_baseline_app` in `prod_orchestrator.rs` (bitcoin-knots, electrumx, lnd, mempool,
mempool-api, archy-mempool-db, filebrowser, fedimint-clientd) self-heals when its container is
missing — including right after an explicit uninstall — because the in-memory `disabled` set used
to suppress that is unconditionally wiped by `load_manifests()`, which runs once per archipelago
startup/reboot, immediately before the boot reconciler's first pass. Fix: a durable
`user-uninstalled.json` marker (mirrors the existing `user_stopped` mechanism in
`crash_recovery.rs`) checked at the same single reconcile choke point in
`ensure_running_with_mode`, set on successful `remove()`, cleared on `install()`/`start()`.
Test `reconcile_existing_respects_durable_user_uninstalled_marker_for_baseline_apps` passes;
`cargo test --workspace` green (873 tests). Low collision risk confirmed — the mechanism is
generic (applies to all baseline apps, not bitcoin-multi-version-specific) and the
`bitcoin-version-bulletproof` branch/worktree had no uncommitted changes in these files at the
time this was written. Not yet committed/pushed — pending user go-ahead.
- **[NON-MESH, touches bitcoin lifecycle]** Manually stopping Bitcoin causes it to auto-restart — a
user-initiated `package.stop` should NOT be treated as a crash by the auto-restart/health-monitor
logic. Investigated 2026-07-01: both live restart paths (`prod_orchestrator.rs`
`ensure_running_with_mode` and the legacy `health_monitor.rs` loop) already check the durable
`user_stopped` marker before restarting and look correctly wired on current `main` — no live
repro path found in code. Likely the reporting node's deployed binary predates a fix already on
`main`; needs the node identity + build/commit to confirm before further action.
- **[NON-MESH] FIXED 2026-07-01, LIVE ON `.228`** — `.228` Bitcoin RPC was connection-refused
("waiting for the Bitcoin RPC listener"). Root cause: the queued `bitcoin-knots-reindex` swap from
the bitcoin-rollout handover (`project_bitcoin_rollout_handover.md`) was never finished — the
detached reindex container (RPC intentionally off) had been fully synced and idling for 2 days
(height 956191, `progress=1.000000`). Executed the queued swap: stopped+removed
`bitcoin-knots-reindex`, started the managed `bitcoin-knots` service via RPC. Confirmed healthy:
v29.3.knots20260210, connected to peers, tip advanced to 956193, RPC listening on 8332.
**Follow-up same day:** user asked to confirm the version, since the UI/catalog said "latest" —
turned out the container was running a **4-month-old cached `:latest` image**
(`v29.3.knots20260210`) while the actual newest release (`29.3.knots20260508`) was already pulled
locally 2 days earlier but never applied. Root-caused why: `installed_version()` in
`set_config.rs` (`package.versions`/`package.set-config`) reported the literal image **tag string**
used to create the container (`"latest"`), not the content actually running — a stale local
`:latest` cache reports "latest" forever regardless of what `latest` has since moved to. **FIXED**:
when the resolved tag is a floating one (`latest`/`stable`/`release`/`main`), `installed_version()`
now asks the Bitcoin backend directly (`podman exec <name> bitcoind --version`, parsed via new
`parse_bitcoind_version_output`) instead of trusting the tag literal. 5 new tests in
`set_config.rs` (`floating_tag_detects_generic_channel_names`, `parses_knots_version_line`,
`parses_core_version_line`, `parse_returns_none_when_output_has_no_version_marker`,
`image_tag_keeps_registry_port_colon`) all pass. No frontend change needed — `AppSidebar.vue`
("Running Version" in the Version & Updates card) already renders `versionInfo.installedVersion`
verbatim, so it will show the real version once this backend fix ships. Then used the existing
bulletproof switch mechanism itself — `package.set-config {id: "bitcoin-knots", version:
"29.3.knots20260508"}` (an upgrade, so no downgrade-confirm gate) — to move `.228` onto the real
latest image. Confirmed: `bitcoind --version` now reports `v29.3.knots20260508`, no reindex
triggered, tip advancing normally. Not yet committed/pushed — pending user go-ahead, same batch as
the uninstall-durability fix above.
- **[NON-MESH, untriaged]** `.198` — `bitcoin-knots` RPC is saturated: logs flooded with "Request
rejected because http work queue depth exceeded" despite `-rpcworkqueue=256` already applied
(confirmed via `podman inspect`/entrypoint). This cascades into fedimint: `fedimint` /
`fedimint-gateway` / `fedimint-clientd` have been stuck in `(starting)` for 3646h because their
RPC calls to `bitcoin-knots` time out (45s) — this is almost certainly what the user meant by
"fedimint guardian keeps going down" (not `.228`, whose fedimint stack looks healthy). Root cause
of the saturation itself not yet found — suspect a multi-service retry storm (health_monitor +
fedimint x2 + electrumx + mempool + UI all polling without backoff) compounding under any bitcoind
slowdown, but not confirmed.
- **[NON-MESH, untriaged]** `.198` — portainer is completely absent from `podman ps -a` (not just
crashed/stopped — no container record at all). `.228`'s portainer is healthy for comparison. No
`/var/lib/archipelago/install.log` found on `.198` to check install history; needs a
package_data/state check via RPC or `journalctl` for the archipelago service.