fix(orchestrator): durable uninstall marker for baseline apps + archival-bitcoin/version-report gaps
- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
gap where installing it standalone (a legitimate direct orchestrator-install
target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
etc.) resurrecting itself after an explicit uninstall survives a restart or
reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
("latest") as the reported running version -- a stale local :latest cache
reported "latest" forever regardless of what latest had moved on to. Now
falls back to asking the Bitcoin backend directly via `bitcoind --version`
when the tag is floating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
de8b2bb812
commit
5b7cd5d5d0
@@ -1010,3 +1010,79 @@ this match.
|
||||
- Definition of done: a new text sent from stock Meshtastic `3ccc` appears in `.116`
|
||||
`mesh.messages` as an incoming LoRa message without a browser refresh, and `.116` -> `3ccc`
|
||||
visibly arrives in the Meshtastic app.
|
||||
|
||||
## 11. Arch Issues (reported 2026-07-01, untriaged)
|
||||
|
||||
User-reported, raw, not yet root-caused. Split by owner — **do not fix the mesh items from the
|
||||
non-mesh thread**; they route to the mesh/Reticulum agent (§10d owner).
|
||||
|
||||
- **[MESH — routes to §10d owner]** Transport-type label on mesh is delayed / requires a browser
|
||||
refresh to show. Note: §10d (2026-06-30) already claims this was fixed ("Post-send message
|
||||
refresh now polls briefly so FIPS/Tor/LoRa pills do not require a manual browser refresh") — this
|
||||
report means it has regressed or the fix didn't fully land/deploy. Needs re-verification by the
|
||||
mesh owner, not a re-fix from scratch. (The "mesh"-tag-should-read-"LoRa" report that used to be
|
||||
listed alongside this was dropped 2026-07-01 — user is OK with current behavior there.)
|
||||
- **[NON-MESH]** Indeedhub won't install on Arch Dev (node identity TBD — likely `.116`; confirm).
|
||||
Untriaged.
|
||||
- **[NON-MESH, touches bitcoin lifecycle] ROOT-CAUSED + FIX WRITTEN 2026-07-01** — Uninstalling
|
||||
Bitcoin didn't stick: the container came back in My Apps and restarted IBD. Root cause:
|
||||
`is_required_baseline_app` in `prod_orchestrator.rs` (bitcoin-knots, electrumx, lnd, mempool,
|
||||
mempool-api, archy-mempool-db, filebrowser, fedimint-clientd) self-heals when its container is
|
||||
missing — including right after an explicit uninstall — because the in-memory `disabled` set used
|
||||
to suppress that is unconditionally wiped by `load_manifests()`, which runs once per archipelago
|
||||
startup/reboot, immediately before the boot reconciler's first pass. Fix: a durable
|
||||
`user-uninstalled.json` marker (mirrors the existing `user_stopped` mechanism in
|
||||
`crash_recovery.rs`) checked at the same single reconcile choke point in
|
||||
`ensure_running_with_mode`, set on successful `remove()`, cleared on `install()`/`start()`.
|
||||
Test `reconcile_existing_respects_durable_user_uninstalled_marker_for_baseline_apps` passes;
|
||||
`cargo test --workspace` green (873 tests). Low collision risk confirmed — the mechanism is
|
||||
generic (applies to all baseline apps, not bitcoin-multi-version-specific) and the
|
||||
`bitcoin-version-bulletproof` branch/worktree had no uncommitted changes in these files at the
|
||||
time this was written. Not yet committed/pushed — pending user go-ahead.
|
||||
- **[NON-MESH, touches bitcoin lifecycle]** Manually stopping Bitcoin causes it to auto-restart — a
|
||||
user-initiated `package.stop` should NOT be treated as a crash by the auto-restart/health-monitor
|
||||
logic. Investigated 2026-07-01: both live restart paths (`prod_orchestrator.rs`
|
||||
`ensure_running_with_mode` and the legacy `health_monitor.rs` loop) already check the durable
|
||||
`user_stopped` marker before restarting and look correctly wired on current `main` — no live
|
||||
repro path found in code. Likely the reporting node's deployed binary predates a fix already on
|
||||
`main`; needs the node identity + build/commit to confirm before further action.
|
||||
- **[NON-MESH] FIXED 2026-07-01, LIVE ON `.228`** — `.228` Bitcoin RPC was connection-refused
|
||||
("waiting for the Bitcoin RPC listener"). Root cause: the queued `bitcoin-knots-reindex` swap from
|
||||
the bitcoin-rollout handover (`project_bitcoin_rollout_handover.md`) was never finished — the
|
||||
detached reindex container (RPC intentionally off) had been fully synced and idling for 2 days
|
||||
(height 956191, `progress=1.000000`). Executed the queued swap: stopped+removed
|
||||
`bitcoin-knots-reindex`, started the managed `bitcoin-knots` service via RPC. Confirmed healthy:
|
||||
v29.3.knots20260210, connected to peers, tip advanced to 956193, RPC listening on 8332.
|
||||
**Follow-up same day:** user asked to confirm the version, since the UI/catalog said "latest" —
|
||||
turned out the container was running a **4-month-old cached `:latest` image**
|
||||
(`v29.3.knots20260210`) while the actual newest release (`29.3.knots20260508`) was already pulled
|
||||
locally 2 days earlier but never applied. Root-caused why: `installed_version()` in
|
||||
`set_config.rs` (`package.versions`/`package.set-config`) reported the literal image **tag string**
|
||||
used to create the container (`"latest"`), not the content actually running — a stale local
|
||||
`:latest` cache reports "latest" forever regardless of what `latest` has since moved to. **FIXED**:
|
||||
when the resolved tag is a floating one (`latest`/`stable`/`release`/`main`), `installed_version()`
|
||||
now asks the Bitcoin backend directly (`podman exec <name> bitcoind --version`, parsed via new
|
||||
`parse_bitcoind_version_output`) instead of trusting the tag literal. 5 new tests in
|
||||
`set_config.rs` (`floating_tag_detects_generic_channel_names`, `parses_knots_version_line`,
|
||||
`parses_core_version_line`, `parse_returns_none_when_output_has_no_version_marker`,
|
||||
`image_tag_keeps_registry_port_colon`) all pass. No frontend change needed — `AppSidebar.vue`
|
||||
("Running Version" in the Version & Updates card) already renders `versionInfo.installedVersion`
|
||||
verbatim, so it will show the real version once this backend fix ships. Then used the existing
|
||||
bulletproof switch mechanism itself — `package.set-config {id: "bitcoin-knots", version:
|
||||
"29.3.knots20260508"}` (an upgrade, so no downgrade-confirm gate) — to move `.228` onto the real
|
||||
latest image. Confirmed: `bitcoind --version` now reports `v29.3.knots20260508`, no reindex
|
||||
triggered, tip advancing normally. Not yet committed/pushed — pending user go-ahead, same batch as
|
||||
the uninstall-durability fix above.
|
||||
- **[NON-MESH, untriaged]** `.198` — `bitcoin-knots` RPC is saturated: logs flooded with "Request
|
||||
rejected because http work queue depth exceeded" despite `-rpcworkqueue=256` already applied
|
||||
(confirmed via `podman inspect`/entrypoint). This cascades into fedimint: `fedimint` /
|
||||
`fedimint-gateway` / `fedimint-clientd` have been stuck in `(starting)` for 36–46h because their
|
||||
RPC calls to `bitcoin-knots` time out (45s) — this is almost certainly what the user meant by
|
||||
"fedimint guardian keeps going down" (not `.228`, whose fedimint stack looks healthy). Root cause
|
||||
of the saturation itself not yet found — suspect a multi-service retry storm (health_monitor +
|
||||
fedimint x2 + electrumx + mempool + UI all polling without backoff) compounding under any bitcoind
|
||||
slowdown, but not confirmed.
|
||||
- **[NON-MESH, untriaged]** `.198` — portainer is completely absent from `podman ps -a` (not just
|
||||
crashed/stopped — no container record at all). `.228`'s portainer is healthy for comparison. No
|
||||
`/var/lib/archipelago/install.log` found on `.198` to check install history; needs a
|
||||
package_data/state check via RPC or `journalctl` for the archipelago service.
|
||||
|
||||
Reference in New Issue
Block a user