diff --git a/.gitignore b/.gitignore index e2e7fdcf..9d9e082a 100644 --- a/.gitignore +++ b/.gitignore @@ -141,3 +141,4 @@ uploads/ /docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md /image-recipe/INTEGRATION-GUIDE.md /docs/multinode-testing-plan.md +/docs/bitcoin-version-bulletproof-rollout.md diff --git a/docs/1.8.0-RELEASE-HARDENING-PLAN.md b/docs/1.8.0-RELEASE-HARDENING-PLAN.md index e1a204fd..550c6e7b 100644 --- a/docs/1.8.0-RELEASE-HARDENING-PLAN.md +++ b/docs/1.8.0-RELEASE-HARDENING-PLAN.md @@ -391,7 +391,7 @@ media (latest artifact only one minor behind). **`1.8.0-alpha`**. Remaining work is the mechanical bump + `create-release.sh` run when the gate criteria are met. - [ ] 🟒 **Bitcoin multi-version fleet OTA** β€” DECIDED (user, 2026-07-08): timing doesn't - matter; fold the branch into the next fleet OTA (`docs/bitcoin-version-bulletproof-rollout.md`). + matter; fold the branch into the next fleet OTA. - [x] ~~β›”πŸŸ’ **3ccc stock-Meshtastic RF validation**~~ β€” DROPPED per user 2026-07-08; the code fix stays in, no live-radio validation will be scheduled. diff --git a/docs/bitcoin-multi-version-design.md b/docs/bitcoin-multi-version-design.md index 8eb1daf8..a91db710 100644 --- a/docs/bitcoin-multi-version-design.md +++ b/docs/bitcoin-multi-version-design.md @@ -1,91 +1,7 @@ # Bitcoin Multi-Version Support β€” Design - - -**Status:** design (2026-06-22) +**Status:** implemented β€” all four phases shipped (catalog schema, install-time selection, in-app switch + auto-update toggle, and the verified image build pipeline). Downgrades are guarded: the update path never offers a lower version than what is running. **Goal:** let a user choose *which* version of Bitcoin Core / Bitcoin Knots to install (latest pre-selected, older versions in a dropdown), and later switch versions or opt into auto-update β€” all manifest/catalog-driven, all served from @@ -94,13 +10,8 @@ changes. See also: [`docs/registry-manifest-design.md`](registry-manifest-design.md) (catalog distribution + signing this builds on), -the production test gate (which must be -green first), `MEMORY β†’ project_decoupled_app_updates`, -`MEMORY β†’ project_manifest_driven_north_star`. +and the production test gate (which must be green first). -> **Scheduling:** this is net-new scope. It lands **after** the production test -> gate (`tests/lifecycle/run-20x.sh`) is green on `.228` + `.198`. The data- -> preservation invariant (downgrade vs. chainstate) is the highest risk here. --- @@ -260,7 +171,7 @@ free. `version`/`image` stay as the default for back-compat. - **Floating tags** (`latest`) are never advertised as a selectable "version" and never counted as an available update (already handled by `available_update_for_app`). -- **Verify on a real node** (`.228` then `.198`) and pass `run-20x` before any +- **Verify on a real node** and pass the lifecycle gate before any tag. --- diff --git a/docs/bitcoin-version-bulletproof-rollout.md b/docs/bitcoin-version-bulletproof-rollout.md deleted file mode 100644 index 2c82f13a..00000000 --- a/docs/bitcoin-version-bulletproof-rollout.md +++ /dev/null @@ -1,131 +0,0 @@ -# Bitcoin Multi-Version β€” Bulletproofing & Rollout (handoff) - -> **Status 2026-06-29:** code + images + catalog + frontend DONE on branch -> `bitcoin-version-bulletproof` (base commit `095a76cd`, plus the catalog-generator -> + handoff follow-ups). **.228 is the test node**: binary + frontend + catalog are -> live there; its Knots chainstate is mid-**reindex recovery** (see Β§5). The fleet -> rollout (OTA binary+frontend, mirror catalog publish, `:latest` repoint) is the -> **coordinated step the other agent owns** β€” see Β§4. Pairs with -> `docs/bitcoin-multi-version-design.md` (the original design). - -## 1. What was broken (root causes) - -User report: "switched Knots to `v29.3.knots20260508`, version didn't update in the UI." -Three **stacked** bugs, plus a data-corruption hazard: - -1. **Reconciler reverted the pin.** `prod_orchestrator::sync_quadlet_unit` re-rendered the - quadlet every reconcile tick using the manifest's `:latest`, ignoring the per-app - pinned version β†’ any switch silently reverted within one tick. -2. **Entrypoint render bug.** The renderer folded the manifest `entrypoint: ["sh","-lc"]` - into `Exec=`. That only works when the image ENTRYPOINT is a passthrough shell wrapper. - The versioned images use `ENTRYPOINT ["bitcoind"]`, so `Exec=sh -lc …` became - `bitcoind sh -lc …` β†’ `unexpected token 'sh'` β†’ crash loop. -3. **Image USER divergence.** The versioned images were built `USER bitcoin` (uid 1000); - the legacy `:latest` ran as **root**. Chain data is owned by the `data_uid` - (host 100101 / container uid 102). Root reads it via `CAP_DAC_OVERRIDE` (granted in the - manifest); uid-1000 cannot β†’ `Error initializing block database`. -4. **Data hazard (already hit on .228).** Repeated failed starts under mixed UIDs left - bitcoind's two LevelDBs (`blocks/index/` + `chainstate/`) truncated to KB stubs while - the raw `blocks/blk*.dat` (797 GB) stayed intact. Recovery = `bitcoind -reindex` from - local blocks (no re-download). The uniform-root image fix (below) removes the mixed-UID - cause going forward; the proper switch flow was already data-safe (600s stop grace, - clean stopβ†’rmβ†’recreate, conflict-stops the other impl β€” they share port 8332 + datadir - `/var/lib/archipelago/bitcoin`). - -## 2. What was fixed (all on the branch) - -- **Renderer** (`core/archipelago/src/container/`): - - `prod_orchestrator.rs`: factored `resolve_catalog_image()` (catalog/pinned-version β†’ - image) and call it in BOTH `install_fresh` and `sync_quadlet_unit` β€” the pin now - survives reconcile. - - `quadlet.rs`: emit a real `Entrypoint=` + `Exec=` instead of folding; - `exec_changed` now also diffs `Entrypoint=` so the recreate fires. Validated against - the live podman 5.4.2 quadlet generator. -- **Images** (`scripts/build-bitcoin-image.sh`, `apps/bitcoin-{knots,core}/Dockerfile`): - removed `USER bitcoin` β†’ run as **container-root** like legacy (still 100% rootless: - container-root maps to the unprivileged host service user; `CAP_DAC_OVERRIDE` from the - manifest lets bitcoind read the `data_uid`-owned datadir). **All** images rebuilt root + - pushed to the mirror (`source.archipelago-foundation.org/lfg2025`): - - Knots: `29.3.knots20260508`, `29.3.knots20260507`, `29.3.knots20260210`, `29.2.knots20251110` - - Core: `25.2 26.2 27.2 28.4 29.2 29.3 30.2 31.0` + `latest` (β†’31.0) -- **Catalog** (`scripts/generate-app-catalog.sh` VERSIONS map + regenerated - `releases/app-catalog.json`): Knots & Core `versions[]` populated; the generator now - forces top-level `version` == the `default` entry's version (the `169ff2e2` invariant) - regardless of the manifest version. Knots `latest` entry points at the newest **dated** - image (`29.3.knots20260508`) so "Always use latest" = newest on fixed-binary nodes. -- **Frontend** (`neode-ui/`): - - `AppSidebar.vue`: rename the latest option to **"Always use the latest version"** - (no `v` prefix), fix right padding, and `pickSelection()` guarantees the bound value is - a real option (fixes the blank dropdown). - - New `components/InstallVersionModal.vue`: full-screen version chooser shown from the - App Store / Discover **card** install button for multi-version apps β€” app icon + - "Install ", latest pre-selected. Wired in `Discover.vue handleInstall`. - - i18n keys: `appDetails.alwaysUseLatestVersion`, `marketplace.installModalTitle/Hint`. - -## 3. Current live state on .228 (test node) - -- Binary with both renderer fixes: **deployed** (`/usr/local/bin/archipelago`). -- New frontend bundle: **deployed** to `/opt/archipelago/web-ui` (hard-refresh to see it). -- Updated catalog: placed at `/var/lib/archipelago/app-catalog.json` (local override β€” - will refresh from the mirror's OLDER copy at the next hourly fetch until Β§4 publishes it). -- Knots: `bitcoin-knots` service held **stopped** (`package.stop`, user_stopped); - a detached `bitcoin-knots-reindex` container is rebuilding the index+UTXO (Β§5). - -## 4. Remaining β€” coordinated fleet rollout (OTHER AGENT) - -Do this together with the other workstream's release, AFTER both are ready: - -1. **Merge** branch `bitcoin-version-bulletproof` into the release line. -2. **Build + OTA** the binary + frontend (these carry the renderer fix + UI). The renderer - fix is a **hard prerequisite** for the new images everywhere β€” see fleet-safety below. -3. **Publish the catalog** to the mirror (push `releases/app-catalog.json` to gitea-vps2 - `main`, the raw URL nodes fetch hourly). The current catalog is **fleet-safe even before - the binary lands**: unpinned/auto-update nodes resolve via the manifest's floating - `:latest` (still the legacy image); only explicit version selection (needs the new UI) - uses the new root images. -4. **Only AFTER the binary is fleet-wide:** optionally repoint the `bitcoin-knots:latest` - tag β†’ `29.3.knots20260508` (root) and simplify the catalog `latest` entry back to the - `:latest` tag. **Do NOT repoint `:latest` before then** β€” old-binary nodes fold - `Exec=sh -lc …` and would crash on an `ENTRYPOINT ["bitcoind"]` image. (Core never - worked on old binaries β€” it always shipped `ENTRYPOINT ["bitcoind"]` β€” so Core has no - such constraint.) -5. **Verify the full switch matrix** on a healthy node (Β§6). - -## 5. Finishing .228's reindex (OTHER AGENT owns this β€” not babysat by the original author) - -The detached `bitcoin-knots-reindex` container runs the new **root** `29.3.knots20260508` -image with `-reindex -server=0` against `/var/lib/archipelago/bitcoin`. It holds the datadir -lock, so the managed service (held stopped) can't collide. When it has connected blocks up -to ~the prior tip (height β‰₯ ~955800) it's done; then: - -```sh -# on .228 (SSH/sudo/UI pw all: ) -podman stop -t 600 bitcoin-knots-reindex && podman rm bitcoin-knots-reindex -# start the managed service via RPC (sets desired=running, clears user_stopped): -# package.start {id: bitcoin-knots} (POST https://127.0.0.1/rpc/v1, CSRF: echo csrf_token cookie as X-CSRF-Token) -# verify: -podman exec bitcoin-knots sh -lc '$(command -v bitcoind) --version | head -1' # β†’ v29.3.knots20260508 -# RPC up β†’ the Bitcoin UI populates; it syncs the gap to tip. -``` -The "Bitcoin RPC connection refused (127.0.0.1:8332)" the UI shows is EXPECTED until this -swap (reindex runs with RPC off). - -## 6. Switch-matrix test plan (what "bulletproof" must prove) - -On a healthy node, each step must end with bitcoind running + RPC answering + syncing, with -NO `Error initializing block database` and NO data loss: -- Knots: switch `latest` β†’ `29.3.knots20260507` β†’ `29.3.knots20260210` β†’ back to `latest`. -- Core: install `latest`; switch `31.0` β†’ `28.4.0`. -- **Knots ↔ Core** (shared datadir/port): Knotsβ†’Core upgrade path (Core β‰₯ data version) and - the reverse. **Cross-major DOWNGRADES** (e.g. 29.x data β†’ Core 28.4) legitimately need a - reindex β€” the UI already surfaces a downgrade warning; confirm it does and that confirming - reindexes cleanly rather than crash-looping. -- Reboot survival after each switch. - -## 7. Notes / assumptions - -- **"29.2"** in the request doesn't exist as a Knots build (404 upstream); added as **Bitcoin - Core 29.2** (exists). Revisit if a Knots 29.2 was meant. -- Reindex is unavoidable ONLY because .228's index was already corrupted by the pre-fix - crash loop; a normal switch on the fixed binary does NOT reindex. -- Creds for .228: SSH/sudo + UI/RPC all ``. diff --git a/scripts/generate-app-catalog.sh b/scripts/generate-app-catalog.sh index 73a933c6..c0f8f72a 100755 --- a/scripts/generate-app-catalog.sh +++ b/scripts/generate-app-catalog.sh @@ -203,7 +203,7 @@ VERSIONS = { # newest build on fixed-binary nodes, while UNPINNED nodes still resolve via # the manifest's floating :latest tag (kept on the legacy image until the # entrypoint-render fix is fleet-deployed β€” see - # docs/bitcoin-version-bulletproof-rollout.md). + # the bitcoin multi-version design). "bitcoin-knots": [ {"version": "latest", "image": f"{REGISTRY}/bitcoin-knots:29.3.knots20260508", "default": True},