archipelagoandClaude Opus 5
8e814ca06a
Demo images / Build & push demo images (push) Failing after 2m22s
Replaces the registry host across 86 files: 309 references, covering all 40 app manifests, the orchestrator and container crates, the release and catalog scripts, both demo-images workflows, the ISO builder, demo-deploy, and the frontend marketplace data. Verified the domain actually serves the registry before rewriting anything, rather than assuming the web host implies the registry: - TLS verifies clean, HTTP/2 on the web root - an anonymous token grants a manifest fetch (HTTP 200) with no credentials - skopeo inspect --no-creds resolves an image and lists its tags That last check is the one that matters: an outside developer with no account can now pull, which was the functional blocker for publishing at all. Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop crossing the network in the clear. Deliberately NOT rewritten: - The public FIPS anchor on port 8444. It is a functional network endpoint every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded seeds than to leaked infrastructure. The domain does resolve to the same host, so it could become a hostname, but that adds a DNS dependency to the path used precisely when things are broken. Worth a deliberate decision, not a side effect of this change. - The companion APK on port 2100. The domain returns 404 for that path, so rewriting it would swap a working URL for a broken one. The Releases page does serve (200), which is where the plan already wants those binaries. - releases/app-catalog.json, releases/manifest.json and release-manifest.json. These carry `signature` and `signed_by`; editing their contents invalidates the signature and the fleet refuses artifacts that fail verification. They were rewritten in a first pass and reverted — they must be regenerated and re-signed through the signing ceremony instead, which needs the mnemonic. So the catalog still advertises the old host until that ceremony runs. Nodes resolve images through the signed catalog, not the on-disk manifests, so this commit alone does not change what a node pulls. Verified: archipelago-container 75/75; every manifest still parses with a top-level app block; no signed artifact modified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
132 lines
8.3 KiB
Markdown
132 lines
8.3 KiB
Markdown
# Bitcoin Multi-Version — Bulletproofing & Rollout (handoff)
|
|
|
|
> **Status 2026-06-29:** code + images + catalog + frontend DONE on branch
|
|
> `bitcoin-version-bulletproof` (base commit `095a76cd`, plus the catalog-generator
|
|
> + handoff follow-ups). **.228 is the test node**: binary + frontend + catalog are
|
|
> live there; its Knots chainstate is mid-**reindex recovery** (see §5). The fleet
|
|
> rollout (OTA binary+frontend, mirror catalog publish, `:latest` repoint) is the
|
|
> **coordinated step the other agent owns** — see §4. Pairs with
|
|
> `docs/bitcoin-multi-version-design.md` (the original design).
|
|
|
|
## 1. What was broken (root causes)
|
|
|
|
User report: "switched Knots to `v29.3.knots20260508`, version didn't update in the UI."
|
|
Three **stacked** bugs, plus a data-corruption hazard:
|
|
|
|
1. **Reconciler reverted the pin.** `prod_orchestrator::sync_quadlet_unit` re-rendered the
|
|
quadlet every reconcile tick using the manifest's `:latest`, ignoring the per-app
|
|
pinned version → any switch silently reverted within one tick.
|
|
2. **Entrypoint render bug.** The renderer folded the manifest `entrypoint: ["sh","-lc"]`
|
|
into `Exec=`. That only works when the image ENTRYPOINT is a passthrough shell wrapper.
|
|
The versioned images use `ENTRYPOINT ["bitcoind"]`, so `Exec=sh -lc …` became
|
|
`bitcoind sh -lc …` → `unexpected token 'sh'` → crash loop.
|
|
3. **Image USER divergence.** The versioned images were built `USER bitcoin` (uid 1000);
|
|
the legacy `:latest` ran as **root**. Chain data is owned by the `data_uid`
|
|
(host 100101 / container uid 102). Root reads it via `CAP_DAC_OVERRIDE` (granted in the
|
|
manifest); uid-1000 cannot → `Error initializing block database`.
|
|
4. **Data hazard (already hit on .228).** Repeated failed starts under mixed UIDs left
|
|
bitcoind's two LevelDBs (`blocks/index/` + `chainstate/`) truncated to KB stubs while
|
|
the raw `blocks/blk*.dat` (797 GB) stayed intact. Recovery = `bitcoind -reindex` from
|
|
local blocks (no re-download). The uniform-root image fix (below) removes the mixed-UID
|
|
cause going forward; the proper switch flow was already data-safe (600s stop grace,
|
|
clean stop→rm→recreate, conflict-stops the other impl — they share port 8332 + datadir
|
|
`/var/lib/archipelago/bitcoin`).
|
|
|
|
## 2. What was fixed (all on the branch)
|
|
|
|
- **Renderer** (`core/archipelago/src/container/`):
|
|
- `prod_orchestrator.rs`: factored `resolve_catalog_image()` (catalog/pinned-version →
|
|
image) and call it in BOTH `install_fresh` and `sync_quadlet_unit` — the pin now
|
|
survives reconcile.
|
|
- `quadlet.rs`: emit a real `Entrypoint=<first>` + `Exec=<rest+cmd>` instead of folding;
|
|
`exec_changed` now also diffs `Entrypoint=` so the recreate fires. Validated against
|
|
the live podman 5.4.2 quadlet generator.
|
|
- **Images** (`scripts/build-bitcoin-image.sh`, `apps/bitcoin-{knots,core}/Dockerfile`):
|
|
removed `USER bitcoin` → run as **container-root** like legacy (still 100% rootless:
|
|
container-root maps to the unprivileged host service user; `CAP_DAC_OVERRIDE` from the
|
|
manifest lets bitcoind read the `data_uid`-owned datadir). **All** images rebuilt root +
|
|
pushed to the mirror (`source.archipelago-foundation.org/lfg2025`):
|
|
- Knots: `29.3.knots20260508`, `29.3.knots20260507`, `29.3.knots20260210`, `29.2.knots20251110`
|
|
- Core: `25.2 26.2 27.2 28.4 29.2 29.3 30.2 31.0` + `latest` (→31.0)
|
|
- **Catalog** (`scripts/generate-app-catalog.sh` VERSIONS map + regenerated
|
|
`releases/app-catalog.json`): Knots & Core `versions[]` populated; the generator now
|
|
forces top-level `version` == the `default` entry's version (the `169ff2e2` invariant)
|
|
regardless of the manifest version. Knots `latest` entry points at the newest **dated**
|
|
image (`29.3.knots20260508`) so "Always use latest" = newest on fixed-binary nodes.
|
|
- **Frontend** (`neode-ui/`):
|
|
- `AppSidebar.vue`: rename the latest option to **"Always use the latest version"**
|
|
(no `v` prefix), fix right padding, and `pickSelection()` guarantees the bound value is
|
|
a real option (fixes the blank dropdown).
|
|
- New `components/InstallVersionModal.vue`: full-screen version chooser shown from the
|
|
App Store / Discover **card** install button for multi-version apps — app icon +
|
|
"Install <name>", latest pre-selected. Wired in `Discover.vue handleInstall`.
|
|
- i18n keys: `appDetails.alwaysUseLatestVersion`, `marketplace.installModalTitle/Hint`.
|
|
|
|
## 3. Current live state on .228 (test node)
|
|
|
|
- Binary with both renderer fixes: **deployed** (`/usr/local/bin/archipelago`).
|
|
- New frontend bundle: **deployed** to `/opt/archipelago/web-ui` (hard-refresh to see it).
|
|
- Updated catalog: placed at `/var/lib/archipelago/app-catalog.json` (local override —
|
|
will refresh from the mirror's OLDER copy at the next hourly fetch until §4 publishes it).
|
|
- Knots: `bitcoin-knots` service held **stopped** (`package.stop`, user_stopped);
|
|
a detached `bitcoin-knots-reindex` container is rebuilding the index+UTXO (§5).
|
|
|
|
## 4. Remaining — coordinated fleet rollout (OTHER AGENT)
|
|
|
|
Do this together with the other workstream's release, AFTER both are ready:
|
|
|
|
1. **Merge** branch `bitcoin-version-bulletproof` into the release line.
|
|
2. **Build + OTA** the binary + frontend (these carry the renderer fix + UI). The renderer
|
|
fix is a **hard prerequisite** for the new images everywhere — see fleet-safety below.
|
|
3. **Publish the catalog** to the mirror (push `releases/app-catalog.json` to gitea-vps2
|
|
`main`, the raw URL nodes fetch hourly). The current catalog is **fleet-safe even before
|
|
the binary lands**: unpinned/auto-update nodes resolve via the manifest's floating
|
|
`:latest` (still the legacy image); only explicit version selection (needs the new UI)
|
|
uses the new root images.
|
|
4. **Only AFTER the binary is fleet-wide:** optionally repoint the `bitcoin-knots:latest`
|
|
tag → `29.3.knots20260508` (root) and simplify the catalog `latest` entry back to the
|
|
`:latest` tag. **Do NOT repoint `:latest` before then** — old-binary nodes fold
|
|
`Exec=sh -lc …` and would crash on an `ENTRYPOINT ["bitcoind"]` image. (Core never
|
|
worked on old binaries — it always shipped `ENTRYPOINT ["bitcoind"]` — so Core has no
|
|
such constraint.)
|
|
5. **Verify the full switch matrix** on a healthy node (§6).
|
|
|
|
## 5. Finishing .228's reindex (OTHER AGENT owns this — not babysat by the original author)
|
|
|
|
The detached `bitcoin-knots-reindex` container runs the new **root** `29.3.knots20260508`
|
|
image with `-reindex -server=0` against `/var/lib/archipelago/bitcoin`. It holds the datadir
|
|
lock, so the managed service (held stopped) can't collide. When it has connected blocks up
|
|
to ~the prior tip (height ≥ ~955800) it's done; then:
|
|
|
|
```sh
|
|
# on .228 (SSH/sudo/UI pw all: <FLEET_PW>)
|
|
podman stop -t 600 bitcoin-knots-reindex && podman rm bitcoin-knots-reindex
|
|
# start the managed service via RPC (sets desired=running, clears user_stopped):
|
|
# package.start {id: bitcoin-knots} (POST https://127.0.0.1/rpc/v1, CSRF: echo csrf_token cookie as X-CSRF-Token)
|
|
# verify:
|
|
podman exec bitcoin-knots sh -lc '$(command -v bitcoind) --version | head -1' # → v29.3.knots20260508
|
|
# RPC up → the Bitcoin UI populates; it syncs the gap to tip.
|
|
```
|
|
The "Bitcoin RPC connection refused (127.0.0.1:8332)" the UI shows is EXPECTED until this
|
|
swap (reindex runs with RPC off).
|
|
|
|
## 6. Switch-matrix test plan (what "bulletproof" must prove)
|
|
|
|
On a healthy node, each step must end with bitcoind running + RPC answering + syncing, with
|
|
NO `Error initializing block database` and NO data loss:
|
|
- Knots: switch `latest` → `29.3.knots20260507` → `29.3.knots20260210` → back to `latest`.
|
|
- Core: install `latest`; switch `31.0` → `28.4.0`.
|
|
- **Knots ↔ Core** (shared datadir/port): Knots→Core upgrade path (Core ≥ data version) and
|
|
the reverse. **Cross-major DOWNGRADES** (e.g. 29.x data → Core 28.4) legitimately need a
|
|
reindex — the UI already surfaces a downgrade warning; confirm it does and that confirming
|
|
reindexes cleanly rather than crash-looping.
|
|
- Reboot survival after each switch.
|
|
|
|
## 7. Notes / assumptions
|
|
|
|
- **"29.2"** in the request doesn't exist as a Knots build (404 upstream); added as **Bitcoin
|
|
Core 29.2** (exists). Revisit if a Knots 29.2 was meant.
|
|
- Reindex is unavoidable ONLY because .228's index was already corrupted by the pre-fix
|
|
crash loop; a normal switch on the fixed binary does NOT reindex.
|
|
- Creds for .228: SSH/sudo + UI/RPC all `<FLEET_PW>`.
|