archipelagoandClaude Opus 5
6ba0599639
Demo images / Build & push demo images (push) Failing after 2m13s
The repo is source code and guidelines only. Nothing about how Archipelago's own fleet is run, or how the team works, stays in it. Untracked (kept on disk, gitignored) — 250 files: - .planning/ (199) and loop/ — internal development process - fleet operations tooling that targets specific nodes: deploy-to-target, deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server, setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair, image-recipe/sync-from-live.sh - image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of which are live-server workflow and fleet node inventories - the Phase 10 on-node verification and evidence records, which cite .planning/ as their evidence base KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10 docs and then put back: it is cited as normative rationale from ten places in the codebase, including core/clippy.toml, which bans rand::thread_rng and points at it for the reason. That makes it a guideline, not an internal record. Node names removed from source (48 occurrences across comments, manifests and test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt, zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the date, which is what carried the meaning; the machine name did not. Three of those were live test values rather than comments and were replaced with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name. An earlier pass substituted "a test node" into a hostname assertion, producing an invalid hostname; caught and fixed as test-node.local. Wipe mechanism: .local-only/manifest.txt inventories every local-only path and .local-only/wipe.sh deletes them on one confirmation, refusing to touch anything git still tracks. Both are themselves untracked, so the public repo does not carry a map of internal filenames. Verified: cargo check -p archipelago --all-features clean; archipelago-container 75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link in tracked markdown resolves (0 broken). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
154 lines
8.4 KiB
Markdown
154 lines
8.4 KiB
Markdown
# The Bitcoin RPC proxy that stayed open after it was fixed
|
||
|
||
**Status:** code fix committed (`f6b5245b`); on-node verification recorded below.
|
||
**Found:** 2026-08-02, a test node, while verifying `a05956c4` instead of assuming it.
|
||
**Severity:** critical on any affected node — unauthenticated control of Bitcoin Core RPC
|
||
through a proxy that injects the node's own credentials.
|
||
|
||
## Why this document exists
|
||
|
||
`a05956c4` closed two unauthenticated endpoints on the wallet UI ports. Its commit message
|
||
stated:
|
||
|
||
> The nginx template is `include_str!`'d and re-rendered on every reconcile pass, so this
|
||
> ships atomically with the binary.
|
||
|
||
That is true for most nodes and false for a specific, silent, and not-rare state. The half
|
||
that landed correctly (LND) made the half that did not (Bitcoin RPC) *harder* to notice,
|
||
because a spot check of the LND endpoint returns a clean `401` and reads as "patched".
|
||
|
||
## What was observed
|
||
|
||
Node running the fixed binary (installed 17:21, contains the new template — `auth_request`
|
||
present in the binary at 4 occurrences). All probes from the node's own LAN address, no
|
||
cookies, no credentials:
|
||
|
||
| Probe | Result |
|
||
|---|---|
|
||
| `GET http://192.168.63.240:18083/lnd-connect-info` | `401`, 24 bytes, `{"error":"Unauthorized"}` — **closed** |
|
||
| `POST http://192.168.63.240:8334/bitcoin-rpc/` (`getblockcount`) | `200` — `{"result":960774,"error":null}` — **OPEN** |
|
||
| `OPTIONS http://192.168.63.240:8334/bitcoin-rpc/` | `204` with `Access-Control-Allow-Origin: *` — **OPEN** |
|
||
|
||
The rendered config on disk, `/var/lib/archipelago/bitcoin-ui/nginx.conf`, was dated
|
||
**2026-06-30** — the pre-fix version, with no `auth_request` and with the wildcard CORS
|
||
header the fix removes.
|
||
|
||
## Root cause
|
||
|
||
Three facts have to be true at once, and on this node they were:
|
||
|
||
1. `bitcoin-ui` is listed in the node's durable `user-uninstalled` marker
|
||
(`/var/lib/archipelago/user-uninstalled.json`).
|
||
2. `reconcile_app` returns on that marker (`prod_orchestrator.rs:1956`) **before** reaching
|
||
`run_pre_start_hooks`, which is the only thing that renders the nginx config.
|
||
3. The container keeps running anyway, because it is owned by **systemd via a Quadlet
|
||
unit** — `archy-bitcoin-ui.service`, `active`, restarted 17:25 after the daemon restart —
|
||
not by the reconciler that is refusing to touch it.
|
||
|
||
So: *a container systemd keeps alive, that the orchestrator has stopped reconciling, never
|
||
receives a config fix shipped inside the binary.* The marker means "must stay removed", but
|
||
nothing enforces removal against systemd, and the orchestrator treats the marker as
|
||
permission to stop looking.
|
||
|
||
This is not a one-app accident. On the same node `archy-electrs-ui` is in the identical
|
||
state (uninstalled marker + active Quadlet unit + `Up 10 days`). It serves only a static
|
||
page with no credential-injecting proxy, so its exposure is low — but it would miss any
|
||
future config fix the same way.
|
||
|
||
## Why it matters beyond this node
|
||
|
||
An OTA carrying `a05956c4` would have closed the LND leak everywhere and silently failed to
|
||
close the Bitcoin RPC proxy on every node in this state — while making those nodes *look*
|
||
patched to exactly the check an operator would run first. That is the most misleading
|
||
possible outcome of shipping a security fix.
|
||
|
||
## The fix
|
||
|
||
`f6b5245b`: a container that is actually running is a live attack surface whatever a marker
|
||
says about it, so its security-relevant config is reconciled even behind the marker, and the
|
||
container is restarted so nginx loads it.
|
||
|
||
Deliberately narrow:
|
||
|
||
- Nothing is created, pulled, built, started or resurrected. The "must stay removed"
|
||
contract can only weaken for a container that is **already running**, which by definition
|
||
means it was never removed.
|
||
- A hook error is swallowed, not propagated — an app the user uninstalled must not be able
|
||
to fail the reconcile pass for every app after it.
|
||
- The pre-existing marker test passes unchanged; that is what proves the removal contract
|
||
survived. A new regression test pins the whole chain: stale conf in, gate present out,
|
||
container restarted, nothing created.
|
||
|
||
## What actually closed it on a test node — and what that does NOT prove
|
||
|
||
Sequence, from file mtimes, container start times and the daemon journal:
|
||
|
||
| Time (EDT) | Event |
|
||
|---|---|
|
||
| 18:33 | Probe: `POST /bitcoin-rpc/` → `200` with a real block height. Exposure confirmed live. |
|
||
| 18:36 | A **separate rebuild of bitcoin-ui**, done outside this work, rendered the fixed conf and recreated `archy-bitcoin-ui`. `:8334` closes here. |
|
||
| 19:06 | The binary carrying `f6b5245b` is installed and the daemon restarted. |
|
||
| 19:12 | Probe: `POST /bitcoin-rpc/` → `401`. `OPTIONS` now returns `Access-Control-Allow-Origin: http://192.168.63.240:8334`, not `*`. |
|
||
|
||
So the node is closed, and the fixed template is proven to work end to end on real
|
||
hardware — but **the reconcile fix itself was never exercised.** By the time it was
|
||
deployed, the state it repairs had already been cleared by the unrelated rebuild. The
|
||
`401` proves `a05956c4`'s template; it does not prove the delivery path `f6b5245b` adds.
|
||
|
||
That distinction is the whole point of this document, so it is recorded rather than
|
||
rounded off: `bitcoin-ui` is *still* in the node's `user-uninstalled` marker, meaning the
|
||
next time its config needs to change, this node depends on `f6b5245b` — untested — or on
|
||
someone happening to rebuild the app again.
|
||
|
||
Tracked as broken window 15 — **since closed by the controlled test below.**
|
||
|
||
## Proving the delivery path on real hardware
|
||
|
||
Run on a test node, 2026-08-02 20:00–20:03 EDT, with operator approval. The point was to
|
||
prove the thing the incidental rebuild had made unprovable: that **reconcile itself**
|
||
repairs this state, unaided.
|
||
|
||
The daemon was stopped first, so the reconciler could not repair the state before the
|
||
re-exposure had been confirmed — otherwise a passing probe would prove nothing about
|
||
which mechanism produced it.
|
||
|
||
| Step | Action | Observed |
|
||
|---|---|---|
|
||
| 1 | Install a faithfully stale conf (no `auth_request`, credential-injecting `proxy_pass`, `Allow-Origin: *`) and restart the container | — |
|
||
| 2 | Probe with no cookies | `POST /bitcoin-rpc/` → **`200`**, `{"result":960790}`; `Allow-Origin: *`. **Genuinely re-exposed** |
|
||
| 3 | Start the daemon (20:00:36) and touch nothing further | — |
|
||
| 4 | Reconcile pass at **20:02:19** | `bitcoin_ui: nginx.conf rendered auth_hash=51f2b5af`, then `WARN prod_orchestrator: rewrote config for a user-uninstalled app whose container is still RUNNING (systemd/Quadlet keeps it alive independently of reconcile) — restarting so it picks the new config up app_id=bitcoin-ui container=archy-bitcoin-ui` |
|
||
| 5 | Probe again | `POST /bitcoin-rpc/` → **`401`**; `Allow-Origin: http://192.168.63.240:8334` |
|
||
| 6 | Compare state | Conf **byte-identical** to the pre-test known-good; container healthy |
|
||
|
||
Step 2 is what makes steps 4–6 mean anything: without a confirmed `200`, the later `401`
|
||
would be consistent with the state never having been broken at all.
|
||
|
||
Both halves are now proven on hardware: `a05956c4`'s template (the gate works) and
|
||
`f6b5245b`'s delivery path (the gate arrives at a container the reconciler had been
|
||
skipping).
|
||
|
||
## Credential rotation — decided against, 2026-08-02
|
||
|
||
The operator's call, recorded here so it is not silently re-litigated: **no LND macaroon
|
||
rotation, and no Bitcoin RPC password rotation.** The reasoning was that there is no
|
||
evidence of exploitation and the vulnerability is being closed rather than lived with.
|
||
|
||
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. It has been
|
||
exercised in detect mode only, and has never rotated anything on any node. Its ordering
|
||
guard (refuses to rotate on a binary lacking the fix) remains the right shape for whenever
|
||
rotation is wanted — including for the Bitcoin RPC password, which has no equivalent tool
|
||
yet.
|
||
|
||
What this decision accepts: any macaroon or RPC password read through either hole before
|
||
it was closed stays valid. That is a deliberate, informed trade, not an oversight.
|
||
|
||
## Operator note
|
||
|
||
Deploying the fix rewrites the config and restarts `archy-bitcoin-ui` (a brief Bitcoin UI
|
||
interruption, nothing else). Any node that ever had `bitcoin-ui` uninstalled while its
|
||
Quadlet unit stayed active should be re-probed with the `POST /bitcoin-rpc/` check above —
|
||
a `401` is the pass condition. Treat the Bitcoin RPC password on any node that answered
|
||
`200` as known to anyone who could reach that port, and rotate it **after** the fix is
|
||
deployed, never before.
|