archipelagoandClaude Opus 5
e205f2c34a
Controlled test on archi-dev-box with operator approval. The daemon was
stopped first so the reconciler could not repair the state before the
re-exposure was confirmed — without a confirmed 200, the later 401 would
be consistent with the state never having been broken at all.
1. stale conf installed + container restarted -> POST /bitcoin-rpc/
returned 200 with a real block height and Allow-Origin: *
2. daemon started 20:00:36, nothing else touched
3. 20:02:19 reconcile rendered the conf and logged the expected warn
line naming bitcoin-ui/archy-bitcoin-ui, then restarted it
4. POST -> 401, Allow-Origin origin-scoped
5. conf byte-identical to the pre-test known-good, container healthy
Both halves are now proven on real hardware: a05956c4's template (the
gate works) and f6b5245b's delivery path (the gate reaches a container
the reconciler had been skipping).
Also records the operator's decision AGAINST credential rotation — no
macaroon, no Bitcoin RPC password — with the trade it accepts stated
plainly, so it is not silently re-litigated later.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
154 lines
8.4 KiB
Markdown
154 lines
8.4 KiB
Markdown
# The Bitcoin RPC proxy that stayed open after it was fixed
|
||
|
||
**Status:** code fix committed (`f6b5245b`); on-node verification recorded below.
|
||
**Found:** 2026-08-02, archi-dev-box, while verifying `a05956c4` instead of assuming it.
|
||
**Severity:** critical on any affected node — unauthenticated control of Bitcoin Core RPC
|
||
through a proxy that injects the node's own credentials.
|
||
|
||
## Why this document exists
|
||
|
||
`a05956c4` closed two unauthenticated endpoints on the wallet UI ports. Its commit message
|
||
stated:
|
||
|
||
> The nginx template is `include_str!`'d and re-rendered on every reconcile pass, so this
|
||
> ships atomically with the binary.
|
||
|
||
That is true for most nodes and false for a specific, silent, and not-rare state. The half
|
||
that landed correctly (LND) made the half that did not (Bitcoin RPC) *harder* to notice,
|
||
because a spot check of the LND endpoint returns a clean `401` and reads as "patched".
|
||
|
||
## What was observed
|
||
|
||
Node running the fixed binary (installed 17:21, contains the new template — `auth_request`
|
||
present in the binary at 4 occurrences). All probes from the node's own LAN address, no
|
||
cookies, no credentials:
|
||
|
||
| Probe | Result |
|
||
|---|---|
|
||
| `GET http://192.168.63.240:18083/lnd-connect-info` | `401`, 24 bytes, `{"error":"Unauthorized"}` — **closed** |
|
||
| `POST http://192.168.63.240:8334/bitcoin-rpc/` (`getblockcount`) | `200` — `{"result":960774,"error":null}` — **OPEN** |
|
||
| `OPTIONS http://192.168.63.240:8334/bitcoin-rpc/` | `204` with `Access-Control-Allow-Origin: *` — **OPEN** |
|
||
|
||
The rendered config on disk, `/var/lib/archipelago/bitcoin-ui/nginx.conf`, was dated
|
||
**2026-06-30** — the pre-fix version, with no `auth_request` and with the wildcard CORS
|
||
header the fix removes.
|
||
|
||
## Root cause
|
||
|
||
Three facts have to be true at once, and on this node they were:
|
||
|
||
1. `bitcoin-ui` is listed in the node's durable `user-uninstalled` marker
|
||
(`/var/lib/archipelago/user-uninstalled.json`).
|
||
2. `reconcile_app` returns on that marker (`prod_orchestrator.rs:1956`) **before** reaching
|
||
`run_pre_start_hooks`, which is the only thing that renders the nginx config.
|
||
3. The container keeps running anyway, because it is owned by **systemd via a Quadlet
|
||
unit** — `archy-bitcoin-ui.service`, `active`, restarted 17:25 after the daemon restart —
|
||
not by the reconciler that is refusing to touch it.
|
||
|
||
So: *a container systemd keeps alive, that the orchestrator has stopped reconciling, never
|
||
receives a config fix shipped inside the binary.* The marker means "must stay removed", but
|
||
nothing enforces removal against systemd, and the orchestrator treats the marker as
|
||
permission to stop looking.
|
||
|
||
This is not a one-app accident. On the same node `archy-electrs-ui` is in the identical
|
||
state (uninstalled marker + active Quadlet unit + `Up 10 days`). It serves only a static
|
||
page with no credential-injecting proxy, so its exposure is low — but it would miss any
|
||
future config fix the same way.
|
||
|
||
## Why it matters beyond this node
|
||
|
||
An OTA carrying `a05956c4` would have closed the LND leak everywhere and silently failed to
|
||
close the Bitcoin RPC proxy on every node in this state — while making those nodes *look*
|
||
patched to exactly the check an operator would run first. That is the most misleading
|
||
possible outcome of shipping a security fix.
|
||
|
||
## The fix
|
||
|
||
`f6b5245b`: a container that is actually running is a live attack surface whatever a marker
|
||
says about it, so its security-relevant config is reconciled even behind the marker, and the
|
||
container is restarted so nginx loads it.
|
||
|
||
Deliberately narrow:
|
||
|
||
- Nothing is created, pulled, built, started or resurrected. The "must stay removed"
|
||
contract can only weaken for a container that is **already running**, which by definition
|
||
means it was never removed.
|
||
- A hook error is swallowed, not propagated — an app the user uninstalled must not be able
|
||
to fail the reconcile pass for every app after it.
|
||
- The pre-existing marker test passes unchanged; that is what proves the removal contract
|
||
survived. A new regression test pins the whole chain: stale conf in, gate present out,
|
||
container restarted, nothing created.
|
||
|
||
## What actually closed it on archi-dev-box — and what that does NOT prove
|
||
|
||
Sequence, from file mtimes, container start times and the daemon journal:
|
||
|
||
| Time (EDT) | Event |
|
||
|---|---|
|
||
| 18:33 | Probe: `POST /bitcoin-rpc/` → `200` with a real block height. Exposure confirmed live. |
|
||
| 18:36 | A **separate rebuild of bitcoin-ui**, done outside this work, rendered the fixed conf and recreated `archy-bitcoin-ui`. `:8334` closes here. |
|
||
| 19:06 | The binary carrying `f6b5245b` is installed and the daemon restarted. |
|
||
| 19:12 | Probe: `POST /bitcoin-rpc/` → `401`. `OPTIONS` now returns `Access-Control-Allow-Origin: http://192.168.63.240:8334`, not `*`. |
|
||
|
||
So the node is closed, and the fixed template is proven to work end to end on real
|
||
hardware — but **the reconcile fix itself was never exercised.** By the time it was
|
||
deployed, the state it repairs had already been cleared by the unrelated rebuild. The
|
||
`401` proves `a05956c4`'s template; it does not prove the delivery path `f6b5245b` adds.
|
||
|
||
That distinction is the whole point of this document, so it is recorded rather than
|
||
rounded off: `bitcoin-ui` is *still* in the node's `user-uninstalled` marker, meaning the
|
||
next time its config needs to change, this node depends on `f6b5245b` — untested — or on
|
||
someone happening to rebuild the app again.
|
||
|
||
Tracked as broken window 15 — **since closed by the controlled test below.**
|
||
|
||
## Proving the delivery path on real hardware
|
||
|
||
Run on archi-dev-box, 2026-08-02 20:00–20:03 EDT, with operator approval. The point was to
|
||
prove the thing the incidental rebuild had made unprovable: that **reconcile itself**
|
||
repairs this state, unaided.
|
||
|
||
The daemon was stopped first, so the reconciler could not repair the state before the
|
||
re-exposure had been confirmed — otherwise a passing probe would prove nothing about
|
||
which mechanism produced it.
|
||
|
||
| Step | Action | Observed |
|
||
|---|---|---|
|
||
| 1 | Install a faithfully stale conf (no `auth_request`, credential-injecting `proxy_pass`, `Allow-Origin: *`) and restart the container | — |
|
||
| 2 | Probe with no cookies | `POST /bitcoin-rpc/` → **`200`**, `{"result":960790}`; `Allow-Origin: *`. **Genuinely re-exposed** |
|
||
| 3 | Start the daemon (20:00:36) and touch nothing further | — |
|
||
| 4 | Reconcile pass at **20:02:19** | `bitcoin_ui: nginx.conf rendered auth_hash=51f2b5af`, then `WARN prod_orchestrator: rewrote config for a user-uninstalled app whose container is still RUNNING (systemd/Quadlet keeps it alive independently of reconcile) — restarting so it picks the new config up app_id=bitcoin-ui container=archy-bitcoin-ui` |
|
||
| 5 | Probe again | `POST /bitcoin-rpc/` → **`401`**; `Allow-Origin: http://192.168.63.240:8334` |
|
||
| 6 | Compare state | Conf **byte-identical** to the pre-test known-good; container healthy |
|
||
|
||
Step 2 is what makes steps 4–6 mean anything: without a confirmed `200`, the later `401`
|
||
would be consistent with the state never having been broken at all.
|
||
|
||
Both halves are now proven on hardware: `a05956c4`'s template (the gate works) and
|
||
`f6b5245b`'s delivery path (the gate arrives at a container the reconciler had been
|
||
skipping).
|
||
|
||
## Credential rotation — decided against, 2026-08-02
|
||
|
||
The operator's call, recorded here so it is not silently re-litigated: **no LND macaroon
|
||
rotation, and no Bitcoin RPC password rotation.** The reasoning was that there is no
|
||
evidence of exploitation and the vulnerability is being closed rather than lived with.
|
||
|
||
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. It has been
|
||
exercised in detect mode only, and has never rotated anything on any node. Its ordering
|
||
guard (refuses to rotate on a binary lacking the fix) remains the right shape for whenever
|
||
rotation is wanted — including for the Bitcoin RPC password, which has no equivalent tool
|
||
yet.
|
||
|
||
What this decision accepts: any macaroon or RPC password read through either hole before
|
||
it was closed stays valid. That is a deliberate, informed trade, not an oversight.
|
||
|
||
## Operator note
|
||
|
||
Deploying the fix rewrites the config and restarts `archy-bitcoin-ui` (a brief Bitcoin UI
|
||
interruption, nothing else). Any node that ever had `bitcoin-ui` uninstalled while its
|
||
Quadlet unit stayed active should be re-probed with the `POST /bitcoin-rpc/` check above —
|
||
a `401` is the pass condition. Treat the Bitcoin RPC password on any node that answered
|
||
`200` as known to anyone who could reach that port, and rotate it **after** the fix is
|
||
deployed, never before.
|