lfg2025/archy

Fork 0

Files

T

archipelagoandClaude Opus 5 d15cd58d7f

Demo images / Build & push demo images (push) Successful in 3m34s

Details

feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay

Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.

Three findings from performing a real rotation on a dev node, each fixed here:

1. BTCPay was left holding a dead credential, silently. Its connection string
   carries the macaroon INLINE (LND's datadir is owned by its container subuid,
   so btcpay cannot bind-mount the file), and the daemon only regenerates that
   secret when LND's TLS cert thumbprint changes — which macaroon rotation does
   not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
   failing, nothing anywhere saying why.

2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
   change visible as env drift, but the reconcile loop runs `ExistingOnly` at
   boot AND periodically, and there it deliberately leaves running
   restart-sensitive apps untouched — observed once per tick for half an hour on
   the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
   default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
   reasoning: restart sensitivity protects apps that are working, and this one is
   working only in appearance. The shell script cannot reach an in-process flag,
   so it removes the container and lets desired-state recovery rebuild it.

3. LND stayed locked forever on a loaded node. The unlocker is only served after
   channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
   containers; the unlock helper gave up at ~60s. That is not a harmless retry —
   reconcile records the post-start hook as failed, restarts LND, and the slow
   open begins again, so the wallet never opens and every LND-dependent app stays
   broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
   still exits on the first pass via `all_rejected`.

Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
  feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
  code path from "rotate my credentials" to `recreate_wallet_destructively`. A
  wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
  which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.

Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-08 07:45:51 -04:00

9.0 KiB

Raw Blame History

The Bitcoin RPC proxy that stayed open after it was fixed

Status: code fix committed (f6b5245b); on-node verification recorded below. Found: 2026-08-02, a test node, while verifying a05956c4 instead of assuming it. Severity: critical on any affected node — unauthenticated control of Bitcoin Core RPC through a proxy that injects the node's own credentials.

Why this document exists

a05956c4 closed two unauthenticated endpoints on the wallet UI ports. Its commit message stated:

The nginx template is include_str!'d and re-rendered on every reconcile pass, so this ships atomically with the binary.

That is true for most nodes and false for a specific, silent, and not-rare state. The half that landed correctly (LND) made the half that did not (Bitcoin RPC) harder to notice, because a spot check of the LND endpoint returns a clean 401 and reads as "patched".

What was observed

Node running the fixed binary (installed 17:21, contains the new template — auth_request present in the binary at 4 occurrences). All probes from the node's own LAN address, no cookies, no credentials:

Probe	Result
`GET http://192.0.2.240:18083/lnd-connect-info`	`401`, 24 bytes, `{"error":"Unauthorized"}` — closed
`POST http://192.0.2.240:8334/bitcoin-rpc/` (`getblockcount`)	`200` — `{"result":960774,"error":null}` — OPEN
`OPTIONS http://192.0.2.240:8334/bitcoin-rpc/`	`204` with `Access-Control-Allow-Origin: ` — OPEN*

The rendered config on disk, /var/lib/archipelago/bitcoin-ui/nginx.conf, was dated 2026-06-30 — the pre-fix version, with no auth_request and with the wildcard CORS header the fix removes.

Root cause

Three facts have to be true at once, and on this node they were:

bitcoin-ui is listed in the node's durable user-uninstalled marker (/var/lib/archipelago/user-uninstalled.json).
reconcile_app returns on that marker (prod_orchestrator.rs:1956) before reaching run_pre_start_hooks, which is the only thing that renders the nginx config.
The container keeps running anyway, because it is owned by systemd via a Quadlet unit — archy-bitcoin-ui.service, active, restarted 17:25 after the daemon restart — not by the reconciler that is refusing to touch it.

So: a container systemd keeps alive, that the orchestrator has stopped reconciling, never receives a config fix shipped inside the binary. The marker means "must stay removed", but nothing enforces removal against systemd, and the orchestrator treats the marker as permission to stop looking.

This is not a one-app accident. On the same node archy-electrs-ui is in the identical state (uninstalled marker + active Quadlet unit + Up 10 days). It serves only a static page with no credential-injecting proxy, so its exposure is low — but it would miss any future config fix the same way.

Why it matters beyond this node

An OTA carrying a05956c4 would have closed the LND leak everywhere and silently failed to close the Bitcoin RPC proxy on every node in this state — while making those nodes look patched to exactly the check an operator would run first. That is the most misleading possible outcome of shipping a security fix.

The fix

f6b5245b: a container that is actually running is a live attack surface whatever a marker says about it, so its security-relevant config is reconciled even behind the marker, and the container is restarted so nginx loads it.

Deliberately narrow:

Nothing is created, pulled, built, started or resurrected. The "must stay removed" contract can only weaken for a container that is already running, which by definition means it was never removed.
A hook error is swallowed, not propagated — an app the user uninstalled must not be able to fail the reconcile pass for every app after it.
The pre-existing marker test passes unchanged; that is what proves the removal contract survived. A new regression test pins the whole chain: stale conf in, gate present out, container restarted, nothing created.

What actually closed it on a test node — and what that does NOT prove

Sequence, from file mtimes, container start times and the daemon journal:

Time (EDT)	Event
18:33	Probe: `POST /bitcoin-rpc/` → `200` with a real block height. Exposure confirmed live.
18:36	A separate rebuild of bitcoin-ui, done outside this work, rendered the fixed conf and recreated `archy-bitcoin-ui`. `:8334` closes here.
19:06	The binary carrying `f6b5245b` is installed and the daemon restarted.
19:12	Probe: `POST /bitcoin-rpc/` → `401`. `OPTIONS` now returns `Access-Control-Allow-Origin: http://192.0.2.240:8334`, not `*`.

So the node is closed, and the fixed template is proven to work end to end on real hardware — but the reconcile fix itself was never exercised. By the time it was deployed, the state it repairs had already been cleared by the unrelated rebuild. The 401 proves a05956c4's template; it does not prove the delivery path f6b5245b adds.

That distinction is the whole point of this document, so it is recorded rather than rounded off: bitcoin-ui is still in the node's user-uninstalled marker, meaning the next time its config needs to change, this node depends on f6b5245b — untested — or on someone happening to rebuild the app again.

Tracked as broken window 15 — since closed by the controlled test below.

Proving the delivery path on real hardware

Run on a test node, 2026-08-02 20:00–20:03 EDT, with operator approval. The point was to prove the thing the incidental rebuild had made unprovable: that reconcile itself repairs this state, unaided.

The daemon was stopped first, so the reconciler could not repair the state before the re-exposure had been confirmed — otherwise a passing probe would prove nothing about which mechanism produced it.

Step	Action	Observed
1	Install a faithfully stale conf (no `auth_request`, credential-injecting `proxy_pass`, `Allow-Origin: *`) and restart the container	—
2	Probe with no cookies	`POST /bitcoin-rpc/` → `200`, `{"result":960790}`; `Allow-Origin: `. Genuinely re-exposed*
3	Start the daemon (20:00:36) and touch nothing further	—
4	Reconcile pass at 20:02:19	`bitcoin_ui: nginx.conf rendered auth_hash=51f2b5af`, then `WARN prod_orchestrator: rewrote config for a user-uninstalled app whose container is still RUNNING (systemd/Quadlet keeps it alive independently of reconcile) — restarting so it picks the new config up app_id=bitcoin-ui container=archy-bitcoin-ui`
5	Probe again	`POST /bitcoin-rpc/` → `401`; `Allow-Origin: http://192.0.2.240:8334`
6	Compare state	Conf byte-identical to the pre-test known-good; container healthy

Step 2 is what makes steps 4–6 mean anything: without a confirmed 200, the later 401 would be consistent with the state never having been broken at all.

Both halves are now proven on hardware: a05956c4's template (the gate works) and f6b5245b's delivery path (the gate arrives at a container the reconciler had been skipping).

Credential rotation — decided against, 2026-08-02

The operator's call, recorded here so it is not silently re-litigated: no LND macaroon rotation, and no Bitcoin RPC password rotation. The reasoning was that there is no evidence of exploitation and the vulnerability is being closed rather than lived with.

scripts/security/rotate-lnd-macaroon.sh stays in the tree as a tool. Its ordering guard (refuses to rotate on a binary lacking the fix) remains the right shape for whenever rotation is wanted — including for the Bitcoin RPC password, which has no equivalent tool yet.

Amended 2026-08-08. This section said the script "has never rotated anything on any node"; that is no longer true. A rotation was performed on a development node while responding to the BTCPay Server advisory (that node had been running an affected btcpayserver:2.3.9), and it exposed a gap the script did not cover: BTCPay's inline copy of the macaroon was left stranded, so its Lightning payments failed silently while both apps reported healthy. Rotation is now a first-class, password-confirmed dashboard action that repairs that copy as part of the run — see LND-MACAROON-ROTATION.md. The fleet decision recorded above is unchanged: no fleet-wide rotation for this leak.

What this decision accepts: any macaroon or RPC password read through either hole before it was closed stays valid. That is a deliberate, informed trade, not an oversight.

Operator note

Deploying the fix rewrites the config and restarts archy-bitcoin-ui (a brief Bitcoin UI interruption, nothing else). Any node that ever had bitcoin-ui uninstalled while its Quadlet unit stayed active should be re-probed with the POST /bitcoin-rpc/ check above — a 401 is the pass condition. Treat the Bitcoin RPC password on any node that answered 200 as known to anyone who could reach that port, and rotate it after the fix is deployed, never before.

9.0 KiB Raw Blame History Unescape Escape