feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay

Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.

Three findings from performing a real rotation on a dev node, each fixed here:

1. BTCPay was left holding a dead credential, silently. Its connection string
   carries the macaroon INLINE (LND's datadir is owned by its container subuid,
   so btcpay cannot bind-mount the file), and the daemon only regenerates that
   secret when LND's TLS cert thumbprint changes — which macaroon rotation does
   not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
   failing, nothing anywhere saying why.

2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
   change visible as env drift, but the reconcile loop runs `ExistingOnly` at
   boot AND periodically, and there it deliberately leaves running
   restart-sensitive apps untouched — observed once per tick for half an hour on
   the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
   default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
   reasoning: restart sensitivity protects apps that are working, and this one is
   working only in appearance. The shell script cannot reach an in-process flag,
   so it removes the container and lets desired-state recovery rebuild it.

3. LND stayed locked forever on a loaded node. The unlocker is only served after
   channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
   containers; the unlock helper gave up at ~60s. That is not a harmless retry —
   reconcile records the post-start hook as failed, restarts LND, and the slow
   open begins again, so the wallet never opens and every LND-dependent app stays
   broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
   still exits on the first pass via `all_rejected`.

Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
  feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
  code path from "rotate my credentials" to `recreate_wallet_destructively`. A
  wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
  which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.

Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/security/LND-MACAROON-ROTATION.md

@@ -0,0 +1,166 @@
+# Rotating this node's Lightning credentials
+
+A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
+node's wallet. There is no revocation list and no expiry. If a macaroon is ever
+read by something you do not control — a leaked endpoint, a screenshot, a phone
+that has since been lost, an app that ran a version with a published
+vulnerability — that ability persists until the macaroons are rotated.
+
+Rotation is therefore a **routine operator action**, not an emergency procedure.
+Two paths do the same work:
+
+| Path | Use when |
+|---|---|
+| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
+| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |
+
+## What rotation actually does
+
+LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
+that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
+root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
+issued before that moment — including any an attacker holds — stops verifying.
+
+## Why your funds and channels survive
+
+Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
+state in `channel.db`; channels are secured by the node's identity and channel
+keys, none of which are derived from the macaroon root key. Neither database is
+opened, moved or deleted.
+
+Both paths **prove** this rather than asserting it: they record the node's
+identity pubkey and its channel census before rotating, and refuse to report
+success if either differs afterwards.
+
+Two details in that check are deliberate and should not be "tightened":
+
+- **Channels are compared as a total, not as `num_active_channels`.** The active
+  count only counts channels whose peer is currently online, so it legitimately
+  dips for minutes after *any* restart while peers reconnect. Asserting on it
+  alone would abort a perfectly healthy rotation.
+- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
+  progress inside it, so the file changes on every start. Asserting byte-identity
+  would fire a frightening false alarm on a completely healthy rotation.
+
+## What it never does
+
+- No macaroon **content** reaches a response, an error, a log line, or the
+  progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
+  byte count — enough to prove the material changed without disclosing it to
+  whoever is reading the screen.
+- No path from "rotate my credentials" to "delete my wallet". LND's boot path
+  self-heals a wallet no candidate password can open by wiping and recreating it;
+  correct for an unattended boot, catastrophic here. Rotation unlocks through
+  `container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
+  this node does not hold surfaces as a **failed rotation** with the wallet
+  intact.
+
+## The BTCPay coupling — the part that bites
+
+**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
+self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
+cannot bind-mount the macaroon file (EACCES across the userns boundary). The
+connection string therefore carries the macaroon as hex:
+
+```
+type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
+```
+
+delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
+that copy becomes a dead credential. Nothing notices on its own, because the
+daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
+and macaroon rotation does not touch the cert.
+
+The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
+healthy, and every Lightning invoice BTCPay tries to create fails.**
+
+Repair needs two things, and one without the other is cosmetic:
+
+1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
+   This is what makes the change visible: `secret_env_hash` is derived from the
+   resolved secret contents, so a changed file reads as label drift on the
+   running container.
+2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
+   and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
+   periodic alike — where env drift on a restart-sensitive app is detected and
+   then deliberately skipped. Rewriting the secret alone therefore changes
+   nothing that is running. Observed directly on a development node, once per
+   tick, for half an hour:
+
+   ```
+   container drift detected during boot reconcile; leaving running
+   restart-sensitive app untouched app_id=btcpay-server
+   ```
+
+   The dashboard path calls
+   `ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
+   the flag the drift check consults to override restart-sensitivity. It is the
+   same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
+   identical: restart sensitivity protects apps that are *working*, and this one
+   is working only in appearance.
+
+**The shell script cannot set that in-process flag**, so it does the equivalent
+from outside: it deletes the secret (the daemon regenerates it within a tick),
+then removes the `btcpay-server` container so the orchestrator's own
+desired-state recovery rebuilds it around unchanged data. That recovery is what
+makes this safe rather than a hand-rolled remove-and-run — it fires because the
+app is still installed and was in the last running-containers snapshot. The
+script then prints the commands to confirm it actually happened, because a
+failure here is invisible.
+
+## Slow nodes: the unlock budget
+
+LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
+at all, and on a busy node that is genuinely slow — **2m38s measured on a box
+running 30 containers**. The unlock helper used to give up after ~60s, which on
+such a node could never succeed.
+
+That timeout was not a harmless retry. Reconcile records the post-start hook as
+failed, restarts LND, and the slow database open starts over: a restart loop that
+leaves the wallet permanently locked and every LND-dependent app (BTCPay's
+internal node included) broken, on exactly the nodes least able to afford it.
+
+The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
+longer costs nothing, because a genuinely wrong password still exits on the first
+pass through the candidate list — the `all_rejected` fast path is untouched.
+
+## Verifying a rotation
+
+The dashboard shows all of this. From a shell:
+
+```bash
+# 1. Fingerprint changed (digest only — never print the macaroon)
+sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon
+
+# 2. Same node, same channels
+podman exec lnd lncli --network=mainnet getinfo \
+  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
+      d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'
+
+# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
+CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
+SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
+[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"
+
+# 4. BTCPay was actually recreated (a silent failure looks like success)
+podman inspect btcpay-server --format '{{.Created}}'
+```
+
+Check 3 is the one people skip, and it is the one that fails.
+
+## Afterwards
+
+- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
+  the dashboard and scan the pairing QR again; it serves the new macaroon.
+- **Delete the backup once re-pairing is done.** Both paths back the old material
+  up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
+  is recoverable. That directory holds the **old root key** and is still
+  sensitive: `sudo rm -rf <path>`.
+
+## Related
+
+- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
+  rotation necessary, and the operator decision not to rotate the fleet for it.
+- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
+  ordering guard (it refuses to rotate on a binary that still leaks
+  `/lnd-connect-info`, since the new macaroon would leak within seconds).