feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay
Demo images / Build & push demo images (push) Successful in 3m34s

Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.

Three findings from performing a real rotation on a dev node, each fixed here:

1. BTCPay was left holding a dead credential, silently. Its connection string
   carries the macaroon INLINE (LND's datadir is owned by its container subuid,
   so btcpay cannot bind-mount the file), and the daemon only regenerates that
   secret when LND's TLS cert thumbprint changes — which macaroon rotation does
   not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
   failing, nothing anywhere saying why.

2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
   change visible as env drift, but the reconcile loop runs `ExistingOnly` at
   boot AND periodically, and there it deliberately leaves running
   restart-sensitive apps untouched — observed once per tick for half an hour on
   the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
   default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
   reasoning: restart sensitivity protects apps that are working, and this one is
   working only in appearance. The shell script cannot reach an in-process flag,
   so it removes the container and lets desired-state recovery rebuild it.

3. LND stayed locked forever on a loaded node. The unlocker is only served after
   channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
   containers; the unlock helper gave up at ~60s. That is not a harmless retry —
   reconcile records the post-start hook as failed, restarts LND, and the slow
   open begins again, so the wallet never opens and every LND-dependent app stays
   broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
   still exits on the first pass via `all_rejected`.

Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
  feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
  code path from "rotate my credentials" to `recreate_wallet_destructively`. A
  wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
  which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.

Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-08 07:45:51 -04:00
co-authored by Claude Opus 5
parent b7e57ca9cf
commit d15cd58d7f
13 changed files with 1628 additions and 13 deletions
+58
View File
@@ -268,6 +268,57 @@ if [ -n "$FAIL" ]; then
die "rotation verification FAILED:$FAIL — old material is in $BACKUP"
fi
# ── BTCPay's inline copy ──────────────────────────────────────────────
# BTCPay reaches the internal LND node with a connection string that carries
# the macaroon INLINE as hex, not as a file path: LND's datadir is owned by its
# container's mapped uid, so btcpay cannot bind-mount the file. That copy is
# therefore now a dead credential, and nothing else will notice — the daemon
# only regenerates this secret when LND's TLS *cert* thumbprint changes, which
# macaroon rotation does not touch. The node keeps looking healthy (btcpay up,
# LND up) while every Lightning invoice BTCPay tries to create fails.
#
# Deleting the secret file gets the daemon to regenerate it from the new
# macaroon on its next reconcile tick. That is necessary but NOT sufficient, and
# the difference matters: the RUNNING container still holds the dead value, and
# the periodic reconciler only ever runs in `ExistingOnly` mode, where env drift
# on a restart-sensitive app (btcpay-server is one) is detected and then
# deliberately skipped — "leaving running restart-sensitive app untouched". So
# the container has to be recreated on purpose. The dashboard path
# (Settings → Lightning credentials) does this itself by flagging the app as
# credential-rotated; a shell script cannot reach that in-process flag, so it
# removes the container instead and lets the orchestrator's own desired-state
# recovery rebuild it around unchanged data, ports and volumes.
#
# Nothing is printed but a path — never the value.
BTCPAY_SECRET="/var/lib/archipelago/secrets/btcpay-lnd-connection"
BTCPAY_NOTE=no
if sudo test -f "$BTCPAY_SECRET"; then
if sudo rm -f "$BTCPAY_SECRET"; then
say
say "btcpay : removed its stale connection string ($BTCPAY_SECRET)."
say " The daemon regenerates it from the new macaroon within a minute."
BTCPAY_NOTE=yes
if podman container exists btcpay-server 2>/dev/null; then
say " Recreating btcpay-server so it stops using the dead one."
podman stop btcpay-server >/dev/null 2>&1 || true
if podman rm -f btcpay-server >/dev/null 2>&1; then
say " Removed; the orchestrator rebuilds it around its existing"
say " data (it was running, so desired-state recovery restores it)."
else
say " ⚠ could not remove btcpay-server. Its Lightning payments will"
say " fail until it is recreated."
BTCPAY_NOTE=warn
fi
fi
else
say
say "btcpay : ⚠ could not remove $BTCPAY_SECRET. BTCPay is still holding"
say " the OLD macaroon, so its Lightning payments will fail until"
say " that file is deleted and btcpay-server is recreated."
BTCPAY_NOTE=warn
fi
fi
say
say "✅ Rotated. Every macaroon issued before now no longer verifies."
say
@@ -278,6 +329,13 @@ say " and scan the new pairing QR; it serves the new macaroon."
say
say " Your funds and channels are untouched: the node kept its identity and"
say " no channel was closed."
if [ "${BTCPAY_NOTE:-no}" != no ]; then
say
say " CONFIRM BTCPAY CAME BACK. A silent failure here looks identical to success:"
say " btcpay stays up and healthy while every Lightning payment it tries fails."
say " podman inspect btcpay-server --format '{{.Created}}' # should be just now"
say " sudo test -f $BTCPAY_SECRET && echo regenerated"
fi
say
say " Once every client is re-paired, delete the backup — it holds the OLD"
say " root key, which is still sensitive:"