Commit Graph
2 Commits
Author SHA1 Message Date
archipelagoandClaude Opus 5 cfa6c6cb0d fix(lnd): hold LND's lifecycle lock across a rotation; mock the rotation RPCs
Demo images / Build & push demo images (push) Successful in 3m55s
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.

Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.

It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.

Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.

Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.

Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 09:33:10 -04:00
archipelagoandClaude Opus 5 d15cd58d7f feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay
Demo images / Build & push demo images (push) Successful in 3m34s
Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.

Three findings from performing a real rotation on a dev node, each fixed here:

1. BTCPay was left holding a dead credential, silently. Its connection string
   carries the macaroon INLINE (LND's datadir is owned by its container subuid,
   so btcpay cannot bind-mount the file), and the daemon only regenerates that
   secret when LND's TLS cert thumbprint changes — which macaroon rotation does
   not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
   failing, nothing anywhere saying why.

2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
   change visible as env drift, but the reconcile loop runs `ExistingOnly` at
   boot AND periodically, and there it deliberately leaves running
   restart-sensitive apps untouched — observed once per tick for half an hour on
   the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
   default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
   reasoning: restart sensitivity protects apps that are working, and this one is
   working only in appearance. The shell script cannot reach an in-process flag,
   so it removes the container and lets desired-state recovery rebuild it.

3. LND stayed locked forever on a loaded node. The unlocker is only served after
   channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
   containers; the unlock helper gave up at ~60s. That is not a harmless retry —
   reconcile records the post-start hook as failed, restarts LND, and the slow
   open begins again, so the wallet never opens and every LND-dependent app stays
   broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
   still exits on the first pass via `all_rejected`.

Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
  feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
  code path from "rotate my credentials" to `recreate_wallet_destructively`. A
  wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
  which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.

Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 07:45:51 -04:00