fix(lnd): hold LND's lifecycle lock across a rotation; mock the rotation RPCs

Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.

Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.

It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.

Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.

Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.

Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

core/archipelago/src/api/rpc/lnd/macaroons.rs

@@ -518,8 +518,46 @@ impl RpcHandler {
 
 // ── The rotation itself ──────────────────────────────────────────────────────
 
+/// Hold LND's lifecycle lock for the whole rotation, then do the work.
+///
+/// Between "stop LND" and "start LND" this owns a stopped container with its
+/// credential material deleted — the single worst moment for another actor to
+/// step in. Two would, unasked: the health monitor restarts any container it
+/// finds stopped, and the reconciler starts one whose unit is enabled. Either
+/// brings LND back up mid-deletion, and LND re-mints `macaroons.db` on unlock —
+/// so the deletion loop would race a live process writing that file, or
+/// "succeed" against material that had already been regenerated, leaving the
+/// operator told they had rotated while the old root key was still in service.
+///
+/// `app_ops::op_lock` is the mechanism both of those actors already consult
+/// (`lifecycle_op_in_flight`, via `lifecycle_op_covers_container` in the health
+/// monitor), and it also serialises against the package.start/stop/restart
+/// workers, so an operator hitting "Restart" on Lightning mid-rotation queues
+/// instead of interleaving.
+///
+/// Chosen over the `user-stopped` marker that `recreate_wallet_destructively`
+/// uses for its own window: that marker is a file on disk, so a rotation that
+/// died between marking and clearing would leave Lightning suppressed
+/// *permanently*, fixable only by finding and editing JSON on the node. This
+/// guard releases when it drops, on every path including a panic.
 async fn run_rotation(
     orchestrator: Option<Arc<dyn crate::container::ContainerOrchestrator>>,
+) -> Result<()> {
+    let lock = crate::app_ops::op_lock(LND_CONTAINER);
+    // Fail fast rather than queue. This is a button someone just pressed: a
+    // silent wait behind a start/stop/restart that may itself take minutes reads
+    // as "nothing happened", and the honest answer is short.
+    let _guard = lock.try_lock().map_err(|_| {
+        anyhow::anyhow!(
+            "another Lightning start/stop/restart is in progress on this node — \
+             wait for it to finish and try again"
+        )
+    })?;
+    rotate_with_lnd_pinned(orchestrator).await
+}
+
+async fn rotate_with_lnd_pinned(
+    orchestrator: Option<Arc<dyn crate::container::ContainerOrchestrator>>,
 ) -> Result<()> {
     // 1. Preflight — establish what must survive, while LND can still be asked.
     with_progress(|p| p.set("preflight", StepState::Running, None));
@@ -584,7 +622,8 @@ async fn run_rotation(
         );
     });
 
-    // 3. Stop.
+    // 3. Stop. Nothing may restart LND from here until step 5 — see the lock
+    //    `run_rotation` holds around this whole function.
     with_progress(|p| p.set("stop", StepState::Running, None));
     stop_lnd().await.context("stopping LND")?;
     with_progress(|p| p.set("stop", StepState::Done, None));

docs/security/LND-MACAROON-ROTATION.md

@@ -55,6 +55,31 @@ Two details in that check are deliberate and should not be "tightened":
   this node does not hold surfaces as a **failed rotation** with the wallet
   intact.
 
+## Nothing else may touch LND mid-rotation
+
+Between "stop LND" and "start LND" the rotation owns a stopped container whose
+credential material is being deleted. Two background actors would step in there
+unasked: the **health monitor** restarts any container it finds stopped, and the
+**reconciler** starts one whose unit is enabled. Either brings LND back up
+mid-deletion — and LND re-mints `macaroons.db` on unlock, so the deletion loop
+would race a live process writing that file, or "succeed" against material that
+had already been regenerated. The operator would be told they had rotated while
+the old root key was still in service.
+
+The rotation therefore holds `app_ops::op_lock("lnd")` for its whole duration.
+That is the lock both actors already consult (`lifecycle_op_in_flight`, reached
+in the health monitor via `lifecycle_op_covers_container`), and it also
+serialises against the `package.start`/`stop`/`restart` workers, so an operator
+hitting "Restart" on Lightning mid-rotation queues rather than interleaving. A
+rotation requested while one of those is running fails fast with a short
+explanation instead of waiting silently.
+
+Deliberately **not** the `user-stopped` marker that `recreate_wallet_destructively`
+uses for its own window: that marker is a file on disk, so a rotation that died
+between marking and clearing would leave Lightning suppressed *permanently* —
+fixable only by finding and editing JSON on the node. The lock guard releases
+when it drops, on every path including a panic.
+
 ## The BTCPay coupling — the part that bites
 
 **BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot

neode-ui/mock-backend.js

@@ -3704,6 +3704,48 @@ app.post('/rpc/v1', (req, res) => {
         })
       }
 
+      // Lightning credential rotation (Settings → Lightning credentials).
+      // Stateful so the dev preview can exercise the whole arc without a node:
+      // request a rotation and the steps advance on each poll, ending with the
+      // BTCPay reconnect. Digests only — the real daemon never returns a
+      // macaroon and neither does this.
+      case 'lnd.macaroon-status': {
+        return res.json({
+          result: {
+            installed: true,
+            admin_macaroon_sha256: '52219e90aeba8ac6a98fdac1cc754fe0a2e8407ae6758ffe2cf92039503f5575',
+            issued_at: '2026-08-08 06:03:11',
+            identity_pubkey: '03a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456',
+            channels_open: 5,
+            channels_pending: 1,
+            lnd_error: null,
+            btcpay_uses_internal_lnd: true,
+            // Flip to false to see the "BTCPay is holding an old credential"
+            // warning this feature exists to prevent.
+            btcpay_credential_current: true,
+            rotation: macaroonRotationSnapshot(),
+          },
+        })
+      }
+
+      case 'lnd.rotate-macaroons': {
+        if (!params?.password) {
+          return res.json({ error: { code: -1, message: 'Node password required to rotate Lightning credentials' } })
+        }
+        if (params.password !== userState.passwordHash && params.password !== MOCK_PASSWORD) {
+          return res.json({ error: { code: -1, message: 'Password verification failed' } })
+        }
+        if (macaroonRotation.running) {
+          return res.json({ error: { code: -1, message: 'A macaroon rotation is already running on this node' } })
+        }
+        startMacaroonRotation()
+        return res.json({ result: { status: 'started' } })
+      }
+
+      case 'lnd.macaroon-rotation-progress': {
+        return res.json({ result: macaroonRotationSnapshot() })
+      }
+
       case 'lnd.gettransactions': {
         const pending = walletState.transactions.filter(tx => tx.direction === 'incoming' && tx.num_confirmations < 3).length
         return res.json({
@@ -6075,6 +6117,66 @@ const walletState = sessionBucketProxy('walletState')
 const userState = sessionBucketProxy('userState')
 const mockState = sessionBucketProxy('mockState')
 
+// ── Lightning macaroon rotation (mock) ──────────────────────────────────────
+// Mirrors the daemon's `RotationProgress`: same step keys and the same five
+// states, so the Settings section can be driven end-to-end without a node.
+// Advances one step per poll rather than on a timer, which keeps it
+// deterministic and makes each intermediate state actually observable.
+const MACAROON_ROTATION_STEPS = [
+  ['preflight', 'Check LND is healthy and record what must survive', '5 channel(s) open, 1 pending — these must be identical afterwards'],
+  ['backup', 'Back up the current macaroon material', '8 file(s) copied to /var/lib/archipelago/lnd/macaroon-rotation-20260808T120000Z'],
+  ['stop', 'Stop Lightning', null],
+  ['remove', 'Remove the old root key and issued macaroons', '8 file(s) removed'],
+  ['start', 'Start Lightning and unlock the wallet', 'Lightning is up with freshly minted credentials'],
+  ['verify', 'Confirm the node and its channels are unchanged', 'same node, same 5 channel(s)'],
+  ['btcpay', 'Reconnect BTCPay Server to the new credentials', 'Connection string updated. BTCPay restarts itself within a minute or two to pick it up.'],
+]
+
+const macaroonRotation = { running: false, done: 0, ok: null, startedAt: null, finishedAt: null }
+
+function startMacaroonRotation() {
+  macaroonRotation.running = true
+  macaroonRotation.done = 0
+  macaroonRotation.ok = null
+  macaroonRotation.startedAt = new Date().toISOString()
+  macaroonRotation.finishedAt = null
+}
+
+function macaroonRotationSnapshot() {
+  if (macaroonRotation.running) {
+    macaroonRotation.done += 1
+    if (macaroonRotation.done >= MACAROON_ROTATION_STEPS.length) {
+      macaroonRotation.done = MACAROON_ROTATION_STEPS.length
+      macaroonRotation.running = false
+      macaroonRotation.ok = true
+      macaroonRotation.finishedAt = new Date().toISOString()
+    }
+  }
+  const started = macaroonRotation.startedAt !== null
+  return {
+    running: macaroonRotation.running,
+    ok: macaroonRotation.ok,
+    started_at: macaroonRotation.startedAt,
+    finished_at: macaroonRotation.finishedAt,
+    error: null,
+    steps: MACAROON_ROTATION_STEPS.map(([key, label, detail], i) => {
+      let state = 'pending'
+      if (started) {
+        if (i < macaroonRotation.done) state = 'done'
+        else if (i === macaroonRotation.done && macaroonRotation.running) state = 'running'
+      }
+      return { key, label, state, detail: state === 'done' ? detail : null }
+    }),
+    backup_path: started ? '/var/lib/archipelago/lnd/macaroon-rotation-20260808T120000Z' : null,
+    identity_pubkey: started ? '03a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456' : null,
+    channels_before: started ? 5 : null,
+    channels_after: macaroonRotation.ok ? 5 : null,
+    new_admin_macaroon_sha256: macaroonRotation.ok
+      ? '8c19b99de4a8f5c3145d8189500089829174909ca09b48f55e0464239bd8d412'
+      : null,
+  }
+}
+
 // Seed for the per-session Tor services demo state (tor.list-services /
 // tor.create-service / tor.delete-service round-trip against this).
 function defaultTorServices() {