feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay

Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.

Three findings from performing a real rotation on a dev node, each fixed here:

1. BTCPay was left holding a dead credential, silently. Its connection string
   carries the macaroon INLINE (LND's datadir is owned by its container subuid,
   so btcpay cannot bind-mount the file), and the daemon only regenerates that
   secret when LND's TLS cert thumbprint changes — which macaroon rotation does
   not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
   failing, nothing anywhere saying why.

2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
   change visible as env drift, but the reconcile loop runs `ExistingOnly` at
   boot AND periodically, and there it deliberately leaves running
   restart-sensitive apps untouched — observed once per tick for half an hour on
   the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
   default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
   reasoning: restart sensitivity protects apps that are working, and this one is
   working only in appearance. The shell script cannot reach an in-process flag,
   so it removes the container and lets desired-state recovery rebuild it.

3. LND stayed locked forever on a loaded node. The unlocker is only served after
   channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
   containers; the unlock helper gave up at ~60s. That is not a harmless retry —
   reconcile records the post-start hook as failed, restarts LND, and the slow
   open begins again, so the wallet never opens and every LND-dependent app stays
   broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
   still exits on the first pass via `all_rejected`.

Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
  feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
  code path from "rotate my credentials" to `recreate_wallet_destructively`. A
  wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
  which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.

Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

core/archipelago/src/container/lnd.rs

@@ -22,6 +22,11 @@ const WALLET_PASSWORD_SECRET: &str = "/var/lib/archipelago/secrets/lnd-wallet-pa
 /// never use it, and the login-path migration rotates away from it.
 const LEGACY_WALLET_PASSWORD: &str = "hellohello";
 
+/// How many one-second passes `unlock_existing_wallet_via_rest` will make while
+/// LND's unlocker is still not listening (~10 minutes). See the comment at the
+/// retry loop for why this is measured in minutes rather than seconds.
+const UNLOCK_NOT_READY_ATTEMPTS: u32 = 600;
+
 #[derive(Debug, Clone)]
 pub struct EnsurePaths {
     pub data_dir: PathBuf,
@@ -345,7 +350,18 @@ async fn unlock_existing_wallet_via_rest() -> Result<bool> {
     // *actively rejected* (invalid passphrase), retrying can't help — fail fast
     // with a clear message instead of hanging the boot path for 60s+ (the wallet
     // was created with a password this node doesn't have → migration/recovery).
-    for _ in 0..60 {
+    //
+    // The not-ready budget is deliberately generous. LND opens channel.db,
+    // graph.db and wallet.db before it starts serving the unlocker at all, and
+    // on a busy node that is genuinely slow — observed at 2m38s on a box running
+    // 30 containers, where a 60s budget could never succeed. Timing out here is
+    // not a harmless retry: reconcile records the post-start hook as failed,
+    // which restarts LND, which starts the slow database open over again. The
+    // result is a restart loop that leaves the wallet permanently locked and
+    // every LND-dependent app (BTCPay's internal node included) broken, on
+    // exactly the nodes least able to afford it. Waiting longer costs nothing —
+    // a wrong password still exits on the first pass via `all_rejected`.
+    for _ in 0..UNLOCK_NOT_READY_ATTEMPTS {
         let mut all_rejected = true;
         for pw in &candidates {
             match try_unlock_once(&client, pw).await {
@@ -364,7 +380,28 @@ async fn unlock_existing_wallet_via_rest() -> Result<bool> {
         }
         tokio::time::sleep(std::time::Duration::from_secs(1)).await;
     }
-    anyhow::bail!("LND wallet unlock timed out waiting for the unlocker to become ready")
+    anyhow::bail!(
+        "LND wallet unlock timed out after ~{}s waiting for the unlocker to become ready",
+        UNLOCK_NOT_READY_ATTEMPTS
+    )
+}
+
+/// Unlock an existing wallet WITHOUT the destructive fallback.
+///
+/// `ensure_wallet_initialized` wipes and recreates a wallet no candidate
+/// password can open — correct for a boot path that must self-heal, and exactly
+/// wrong for macaroon rotation, which restarts LND against a wallet the operator
+/// still wants. Rotation calls this instead, so there is no code path from
+/// "rotate my credentials" to "delete my wallet": a rejected password surfaces
+/// as an error the caller reports, never as a wipe.
+pub(crate) async fn unlock_existing_wallet_no_wipe() -> Result<()> {
+    match unlock_existing_wallet().await? {
+        true => Ok(()),
+        false => anyhow::bail!(
+            "LND rejected every candidate wallet password — refusing to touch the wallet. \
+             The wallet is intact and still locked; its password is not one this node holds."
+        ),
+    }
 }
 
 /// Current LND wallet state via the unauthenticated `/v1/state` endpoint
@@ -718,29 +755,73 @@ const BTCPAY_LND_CONNECTION_SECRET: &str = "btcpay-lnd-connection";
 /// btcpay's secret_env entry is `optional`, so it simply starts without an
 /// internal Lightning node and picks it up on a later reconcile tick.
 /// Rewrites when the pinned cert thumbprint no longer matches (LND TLS cert
-/// rotation). Macaroon rotation without cert rotation is not auto-detected
+/// rotation). Macaroon rotation without cert rotation is not auto-detected here
 /// (reading the macaroon needs sudo; probing it every tick is not worth the
-/// churn) — delete the secret file once to force regeneration.
+/// churn) — the rotation path calls `rewrite_btcpay_lnd_connection_secret`
+/// instead, and deleting the secret file also forces regeneration.
 pub async fn ensure_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path) -> Result<()> {
+    build_btcpay_lnd_connection_secret(secrets_dir, false)
+        .await
+        .map(|_| ())
+}
+
+/// Rewrite the BTCPay→LND connection secret unconditionally, ignoring the
+/// cert-thumbprint fast path.
+///
+/// Rotating LND's macaroons invalidates the one embedded in this secret, and it
+/// is embedded *inline* rather than referenced by path — LND's datadir is owned
+/// by its container subuid, so btcpay cannot bind-mount the file and the string
+/// cannot self-heal. Nothing else notices: the TLS cert is untouched by macaroon
+/// rotation, so `ensure_…` takes its fast path forever and BTCPay keeps
+/// presenting a credential LND no longer honours. A node in that state looks
+/// entirely healthy — btcpay is up, LND is up — while every Lightning invoice it
+/// tries to create fails.
+///
+/// Writing the new value makes the change *visible*: `secret_env_hash` is
+/// derived from the resolved secret contents, so a changed file reads as label
+/// drift on the running container. It is not sufficient on its own — btcpay is
+/// restart-sensitive, and boot reconcile deliberately leaves running
+/// restart-sensitive apps untouched on drift. The caller must also call
+/// `ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
+/// the carve-out for exactly this case: a container that is up and healthy while
+/// holding a credential that no longer works. The orchestrator's own recreate
+/// path then rebuilds it around an unchanged data directory. No teardown here,
+/// deliberately — a hand-rolled remove-and-run is the anti-pattern CLAUDE.md
+/// names.
+///
+/// Returns `false` when LND isn't provisioned enough to derive a value.
+pub async fn rewrite_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path) -> Result<bool> {
+    build_btcpay_lnd_connection_secret(secrets_dir, true).await
+}
+
+/// Shared body. `force` skips the "already pins the current cert" fast path.
+/// Returns whether a value was written.
+async fn build_btcpay_lnd_connection_secret(
+    secrets_dir: &std::path::Path,
+    force: bool,
+) -> Result<bool> {
     let cert_path = format!("{DEFAULT_DATA_DIR}/tls.cert");
     let pem = match fs::read_to_string(&cert_path).await {
         Ok(s) => s,
-        Err(_) => return Ok(()), // LND not installed/provisioned yet
+        Err(_) => return Ok(false), // LND not installed/provisioned yet
     };
     let thumbprint = cert_sha256_thumbprint(&pem).context("computing LND tls.cert thumbprint")?;
 
     let target = secrets_dir.join(BTCPAY_LND_CONNECTION_SECRET);
     // Fast path (no sudo): existing secret already pins the current cert.
-    if let Ok(existing) = fs::read_to_string(&target).await {
-        if !existing.trim().is_empty() && existing.contains(&format!("certthumbprint={thumbprint}"))
-        {
-            return Ok(());
+    if !force {
+        if let Ok(existing) = fs::read_to_string(&target).await {
+            if !existing.trim().is_empty()
+                && existing.contains(&format!("certthumbprint={thumbprint}"))
+            {
+                return Ok(false);
+            }
         }
     }
 
     let macaroon_path = format!("{DEFAULT_DATA_DIR}/data/chain/bitcoin/mainnet/admin.macaroon");
     if !file_exists_as_root(&macaroon_path).await {
-        return Ok(()); // wallet not created yet; next tick retries
+        return Ok(false); // wallet not created yet; next tick retries
     }
     let macaroon = read_file_as_root(&macaroon_path).await?;
     let value = format!(
@@ -749,7 +830,30 @@ pub async fn ensure_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path)
         thumbprint
     );
     crate::container::secrets::write_secret_file(&target, &value)
-        .context("writing btcpay-lnd-connection secret")
+        .context("writing btcpay-lnd-connection secret")?;
+    Ok(true)
+}
+
+/// Does the on-disk BTCPay connection secret still carry the macaroon LND is
+/// currently issuing? `None` when there is nothing to compare — no secret file
+/// (BTCPay has no internal node configured) or no macaroon (LND unprovisioned).
+///
+/// Compares only hex text that is already on this host; the value is never
+/// logged, returned over RPC, or placed in an error.
+pub(crate) async fn btcpay_lnd_connection_is_current(
+    secrets_dir: &std::path::Path,
+) -> Option<bool> {
+    let target = secrets_dir.join(BTCPAY_LND_CONNECTION_SECRET);
+    let existing = fs::read_to_string(&target).await.ok()?;
+    let embedded = existing
+        .split("macaroon=")
+        .nth(1)?
+        .split(';')
+        .next()?
+        .to_string();
+    let macaroon_path = format!("{DEFAULT_DATA_DIR}/data/chain/bitcoin/mainnet/admin.macaroon");
+    let current = read_file_as_root(&macaroon_path).await.ok()?;
+    Some(embedded.eq_ignore_ascii_case(&hex::encode(current)))
 }
 
 /// SHA256 over the DER certificate body (matches

core/archipelago/src/container/prod_orchestrator.rs

@@ -4449,6 +4449,18 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
             ContainerState::Unknown(s) => format!("unknown:{s}"),
         })
     }
+
+    async fn mark_credential_rotated(&self, app_id: &str) {
+        self.credential_rotated
+            .lock()
+            .await
+            .insert(app_id.to_string());
+        tracing::info!(
+            app_id = %app_id,
+            "a credential this app consumes was rotated — its running container will be recreated \
+             on the next drift check even though the app is restart-sensitive"
+        );
+    }
 }
 
 /// Compare a manifest's command against a live container's, over the

core/archipelago/src/container/traits.rs

@@ -73,4 +73,21 @@ pub trait ContainerOrchestrator: Send + Sync {
 
     /// Coarse health summary: "healthy", "unhealthy", "starting", "paused", "unknown".
     async fn health(&self, app_id: &str) -> Result<String>;
+
+    /// Declare that a credential this app consumes has just been rotated, so
+    /// the running container is now holding an invalid one.
+    ///
+    /// Restart-sensitivity normally protects apps like `btcpay-server` from
+    /// being recreated on drift — correct when the running container is
+    /// working, and exactly wrong when it is working only in appearance. After
+    /// an LND macaroon rotation, BTCPay is up and healthy while every Lightning
+    /// operation it attempts fails against a credential LND no longer honours;
+    /// leaving it untouched perpetuates the breakage rather than protecting
+    /// anything. This is the same carve-out FED-07 uses for the Fedimint
+    /// gateway, reached from the RPC layer instead of from inside a reconcile.
+    ///
+    /// Consumed by the next drift check, which recreates the container around
+    /// its unchanged data directory, ports and volumes. Default no-op: an
+    /// orchestrator without restart-sensitivity has nothing to override.
+    async fn mark_credential_rotated(&self, _app_id: &str) {}
 }