feat(lnd): channel-peer watchdog — a dropped peer link heals itself

LND normally reconnects channel peers after a restart, but not reliably:
after long or repeated downtime (an app update, a node reboot,
reconciler churn) the peer link can stay down for hours while BOTH
endpoints keep the channel flagged disabled in the routing graph. The
node looks perfectly healthy, the wallet shows balance, and every
payment in either direction fails "no route to the recipient" —
observed live on framework-pt (2026-09-01): its only channel sat
disabled on both policy sides for ~17 hours after the LND 0.21.2
update, while shorty had 583k spendable and the user was told, by a
mis-mapped modal, that they had 'no payment channel'.

The channel graph is desired state — every open channel should have a
live peer connection. A daemon-side watchdog now enforces it:

- every 2 minutes, list channels + peers over LND REST
- for each channel whose remote peer is not connected, look the peer's
  advertised addresses up in the public graph and dial one
- per-peer retries throttled to 10 minutes so an unreachable peer is
  not hammered; 'already connected' counts as done; a peer with no
  advertised address is logged once per pass (cannot be dialed)
- no-ops quietly on nodes without LND (missing macaroon) and while a
  wallet is locked (503 body has no channels)

Unit tests pin the selection against the live REST shapes
(remote_pubkey in /v1/channels vs pub_key in /v1/peers).

v1.8.10 CHANGELOG + What's New entries staged so the next release run
is clean first time.

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## v1.8.10-alpha (2026-09-02)
+
+- **A channel that drops its peer link now heals itself — on every node.** Restarting LND (an app update, a reboot, container churn) can leave a channel's peer connection down for hours while both endpoints keep the channel flagged disabled in the routing graph: the node looks perfectly healthy, the wallet shows balance, and every payment in either direction fails "no route to the recipient". Observed live: a node's only channel sat unroutable for ~17 hours after the LND 0.21.2 update, with no sign of it in any dashboard. The daemon now watches the channel graph as desired state — every open channel should have a live peer — and reconnects any that don't, using the peer's advertised addresses. Nodes without LND are untouched; an unreachable peer is retried gently, not hammered.
+
+- **The Lightning wallet states the node's real funding state instead of "you have no channel."** Trying to send while a freshly opened channel was still waiting for on-chain confirmations — or when all its balance sits on the far side — raised a modal that claimed the node had NO channel at all (the outbound sum is legitimately zero in both states), pointed the user at opening a second channel, and — for payment routing failures — even showed the *receiving* copy. The funding gate now reads the channel list it already fetched: a confirming channel gets "it unlocks automatically once confirmed, nothing is needed from you", a far-side balance gets "you can receive, but there's nothing to send right now", a routing/liquidity payment failure says so instead of claiming channel problems, and only a genuinely channel-less node keeps the open-one guidance.
+
 ## v1.8.9-alpha (2026-09-01)
 
 - **Lightning sends work again after the LND 0.21.2 update.** LND 0.21 removed the old synchronous payment route the node's backend paid through (`/v1/channels/transactions`) — every Lightning send answered the literal "Not Found" and the wallet showed "Payment failed: Not Found". The backend now pays through the supported Router.SendPaymentV2 route, keeps the same settle-then-report behaviour (a slow multi-hop payment is still tracked to completion, never falsely declared failed), and translates LND's failure reasons into plain advice. A new gate test speaks the payment route directly against the running LND, so an image/backend skew like this can never ship silently again.

core/archipelago/src/container/lnd.rs

@@ -131,6 +131,10 @@ const LND_STATE_DIRS: &[&str] = &[
 /// container, not a Quadlet unit, so it is restarted via `podman`, not systemctl.
 const LND_CONTAINER: &str = "lnd";
 
+/// Canonical on-host admin macaroon — same path the RPC layer reads.
+const LND_ADMIN_MACAROON: &str =
+    "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon";
+
 /// Archipelago data dir (default; not overridden in prod). Holds the
 /// `user-stopped.json` that gates health-monitor auto-restart.
 const ARCHY_DATA_DIR: &str = "/var/lib/archipelago";
@@ -872,6 +876,188 @@ fn cert_sha256_thumbprint(pem: &str) -> Result<String> {
     Ok(hex::encode_upper(Sha256::digest(&der)))
 }
 
+// ── Channel-peer watchdog ──────────────────────────────────────────────────
+
+/// Every open channel's remote peer that is NOT currently connected.
+/// Pure over LND's REST JSON so the selection can be unit-tested.
+///
+/// `/v1/peers` uses `pub_key`; `/v1/channels` uses `remote_pubkey` — the
+/// asymmetry is LND's, not ours.
+fn select_reconnect_targets(
+    channels: &serde_json::Value,
+    peers: &serde_json::Value,
+) -> Vec<String> {
+    let connected: std::collections::HashSet<&str> = peers
+        .get("peers")
+        .and_then(|p| p.as_array())
+        .map(|arr| {
+            arr.iter()
+                .filter_map(|p| p.get("pub_key").and_then(|v| v.as_str()))
+                .collect()
+        })
+        .unwrap_or_default();
+    let mut targets: Vec<String> = channels
+        .get("channels")
+        .and_then(|c| c.as_array())
+        .map(|arr| {
+            arr.iter()
+                .filter_map(|c| c.get("remote_pubkey").and_then(|v| v.as_str()))
+                .filter(|pk| !connected.contains(pk))
+                .map(str::to_string)
+                .collect()
+        })
+        .unwrap_or_default();
+    targets.sort();
+    targets.dedup();
+    targets
+}
+
+/// Reconnect peers of open channels that LND has not re-established on its
+/// own. Returns the number of peers reconnected this pass.
+///
+/// LND normally reconnects channel peers after a restart — but not reliably:
+/// when the restart outages are long or repeated (an app update, a node
+/// reboot, reconciler churn), the peer link can stay down for hours while
+/// BOTH endpoints keep flagging the channel `disabled` in the routing
+/// graph. The node itself looks perfectly healthy and every payment in
+/// either direction fails "no route to the recipient" — observed live on
+/// framework-pt (2026-09-01): its only channel sat disabled on both policy
+/// sides for ~17h after the LND 0.21.2 update, while the wallet showed
+/// plenty of outbound. The channel graph is desired state; this keeps it.
+///
+/// Quietly returns Ok(0) when LND is not installed or its wallet is locked —
+/// that is every node without LND, on every pass.
+///
+/// `last_attempt` throttles retries per peer (`min_retry`) so an unreachable
+/// peer is not hammered every pass; the caller owns the map so the pass
+/// itself stays stateless and testable.
+pub(crate) async fn reconnect_disconnected_channel_peers(
+    last_attempt: &mut std::collections::HashMap<String, std::time::Instant>,
+    min_retry: std::time::Duration,
+) -> Result<usize> {
+    let Ok(macaroon) = read_file_as_root(LND_ADMIN_MACAROON).await else {
+        return Ok(0); // LND not installed (or not initialized yet)
+    };
+    let macaroon_hex = hex::encode(macaroon);
+    let client = reqwest::Client::builder()
+        .no_proxy()
+        .timeout(std::time::Duration::from_secs(8))
+        .danger_accept_invalid_certs(true)
+        .build()
+        .context("building LND REST client for the channel-peer watchdog")?;
+
+    let channels: serde_json::Value = client
+        .get(format!("{LND_REST_BASE_URL}/v1/channels"))
+        .header("Grpc-Metadata-macaroon", &macaroon_hex)
+        .send()
+        .await
+        .context("LND REST: listing channels for the peer watchdog")?
+        .json()
+        .await
+        .context("parsing LND channel list")?;
+    // A locked wallet answers 503 with an error body — it parses as JSON
+    // with no "channels" key, which selects nothing. That is a quiet pass.
+    let peers: serde_json::Value = client
+        .get(format!("{LND_REST_BASE_URL}/v1/peers"))
+        .header("Grpc-Metadata-macaroon", &macaroon_hex)
+        .send()
+        .await
+        .context("LND REST: listing peers for the peer watchdog")?
+        .json()
+        .await
+        .context("parsing LND peer list")?;
+
+    let mut reconnected = 0usize;
+    for pubkey in select_reconnect_targets(&channels, &peers) {
+        if last_attempt
+            .get(&pubkey)
+            .is_some_and(|t| t.elapsed() < min_retry)
+        {
+            continue;
+        }
+        last_attempt.insert(pubkey.clone(), std::time::Instant::now());
+
+        // Where does the peer live? Its advertised addresses in the public
+        // graph. A peer with none (fully private) cannot be dialed from here
+        // — LND itself may still find it; we only log the gap once per pass.
+        // Unknown to the public graph (or the graph query failed) — nothing
+        // to dial on.
+        let Ok(node) = client
+            .get(format!("{LND_REST_BASE_URL}/v1/graph/node/{pubkey}"))
+            .header("Grpc-Metadata-macaroon", &macaroon_hex)
+            .send()
+            .await
+            .and_then(|r| r.error_for_status())
+        else {
+            continue;
+        };
+        let Ok(node) = node.json::<serde_json::Value>().await else {
+            continue;
+        };
+        let addresses: Vec<String> = node
+            .get("node")
+            .and_then(|n| n.get("addresses"))
+            .and_then(|a| a.as_array())
+            .map(|arr| {
+                arr.iter()
+                    .filter_map(|a| a.get("addr").and_then(|v| v.as_str()))
+                    .map(str::to_string)
+                    .collect()
+            })
+            .unwrap_or_default();
+        if addresses.is_empty() {
+            tracing::warn!(
+                peer = %pubkey,
+                "LND channel peer is disconnected and advertises no address — cannot dial it; payments through this channel stay unroutable"
+            );
+            continue;
+        }
+
+        for addr in addresses {
+            let Some((host, port)) = addr.rsplit_once(':') else {
+                continue;
+            };
+            let Ok(port) = port.parse::<u32>() else {
+                continue;
+            };
+            let body = serde_json::json!({
+                "perm": false,
+                "timeout": "15s",
+                "addr": { "pubkey": pubkey, "host": host, "port": port },
+            });
+            match client
+                .post(format!("{LND_REST_BASE_URL}/v1/peers"))
+                .header("Grpc-Metadata-macaroon", &macaroon_hex)
+                .json(&body)
+                .send()
+                .await
+            {
+                Ok(resp) if resp.status().is_success() => {
+                    reconnected += 1;
+                    tracing::info!(
+                        peer = %pubkey,
+                        addr = %addr,
+                        "reconnected a disconnected channel peer (channel was unroutable)"
+                    );
+                    break;
+                }
+                Ok(resp) => {
+                    let msg = resp.text().await.unwrap_or_default();
+                    // Already connected between our list call and now — success.
+                    if msg.contains("already connected") {
+                        break;
+                    }
+                    tracing::debug!(peer = %pubkey, addr = %addr, %msg, "channel-peer connect attempt failed");
+                }
+                Err(e) => {
+                    tracing::debug!(peer = %pubkey, addr = %addr, error = %e, "channel-peer connect attempt failed");
+                }
+            }
+        }
+    }
+    Ok(reconnected)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -985,4 +1171,35 @@ mod tests {
         let cands = unlock_password_candidates().await;
         assert!(cands.iter().any(|p| p == LEGACY_WALLET_PASSWORD));
     }
+
+    #[test]
+    fn reconnect_targets_pick_disconnected_channel_peers_only() {
+        // Shape captured from a live node: /v1/channels uses remote_pubkey,
+        // /v1/peers uses pub_key, and an offline channel's peer is simply
+        // absent from the peer list — that absence is the whole signal.
+        let channels = serde_json::json!({
+            "channels": [
+                { "remote_pubkey": "AAA", "active": true },
+                { "remote_pubkey": "BBB", "active": false },
+                { "remote_pubkey": "AAA" }
+            ]
+        });
+        let peers = serde_json::json!({ "peers": [ { "pub_key": "AAA" } ] });
+
+        let targets = select_reconnect_targets(&channels, &peers);
+        assert_eq!(targets, vec!["BBB".to_string()]);
+    }
+
+    #[test]
+    fn reconnect_targets_empty_without_channels_or_peers() {
+        // No LND wallet (503 error body), locked wallet, or an empty node:
+        // selects nothing, quietly.
+        let error_body = serde_json::json!({ "message": "locked" });
+        assert!(select_reconnect_targets(&error_body, &serde_json::json!({})).is_empty());
+        assert!(select_reconnect_targets(
+            &serde_json::json!({ "channels": [] }),
+            &serde_json::json!({ "peers": [] })
+        )
+        .is_empty());
+    }
 }

core/archipelago/src/server.rs

@@ -841,6 +841,37 @@ impl Server {
             });
         }
 
+        // LND channel-peer watchdog — every 2 minutes, reconnect the peers
+        // of open channels that LND has not re-established on its own. LND's
+        // reconnect logic gives up with a long backoff after repeated or
+        // extended downtime (an app update, a reboot, reconciler churn), and
+        // while the peer link is down BOTH endpoints keep the channel flagged
+        // `disabled` in the routing graph — payments fail "no route" in both
+        // directions while the node itself looks perfectly healthy. The
+        // channel graph is desired state; this keeps it (framework-pt,
+        // 2026-09-01: only channel unroutable ~17h after the 0.21.2 update).
+        // No-ops quietly on nodes without LND. Per-peer retries are throttled
+        // to 10 minutes so an unreachable peer is not hammered every pass.
+        {
+            tokio::spawn(async move {
+                let mut interval = tokio::time::interval(Duration::from_secs(120));
+                let mut last_attempt: HashMap<String, Instant> = HashMap::new();
+                loop {
+                    interval.tick().await;
+                    match crate::container::lnd::reconnect_disconnected_channel_peers(
+                        &mut last_attempt,
+                        Duration::from_secs(600),
+                    )
+                    .await
+                    {
+                        Ok(0) => {}
+                        Ok(n) => info!(n, "LND channel-peer watchdog reconnected channel peers"),
+                        Err(e) => debug!("LND channel-peer watchdog (non-fatal): {}", e),
+                    }
+                }
+            });
+        }
+
         // FIPS seed-anchor apply loop — every 5 minutes we re-push the
         // configured seed anchors into the running fips daemon via
         // `fipsctl connect`. This keeps the mesh bootstrap resilient:

docs/incident-2026-09-01.md

@@ -62,15 +62,22 @@ built and verified to embed the alias fix. `cargo fmt` applied.
 
 ## E. Follow-ups discovered during the incident (ride the NEXT release, v1.8.10+)
 
-- **Funding-modal honesty fix landed after the v1.8.9 tag** (1464b1b2): the
+- **LND channel-peer watchdog** (this release's headline platform fix): every
+  2 minutes the daemon reconnects peers of open channels that LND has not
+  re-established on its own (per-peer retry throttled to 10 minutes), using
+  the peer's advertised addresses from the public graph. Kills the whole
+  class this incident exposed — a channel unroutable ~17h after an LND update
+  while both nodes looked healthy. Unit tests pin the selection logic over the
+  live REST shapes.
+- **Funding-modal honesty fix** (1464b1b2): the
   Lightning "no channel" modal now states the node's real state — pending
   channel confirming / balance on the far side / payment couldn't route /
-  genuinely no channels. Ships in the next release; needs its own
-  create-release run (one more mnemonic paste). The CHANGELOG entry for
-  v1.8.10 should carry it. Note the stale-direction defect it fixes: the
+  genuinely no channels. Note the stale-direction defect it fixes: the
   payment-failure mapper never set the direction, so a SEND failure showed
   the RECEIVE-branch copy ("Receiving needs inbound liquidity…") — the exact
   modal users saw while their node had a healthy 583k-outbound channel.
+  Both fixes have their v1.8.10 CHANGELOG + What's New entries staged so the
+  next `create-release.sh 1.8.10-alpha` runs clean first time.
 - Nodes poll for OTA updates on `daily_check` — after publishing, tell the
   user to hit Update rather than wait for the next check.
 - `origin` remote had a stale pushurl with a dead token (pushes failed);

neode-ui/src/views/settings/AccountInfoSection.vue

@@ -362,6 +362,17 @@ init()
               </button>
             </div>
             <div class="overflow-y-auto flex-1 min-h-0 space-y-6 pr-1">
+              <!-- v1.8.10-alpha -->
+              <div>
+                <div class="flex items-center gap-2 mb-3">
+                  <span class="text-xs font-mono px-2 py-0.5 rounded bg-orange-500/20 text-orange-300">v1.8.10-alpha</span>
+                  <span class="text-xs text-white/40">September 2, 2026</span>
+                </div>
+                <div class="space-y-3 text-sm text-white/80 pl-3 border-l border-white/10">
+                  <p><strong>A channel that drops its peer link now heals itself — on every node.</strong> Restarting LND (an app update, a reboot, container churn) can leave a channel's peer connection down for hours while both endpoints keep the channel flagged disabled in the routing graph: the node looks perfectly healthy, the wallet shows balance, and every payment in either direction fails "no route to the recipient". The daemon now watches the channel graph as desired state — every open channel should have a live peer — and reconnects any that don't. Nodes without LND are untouched; an unreachable peer is retried gently.</p>
+                  <p><strong>The Lightning wallet says what's actually wrong, instead of "you have no channel".</strong> Trying to send while a channel you just opened was still confirming — or when all its balance sits on the far side — produced a modal claiming you had no channel at all, and payment routing failures even showed the receiving copy. The gate now reads your real channel list: a confirming channel gets "it unlocks automatically once confirmed, nothing is needed from you", a far-side balance gets "you can receive, but there's nothing to send right now", and only a genuinely channel-less node is sent to open one.</p>
+                </div>
+              </div>
               <!-- v1.8.9-alpha -->
               <div>
                 <div class="flex items-center gap-2 mb-3">