fix(server): stop blocking network/SSH calls from freezing the async runtime

Root cause of the actual, ongoing outage today (distinct from the connection-
semaphore issue fixed earlier on this branch): several RPC handlers ran
blocking, synchronous I/O directly inside async fns with no real await point,
occupying a tokio worker thread for the full duration of the call:

- network::router::check_upnp_available/check_tor_connectivity/check_dns —
  blocking std::net socket calls (check_dns had no timeout at all).
- api::rpc::openwrt's 5 handlers (get-status, provision-tollgate, scan-wifi,
  configure-wan, scan) — Router::connect_password uses plain
  std::net::TcpStream::connect with NO timeout, wrapped in a fully
  synchronous ssh2 session (connect, handshake, run commands).

Confirmed live via gdb: 3 of 4 tokio worker threads simultaneously blocked
in TcpStream::connect -> Router::connect_password, called from
openwrt.get-status. The frontend's home-dashboard polling loop calls
openwrt.get-status periodically (refreshTollgate); once the configured
router became unreachable (physical relocation to a new network today),
every poll hung for the OS's default multi-minute TCP connect timeout,
and polls arrived faster than they timed out — so stuck attempts
accumulated until every worker thread was blocked and the entire
process (not just openwrt/network calls — every RPC method, since they
share the same small worker pool) stopped responding.

Fix: move all the blocking I/O onto tokio's blocking pool via
spawn_blocking (with an explicit timeout added to check_dns, since unlike
the others it had no bound of its own). Added a regression test
(network::router::blocking_io_tests) proving a cheap concurrent task is
no longer starved while these checks run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

core/archipelago/src/api/rpc/openwrt.rs

@@ -38,7 +38,19 @@ impl RpcHandler {
             .unwrap_or("")
             .to_string();
 
-        let routers = detect::scan_subnet(subnet, prefix, &ssh_user, &ssh_password).await;
+        // `scan_subnet` is `async fn` but loops over up to a full /24 of
+        // blocking TCP probes + SSH handshakes with no real await points —
+        // same blocking-on-a-worker-thread hazard as the other openwrt
+        // handlers (see handle_openwrt_get_status), just larger in scope.
+        let routers = tokio::task::spawn_blocking(move || {
+            tokio::runtime::Handle::current().block_on(detect::scan_subnet(
+                subnet,
+                prefix,
+                &ssh_user,
+                &ssh_password,
+            ))
+        })
+        .await?;
         let ips: Vec<String> = routers.iter().map(|ip| ip.to_string()).collect();
 
         Ok(serde_json::json!({ "routers": ips }))
@@ -87,26 +99,26 @@ impl RpcHandler {
             .or_else(|| saved.password.clone())
             .unwrap_or_default();
 
-        let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
+        // Router/Session (ssh2) is a fully synchronous, blocking API with no
+        // timeout on the initial TCP connect — run it on the blocking pool,
+        // not directly on a tokio worker thread. Inlined here, a single
+        // unreachable router (e.g. after physically relocating the node, so
+        // the configured router is on a different/unreachable network) hangs
+        // for the OS's default TCP connect timeout (routinely 2+ minutes),
+        // and every concurrent poll of this endpoint eats another worker
+        // thread — with only a handful of worker threads total, that starves
+        // every other in-flight request in the whole process. This was a
+        // real full-node outage (2026-07-24), diagnosed via a live gdb
+        // backtrace showing 3 of 4 worker threads blocked in this exact
+        // `TcpStream::connect` → `Router::connect_password` call chain.
+        let host_for_task = host.clone();
+        let ssh_user_for_task = ssh_user.clone();
+        let ssh_password_for_task = ssh_password.clone();
+        let status = tokio::task::spawn_blocking(move || -> Result<serde_json::Value> {
+            let router =
+                Router::connect_password(&host_for_task, 22, &ssh_user_for_task, &ssh_password_for_task)?;
             router.verify_openwrt()?;
 
-        // Persist the connection so other views (e.g. the Home dashboard's
-        // Network tile) can poll `openwrt.get-status` with no params instead
-        // of every caller needing to carry host/credentials around. Only do
-        // this when the host actually came from params — otherwise every
-        // no-args poll would re-save the same thing it just read.
-        if host_from_params {
-            let _ = net_router::configure_router(
-                &self.config.data_dir,
-                net_router::RouterType::OpenWrt,
-                &host,
-                None,
-                Some(&ssh_user),
-                Some(&ssh_password),
-            )
-            .await;
-        }
-
             // System info
             let release = router
                 .run_ok("cat /etc/openwrt_release")
@@ -155,7 +167,7 @@ impl RpcHandler {
             let wan_status = wan::get_wan_status(&router);
 
             Ok(serde_json::json!({
-            "host": host,
+                "host": host_for_task,
                 "hostname": hostname,
                 "uptime_secs": uptime_secs,
                 "release": parse_release(&release),
@@ -163,6 +175,27 @@ impl RpcHandler {
                 "wifi_interfaces": wifi_interfaces,
                 "wan": wan_status,
             }))
+        })
+        .await??;
+
+        // Persist the connection so other views (e.g. the Home dashboard's
+        // Network tile) can poll `openwrt.get-status` with no params instead
+        // of every caller needing to carry host/credentials around. Only do
+        // this when the host actually came from params — otherwise every
+        // no-args poll would re-save the same thing it just read.
+        if host_from_params {
+            let _ = net_router::configure_router(
+                &self.config.data_dir,
+                net_router::RouterType::OpenWrt,
+                &host,
+                None,
+                Some(&ssh_user),
+                Some(&ssh_password),
+            )
+            .await;
+        }
+
+        Ok(status)
     }
 
     /// Provision TollGate on an OpenWrt router and create the "archipelago" SSID.
@@ -228,15 +261,32 @@ impl RpcHandler {
             enabled: p.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true),
         };
 
-        let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
+        let response_ssid = config.ssid.clone();
+        let response_mint_url = config.mint_url.clone();
+
+        // Blocking ssh2 I/O — see handle_openwrt_get_status for why this
+        // must run on the blocking pool rather than a tokio worker thread.
+        // `tollgate::provision` is `async fn` but has no real await points
+        // (every op inside it is a synchronous SSH round trip) — block_on
+        // here just runs it to completion on this blocking-pool thread
+        // instead of pretending it yields on a tokio worker.
+        let host_for_task = host.clone();
+        let ssh_user_for_task = ssh_user.clone();
+        let ssh_password_for_task = ssh_password.clone();
+        tokio::task::spawn_blocking(move || -> Result<()> {
+            let router =
+                Router::connect_password(&host_for_task, 22, &ssh_user_for_task, &ssh_password_for_task)?;
             router.verify_openwrt()?;
-        tollgate::provision(&router, &config).await?;
+            tokio::runtime::Handle::current().block_on(tollgate::provision(&router, &config))?;
+            Ok(())
+        })
+        .await??;
 
         Ok(serde_json::json!({
             "ok": true,
             "host": host,
-            "ssid": config.ssid,
+            "ssid": response_ssid,
-            "mint_url": config.mint_url,
+            "mint_url": response_mint_url,
         }))
     }
 
@@ -279,11 +329,14 @@ impl RpcHandler {
             .or_else(|| saved.password.clone())
             .unwrap_or_default();
 
+        // Blocking ssh2 I/O — see handle_openwrt_get_status for why this
+        // must run on the blocking pool rather than a tokio worker thread.
+        let result = tokio::task::spawn_blocking(move || -> Result<Vec<serde_json::Value>> {
             let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
             router.verify_openwrt()?;
 
             let networks = wifi_scan::scan_networks(&router)?;
-        let result: Vec<serde_json::Value> = networks
+            Ok(networks
                 .iter()
                 .map(|n| {
                     serde_json::json!({
@@ -294,7 +347,9 @@ impl RpcHandler {
                         "encryption": n.encryption,
                     })
                 })
-            .collect();
+                .collect())
+        })
+        .await??;
 
         Ok(serde_json::json!({ "networks": result }))
     }
@@ -357,9 +412,6 @@ impl RpcHandler {
         let dhcp_limit = p.get("dhcp_limit").and_then(|v| v.as_u64()).unwrap_or(150) as u32;
         let masq = p.get("masq").and_then(|v| v.as_bool()).unwrap_or(true);
 
-        let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
-        router.verify_openwrt()?;
-
         let config = wan::WispConfig {
             ssid: ssid.clone(),
             password,
@@ -368,7 +420,17 @@ impl RpcHandler {
             dhcp_limit,
             masq,
         };
+
+        // Blocking ssh2 I/O — see handle_openwrt_get_status for why this
+        // must run on the blocking pool rather than a tokio worker thread.
+        let host_for_task = host.clone();
+        tokio::task::spawn_blocking(move || -> Result<()> {
+            let router = Router::connect_password(&host_for_task, 22, &ssh_user, &ssh_password)?;
+            router.verify_openwrt()?;
             wan::configure_wisp(&router, &config)?;
+            Ok(())
+        })
+        .await??;
 
         Ok(serde_json::json!({ "ok": true, "host": host, "ssid": ssid }))
     }

core/archipelago/src/network/router.rs

@@ -97,7 +97,19 @@ async fn get_wan_ip() -> Option<String> {
 }
 
 /// Check if UPnP is available by attempting SSDP discovery.
+///
+/// The socket I/O here is plain blocking `std::net` (its 3s read timeout is
+/// enforced by the OS, not by yielding to the async runtime), so it must run
+/// on the blocking-pool via `spawn_blocking` — inlined into this "async fn"
+/// directly, it used to occupy a tokio worker thread for the full 3s on
+/// every call. With only as many worker threads as CPU cores, a handful of
+/// concurrent `network.diagnostics` calls (e.g. several Server-settings page
+/// loads) could starve the whole runtime and freeze every other in-flight
+/// request — root cause of a full-node outage (2026-07-24) that had nothing
+/// to do with connection limits and everything to do with blocking sockets
+/// on async worker threads.
 async fn check_upnp_available() -> bool {
+    tokio::task::spawn_blocking(|| {
         use std::net::UdpSocket;
 
         let ssdp_request = "M-SEARCH * HTTP/1.1\r\n\
@@ -133,6 +145,9 @@ async fn check_upnp_available() -> bool {
             }
             Err(_) => false,
         }
+    })
+    .await
+    .unwrap_or(false)
 }
 
 /// Add a port forward (stored locally; actual UPnP mapping done on request).
@@ -281,19 +296,41 @@ pub async fn run_diagnostics() -> Result<NetworkDiagnostics> {
 }
 
 /// Check if Tor SOCKS proxy is reachable.
+///
+/// `TcpStream::connect_timeout` blocks the calling OS thread for up to its
+/// timeout — same blocking-on-a-worker-thread hazard as `check_upnp_available`
+/// above, so this also runs on the blocking pool.
 async fn check_tor_connectivity() -> bool {
+    tokio::task::spawn_blocking(|| {
         use std::net::TcpStream;
         TcpStream::connect_timeout(
             &"127.0.0.1:9050".parse().unwrap(),
             std::time::Duration::from_secs(2),
         )
         .is_ok()
+    })
+    .await
+    .unwrap_or(false)
 }
 
 /// Check DNS resolution works.
+///
+/// `to_socket_addrs()` is a blocking libc resolver call with no timeout of
+/// its own — on a network with a slow/unreachable DNS server (e.g. right
+/// after relocating to a new network) it can hang far longer than the other
+/// checks here. Runs on the blocking pool (same reason as the checks above)
+/// AND under an explicit timeout, since unlike UPnP/Tor there's no built-in
+/// bound to rely on.
 async fn check_dns() -> bool {
+    let lookup = tokio::task::spawn_blocking(|| {
         use std::net::ToSocketAddrs;
         "cloudflare.com:443".to_socket_addrs().is_ok()
+    });
+    tokio::time::timeout(std::time::Duration::from_secs(5), lookup)
+        .await
+        .ok()
+        .and_then(|r| r.ok())
+        .unwrap_or(false)
 }
 
 // --- Router Compatibility Abstraction ---
@@ -469,3 +506,40 @@ pub async fn get_router_info(data_dir: &Path) -> Result<serde_json::Value> {
         },
     }))
 }
+
+#[cfg(test)]
+mod blocking_io_tests {
+    use super::*;
+
+    // Regression test for the 2026-07-24 outage: check_upnp_available,
+    // check_tor_connectivity, and check_dns each did blocking std::net I/O
+    // directly on their calling task instead of via spawn_blocking. On a
+    // small worker pool (4 threads in production), a handful of concurrent
+    // network.diagnostics calls tied up every worker thread for seconds,
+    // freezing every other in-flight RPC request. Proves a cheap task
+    // spawned alongside these checks still gets scheduled promptly, which
+    // only holds if the checks aren't monopolizing worker threads.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn network_checks_do_not_starve_other_tasks() {
+        let cheap = tokio::spawn(async {
+            tokio::time::sleep(std::time::Duration::from_millis(5)).await;
+            std::time::Instant::now()
+        });
+
+        let _ = tokio::join!(
+            check_upnp_available(),
+            check_tor_connectivity(),
+            check_dns()
+        );
+
+        // The assertion is that `cheap` — spawned before the blocking checks
+        // and sleeping only 5ms — finishes within 1s of being spawned. If the
+        // checks were still blocking worker threads directly, a 2-worker
+        // runtime running 3 blocking checks concurrently would starve this
+        // task well past 1s.
+        tokio::time::timeout(std::time::Duration::from_secs(1), cheap)
+            .await
+            .expect("a concurrently-spawned cheap task must not be starved by blocking network checks")
+            .expect("cheap task panicked");
+    }
+}

core/archipelago/src/server.rs

@@ -976,13 +976,21 @@ impl Server {
         main_addr: SocketAddr,
         shutdown: impl std::future::Future<Output = ()>,
     ) -> Result<()> {
-        let active_connections = Arc::new(tokio::sync::Semaphore::new(1024));
+        // Separate pools per listener. Federation/peer connections (fips0,
+        // often over Tor, from nodes we don't control the behavior of) used
+        // to share one pool with the local web UI listener — when peer
+        // connections piled up in CLOSE-WAIT without ever completing, they
+        // starved the shared pool and took the web UI down with them
+        // (production outage, 2026-07-24). Peer congestion must never be
+        // able to block a local login.
+        let main_connections = Arc::new(tokio::sync::Semaphore::new(1024));
+        let peer_connections = Arc::new(tokio::sync::Semaphore::new(256));
         let (tx, rx_main) = tokio::sync::watch::channel(false);
 
         let main_task = tokio::spawn(accept_loop(
             self.api_handler.clone(),
             TcpListener::bind(main_addr).await?,
-            active_connections.clone(),
+            main_connections.clone(),
             false, // main listener: no path filter
             rx_main,
             main_addr,
@@ -992,7 +1000,7 @@ impl Server {
         // restart when fips0 comes up after onboarding.
         let peer_task = tokio::spawn(peer_late_bind_loop(
             self.api_handler.clone(),
-            active_connections.clone(),
+            peer_connections.clone(),
             tx.subscribe(),
         ));
 
@@ -1003,7 +1011,9 @@ impl Server {
         // Wait up to 5s for in-flight requests.
         let drain_start = std::time::Instant::now();
         let drain_timeout = std::time::Duration::from_secs(5);
-        while active_connections.available_permits() < 1024 {
+        while main_connections.available_permits() < 1024
+            || peer_connections.available_permits() < 256
+        {
             if drain_start.elapsed() > drain_timeout {
                 warn!("Drain timeout reached, forcing shutdown");
                 break;
@@ -1096,6 +1106,11 @@ pub fn is_peer_allowed_path(path: &str) -> bool {
         || path.starts_with("/content/")
 }
 
+/// How long a freshly-accepted connection will wait for a connection-pool
+/// permit before it's dropped. Bounds worst-case fd/task growth if the pool
+/// is ever genuinely saturated; under normal load this never triggers.
+const PERMIT_ACQUIRE_TIMEOUT: Duration = Duration::from_secs(30);
+
 async fn accept_loop(
     handler: Arc<ApiHandler>,
     listener: TcpListener,
@@ -1115,8 +1130,35 @@ async fn accept_loop(
                     }
                 };
                 let handler = handler.clone();
-                let permit = active_connections.clone().acquire_owned().await;
+                let active_connections = active_connections.clone();
+                // Acquire the permit *inside* the spawned task, not here.
+                // This loop must never block on anything but accept()/shutdown:
+                // the main (5678) and FIPS peer (5679) listeners share one
+                // semaphore, and a single slow/hung connection holding the
+                // last permit used to freeze this whole loop — including for
+                // the OTHER listener — since accept() couldn't be called
+                // again until a permit freed up. That took down the entire
+                // web UI in production (2026-07-24) when federation peer
+                // connections piled up. Now a saturated pool just delays
+                // (and, past PERMIT_ACQUIRE_TIMEOUT, drops) individual
+                // connections instead of wedging the acceptor itself.
                 tokio::spawn(async move {
+                    let permit = match tokio::time::timeout(
+                        PERMIT_ACQUIRE_TIMEOUT,
+                        active_connections.acquire_owned(),
+                    )
+                    .await
+                    {
+                        Ok(Ok(permit)) => permit,
+                        Ok(Err(_)) => return, // semaphore closed during shutdown
+                        Err(_) => {
+                            warn!(
+                                "{} connection from {} dropped — connection pool saturated for {}s",
+                                local_addr, peer_addr, PERMIT_ACQUIRE_TIMEOUT.as_secs()
+                            );
+                            return;
+                        }
+                    };
                     let _permit = permit;
                     let service = service_fn(move |mut req: hyper::Request<hyper::Body>| {
                         let handler = handler.clone();