Root cause of the actual, ongoing outage today (distinct from the connection-
semaphore issue fixed earlier on this branch): several RPC handlers ran
blocking, synchronous I/O directly inside async fns with no real await point,
occupying a tokio worker thread for the full duration of the call:
- network::router::check_upnp_available/check_tor_connectivity/check_dns —
blocking std::net socket calls (check_dns had no timeout at all).
- api::rpc::openwrt's 5 handlers (get-status, provision-tollgate, scan-wifi,
configure-wan, scan) — Router::connect_password uses plain
std::net::TcpStream::connect with NO timeout, wrapped in a fully
synchronous ssh2 session (connect, handshake, run commands).
Confirmed live via gdb: 3 of 4 tokio worker threads simultaneously blocked
in TcpStream::connect -> Router::connect_password, called from
openwrt.get-status. The frontend's home-dashboard polling loop calls
openwrt.get-status periodically (refreshTollgate); once the configured
router became unreachable (physical relocation to a new network today),
every poll hung for the OS's default multi-minute TCP connect timeout,
and polls arrived faster than they timed out — so stuck attempts
accumulated until every worker thread was blocked and the entire
process (not just openwrt/network calls — every RPC method, since they
share the same small worker pool) stopped responding.
Fix: move all the blocking I/O onto tokio's blocking pool via
spawn_blocking (with an explicit timeout added to check_dns, since unlike
the others it had no bound of its own). Added a regression test
(network::router::blocking_io_tests) proving a cheap concurrent task is
no longer starved while these checks run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ssmithx
7bea2f254d