Compare commits

...

18 Commits

Author SHA1 Message Date
7bea2f254d fix(server): stop blocking network/SSH calls from freezing the async runtime
Root cause of the actual, ongoing outage today (distinct from the connection-
semaphore issue fixed earlier on this branch): several RPC handlers ran
blocking, synchronous I/O directly inside async fns with no real await point,
occupying a tokio worker thread for the full duration of the call:

- network::router::check_upnp_available/check_tor_connectivity/check_dns —
  blocking std::net socket calls (check_dns had no timeout at all).
- api::rpc::openwrt's 5 handlers (get-status, provision-tollgate, scan-wifi,
  configure-wan, scan) — Router::connect_password uses plain
  std::net::TcpStream::connect with NO timeout, wrapped in a fully
  synchronous ssh2 session (connect, handshake, run commands).

Confirmed live via gdb: 3 of 4 tokio worker threads simultaneously blocked
in TcpStream::connect -> Router::connect_password, called from
openwrt.get-status. The frontend's home-dashboard polling loop calls
openwrt.get-status periodically (refreshTollgate); once the configured
router became unreachable (physical relocation to a new network today),
every poll hung for the OS's default multi-minute TCP connect timeout,
and polls arrived faster than they timed out — so stuck attempts
accumulated until every worker thread was blocked and the entire
process (not just openwrt/network calls — every RPC method, since they
share the same small worker pool) stopped responding.

Fix: move all the blocking I/O onto tokio's blocking pool via
spawn_blocking (with an explicit timeout added to check_dns, since unlike
the others it had no bound of its own). Added a regression test
(network::router::blocking_io_tests) proving a cheap concurrent task is
no longer starved while these checks run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-24 14:23:58 +00:00
fac2682268 fix(server): stop federation peer congestion from freezing the web UI
The main (5678) and FIPS peer/federation (5679) listeners shared one
1024-permit connection semaphore, and accept_loop awaited that permit
before spawning the connection handler — inside the accept loop itself.
When federation peer connections piled up in CLOSE-WAIT without ever
completing, they exhausted the shared pool and froze accept() entirely,
taking the web UI down with them (production outage, 2026-07-24: no
login possible, auth.isSetup/server.echo hung indefinitely even hitting
the backend directly).

Two changes:
- Acquire the permit inside the spawned task (with a bounded 30s wait)
  instead of in the accept loop, so a saturated pool can no longer
  block accept() from continuing to accept new connections.
- Give the peer listener its own semaphore, separate from the main
  listener's, so peer/federation congestion can never starve local
  web UI connections again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-24 12:26:06 +00:00
dcb0618012 Merge remote-tracking branch 'origin/main' into archy-hwconfig 2026-07-23 23:55:40 +00:00
8403f2233e probing issues with flashing 2026-07-23 23:43:09 +00:00
078f3b3619 pulled in main 2026-07-23 21:42:02 +00:00
4222a8507c Merge remote-tracking branch 'origin/main' into archy-hwconfig 2026-07-23 21:40:06 +00:00
5799c37111 fix(mesh): DTR/RTS reset settle time was far shorter than real boot time
Every one of Reticulum/Meshcore/Meshtastic's open() deasserts DTR/RTS on
every connection attempt (needed to clear stale line state, but the
transition itself resets ESP32-S3 native-USB boards and CP2102/CH340-
bridged boards wired for Arduino-style auto-reset — acknowledged in the
existing code comments). Each only waited 300ms before expecting a
handshake response — nowhere near real firmware boot time (LoRa radio
init alone routinely takes longer).

A single auto-detect cycle tries multiple protocols in sequence
(Reticulum, then Meshcore, then Meshtastic), each with its own open() and
thus its own reset. With only 300ms of settle per attempt, a board could
plausibly never finish booting from one attempt's reset before the next
attempt's open() reset it again — a self-sustaining "never finishes
booting" loop that would look identical to firmware/hardware flakiness
from the logs, regardless of which firmware family was actually flashed.
Confirmed live 2026-07-23 on both a Heltec V3 (Meshtastic) and V4
(Meshcore): continuous device-side FROM_RADIO_REBOOTED / boot-loop
symptoms with zero config-write-triggered reboots (manage_radio was false
for part of the test), pointing at the connection layer itself rather
than firmware config provisioning.

Bumped the settle delay from 300ms to 2s in Meshcore's and Meshtastic's
open() — full protocol handshakes that need real boot time. Left
reticulum.rs's probe_rnode settle at 300ms deliberately: it's a
cheap/fast KISS-detect gate designed to fail quickly for non-RNode
firmware (documented elsewhere as "~1s"), not a full handshake, and each
subsequent protocol's own open()+settle is what actually needs to cover
real boot time regardless of what probe_rnode did moments before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 19:10:06 +00:00
b14af20d1a fix(mesh): orphaned listener task could race a fresh one on the same port
Traced why a device that's alive and USB-enumerating correctly could still
never complete a single protocol handshake, indefinitely: journal logs
showed genuinely concurrent connection attempts on the same port
(duplicate "Opened serial port"/"Starting X handshake" lines within
microseconds of each other, from what should be sequential probe steps) —
two independent listener sessions were racing on the same tty, each
corrupting the other's reads/writes.

Root cause was in MeshService::stop() combined with mesh::flash's earlier
STOP_LISTENER_TIMEOUT fix: stop() calls `self.listener_handle.take()`
(clearing the field to None immediately) and then awaits the handle with
no bound of its own. When a caller wraps the whole stop() call in a
timeout (as the flash job does, to avoid hanging the RPC response), a slow
listener — mid multi-candidate probe when the shutdown signal arrives —
would cause that outer timeout to cancel the await. But by then
`listener_handle` was already None, so MeshService believed the listener
was stopped, while the actual task kept running, orphaned (dropping a
JoinHandle does not abort the task). A later start() then spawned a
genuinely new listener, racing the orphaned one forever on the same port.

Fixed at the source: stop() now awaits its own handle through a mutable
reference (not by value) with its own bounded timeout, so on timeout it
still owns the handle and can call .abort() on it directly — guaranteeing
the task is actually gone before stop() returns, regardless of how any
caller wraps it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 18:39:33 +00:00
7547d03166 fix(mesh): configure step after a successful flash could race and fail
Traced a real "Operation failed" on the post-flash configure step: the
flash job's own post-completion code unconditionally called
MeshService::start() after configure(), regardless of the persisted
enabled flag. In this incident, the user had toggled mesh off shortly
before the flash (config.enabled saved as false), but the job's forced
start() ran anyway — leaving the listener actually running while
self.config.enabled stayed false. When the user then clicked "Keep As Is"
on the fresh post-flash probe, their mesh.configure call correctly saw a
false→true transition and tried to start the listener itself, hitting
"Mesh listener already running" — the listener had already been
force-started behind the config's back.

Two fixes:
1. The flash job now only restarts the listener if the freshly-loaded
   config still says enabled — respecting a user's own concurrent choice
   to disable mesh instead of silently overriding it.
2. MeshService::start() is now idempotent: finding the listener already
   running is treated as success, not an error. Ensuring the listener is
   running is what every caller actually wants; whichever caller's start()
   wins a race, the other finding it already satisfied is the correct
   outcome, not a failure to surface to the user.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 16:59:41 +00:00
d6019e47a5 fix(mesh): restore esptool's esp32s3 stub instead of routing around it
--erase-all hit the exact same "ROM does not support function erase_flash"
error as a standalone erase_flash — confirmed live: esptool's --erase-all
is implemented as the same full-chip-erase command, not a per-sector loop,
so it needs the stub just as much. The real fix isn't finding another way
around the ROM bootloader's limitations — it's restoring the stub Debian's
package is missing.

Fetched the exact missing file (stub_flasher_32s3.json) from the matching
upstream esptool release tag and installed it directly on archy-x250-dev;
verified live with a read-only `flash_id` that stub mode now loads
("Uploading stub... Running stub..."). Removed --no-stub from flash.rs now
that normal stub-loader mode works correctly (faster and fully-featured vs.
the ROM-only fallback). scripts/self-update.sh now fetches and installs
this same file automatically whenever it's missing, so this isn't a
one-off manual fix tied to a single node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 13:12:15 +00:00
0ca8f25b1b fix(mesh): standalone erase_flash unsupported in --no-stub ROM mode
Real esptool output on a live Heltec V4 exposed the next layer of the
--no-stub fix: "A fatal error occurred: ESP32-S3 ROM does not support
function erase_flash." The ESP32-S3 ROM bootloader only implements
per-sector erase (used internally as write_flash streams data), not a bulk
full-chip-erase opcode — that's a stub-firmware-only feature, and Debian's
esptool package ships without the stub (hence --no-stub in the first
place).

write_flash's own --erase-all flag gets the same "erase everything before
writing" outcome our "always erase before write" default requires, but via
the per-sector mechanism the ROM bootloader actually supports — one esptool
invocation instead of a separate erase_flash + write_flash pair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 12:28:51 +00:00
76d14c3bf9 fix(mesh): flash RPC could hang forever if the listener wouldn't stop
Traced a real "Operation failed" report to its root: MeshService::stop()
ran synchronously inside start_flash_job, BEFORE the background task was
even spawned — so the RPC call itself blocked on it. The mesh listener's
reconnect/multi-candidate-probe loop doesn't check its shutdown signal
between candidates, and in this incident it was mid a Meshcore-then-
Meshtastic probe cycle when the flash request came in, so stop() never
returned. The HTTP request timed out client-side ("connection closed
before message completed" in the server log), the frontend surfaced a
generic "Operation failed", and — worse — the job had already been
inserted into the single-job-guard slot before stop() ran, so with nothing
ever spawned to complete it, every subsequent flash attempt failed with
"already in progress" until a full service restart.

The existing MAX_JOB_DURATION ceiling didn't help here: it only wraps
run_flash(), which is reached AFTER stop() returns — so a hang in stop()
itself was completely unguarded.

Fixed by moving the stop() call inside the spawned background task with
its own 20s timeout. The RPC call now always returns immediately once the
job is registered, and a slow-to-stop listener fails the job cleanly with
a clear message instead of hanging the request and wedging every future
attempt behind it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 10:50:34 +00:00
a1cb83dfb2 fix(mesh): esptool --no-stub + fix retry's broken --baud arg ordering
The improved error logging from the last fix immediately paid off — real
esptool output on the next attempt showed two distinct bugs:

1. Debian's esptool package (4.7.0+dfsg-0.1) ships without the precompiled
   "stub flasher" blobs (stripped for DFSG compliance), so the default
   stub-loader flash mode fails immediately: FileNotFoundError: ... No such
   file or directory: '.../stub_flasher_32s3.json'. Fixed with --no-stub,
   which talks directly to the ROM bootloader instead — the standard
   workaround for this exact Debian packaging gap.

2. The fallback-baud retry appended `--baud 115200` AFTER the subcommand
   token (erase_flash/write_flash), but esptool requires global flags
   before the subcommand — every retry failed with "esptool: error:
   unrecognized arguments: --baud 115200", so the retry logic added
   earlier never actually got a chance to run. esptool_with_retry now
   builds global args (--chip/--port/--baud/--no-stub) and subcommand args
   separately and always concatenates them in the right order, instead of
   relying on call-site ordering of a single flat arg list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 03:21:39 +00:00
ff532465cf fix(mesh): capture actual esptool/rnodeconf output in the failure error
run_streamed's failure path only reported "Command exited with exit status:
N" — the real stderr (already captured line-by-line into job.log_tail for
the UI's live poll) never made it into the error that gets logged to
journald. A real esptool failure just now showed only the bare exit code,
with esptool's actual error text sitting in-memory, visible in the UI but
not diagnosable from the server logs.

Now includes the last 10 log_tail lines in the returned error, so the
actual tool output shows up in journalctl too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 02:51:58 +00:00
39e88529b3 fix(mesh): failed flash silently bounced back to the picker, hiding the error
The step-3 template's condition for showing the picker vs. the progress/
result view was `!flashJob?.active && flashJob?.stage !== 'done'`. That's
true for "no job started yet" (flashJob is null) — but ALSO true for a job
that just FAILED (active:false, stage:'failed'), since 'failed' !== 'done'.
So a failed flash silently reverted to the family/board picker instead of
showing the failure state (error message + log tail) — reported live as
"it went to the download screen and then back to the flash screen" with no
visible error.

Now shows the picker only when no job has been started yet (`!flashJob`);
once a job exists, the progress/result view is always shown, whether it's
still running, succeeded, or failed — the existing error/log-tail markup in
that view already handles all three, it just wasn't being reached.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 02:43:10 +00:00
992bf636e0 fix(mesh): board auto-detect used the wrong signal, false-flagged Heltec V3
The "couldn't confirm the board automatically" warning was checking the
display label text (/v3/i, /v4/i) instead of the same vid:pid table the
backend actually uses to resolve the board. A Heltec V3's CP2102 bridge
chip reports "CP2102 USB to UART Bridge Controller" in its USB strings, not
"Heltec", so meshDeviceImages.ts's fallback label ("LoRa radio (CP2102
serial)") never matched the regex — showing the low-confidence warning (and
leaving the board picker empty) even though the backend's
resolve_flash_board can safely auto-detect V3 via vid:pid 10c4:ea60.

Now checks the same vid:pid directly from detected_device_info, matching
mesh::flash::resolve_flash_board exactly. V4 still has no auto-match entry,
same as the backend — its vid:pid (303a:1001) is the ESP32-S3's generic
native-USB descriptor, not V4-specific, so manual selection is still
required there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 02:31:46 +00:00
beff5dd577 fix(mesh): stop the flash-triggered device boot-loop, fix stuck/wedged jobs
Three real incidents from live-testing the flash feature on archy-x250-dev,
each traced through logs on the node and fixed at the root:

- A failed flash left the mesh listener auto-resuming into a reconnect loop
  whose backoff reset to its 5s minimum on any momentary connection, even
  mid-boot-loop — every retry's open() toggles DTR/RTS, which resets many
  ESP32 boards, so the retries were themselves sustaining the loop. Backoff
  now only resets after a session runs stably for 20s+, and the flash job
  no longer auto-resumes the listener after a failure (only on success,
  after a settle delay).

- The HTTP client's blanket 30s request timeout covered entire downloads
  (Meshtastic's zip is ~170MB), killing large transfers mid-stream; fixed
  with a per-chunk stall timeout instead of a fixed total-transfer cap. But
  the download's initial request had no timeout at all, so a slow-to-start
  server hung it forever — wedging the single-flash-job guard permanently
  ("already in progress" on every subsequent attempt). Fixed with a bounded
  wait for the response to start, plus an absolute ceiling around the whole
  job as a last-resort safety net.

- archy-rnodeconf's PyInstaller freeze broke its own internal esptool
  invocation (sys.executable pointed at the frozen binary itself instead of
  a real interpreter) — fixed via a new runtime hook. esptool's own
  auto-reset-into-bootloader handshake is separately known to be flaky on
  some CP2102/CH340 boards; it now retries once at a conservative baud
  rate instead of failing outright, and failure logs show the full error
  chain instead of just the outer context message.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 02:18:55 +00:00
7d31ca5d65 First commit 2026-07-23 00:33:55 +00:00
24 changed files with 2213 additions and 194 deletions

50
core/Cargo.lock generated
View File

@ -84,6 +84,15 @@ version = "1.0.100"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
[[package]]
name = "arbitrary"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1"
dependencies = [
"derive_arbitrary",
]
[[package]] [[package]]
name = "arc-swap" name = "arc-swap"
version = "1.9.1" version = "1.9.1"
@ -161,6 +170,7 @@ dependencies = [
"uuid", "uuid",
"zbase32", "zbase32",
"zeroize", "zeroize",
"zip",
] ]
[[package]] [[package]]
@ -1175,6 +1185,17 @@ version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
[[package]]
name = "derive_arbitrary"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.114",
]
[[package]] [[package]]
name = "derive_builder" name = "derive_builder"
version = "0.20.2" version = "0.20.2"
@ -6895,8 +6916,37 @@ dependencies = [
"syn 2.0.114", "syn 2.0.114",
] ]
[[package]]
name = "zip"
version = "2.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50"
dependencies = [
"arbitrary",
"crc32fast",
"crossbeam-utils",
"displaydoc",
"flate2",
"indexmap",
"memchr",
"thiserror 2.0.18",
"zopfli",
]
[[package]] [[package]]
name = "zmij" name = "zmij"
version = "1.0.16" version = "1.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dfcd145825aace48cff44a8844de64bf75feec3080e0aa5cdbde72961ae51a65" checksum = "dfcd145825aace48cff44a8844de64bf75feec3080e0aa5cdbde72961ae51a65"
[[package]]
name = "zopfli"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249"
dependencies = [
"bumpalo",
"crc32fast",
"log",
"simd-adler32",
]

View File

@ -108,6 +108,10 @@ bytes = "1"
# Mesh networking (Meshcore serial protocol over USB LoRa radios) # Mesh networking (Meshcore serial protocol over USB LoRa radios)
serial2-tokio = "0.1" serial2-tokio = "0.1"
# LoRa radio firmware flashing: Meshtastic ships per-board images inside a
# per-platform release zip (see mesh/flash.rs).
zip = { version = "2", default-features = false, features = ["deflate"] }
# Double Ratchet key derivation (Phase 3: encrypted mesh messaging) # Double Ratchet key derivation (Phase 3: encrypted mesh messaging)
hkdf = "0.12.4" hkdf = "0.12.4"

View File

@ -388,6 +388,10 @@ impl RpcHandler {
// Mesh networking (Meshcore LoRa) // Mesh networking (Meshcore LoRa)
"mesh.status" => self.handle_mesh_status().await, "mesh.status" => self.handle_mesh_status().await,
"mesh.probe-device" => self.handle_mesh_probe_device(params).await, "mesh.probe-device" => self.handle_mesh_probe_device(params).await,
"mesh.flash-list-firmware" => self.handle_mesh_flash_list_firmware(params).await,
"mesh.flash-device" => self.handle_mesh_flash_device(params).await,
"mesh.flash-status" => self.handle_mesh_flash_status().await,
"mesh.flash-cancel" => self.handle_mesh_flash_cancel().await,
"mesh.peers" => self.handle_mesh_peers().await, "mesh.peers" => self.handle_mesh_peers().await,
"mesh.messages" => self.handle_mesh_messages(params).await, "mesh.messages" => self.handle_mesh_messages(params).await,
"mesh.debug-dump" => self.handle_mesh_debug_dump().await, "mesh.debug-dump" => self.handle_mesh_debug_dump().await,

View File

@ -0,0 +1,132 @@
use super::super::RpcHandler;
use crate::mesh;
use crate::mesh::flash::{self, FlashBoard, FlashJobStatus};
use crate::mesh::types::DeviceType;
use anyhow::Result;
fn parse_family(s: &str) -> Result<DeviceType> {
match s.trim().to_lowercase().as_str() {
"meshcore" => Ok(DeviceType::Meshcore),
"meshtastic" => Ok(DeviceType::Meshtastic),
"reticulum" | "rnode" => Ok(DeviceType::Reticulum),
other => anyhow::bail!("Unknown firmware family: {other} (expected meshcore|meshtastic|reticulum)"),
}
}
fn parse_board(s: &str) -> Result<FlashBoard> {
match s.trim().to_lowercase().as_str() {
"heltec-v3" | "heltec_v3" | "heltecv3" => Ok(FlashBoard::HeltecV3),
"heltec-v4" | "heltec_v4" | "heltecv4" => Ok(FlashBoard::HeltecV4),
other => anyhow::bail!("Unknown board: {other} (expected heltec-v3|heltec-v4)"),
}
}
impl RpcHandler {
/// mesh.flash-list-firmware — resolve the available firmware version(s)
/// for a given family. v1 only ever surfaces "latest".
pub(in crate::api::rpc) async fn handle_mesh_flash_list_firmware(
&self,
params: Option<serde_json::Value>,
) -> Result<serde_json::Value> {
let family = params
.as_ref()
.and_then(|p| p.get("family"))
.and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("Missing family"))?;
let family = parse_family(family)?;
let versions = flash::list_firmware(family).await?;
Ok(serde_json::json!({ "versions": versions }))
}
/// mesh.flash-device — erase and reflash a detected LoRa radio with the
/// latest firmware for the given family, defaulting to a full chip
/// erase before write. `board` is optional: if the port's USB vid:pid
/// unambiguously resolves to a known board, that's used; otherwise the
/// caller must supply it explicitly (see `flash::resolve_flash_board`'s
/// doc comment on why we refuse to guess).
pub(in crate::api::rpc) async fn handle_mesh_flash_device(
&self,
params: Option<serde_json::Value>,
) -> Result<serde_json::Value> {
let path = params
.as_ref()
.and_then(|p| p.get("path"))
.and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("Missing path"))?
.to_string();
let family = params
.as_ref()
.and_then(|p| p.get("family"))
.and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("Missing family"))?;
let family = parse_family(family)?;
let detected = mesh::detect_devices().await;
anyhow::ensure!(
detected.iter().any(|d| d == &path),
"{path} is not a detected mesh-radio candidate port"
);
let board = match params
.as_ref()
.and_then(|p| p.get("board"))
.and_then(|v| v.as_str())
{
Some(explicit) => parse_board(explicit)?,
None => {
let info = mesh::detect_devices_info()
.await
.into_iter()
.find(|d| d.path == path);
info.as_ref()
.and_then(flash::resolve_flash_board)
.ok_or_else(|| {
anyhow::anyhow!(
"Could not auto-detect the board on {path} — specify board explicitly"
)
})?
}
};
flash::start_flash_job(
&self.flash_job,
&self.mesh_service_arc(),
self.config.data_dir.clone(),
path,
board,
family,
)
.await?;
Ok(serde_json::json!({ "started": true }))
}
/// mesh.flash-status — poll the current (or most recent) flash job.
pub(in crate::api::rpc) async fn handle_mesh_flash_status(&self) -> Result<serde_json::Value> {
let job = self.flash_job.read().await;
match job.as_ref() {
Some(j) => {
let status: FlashJobStatus = j.snapshot().await;
let mut value = serde_json::to_value(&status)?;
if let Some(obj) = value.as_object_mut() {
obj.insert("active".into(), (!status.done).into());
}
Ok(value)
}
None => Ok(serde_json::json!({ "active": false })),
}
}
/// mesh.flash-cancel — best-effort; only honored before erase/write has
/// started (see `FlashJob::cancel`'s doc comment).
pub(in crate::api::rpc) async fn handle_mesh_flash_cancel(&self) -> Result<serde_json::Value> {
let job = self.flash_job.read().await;
match job.as_ref() {
Some(j) => {
j.cancel().await?;
Ok(serde_json::json!({ "cancelled": true }))
}
None => anyhow::bail!("No flash job in progress"),
}
}
}

View File

@ -1,5 +1,6 @@
mod assistant; mod assistant;
mod bitcoin_ops; mod bitcoin_ops;
mod flash;
mod messaging; mod messaging;
mod safety; mod safety;
mod status; mod status;

View File

@ -101,12 +101,36 @@ impl RpcHandler {
detected.iter().any(|d| d == &path), detected.iter().any(|d| d == &path),
"{path} is not a detected mesh-radio candidate port" "{path} is not a detected mesh-radio candidate port"
); );
let service = self.mesh_service.read().await; // Refuse to probe while a firmware flash is in flight. Confirmed
let probe = match service.as_ref() { // live 2026-07-23: esptool ("multiple access on port?") and
Some(svc) => svc.probe_device(&path).await?, // rnodeconf (OSError Errno 71 Protocol error on an RTS ioctl) both
// No mesh service yet (radio never enabled) — probe directly. // failed with symptoms consistent with a second process holding the
None => mesh::listener::probe_device(&path).await?, // same serial fd — the flash subprocess runs for minutes outside
}; // our own async runtime, so nothing previously stopped a concurrent
// `mesh.probe-device` call (e.g. the hot-swap modal's own re-probe)
// from opening the identical port at the same time and corrupting
// both operations' handshakes.
if let Some(job) = self.flash_job.read().await.as_ref() {
anyhow::ensure!(
job.snapshot().await.done,
"A firmware flash is in progress — refusing to probe the serial port until it finishes"
);
}
// Only hold the mesh_service lock long enough for the quick
// active-path guard check — NEVER across the actual probe, which
// can take 15-60s across its internal collision retries. Confirmed
// live 2026-07-23: holding this read lock for the full probe starved
// a concurrent firmware-flash job's MeshService::stop() (which needs
// the write lock) well past its own bounded timeout, surfacing as
// "Mesh listener did not release the serial port" even though
// stop() itself was fast.
{
let service = self.mesh_service.read().await;
if let Some(svc) = service.as_ref() {
svc.ensure_probe_allowed(&path).await?;
}
}
let probe = mesh::listener::probe_device(&path).await?;
Ok(serde_json::to_value(probe)?) Ok(serde_json::to_value(probe)?)
} }

View File

@ -89,6 +89,9 @@ pub struct RpcHandler {
endpoint_rate_limiter: EndpointRateLimiter, endpoint_rate_limiter: EndpointRateLimiter,
response_cache: ResponseCache, response_cache: ResponseCache,
mesh_service: Arc<tokio::sync::RwLock<Option<crate::mesh::MeshService>>>, mesh_service: Arc<tokio::sync::RwLock<Option<crate::mesh::MeshService>>>,
/// LoRa radio firmware-flash job state, sibling to `mesh_service` — one
/// job at a time, since flashing needs exclusive access to the port.
flash_job: crate::mesh::flash::FlashJobHandle,
transport_router: Arc<tokio::sync::RwLock<Option<Arc<crate::transport::TransportRouter>>>>, transport_router: Arc<tokio::sync::RwLock<Option<Arc<crate::transport::TransportRouter>>>>,
/// Shared content-addressed blob store. Set by ApiHandler after construction /// Shared content-addressed blob store. Set by ApiHandler after construction
/// so mesh.send-content / mesh.fetch-content RPCs can reach it without a /// so mesh.send-content / mesh.fetch-content RPCs can reach it without a
@ -160,6 +163,7 @@ impl RpcHandler {
endpoint_rate_limiter, endpoint_rate_limiter,
response_cache: ResponseCache::new(5), response_cache: ResponseCache::new(5),
mesh_service: Arc::new(tokio::sync::RwLock::new(None)), mesh_service: Arc::new(tokio::sync::RwLock::new(None)),
flash_job: crate::mesh::flash::new_job_handle(),
transport_router: Arc::new(tokio::sync::RwLock::new(None)), transport_router: Arc::new(tokio::sync::RwLock::new(None)),
blob_store: Arc::new(tokio::sync::RwLock::new(None)), blob_store: Arc::new(tokio::sync::RwLock::new(None)),
self_pubkey_hex: Arc::new(tokio::sync::RwLock::new(None)), self_pubkey_hex: Arc::new(tokio::sync::RwLock::new(None)),

View File

@ -38,7 +38,19 @@ impl RpcHandler {
.unwrap_or("") .unwrap_or("")
.to_string(); .to_string();
let routers = detect::scan_subnet(subnet, prefix, &ssh_user, &ssh_password).await; // `scan_subnet` is `async fn` but loops over up to a full /24 of
// blocking TCP probes + SSH handshakes with no real await points —
// same blocking-on-a-worker-thread hazard as the other openwrt
// handlers (see handle_openwrt_get_status), just larger in scope.
let routers = tokio::task::spawn_blocking(move || {
tokio::runtime::Handle::current().block_on(detect::scan_subnet(
subnet,
prefix,
&ssh_user,
&ssh_password,
))
})
.await?;
let ips: Vec<String> = routers.iter().map(|ip| ip.to_string()).collect(); let ips: Vec<String> = routers.iter().map(|ip| ip.to_string()).collect();
Ok(serde_json::json!({ "routers": ips })) Ok(serde_json::json!({ "routers": ips }))
@ -87,8 +99,84 @@ impl RpcHandler {
.or_else(|| saved.password.clone()) .or_else(|| saved.password.clone())
.unwrap_or_default(); .unwrap_or_default();
let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?; // Router/Session (ssh2) is a fully synchronous, blocking API with no
router.verify_openwrt()?; // timeout on the initial TCP connect — run it on the blocking pool,
// not directly on a tokio worker thread. Inlined here, a single
// unreachable router (e.g. after physically relocating the node, so
// the configured router is on a different/unreachable network) hangs
// for the OS's default TCP connect timeout (routinely 2+ minutes),
// and every concurrent poll of this endpoint eats another worker
// thread — with only a handful of worker threads total, that starves
// every other in-flight request in the whole process. This was a
// real full-node outage (2026-07-24), diagnosed via a live gdb
// backtrace showing 3 of 4 worker threads blocked in this exact
// `TcpStream::connect` → `Router::connect_password` call chain.
let host_for_task = host.clone();
let ssh_user_for_task = ssh_user.clone();
let ssh_password_for_task = ssh_password.clone();
let status = tokio::task::spawn_blocking(move || -> Result<serde_json::Value> {
let router =
Router::connect_password(&host_for_task, 22, &ssh_user_for_task, &ssh_password_for_task)?;
router.verify_openwrt()?;
// System info
let release = router
.run_ok("cat /etc/openwrt_release")
.unwrap_or_default();
let hostname = router
.uci_get("system.@system[0].hostname")
.unwrap_or_else(|_| "unknown".into());
let uptime_secs: u64 = router
.run_ok("cat /proc/uptime")
.unwrap_or_default()
.split_whitespace()
.next()
.and_then(|s| s.split('.').next())
.and_then(|s| s.parse().ok())
.unwrap_or(0);
// TollGate — check via opkg (≤24.x) or binary presence (25.x apk-native).
// The service binary is /usr/bin/tollgate-wrt (per its init.d script),
// not /usr/bin/tollgate-module-basic-go — that's only the opkg/apk
// *package* name, never an on-disk filename.
let tollgate_installed = router
.run("/usr/bin/opkg list-installed 2>/dev/null | grep -q '^tollgate-module-basic-go ' || \
test -f /usr/bin/tollgate-wrt 2>/dev/null")
.map(|(_, code)| code == 0)
.unwrap_or(false);
let tollgate = if tollgate_installed {
serde_json::json!({
"installed": true,
"enabled": router.uci_get("tollgate.main.enabled").map(|v| v == "1").unwrap_or(false),
"metric": router.uci_get("tollgate.main.metric").unwrap_or_default(),
"step_size_ms": router.uci_get("tollgate.main.step_size").ok().and_then(|v| v.parse::<u64>().ok()).unwrap_or(0),
"price_per_step":router.uci_get("tollgate.main.price_per_step").ok().and_then(|v| v.parse::<u64>().ok()).unwrap_or(0),
"min_steps": router.uci_get("tollgate.main.min_steps").ok().and_then(|v| v.parse::<u32>().ok()).unwrap_or(1),
"currency": router.uci_get("tollgate.main.currency").unwrap_or_default(),
"mint_url": router.uci_get("tollgate.main.mint_url").unwrap_or_default(),
})
} else {
serde_json::json!({ "installed": false })
};
// WiFi interfaces
let wifi_raw = router.run_ok("uci show wireless").unwrap_or_default();
let wifi_interfaces = parse_wifi_interfaces(&wifi_raw);
let wan_status = wan::get_wan_status(&router);
Ok(serde_json::json!({
"host": host_for_task,
"hostname": hostname,
"uptime_secs": uptime_secs,
"release": parse_release(&release),
"tollgate": tollgate,
"wifi_interfaces": wifi_interfaces,
"wan": wan_status,
}))
})
.await??;
// Persist the connection so other views (e.g. the Home dashboard's // Persist the connection so other views (e.g. the Home dashboard's
// Network tile) can poll `openwrt.get-status` with no params instead // Network tile) can poll `openwrt.get-status` with no params instead
@ -107,62 +195,7 @@ impl RpcHandler {
.await; .await;
} }
// System info Ok(status)
let release = router
.run_ok("cat /etc/openwrt_release")
.unwrap_or_default();
let hostname = router
.uci_get("system.@system[0].hostname")
.unwrap_or_else(|_| "unknown".into());
let uptime_secs: u64 = router
.run_ok("cat /proc/uptime")
.unwrap_or_default()
.split_whitespace()
.next()
.and_then(|s| s.split('.').next())
.and_then(|s| s.parse().ok())
.unwrap_or(0);
// TollGate — check via opkg (≤24.x) or binary presence (25.x apk-native).
// The service binary is /usr/bin/tollgate-wrt (per its init.d script),
// not /usr/bin/tollgate-module-basic-go — that's only the opkg/apk
// *package* name, never an on-disk filename.
let tollgate_installed = router
.run("/usr/bin/opkg list-installed 2>/dev/null | grep -q '^tollgate-module-basic-go ' || \
test -f /usr/bin/tollgate-wrt 2>/dev/null")
.map(|(_, code)| code == 0)
.unwrap_or(false);
let tollgate = if tollgate_installed {
serde_json::json!({
"installed": true,
"enabled": router.uci_get("tollgate.main.enabled").map(|v| v == "1").unwrap_or(false),
"metric": router.uci_get("tollgate.main.metric").unwrap_or_default(),
"step_size_ms": router.uci_get("tollgate.main.step_size").ok().and_then(|v| v.parse::<u64>().ok()).unwrap_or(0),
"price_per_step":router.uci_get("tollgate.main.price_per_step").ok().and_then(|v| v.parse::<u64>().ok()).unwrap_or(0),
"min_steps": router.uci_get("tollgate.main.min_steps").ok().and_then(|v| v.parse::<u32>().ok()).unwrap_or(1),
"currency": router.uci_get("tollgate.main.currency").unwrap_or_default(),
"mint_url": router.uci_get("tollgate.main.mint_url").unwrap_or_default(),
})
} else {
serde_json::json!({ "installed": false })
};
// WiFi interfaces
let wifi_raw = router.run_ok("uci show wireless").unwrap_or_default();
let wifi_interfaces = parse_wifi_interfaces(&wifi_raw);
let wan_status = wan::get_wan_status(&router);
Ok(serde_json::json!({
"host": host,
"hostname": hostname,
"uptime_secs": uptime_secs,
"release": parse_release(&release),
"tollgate": tollgate,
"wifi_interfaces": wifi_interfaces,
"wan": wan_status,
}))
} }
/// Provision TollGate on an OpenWrt router and create the "archipelago" SSID. /// Provision TollGate on an OpenWrt router and create the "archipelago" SSID.
@ -228,15 +261,32 @@ impl RpcHandler {
enabled: p.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true), enabled: p.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true),
}; };
let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?; let response_ssid = config.ssid.clone();
router.verify_openwrt()?; let response_mint_url = config.mint_url.clone();
tollgate::provision(&router, &config).await?;
// Blocking ssh2 I/O — see handle_openwrt_get_status for why this
// must run on the blocking pool rather than a tokio worker thread.
// `tollgate::provision` is `async fn` but has no real await points
// (every op inside it is a synchronous SSH round trip) — block_on
// here just runs it to completion on this blocking-pool thread
// instead of pretending it yields on a tokio worker.
let host_for_task = host.clone();
let ssh_user_for_task = ssh_user.clone();
let ssh_password_for_task = ssh_password.clone();
tokio::task::spawn_blocking(move || -> Result<()> {
let router =
Router::connect_password(&host_for_task, 22, &ssh_user_for_task, &ssh_password_for_task)?;
router.verify_openwrt()?;
tokio::runtime::Handle::current().block_on(tollgate::provision(&router, &config))?;
Ok(())
})
.await??;
Ok(serde_json::json!({ Ok(serde_json::json!({
"ok": true, "ok": true,
"host": host, "host": host,
"ssid": config.ssid, "ssid": response_ssid,
"mint_url": config.mint_url, "mint_url": response_mint_url,
})) }))
} }
@ -279,22 +329,27 @@ impl RpcHandler {
.or_else(|| saved.password.clone()) .or_else(|| saved.password.clone())
.unwrap_or_default(); .unwrap_or_default();
let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?; // Blocking ssh2 I/O — see handle_openwrt_get_status for why this
router.verify_openwrt()?; // must run on the blocking pool rather than a tokio worker thread.
let result = tokio::task::spawn_blocking(move || -> Result<Vec<serde_json::Value>> {
let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
router.verify_openwrt()?;
let networks = wifi_scan::scan_networks(&router)?; let networks = wifi_scan::scan_networks(&router)?;
let result: Vec<serde_json::Value> = networks Ok(networks
.iter() .iter()
.map(|n| { .map(|n| {
serde_json::json!({ serde_json::json!({
"ssid": n.ssid, "ssid": n.ssid,
"bssid": n.bssid, "bssid": n.bssid,
"signal": n.signal, "signal": n.signal,
"channel": n.channel, "channel": n.channel,
"encryption": n.encryption, "encryption": n.encryption,
})
}) })
}) .collect())
.collect(); })
.await??;
Ok(serde_json::json!({ "networks": result })) Ok(serde_json::json!({ "networks": result }))
} }
@ -357,9 +412,6 @@ impl RpcHandler {
let dhcp_limit = p.get("dhcp_limit").and_then(|v| v.as_u64()).unwrap_or(150) as u32; let dhcp_limit = p.get("dhcp_limit").and_then(|v| v.as_u64()).unwrap_or(150) as u32;
let masq = p.get("masq").and_then(|v| v.as_bool()).unwrap_or(true); let masq = p.get("masq").and_then(|v| v.as_bool()).unwrap_or(true);
let router = Router::connect_password(&host, 22, &ssh_user, &ssh_password)?;
router.verify_openwrt()?;
let config = wan::WispConfig { let config = wan::WispConfig {
ssid: ssid.clone(), ssid: ssid.clone(),
password, password,
@ -368,7 +420,17 @@ impl RpcHandler {
dhcp_limit, dhcp_limit,
masq, masq,
}; };
wan::configure_wisp(&router, &config)?;
// Blocking ssh2 I/O — see handle_openwrt_get_status for why this
// must run on the blocking pool rather than a tokio worker thread.
let host_for_task = host.clone();
tokio::task::spawn_blocking(move || -> Result<()> {
let router = Router::connect_password(&host_for_task, 22, &ssh_user, &ssh_password)?;
router.verify_openwrt()?;
wan::configure_wisp(&router, &config)?;
Ok(())
})
.await??;
Ok(serde_json::json!({ "ok": true, "host": host, "ssid": ssid })) Ok(serde_json::json!({ "ok": true, "host": host, "ssid": ssid }))
} }

View File

@ -0,0 +1,975 @@
// WIP mesh/transport protocol — suppress dead code warnings
#![allow(dead_code)]
//! Firmware flashing for LoRa mesh radios — Heltec V3/V4 in v1, across all
//! three firmware families the mesh module already knows how to detect (see
//! `mesh::types::DeviceType`). Firmware is always fetched from upstream at
//! flash time (never bundled/pinned in the repo), and every flash defaults
//! to a full chip erase before write.
//!
//! MeshCore and Meshtastic are flashed the same way: download a released
//! image, `esptool erase_flash`, then `esptool write_flash 0x0 <image>`.
//! Reticulum/RNode is different: `archy-rnodeconf --autoinstall` owns the
//! whole fetch+erase+flash+EEPROM-bootstrap sequence itself (confirmed live
//! via `archy-rnodeconf --help` — there is no raw esptool path exposed for
//! this family, so we deliberately don't resolve a firmware URL ourselves
//! for Reticulum; rnodeconf already knows how).
use super::serial::DetectedDeviceInfo;
use super::types::DeviceType;
use super::MeshService;
use anyhow::{Context, Result};
use regex::Regex;
use serde::Serialize;
use std::path::{Path, PathBuf};
use std::process::Stdio;
use std::sync::{Arc, OnceLock};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::process::Command;
use tokio::sync::RwLock;
use tracing::{info, warn};
/// Boards supported for v1. Both are ESP32-S3 (a single `--chip esp32s3`
/// esptool target covers both), but ship different USB identities and
/// different per-board firmware assets upstream.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum FlashBoard {
HeltecV3,
HeltecV4,
}
impl FlashBoard {
/// Meshtastic's board id (matches the release manifest's `board` field
/// and its per-board asset naming, e.g. `firmware-heltec-v3-<ver>.factory.bin`).
fn meshtastic_id(self) -> &'static str {
match self {
Self::HeltecV3 => "heltec-v3",
Self::HeltecV4 => "heltec-v4",
}
}
}
/// Map a detected USB vid:pid to a known flashable board, using the same
/// table as `image-recipe/configs/99-mesh-radio.rules`. CP2102 (10c4:ea60)
/// is confirmed there as Heltec V3's USB-UART bridge chip, and is safe to
/// auto-match since that vid:pid is bridge-chip-specific.
///
/// Heltec V4 is NOT auto-matchable and deliberately has no entry here: it
/// was confirmed live (real hardware, 2026-07-23) to use the ESP32-S3's
/// built-in native-USB JTAG/serial peripheral, reporting vid:pid 303a:1001
/// with product string "USB JTAG/serial debug unit" — that descriptor is
/// baked into the chip's ROM and is IDENTICAL across every ESP32-S3 board
/// with native USB enabled, not just Heltec V4. Adding `303a:1001 =>
/// HeltecV4` here would silently misidentify any other native-USB ESP32-S3
/// board (a T3-S3, a bare devkit, etc.) as a V4 and risk writing the wrong
/// board's image. Callers (the RPC layer / frontend) must let the user pick
/// the board manually whenever this returns `None`.
pub fn resolve_flash_board(info: &DetectedDeviceInfo) -> Option<FlashBoard> {
match (info.vid.as_deref(), info.pid.as_deref()) {
(Some("10c4"), Some("ea60")) => Some(FlashBoard::HeltecV3),
_ => None,
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum FlashStage {
Downloading,
Erasing,
Writing,
Autoinstalling,
Done,
Failed,
}
#[derive(Debug, Clone, Serialize)]
pub struct FlashJobStatus {
pub board: FlashBoard,
pub family: DeviceType,
pub path: String,
pub stage: FlashStage,
pub percent: Option<u8>,
pub log_tail: Vec<String>,
pub done: bool,
pub error: Option<String>,
}
const LOG_TAIL_MAX: usize = 200;
/// How long to wait after a successful flash before resuming the mesh
/// listener, so the board finishes its own post-flash boot/reset before we
/// start opening the port (which itself toggles DTR/RTS) again.
const POST_FLASH_SETTLE_DELAY: std::time::Duration = std::time::Duration::from_secs(5);
/// Absolute ceiling on a whole flash job (download + erase + write, or
/// autoinstall), regardless of what it's doing internally. Last-resort
/// safety net so a hang anywhere can't wedge the single-flash-job guard
/// forever — generous enough to never trigger on a legitimately slow
/// multi-hundred-MB transfer.
const MAX_JOB_DURATION: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// How long to wait for MeshService::stop() to release the serial port
/// before giving up. Confirmed live 2026-07-23: the listener's own
/// reconnect/multi-candidate-probe loop doesn't check its shutdown signal
/// between candidates, so stop() can take a while (or, if the loop is
/// wedged, never return) — 20s comfortably covers a normal handshake-probe
/// cycle without leaving a flash request hanging indefinitely if the
/// listener genuinely won't let go.
const STOP_LISTENER_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20);
/// How long to keep retrying the port-free check before giving up.
const PORT_FREE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
/// Confirm nothing else has `path` open by actually opening (and immediately
/// closing) it ourselves. Retries across the timeout since a just-stopped
/// listener's fd can take a moment to actually release even after `stop()`
/// returns (task abort is a request, not an instant guarantee the OS-level
/// resource is gone yet).
async fn wait_for_port_free(path: &str) -> Result<()> {
let deadline = tokio::time::Instant::now() + PORT_FREE_TIMEOUT;
let mut last_err = None;
loop {
match serial2_tokio::SerialPort::open(path, 115200) {
Ok(_) => return Ok(()),
Err(e) => last_err = Some(e),
}
if tokio::time::Instant::now() >= deadline {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
Err(anyhow::anyhow!(
"{path} is still held open by something else after {}s (last error: {}) — refusing to start the flasher against a contended port",
PORT_FREE_TIMEOUT.as_secs(),
last_err.map(|e| e.to_string()).unwrap_or_default()
))
}
/// Live state for the one flash job that can run at a time. A single global
/// slot is sufficient because flashing needs exclusive serial access to the
/// one port being flashed — there is no meaningful concept of two concurrent
/// flash jobs on this node.
pub struct FlashJob {
status: RwLock<FlashJobStatus>,
/// Set once the background task is spawned. Only used while `stage` is
/// still `Downloading` — an interrupted erase/write can leave the chip
/// in a worse state than either finished or unstarted, so cancellation
/// is refused once erase begins (see `cancel()`).
abort_handle: RwLock<Option<tokio::task::AbortHandle>>,
}
impl FlashJob {
fn new(board: FlashBoard, family: DeviceType, path: String) -> Arc<Self> {
Arc::new(Self {
abort_handle: RwLock::new(None),
status: RwLock::new(FlashJobStatus {
board,
family,
path,
stage: FlashStage::Downloading,
percent: None,
log_tail: Vec::new(),
done: false,
error: None,
}),
})
}
pub async fn snapshot(&self) -> FlashJobStatus {
self.status.read().await.clone()
}
async fn set_stage(&self, stage: FlashStage) {
let mut s = self.status.write().await;
s.stage = stage;
s.percent = None;
}
async fn set_percent(&self, percent: u8) {
self.status.write().await.percent = Some(percent.min(100));
}
async fn push_log(&self, line: impl Into<String>) {
let mut s = self.status.write().await;
s.log_tail.push(line.into());
let overflow = s.log_tail.len().saturating_sub(LOG_TAIL_MAX);
if overflow > 0 {
s.log_tail.drain(0..overflow);
}
}
async fn fail(&self, err: &anyhow::Error) {
let mut s = self.status.write().await;
s.stage = FlashStage::Failed;
s.error = Some(format!("{err:#}"));
s.done = true;
}
async fn finish(&self) {
let mut s = self.status.write().await;
s.stage = FlashStage::Done;
s.done = true;
}
/// Best-effort cancel: only honored before erase/write/autoinstall has
/// started (i.e. still in `Downloading`). Once a stage that touches the
/// chip begins, this refuses — interrupting an erase or write can leave
/// the flash in a state worse than either finished or unstarted.
pub async fn cancel(&self) -> Result<()> {
let mut s = self.status.write().await;
if s.done {
anyhow::bail!("Flash job already finished");
}
if s.stage != FlashStage::Downloading {
anyhow::bail!(
"Cannot cancel once {:?} has started — let it finish or fail on its own",
s.stage
);
}
if let Some(handle) = self.abort_handle.write().await.take() {
handle.abort();
}
s.stage = FlashStage::Failed;
s.error = Some("Cancelled by user".to_string());
s.done = true;
Ok(())
}
}
/// Shared handle held by `RpcHandler`, sibling to `mesh_service`.
pub type FlashJobHandle = Arc<RwLock<Option<Arc<FlashJob>>>>;
pub fn new_job_handle() -> FlashJobHandle {
Arc::new(RwLock::new(None))
}
fn firmware_cache_dir(data_dir: &Path) -> PathBuf {
data_dir.join("mesh").join("firmware-cache")
}
/// No blanket `.timeout()` here on purpose: reqwest's request timeout covers
/// the *entire* request including streaming the response body, which would
/// kill a legitimate large download partway through (Meshtastic's esp32s3
/// zip is ~170MB) — not just a hung connection. `download_to_file` instead
/// applies a per-chunk stall timeout, and metadata calls (small JSON
/// responses) get their own short timeout at the call site.
fn github_client() -> Result<reqwest::Client> {
reqwest::Client::builder()
.user_agent("archipelago-mesh-flash")
.connect_timeout(std::time::Duration::from_secs(10))
.build()
.context("Failed to build HTTP client")
}
/// Applied per-chunk while streaming a firmware download — if the transfer
/// stalls (no bytes for this long) it's treated as a failure, but a slow
/// download that's still making progress is never killed just for taking a
/// while.
const DOWNLOAD_STALL_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// Applied to metadata calls (GitHub release JSON) — these are small
/// responses with no reason to ever take this long.
const METADATA_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20);
/// Resolve what firmware is available for a board+family. v1 only ever
/// offers "latest" — MeshCore/Meshtastic latest GitHub release, or, for
/// Reticulum, "latest" meaning "whatever archy-rnodeconf --autoinstall
/// resolves on its own" (it does its own version checking upstream).
pub async fn list_firmware(family: DeviceType) -> Result<Vec<String>> {
match family {
DeviceType::Reticulum => Ok(vec!["latest".to_string()]),
DeviceType::Meshtastic => {
let client = github_client()?;
let release: GithubRelease = client
.get("https://api.github.com/repos/meshtastic/firmware/releases/latest")
.send()
.await
.context("Fetching Meshtastic release list")?
.error_for_status()
.context("Meshtastic releases API error")?
.json()
.await
.context("Parsing Meshtastic release JSON")?;
Ok(vec![release.tag_name])
}
DeviceType::Meshcore => {
let client = github_client()?;
let release: GithubRelease = client
.get("https://api.github.com/repos/meshcore-dev/MeshCore/releases/latest")
.send()
.await
.context("Fetching MeshCore release list")?
.error_for_status()
.context("MeshCore releases API error")?
.json()
.await
.context("Parsing MeshCore release JSON")?;
Ok(vec![release.tag_name])
}
DeviceType::Unknown => anyhow::bail!("Pick a firmware family before listing versions"),
}
}
#[derive(serde::Deserialize)]
struct GithubAsset {
name: String,
browser_download_url: String,
}
#[derive(serde::Deserialize)]
struct GithubRelease {
tag_name: String,
assets: Vec<GithubAsset>,
}
/// Start a flash job in the background. Returns as soon as the job has been
/// registered and the listener released — callers poll `FlashJobHandle` via
/// `mesh.flash-status` for progress. Only one job may be in flight at a time.
pub async fn start_flash_job(
handle: &FlashJobHandle,
mesh_service: &Arc<RwLock<Option<MeshService>>>,
data_dir: PathBuf,
path: String,
board: FlashBoard,
family: DeviceType,
) -> Result<()> {
{
let existing = handle.read().await;
if let Some(job) = existing.as_ref() {
if !job.snapshot().await.done {
anyhow::bail!("A firmware flash is already in progress on this node");
}
}
}
let job = FlashJob::new(board, family, path.clone());
*handle.write().await = Some(Arc::clone(&job));
let bg_job = Arc::clone(&job);
let bg_service = Arc::clone(mesh_service);
let task = tokio::spawn(async move {
// esptool/archy-rnodeconf need exclusive serial access — release
// the listener's hold on the port before touching it. This USED
// TO run synchronously in start_flash_job before the job was even
// spawned, blocking the RPC call itself on s.stop().await — a real
// 2026-07-23 incident: the mesh listener was mid a multi-candidate
// reconnect/probe sequence that doesn't check its shutdown signal
// between candidates, so stop() never returned. The HTTP request
// timed out client-side ("Operation failed"), while the job
// (already inserted into `handle`) was permanently wedged — nothing
// had been spawned yet to ever mark it done, so every later flash
// attempt failed with "already in progress" until a full restart.
// Now this runs inside the spawned task with its own bounded
// timeout, so the RPC call always returns immediately regardless,
// and a slow-to-stop listener fails the job cleanly instead of
// hanging everything downstream of it forever.
let stop_result = tokio::time::timeout(STOP_LISTENER_TIMEOUT, async {
let mut svc = bg_service.write().await;
if let Some(s) = svc.as_mut() {
s.stop().await;
}
})
.await;
if stop_result.is_err() {
let err = anyhow::anyhow!(
"Mesh listener did not release the serial port within {}s — it may still be mid a reconnect attempt. Try again once mesh.status shows the device idle, or restart the archipelago service if this persists.",
STOP_LISTENER_TIMEOUT.as_secs()
);
bg_job.push_log(format!("ERROR: {err:#}")).await;
bg_job.fail(&err).await;
return;
}
// Belt-and-suspenders port-free check. `stop()` above should have
// fully released the port, but esptool/rnodeconf run as external
// subprocesses for minutes outside our own async runtime — if
// ANYTHING else still has it open (a racing probe, a not-yet-dropped
// fd from an aborted task, anything we haven't anticipated), handing
// the port to the flasher anyway risks exactly the corruption
// confirmed live 2026-07-23: esptool's "device disconnected or
// multiple access on port?" and rnodeconf's raw `OSError: [Errno 71]
// Protocol error` on an RTS ioctl are both textbook two-openers-on-
// one-fd symptoms. Verify by actually opening it ourselves — cheap,
// and definitive — before ever starting the flasher.
if let Err(e) = wait_for_port_free(&path).await {
bg_job.push_log(format!("ERROR: {e:#}")).await;
bg_job.fail(&e).await;
return;
}
// Outer ceiling on top of run_flash's own internal timeouts —
// belt-and-suspenders so that no future hang (network, subprocess,
// anything) can ever wedge the single-flash-job guard permanently
// again the way a stuck download did on 2026-07-23 (every
// subsequent mesh.flash-device call failed with "already in
// progress" until the service was restarted). Generous enough that
// a legitimately slow multi-hundred-MB transfer still completes.
let result = match tokio::time::timeout(
MAX_JOB_DURATION,
run_flash(board, family, &data_dir, &path, &bg_job),
)
.await
{
Ok(inner) => inner,
Err(_) => Err(anyhow::anyhow!(
"Flash job exceeded the {}-minute ceiling — aborted",
MAX_JOB_DURATION.as_secs() / 60
)),
};
let succeeded = result.is_ok();
match &result {
Ok(()) => {
bg_job.push_log("Flash completed successfully".to_string()).await;
bg_job.finish().await;
info!(path = %path, board = ?board, family = %family, "LoRa firmware flash succeeded");
}
Err(e) => {
// {:#} (alternate Display) walks the full anyhow context
// chain — plain {} / %e only prints the outermost .context()
// message, which made a real 2026-07-23 esptool failure
// undiagnosable from journalctl alone (just "esptool
// erase_flash failed", no actual esptool stderr).
warn!(path = %path, error = %format!("{e:#}"), "LoRa firmware flash failed");
bg_job.push_log(format!("ERROR: {e:#}")).await;
bg_job.fail(e).await;
}
}
// The board's firmware may now differ from whatever was pinned
// before — clear the pin either way so a later reconnect's strict
// auto-detect order picks up reality instead of getting wedged
// trying the old protocol first.
if let Ok(mut config) = super::load_config(&data_dir).await {
config.device_kind = None;
if let Err(e) = super::save_config(&data_dir, &config).await {
warn!(error = %e, "Failed to clear device_kind pin after flash");
}
}
if !succeeded {
// Deliberately do NOT auto-restart the listener here. A failed
// flash means we can't vouch for the board's state — reopening
// the port immediately (esptool/rnodeconf's own reset sequence
// plus our open() toggling DTR/RTS again right after) risks
// hammering a marginal device with reconnect attempts. Confirmed
// live 2026-07-23: exactly this sequence left a real Heltec V3
// boot-looping for 5+ minutes after a failed flash. Leave mesh
// stopped; the user reconnects explicitly via the hot-swap
// modal/Mesh page once they've confirmed the board is alive.
warn!(
path = %path,
"Leaving mesh listener stopped after failed flash — reconnect manually once the board is confirmed responsive"
);
return;
}
// On success, give the board a moment to finish booting after the
// flash tool's own reset sequence before we start hammering it with
// connection attempts — same reasoning as above, just the
// lower-risk (successful-flash) side of it.
tokio::time::sleep(POST_FLASH_SETTLE_DELAY).await;
let mut svc = bg_service.write().await;
if let Some(s) = svc.as_mut() {
match super::load_config(&data_dir).await {
Ok(config) => {
// Only resume if mesh is actually still enabled per the
// CURRENT persisted config — confirmed live 2026-07-23:
// unconditionally forcing a restart here, regardless of
// `enabled`, overrode a user's own concurrent "disable
// mesh" toggle and left the listener running while
// config said disabled. That inconsistent state is what
// made a later legitimate "Keep As Is" click (which
// correctly tries to start on a false→true transition)
// fail with "already running" — the listener had already
// been force-started behind the config's back.
let should_run = config.enabled;
if let Err(e) = s.configure(config).await {
warn!(error = %e, "Failed to resume mesh listener after flash");
}
if should_run {
if let Err(e) = s.start() {
warn!(error = %e, "Failed to restart mesh listener after flash");
}
}
}
Err(e) => warn!(error = %e, "Failed to load mesh config after flash"),
}
}
});
*job.abort_handle.write().await = Some(task.abort_handle());
Ok(())
}
async fn run_flash(
board: FlashBoard,
family: DeviceType,
data_dir: &Path,
path: &str,
job: &Arc<FlashJob>,
) -> Result<()> {
match family {
DeviceType::Meshtastic | DeviceType::Meshcore => {
let image = fetch_esptool_image(board, family, data_dir, job).await?;
esptool_erase_and_write(path, &image, job).await
}
DeviceType::Reticulum => {
let lora_region = super::load_config(data_dir)
.await
.ok()
.and_then(|c| c.lora_region);
rnodeconf_autoinstall(path, board, lora_region.as_deref(), job).await
}
DeviceType::Unknown => anyhow::bail!("Pick a firmware family before flashing"),
}
}
// ─── MeshCore / Meshtastic: esptool ─────────────────────────────────────
async fn fetch_esptool_image(
board: FlashBoard,
family: DeviceType,
data_dir: &Path,
job: &Arc<FlashJob>,
) -> Result<PathBuf> {
let cache = firmware_cache_dir(data_dir);
tokio::fs::create_dir_all(&cache)
.await
.context("Creating firmware cache dir")?;
let client = github_client()?;
match family {
DeviceType::Meshtastic => fetch_meshtastic_image(&client, board, &cache, job).await,
DeviceType::Meshcore => fetch_meshcore_image(&client, board, &cache, job).await,
_ => anyhow::bail!("{family} is not flashed via esptool"),
}
}
async fn fetch_meshtastic_image(
client: &reqwest::Client,
board: FlashBoard,
cache: &Path,
job: &Arc<FlashJob>,
) -> Result<PathBuf> {
let release: GithubRelease = client
.get("https://api.github.com/repos/meshtastic/firmware/releases/latest")
.timeout(METADATA_TIMEOUT)
.send()
.await
.context("Fetching Meshtastic release list")?
.error_for_status()
.context("Meshtastic releases API error")?
.json()
.await
.context("Parsing Meshtastic release JSON")?;
// Meshtastic bundles all esp32s3 boards' images inside one per-platform
// zip rather than shipping per-board top-level assets — both Heltec V3
// and V4 are esp32s3, so this is the right zip for both (confirmed live
// against v2.7.26.54e0d8d).
let zip_asset = release
.assets
.iter()
.find(|a| a.name.starts_with("firmware-esp32s3-") && a.name.ends_with(".zip"))
.ok_or_else(|| anyhow::anyhow!("No esp32s3 firmware zip in latest Meshtastic release"))?;
let version = zip_asset
.name
.strip_prefix("firmware-esp32s3-")
.and_then(|s| s.strip_suffix(".zip"))
.ok_or_else(|| anyhow::anyhow!("Unexpected Meshtastic asset name: {}", zip_asset.name))?
.to_string();
let zip_path = cache.join(&zip_asset.name);
if tokio::fs::metadata(&zip_path).await.is_err() {
download_to_file(client, &zip_asset.browser_download_url, &zip_path, job).await?;
} else {
job.push_log(format!("Using cached {}", zip_asset.name)).await;
}
// "*.factory.bin" is Meshtastic's full merged image (bootloader +
// partition table + app) meant to be written at offset 0x0 on a freshly
// erased chip — confirmed by inspecting the real zip's contents, as
// opposed to the plain "*.bin" OTA-update image which assumes an
// existing bootloader/partition table already on the chip.
let entry_name = format!(
"firmware-{}-{}.factory.bin",
board.meshtastic_id(),
version
);
let out_path = cache.join(&entry_name);
if tokio::fs::metadata(&out_path).await.is_ok() {
return Ok(out_path);
}
job.push_log(format!(
"Extracting {entry_name} from {}",
zip_asset.name
))
.await;
let zip_path_owned = zip_path.clone();
let entry_name_owned = entry_name.clone();
let out_path_owned = out_path.clone();
tokio::task::spawn_blocking(move || -> Result<()> {
let file = std::fs::File::open(&zip_path_owned).context("Opening downloaded firmware zip")?;
let mut archive = zip::ZipArchive::new(file).context("Reading firmware zip")?;
let mut entry = archive
.by_name(&entry_name_owned)
.with_context(|| format!("{entry_name_owned} not found in firmware zip"))?;
let mut out =
std::fs::File::create(&out_path_owned).context("Creating extracted firmware file")?;
std::io::copy(&mut entry, &mut out).context("Extracting firmware image")?;
Ok(())
})
.await
.context("Firmware extraction task panicked")??;
Ok(out_path)
}
async fn fetch_meshcore_image(
client: &reqwest::Client,
board: FlashBoard,
cache: &Path,
job: &Arc<FlashJob>,
) -> Result<PathBuf> {
let release: GithubRelease = client
.get("https://api.github.com/repos/meshcore-dev/MeshCore/releases/latest")
.timeout(METADATA_TIMEOUT)
.send()
.await
.context("Fetching MeshCore release list")?
.error_for_status()
.context("MeshCore releases API error")?
.json()
.await
.context("Parsing MeshCore release JSON")?;
// Upstream's casing differs between boards (Heltec_v3_... vs
// heltec_v4_...) — match case-insensitively on the exact per-board
// substring so V4 isn't accidentally matched by "heltec_v4_tft_..."
// variants (there's a "_tft_" in between, so a straight substring match
// on "heltec_v4_companion_radio_usb" is already safe).
let needle = match board {
FlashBoard::HeltecV3 => "heltec_v3_companion_radio_usb",
FlashBoard::HeltecV4 => "heltec_v4_companion_radio_usb",
};
let asset = release
.assets
.iter()
.find(|a| {
let lower = a.name.to_lowercase();
lower.contains(needle) && lower.ends_with("-merged.bin")
})
.ok_or_else(|| {
anyhow::anyhow!("No matching MeshCore image in release {}", release.tag_name)
})?;
let out_path = cache.join(&asset.name);
if tokio::fs::metadata(&out_path).await.is_ok() {
job.push_log(format!("Using cached {}", asset.name)).await;
return Ok(out_path);
}
download_to_file(client, &asset.browser_download_url, &out_path, job).await?;
Ok(out_path)
}
async fn download_to_file(
client: &reqwest::Client,
url: &str,
dest: &Path,
job: &Arc<FlashJob>,
) -> Result<()> {
job.set_stage(FlashStage::Downloading).await;
// Bound only the wait for the response to *start* (headers) — NOT a
// request-level `.timeout()`, which would cap the whole body transfer
// again (the bug this replaced: a blanket 30s client timeout killed
// large downloads mid-stream). If the server never responds at all,
// this is what stops the job from hanging forever; the per-chunk stall
// timeout below is what guards the body once streaming starts. Without
// this, a server that accepts the TCP connection but never sends
// headers back hangs this call indefinitely — confirmed live
// 2026-07-23: a stuck `.send()` here wedged the single-flash-job guard
// for good, permanently blocking every subsequent flash attempt with
// "already in progress" until the service was restarted.
let resp = tokio::time::timeout(METADATA_TIMEOUT, client.get(url).send())
.await
.context("Firmware download server did not respond")?
.context("Starting firmware download")?
.error_for_status()
.context("Firmware download returned an error status")?;
let total = resp.content_length();
let tmp = dest.with_extension("part");
let mut file = tokio::fs::File::create(&tmp)
.await
.context("Creating firmware download file")?;
let mut stream = resp.bytes_stream();
let mut downloaded: u64 = 0;
use futures_util::StreamExt;
loop {
let next = tokio::time::timeout(DOWNLOAD_STALL_TIMEOUT, stream.next())
.await
.context("Firmware download stalled")?;
let Some(chunk) = next else { break };
let chunk = chunk.context("Reading firmware download stream")?;
file.write_all(&chunk)
.await
.context("Writing firmware download")?;
downloaded += chunk.len() as u64;
if let Some(total) = total {
if total > 0 {
job.set_percent(((downloaded.saturating_mul(100)) / total) as u8)
.await;
}
}
}
file.flush().await.ok();
tokio::fs::rename(&tmp, dest)
.await
.context("Finalizing firmware download")?;
job.push_log(format!(
"Downloaded {} ({downloaded} bytes)",
dest.display()
))
.await;
Ok(())
}
/// Both Heltec V3 and V4 are ESP32-S3 boards.
const ESPTOOL_CHIP: &str = "esp32s3";
/// esptool's auto-reset-into-bootloader handshake (toggling DTR/RTS in a
/// specific timed pattern) is well-known to be flaky on some CP2102/CH340
/// board+adapter combinations — esptool's own docs recommend retrying at a
/// lower baud rate when this happens. Rather than fail the whole job on the
/// first hiccup, retry once at a conservative baud before giving up.
const ESPTOOL_FALLBACK_BAUD: &str = "115200";
/// `write_flash --erase-all` erases the whole chip before writing, in one
/// esptool invocation. This needs the esp32s3 stub flasher loaded (see
/// esptool_global_args' doc comment) — without it, --erase-all hits the
/// exact same ROM limitation a standalone `erase_flash` does ("ESP32-S3 ROM
/// does not support function erase_flash", confirmed live 2026-07-23), since
/// esptool's --erase-all is implemented as the same full-chip-erase command,
/// not a per-sector loop.
async fn esptool_erase_and_write(path: &str, image: &Path, job: &Arc<FlashJob>) -> Result<()> {
job.set_stage(FlashStage::Writing).await;
let image_str = image.to_string_lossy().to_string();
esptool_with_retry(
path,
&["write_flash", "--erase-all", "0x0", &image_str],
job,
)
.await
.context("esptool write_flash failed")?;
Ok(())
}
/// esptool's global flags (--chip/--port/--baud) MUST precede the subcommand
/// token (erase_flash/write_flash/...) — confirmed live 2026-07-23:
/// appending `--baud 115200` after the subcommand on the retry path
/// produced "esptool: error: unrecognized arguments: --baud 115200" every
/// time, so the fallback-baud retry never actually got a chance to run.
/// Building global args separately from subcommand args keeps this correct
/// by construction instead of relying on call-site ordering.
///
/// Normal stub-loader mode (no --no-stub) needs the esp32s3 stub flasher
/// blob at /usr/lib/python3/dist-packages/esptool/targets/stub_flasher/
/// stub_flasher_32s3.json — Debian's `esptool` package (4.7.0+dfsg-0.1)
/// ships without it (stripped for DFSG compliance: the prebuilt blob has no
/// buildable-from-source path Debian could verify), so scripts/self-update.sh
/// fetches the exact same file from the matching upstream esptool release
/// tag and installs it alongside the apt package (see the esptool install
/// step there). --no-stub (talk directly to the ROM bootloader, skip the
/// stub) was tried first and works for connecting, but the ROM bootloader
/// doesn't implement a full-chip-erase opcode at all — only the stub does —
/// so --no-stub broke our "always erase before write" default outright
/// rather than just being slower. Restoring the real stub file is the
/// correct fix, not routing around its absence.
fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str> {
let mut args = vec!["--chip", ESPTOOL_CHIP, "--port", path];
if let Some(b) = baud {
args.push("--baud");
args.push(b);
}
args
}
async fn esptool_with_retry(path: &str, subcommand: &[&str], job: &Arc<FlashJob>) -> Result<()> {
let mut cmd = Command::new("esptool");
cmd.args(esptool_global_args(path, None));
cmd.args(subcommand);
match run_streamed(cmd, None, job).await {
Ok(()) => Ok(()),
Err(first_err) => {
job.push_log(format!(
"First attempt failed ({first_err:#}); retrying once at {ESPTOOL_FALLBACK_BAUD} baud"
))
.await;
let mut retry = Command::new("esptool");
retry.args(esptool_global_args(path, Some(ESPTOOL_FALLBACK_BAUD)));
retry.args(subcommand);
run_streamed(retry, None, job)
.await
.context(format!("retry also failed (first attempt: {first_err:#})"))
}
}
}
// ─── Reticulum/RNode: archy-rnodeconf ───────────────────────────────────
fn rnodeconf_bin() -> String {
std::env::var("ARCHY_RNODECONF_BIN")
.unwrap_or_else(|_| "/usr/local/bin/archy-rnodeconf".to_string())
}
/// `--autoinstall`'s "which board is this" step is interactive by design —
/// confirmed live against a real Heltec V4 (2026-07-23): even with a board
/// given on the command line, rnodeconf can't always tell V3 from V4 apart
/// (their bootstrap-time USB identity is often generic, same root cause as
/// `resolve_flash_board`'s doc comment), so it always asks. The full prompt
/// sequence observed for a Heltec board that already has *some* RNode
/// firmware installed (the common case — a truly blank chip likely skips
/// straight to the same "Device Selection" menu):
/// 1. numbered device-type menu → answer with the menu number
/// 2. "Hit enter to continue" → answer with a blank line
/// 3. numbered band menu → answer with the menu number
/// 4. "Is the above correct? [y/N]" → answer "y"
/// Feeding all four answers up front (rather than watching stdout for each
/// prompt text) works because the menu is always asked in this fixed order
/// for every board that needs (re)provisioning — verified by driving it
/// through an unprovisioned real V4 end-to-end (erase → flash → EEPROM
/// bootstrap → "Device signature validated" on the next probe).
fn rnodeconf_device_menu_number(board: FlashBoard) -> &'static str {
match board {
FlashBoard::HeltecV3 => "8",
FlashBoard::HeltecV4 => "9",
}
}
/// rnodeconf's band choice is a coarse RF-frontend bootstrap parameter
/// (868/915/923 MHz), not the final operating frequency — that's still
/// configured later via the daemon's interface config, same as today. This
/// is a best-effort mapping from the node's persisted Meshtastic-style
/// region code (see `mesh::meshtastic::region_name_to_code`) down to
/// rnodeconf's 3-way menu; regions with no exact 868/923 match fall back to
/// 915 MHz as the broadest-compatibility default.
fn rnodeconf_band_menu_number(lora_region: Option<&str>) -> &'static str {
match lora_region.map(|s| s.trim().to_uppercase()) {
Some(r) if r.contains("868") => "1",
Some(r) if r.contains("923") => "3",
_ => "2",
}
}
/// `--autoinstall` fetches, erases, flashes, and bootstraps the EEPROM for
/// a detected board as one atomic step (confirmed via `archy-rnodeconf
/// --help` AND a real end-to-end flash on real hardware) — this is the
/// RNode-side equivalent of our "always erase before write" default, since
/// autoinstall doesn't try to preserve any existing on-device state.
async fn rnodeconf_autoinstall(
path: &str,
board: FlashBoard,
lora_region: Option<&str>,
job: &Arc<FlashJob>,
) -> Result<()> {
job.set_stage(FlashStage::Autoinstalling).await;
let bin = rnodeconf_bin();
let mut cmd = if Path::new(&bin).exists() {
Command::new(bin)
} else {
// Dev fallback if only a plain venv/system rnodeconf is on PATH.
Command::new("rnodeconf")
};
cmd.args(["--autoinstall", path]);
let stdin = format!(
"{}\n\n{}\ny\n",
rnodeconf_device_menu_number(board),
rnodeconf_band_menu_number(lora_region)
);
run_streamed(cmd, Some(stdin.into_bytes()), job)
.await
.context("archy-rnodeconf --autoinstall failed")
}
// ─── Subprocess streaming ────────────────────────────────────────────────
fn percent_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"\((\d{1,3})\s*%\)").expect("valid regex"))
}
async fn run_streamed(mut cmd: Command, stdin: Option<Vec<u8>>, job: &Arc<FlashJob>) -> Result<()> {
cmd.stdout(Stdio::piped());
cmd.stderr(Stdio::piped());
if stdin.is_some() {
cmd.stdin(Stdio::piped());
}
// Deliberately NOT kill_on_drop: an interrupted erase/write can leave
// the chip in a worse state than either finished or unstarted (see the
// cancellation-safety note in mesh flashing docs). The job is expected
// to run to completion or fail on its own.
let mut child = cmd.spawn().context("Failed to start subprocess")?;
if let Some(bytes) = stdin {
if let Some(mut child_stdin) = child.stdin.take() {
child_stdin
.write_all(&bytes)
.await
.context("Writing to subprocess stdin")?;
}
}
let mut tasks = Vec::new();
if let Some(stdout) = child.stdout.take() {
let job = Arc::clone(job);
tasks.push(tokio::spawn(async move {
let mut lines = BufReader::new(stdout).lines();
while let Ok(Some(line)) = lines.next_line().await {
if let Some(cap) = percent_regex().captures(&line) {
if let Ok(pct) = cap[1].parse::<u8>() {
job.set_percent(pct).await;
}
}
job.push_log(line).await;
}
}));
}
if let Some(stderr) = child.stderr.take() {
let job = Arc::clone(job);
tasks.push(tokio::spawn(async move {
let mut lines = BufReader::new(stderr).lines();
while let Ok(Some(line)) = lines.next_line().await {
job.push_log(line).await;
}
}));
}
let status = child.wait().await.context("Waiting for subprocess")?;
for t in tasks {
let _ = t.await;
}
if !status.success() {
// Exit status alone isn't diagnosable — the actual esptool/rnodeconf
// stderr (already captured into job.log_tail by the reader tasks
// above) is what actually explains a failure. Confirmed live
// 2026-07-23: a bare "Command exited with exit status: 1" told us
// nothing when esptool's real error was sitting in the log tail the
// whole time, only visible via the UI's live poll, not journald.
let tail: Vec<String> = job
.snapshot()
.await
.log_tail
.iter()
.rev()
.take(10)
.rev()
.cloned()
.collect();
anyhow::bail!("Command exited with {status}\n{}", tail.join("\n"));
}
Ok(())
}

View File

@ -87,6 +87,18 @@ const RECONNECT_DELAY_INIT: Duration = Duration::from_secs(5);
/// Maximum reconnect delay (cap for exponential backoff). /// Maximum reconnect delay (cap for exponential backoff).
const RECONNECT_DELAY_MAX: Duration = Duration::from_secs(60); const RECONNECT_DELAY_MAX: Duration = Duration::from_secs(60);
/// Minimum time a session must run before we trust it enough to reset
/// backoff to the minimum. Without this gate, a device that connects then
/// fails again within a couple of seconds (e.g. mid-boot-loop) never backs
/// off — every retry immediately re-opens the port, which toggles DTR/RTS
/// (resets many ESP32 boards' MCU on native-USB and CP2102/CH340
/// auto-reset-circuit boards alike), turning a device that's merely
/// unstable into a self-sustaining boot loop that outlasts whatever
/// triggered the original instability. Confirmed live 2026-07-23: a Heltec
/// V3 stuck retrying every ~5-15s for 5+ minutes after a failed firmware
/// flash left it in a marginal state.
const STABLE_SESSION_THRESHOLD: Duration = Duration::from_secs(20);
/// Number of consecutive write failures before we consider the device dead /// Number of consecutive write failures before we consider the device dead
/// and trigger a reconnection cycle. /// and trigger a reconnection cycle.
const MAX_CONSECUTIVE_WRITE_FAILURES: u32 = 3; const MAX_CONSECUTIVE_WRITE_FAILURES: u32 = 3;
@ -535,6 +547,10 @@ pub fn spawn_mesh_listener(
let mut shutdown = shutdown; let mut shutdown = shutdown;
let mut cmd_rx = cmd_rx; let mut cmd_rx = cmd_rx;
let mut reconnect_delay = RECONNECT_DELAY_INIT; let mut reconnect_delay = RECONNECT_DELAY_INIT;
// Mutable so a successful auto-detect can pin the firmware kind for
// the rest of this listener's lifetime — see the pin-on-first-success
// block below for why.
let mut device_kind = device_kind;
// Backlog #12 hot-swap re-binding: each run_mesh_session call already // Backlog #12 hot-swap re-binding: each run_mesh_session call already
// builds a fresh device struct (contacts/current_region/etc. all // builds a fresh device struct (contacts/current_region/etc. all
// start empty), so per-device session state is naturally isolated // start empty), so per-device session state is naturally isolated
@ -550,6 +566,7 @@ pub fn spawn_mesh_listener(
return; return;
} }
let session_start = std::time::Instant::now();
match session::run_mesh_session( match session::run_mesh_session(
&state, &state,
&data_dir, &data_dir,
@ -572,13 +589,14 @@ pub fn spawn_mesh_listener(
{ {
Ok(()) => { Ok(()) => {
info!("Mesh session ended cleanly"); info!("Mesh session ended cleanly");
// Session was established before ending — reset backoff // Only trust a session that actually ran for a while —
reconnect_delay = RECONNECT_DELAY_INIT; // see STABLE_SESSION_THRESHOLD's doc comment.
if session_start.elapsed() >= STABLE_SESSION_THRESHOLD {
reconnect_delay = RECONNECT_DELAY_INIT;
}
} }
Err(e) => { Err(e) => {
// Check if session was ever connected (vs failed to open) if session_start.elapsed() >= STABLE_SESSION_THRESHOLD {
let was_connected = state.status.read().await.device_connected;
if was_connected {
reconnect_delay = RECONNECT_DELAY_INIT; reconnect_delay = RECONNECT_DELAY_INIT;
} }
error!("Mesh session error: {} (retry in {:?})", e, reconnect_delay); error!("Mesh session error: {} (retry in {:?})", e, reconnect_delay);
@ -604,6 +622,45 @@ pub fn spawn_mesh_listener(
} }
} }
// Pin the firmware kind after the first successful auto-detect.
// Confirmed live 2026-07-23: with device_kind left unpinned (e.g.
// after clearing a stale pin), EVERY reconnect re-runs the full
// Reticulum→Meshcore→Meshtastic auto-detect cascade — each
// candidate past the first does its own open() with the DTR/RTS
// reset both boards need, so a device correctly identified as
// Meshtastic still gets reset once for the failed Meshcore
// attempt before Meshtastic's own open() resets it again. That
// doubled the reset count on every single reconnect indefinitely,
// not just during initial detection. Once auto-detect has
// identified the device this listener is actually talking to,
// there's no reason to keep guessing on subsequent reconnects —
// pin it, both in this task's own loop (takes effect
// immediately) and on disk (survives a service restart). A
// genuine hot-swap to different firmware is still handled: the
// setup modal's `mesh.probe-device` always re-probes unpinned,
// and the flash flow already clears this pin on its own.
if device_kind.is_none() {
let detected = state.status.read().await.device_type;
if detected != super::types::DeviceType::Unknown {
device_kind = Some(detected);
match super::load_config(&data_dir).await {
Ok(mut cfg) if cfg.device_kind.is_none() => {
cfg.device_kind = Some(detected);
if let Err(e) = super::save_config(&data_dir, &cfg).await {
warn!("Failed to persist auto-detected device_kind: {}", e);
} else {
info!(
kind = %detected,
"Pinned auto-detected firmware kind to avoid repeated multi-protocol resets on reconnect"
);
}
}
Ok(_) => {}
Err(e) => warn!("Failed to load mesh config to persist device_kind: {}", e),
}
}
}
// Update status to disconnected. device_type/firmware_version are // Update status to disconnected. device_type/firmware_version are
// reset too — they were previously left holding the LAST radio's // reset too — they were previously left holding the LAST radio's
// identity, so after a hot-swap the UI showed the old firmware // identity, so after a hot-swap the UI showed the old firmware

View File

@ -274,6 +274,7 @@ async fn auto_detect_and_open(
if paths.is_empty() { if paths.is_empty() {
anyhow::bail!("No serial devices found in /dev"); anyhow::bail!("No serial devices found in /dev");
} }
info!(candidates = ?paths, "Auto-detect candidate ports for this attempt");
for path in &paths { for path in &paths {
debug!(path = %path, "Probing for mesh radio device"); debug!(path = %path, "Probing for mesh radio device");
// Tried FIRST: `ReticulumLink::open()` gates its expensive daemon // Tried FIRST: `ReticulumLink::open()` gates its expensive daemon
@ -487,40 +488,19 @@ async fn open_preferred_path(
}; };
} }
// Reticulum first — see the matching comment on auto_detect_and_open: // Unpinned: don't probe this path ourselves at all. Confirmed live
// its cheap probe_rnode gate fails in ~1s for non-RNode firmware, while // 2026-07-23 — this function used to run its own Reticulum→Meshcore→
// trying Meshcore/Meshtastic first was observed leaving a real RNode // Meshtastic sequence here, and the caller (run_mesh_session) falls
// board unresponsive by the time Reticulum's turn came. // back to `auto_detect_and_open` on any error, which scans every
match ReticulumLink::open( // candidate path (this one included) with the exact same three-protocol
path, // sequence. With a single physical radio — the overwhelmingly common
data_dir, // case — `path` here IS the one candidate `auto_detect_and_open` is
Some(our_ed_pubkey_hex), // about to try, so every unpinned reconnect was resetting the board via
Some(our_x25519_pubkey_hex), // Reticulum/Meshcore/Meshtastic's DTR/RTS toggle TWICE: once here, once
) // again moments later in auto-detect. Bailing immediately (no port
.await // access at all) means auto-detect's single pass is the only one that
{ // ever touches the port when nothing is pinned yet.
Ok(mut dev) => match dev.initialize().await { anyhow::bail!("No device_kind pin — deferring to auto-detect for {path}")
Ok(info) => return Ok((MeshRadioDevice::Reticulum(dev), info)),
Err(e) => {
debug!(path = %path, error = %e, "Preferred path is not a working Reticulum RNode")
}
},
Err(e) => debug!(path = %path, error = %e, "Could not open preferred path as Reticulum"),
}
match MeshcoreDevice::open(path).await {
Ok(mut dev) => match dev.initialize().await {
Ok(info) => return Ok((MeshRadioDevice::Meshcore(dev), info)),
Err(e) => debug!(path = %path, error = %e, "Preferred path is not Meshcore"),
},
Err(e) => debug!(path = %path, error = %e, "Could not open preferred path as Meshcore"),
}
match MeshtasticDevice::open(path).await {
Ok(mut dev) => match dev.initialize().await {
Ok(info) => Ok((MeshRadioDevice::Meshtastic(dev), info)),
Err(e) => Err(e).context("Preferred path is not a working Meshtastic device"),
},
Err(e) => Err(e).context("Could not open preferred path as Meshtastic"),
}
} }
/// Bring up a Reticulum daemon over plain TCP — no physical RNode, no /// Bring up a Reticulum daemon over plain TCP — no physical RNode, no

View File

@ -214,11 +214,20 @@ impl MeshtasticDevice {
path path
))?; ))?;
// See probe_rnode() in reticulum.rs for why: ESP32-S3 native-USB // See probe_rnode() in reticulum.rs for why: ESP32-S3 native-USB
// boards reset on a DTR/RTS transition, so deassert both and settle // boards (and CP2102/CH340-bridged boards wired for Arduino-style
// before the handshake below. // auto-reset) reset on a DTR/RTS transition, so deassert both and
// settle before the handshake below. 300ms is nowhere near a real
// firmware boot time (LoRa radio init alone can take longer) —
// confirmed live 2026-07-23: with every one of Reticulum/Meshcore/
// Meshtastic's open() doing this same reset, a single auto-detect
// cycle trying multiple protocols in sequence kept re-resetting the
// board before it ever finished booting from the PREVIOUS attempt's
// reset, on both a Heltec V3 and V4, regardless of firmware family —
// a self-sustaining "never finishes booting" loop with a boot-time
// root cause hiding behind what looked like a per-protocol failure.
let _ = port.set_dtr(false); let _ = port.set_dtr(false);
let _ = port.set_rts(false); let _ = port.set_rts(false);
tokio::time::sleep(Duration::from_millis(300)).await; tokio::time::sleep(Duration::from_millis(2000)).await;
info!(path = %path, baud = BAUD_RATE, "Opened Meshtastic serial port"); info!(path = %path, baud = BAUD_RATE, "Opened Meshtastic serial port");
Ok(Self { Ok(Self {

View File

@ -8,6 +8,7 @@
pub mod alerts; pub mod alerts;
pub mod bitcoin_relay; pub mod bitcoin_relay;
pub mod crypto; pub mod crypto;
pub mod flash;
pub mod listener; pub mod listener;
pub mod meshtastic; pub mod meshtastic;
pub mod message_types; pub mod message_types;
@ -38,6 +39,14 @@ use tokio::sync::watch;
use tracing::{error, info, warn}; use tracing::{error, info, warn};
const MESH_CONFIG_FILE: &str = "mesh-config.json"; const MESH_CONFIG_FILE: &str = "mesh-config.json";
/// How long `MeshService::stop()` waits for the listener task to notice its
/// shutdown signal and exit gracefully before force-aborting it. See
/// `stop()`'s doc comment for the real incident this guards against: without
/// a hard abort fallback, a slow-to-notice listener could be left running
/// forever, orphaned, racing a later independently-started listener on the
/// same serial port.
const LISTENER_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(15);
const MESH_IGNORED_RADIO_FILE: &str = "mesh-ignored-radio-contacts.json"; const MESH_IGNORED_RADIO_FILE: &str = "mesh-ignored-radio-contacts.json";
const MESH_CONTACTS_FILE: &str = "mesh-contacts.json"; const MESH_CONTACTS_FILE: &str = "mesh-contacts.json";
@ -734,10 +743,18 @@ impl MeshService {
self.server_name = name; self.server_name = name;
} }
/// Start the background mesh listener. /// Start the background mesh listener. Idempotent: if the listener is
/// already running, this is a harmless no-op rather than an error —
/// confirmed live 2026-07-23, a real race between the flash job's own
/// post-flash restart and a concurrent user "Keep As Is" click (both
/// legitimately trying to ensure the listener is running) surfaced this
/// as a user-facing "Mesh listener already running" RPC error. Ensuring
/// the listener is running is the intent every caller actually has;
/// whichever caller's start() happens to win the race, the other
/// finding it already satisfied is success, not failure.
pub fn start(&mut self) -> Result<()> { pub fn start(&mut self) -> Result<()> {
if self.listener_handle.is_some() { if self.listener_handle.is_some() {
anyhow::bail!("Mesh listener already running"); return Ok(());
} }
let (shutdown_tx, shutdown_rx) = watch::channel(false); let (shutdown_tx, shutdown_rx) = watch::channel(false);
@ -967,8 +984,36 @@ impl MeshService {
if let Some(tx) = self.shutdown_tx.take() { if let Some(tx) = self.shutdown_tx.take() {
let _ = tx.send(true); let _ = tx.send(true);
} }
if let Some(handle) = self.listener_handle.take() { if let Some(mut handle) = self.listener_handle.take() {
let _ = handle.await; // Bounded wait for graceful shutdown, with a hard abort as
// fallback — confirmed live 2026-07-23: a caller-side timeout
// wrapping stop() (mesh::flash's STOP_LISTENER_TIMEOUT) cancelled
// this await when the listener was slow to notice its shutdown
// signal (mid multi-candidate probe), but `.take()` above had
// already cleared `listener_handle` to None — so MeshService
// believed it was stopped while the task kept running, orphaned
// (dropping a JoinHandle does not abort the task it points to).
// A later start() then spawned a second, fully independent
// listener session racing the orphaned one on the same serial
// port — neither could ever get a clean response, so every
// mesh.configure/probe against that device failed indefinitely
// even though the device itself was fine.
//
// Awaiting `&mut handle` (not `handle` by value) is what makes
// the fallback possible: the Future is polled through the
// reference, so if the timeout fires, this task's own `handle`
// binding is still ours to call `.abort()` on afterward —
// unlike moving `handle` into the timeout future outright, which
// would drop (and thus orphan) it on timeout with nothing left
// to abort.
if tokio::time::timeout(LISTENER_SHUTDOWN_TIMEOUT, &mut handle)
.await
.is_err()
{
warn!("Mesh listener did not shut down gracefully in time — aborting it");
handle.abort();
let _ = handle.await;
}
} }
if let Some(handle) = self.deadman_handle.take() { if let Some(handle) = self.deadman_handle.take() {
handle.abort(); handle.abort();
@ -1019,19 +1064,17 @@ impl MeshService {
self.state.peers.read().await.values().cloned().collect() self.state.peers.read().await.values().cloned().collect()
} }
/// Probe a serial port for a mesh radio without provisioning or keeping /// Refuse to probe the port the live session currently occupies (the
/// it — powers the hot-swap "device detected" modal's current-details /// probe would steal the serial port from under the session); a
/// view. Refuses to probe the port the live session currently occupies
/// (the probe would steal the serial port from under the session); a
/// detected-but-not-connected port is fair game, accepting a benign race /// detected-but-not-connected port is fair game, accepting a benign race
/// with the reconnect loop (whichever loses just retries). /// with the reconnect loop (whichever loses just retries). Split out from
pub async fn probe_device(&self, path: &str) -> Result<listener::DeviceProbe> { /// the actual probe on purpose — see `probe_device`'s doc comment.
pub async fn ensure_probe_allowed(&self, path: &str) -> Result<()> {
let status = self.state.status.read().await; let status = self.state.status.read().await;
if status.device_connected && status.device_path.as_deref() == Some(path) { if status.device_connected && status.device_path.as_deref() == Some(path) {
anyhow::bail!("{path} is the active mesh radio — already connected"); anyhow::bail!("{path} is the active mesh radio — already connected");
} }
drop(status); Ok(())
listener::probe_device(path).await
} }
/// Get message history. /// Get message history.

View File

@ -58,11 +58,20 @@ impl MeshcoreDevice {
path path
))?; ))?;
// See probe_rnode() in reticulum.rs for why: ESP32-S3 native-USB // See probe_rnode() in reticulum.rs for why: ESP32-S3 native-USB
// boards reset on a DTR/RTS transition, so deassert both and settle // boards (and CP2102/CH340-bridged boards wired for Arduino-style
// before the handshake below. // auto-reset) reset on a DTR/RTS transition, so deassert both and
// settle before the handshake below. 300ms is nowhere near a real
// firmware boot time (LoRa radio init alone can take longer) —
// confirmed live 2026-07-23: with every one of Reticulum/Meshcore/
// Meshtastic's open() doing this same reset, a single auto-detect
// cycle trying multiple protocols in sequence kept re-resetting the
// board before it ever finished booting from the PREVIOUS attempt's
// reset, on both a Heltec V3 and V4, regardless of firmware family —
// a self-sustaining "never finishes booting" loop with a boot-time
// root cause hiding behind what looked like a per-protocol failure.
let _ = port.set_dtr(false); let _ = port.set_dtr(false);
let _ = port.set_rts(false); let _ = port.set_rts(false);
tokio::time::sleep(Duration::from_millis(300)).await; tokio::time::sleep(Duration::from_millis(2000)).await;
info!(path = %path, baud = BAUD_RATE, "Opened serial port"); info!(path = %path, baud = BAUD_RATE, "Opened serial port");
@ -547,14 +556,38 @@ fn likely_non_mesh_serial_device(path: &str) -> bool {
/// Scan for serial devices that could be Meshcore radios. /// Scan for serial devices that could be Meshcore radios.
/// Returns paths to existing serial device files. /// Returns paths to existing serial device files.
///
/// Dedupes by canonical (symlink-resolved) target: `/dev/mesh-radio` is a
/// stable udev symlink to whatever `/dev/ttyUSB*`/`/dev/ttyACM*` node the
/// primary radio currently enumerates as, so both names always pointed at
/// the same candidate list entry and both passed this scan — confirmed live
/// 2026-07-23, this made an already-connected, working radio (connected via
/// its `/dev/mesh-radio` alias) simultaneously appear as a second, separate
/// "detected but unclaimed" device under its raw `/dev/ttyUSBn` name. The
/// hot-swap UI's active-session guard compares path strings, so it didn't
/// recognize the two aliases as the same port, showed the "device detected"
/// modal for a radio that was already set up, and probing it there opened
/// (and DTR/RTS-reset) the exact port the live session was mid-conversation
/// with — a continuous, UI-driven reset loop that only ran while that view
/// was open (matches the reported "stops when I leave, resumes when I come
/// back"). SERIAL_CANDIDATES lists `/dev/mesh-radio` first, so it wins the
/// dedup and is what's reported when both alias and target are present.
pub async fn detect_serial_devices() -> Vec<String> { pub async fn detect_serial_devices() -> Vec<String> {
let mut devices = Vec::new(); let mut devices = Vec::new();
let mut seen_real_paths = std::collections::HashSet::new();
for path in SERIAL_CANDIDATES { for path in SERIAL_CANDIDATES {
if tokio::fs::metadata(path).await.is_ok() { if tokio::fs::metadata(path).await.is_ok() {
if likely_non_mesh_serial_device(path) { if likely_non_mesh_serial_device(path) {
debug!(path = %path, "Skipping known non-mesh serial device"); debug!(path = %path, "Skipping known non-mesh serial device");
continue; continue;
} }
let real_path = tokio::fs::canonicalize(path)
.await
.unwrap_or_else(|_| std::path::PathBuf::from(path));
if !seen_real_paths.insert(real_path.clone()) {
debug!(path = %path, real_path = %real_path.display(), "Skipping duplicate alias for an already-listed device");
continue;
}
devices.push(path.to_string()); devices.push(path.to_string());
} }
} }

View File

@ -97,42 +97,57 @@ async fn get_wan_ip() -> Option<String> {
} }
/// Check if UPnP is available by attempting SSDP discovery. /// Check if UPnP is available by attempting SSDP discovery.
///
/// The socket I/O here is plain blocking `std::net` (its 3s read timeout is
/// enforced by the OS, not by yielding to the async runtime), so it must run
/// on the blocking-pool via `spawn_blocking` — inlined into this "async fn"
/// directly, it used to occupy a tokio worker thread for the full 3s on
/// every call. With only as many worker threads as CPU cores, a handful of
/// concurrent `network.diagnostics` calls (e.g. several Server-settings page
/// loads) could starve the whole runtime and freeze every other in-flight
/// request — root cause of a full-node outage (2026-07-24) that had nothing
/// to do with connection limits and everything to do with blocking sockets
/// on async worker threads.
async fn check_upnp_available() -> bool { async fn check_upnp_available() -> bool {
use std::net::UdpSocket; tokio::task::spawn_blocking(|| {
use std::net::UdpSocket;
let ssdp_request = "M-SEARCH * HTTP/1.1\r\n\ let ssdp_request = "M-SEARCH * HTTP/1.1\r\n\
HOST: 239.255.255.250:1900\r\n\ HOST: 239.255.255.250:1900\r\n\
MAN: \"ssdp:discover\"\r\n\ MAN: \"ssdp:discover\"\r\n\
MX: 2\r\n\ MX: 2\r\n\
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1\r\n\r\n"; ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1\r\n\r\n";
let socket = match UdpSocket::bind("0.0.0.0:0") { let socket = match UdpSocket::bind("0.0.0.0:0") {
Ok(s) => s, Ok(s) => s,
Err(_) => return false, Err(_) => return false,
}; };
if socket if socket
.set_read_timeout(Some(std::time::Duration::from_secs(3))) .set_read_timeout(Some(std::time::Duration::from_secs(3)))
.is_err() .is_err()
{ {
return false; return false;
}
if socket
.send_to(ssdp_request.as_bytes(), "239.255.255.250:1900")
.is_err()
{
return false;
}
let mut buf = [0u8; 2048];
match socket.recv_from(&mut buf) {
Ok((len, _)) => {
let response = String::from_utf8_lossy(&buf[..len]);
response.contains("InternetGatewayDevice") || response.contains("200 OK")
} }
Err(_) => false,
} if socket
.send_to(ssdp_request.as_bytes(), "239.255.255.250:1900")
.is_err()
{
return false;
}
let mut buf = [0u8; 2048];
match socket.recv_from(&mut buf) {
Ok((len, _)) => {
let response = String::from_utf8_lossy(&buf[..len]);
response.contains("InternetGatewayDevice") || response.contains("200 OK")
}
Err(_) => false,
}
})
.await
.unwrap_or(false)
} }
/// Add a port forward (stored locally; actual UPnP mapping done on request). /// Add a port forward (stored locally; actual UPnP mapping done on request).
@ -281,19 +296,41 @@ pub async fn run_diagnostics() -> Result<NetworkDiagnostics> {
} }
/// Check if Tor SOCKS proxy is reachable. /// Check if Tor SOCKS proxy is reachable.
///
/// `TcpStream::connect_timeout` blocks the calling OS thread for up to its
/// timeout — same blocking-on-a-worker-thread hazard as `check_upnp_available`
/// above, so this also runs on the blocking pool.
async fn check_tor_connectivity() -> bool { async fn check_tor_connectivity() -> bool {
use std::net::TcpStream; tokio::task::spawn_blocking(|| {
TcpStream::connect_timeout( use std::net::TcpStream;
&"127.0.0.1:9050".parse().unwrap(), TcpStream::connect_timeout(
std::time::Duration::from_secs(2), &"127.0.0.1:9050".parse().unwrap(),
) std::time::Duration::from_secs(2),
.is_ok() )
.is_ok()
})
.await
.unwrap_or(false)
} }
/// Check DNS resolution works. /// Check DNS resolution works.
///
/// `to_socket_addrs()` is a blocking libc resolver call with no timeout of
/// its own — on a network with a slow/unreachable DNS server (e.g. right
/// after relocating to a new network) it can hang far longer than the other
/// checks here. Runs on the blocking pool (same reason as the checks above)
/// AND under an explicit timeout, since unlike UPnP/Tor there's no built-in
/// bound to rely on.
async fn check_dns() -> bool { async fn check_dns() -> bool {
use std::net::ToSocketAddrs; let lookup = tokio::task::spawn_blocking(|| {
"cloudflare.com:443".to_socket_addrs().is_ok() use std::net::ToSocketAddrs;
"cloudflare.com:443".to_socket_addrs().is_ok()
});
tokio::time::timeout(std::time::Duration::from_secs(5), lookup)
.await
.ok()
.and_then(|r| r.ok())
.unwrap_or(false)
} }
// --- Router Compatibility Abstraction --- // --- Router Compatibility Abstraction ---
@ -469,3 +506,40 @@ pub async fn get_router_info(data_dir: &Path) -> Result<serde_json::Value> {
}, },
})) }))
} }
#[cfg(test)]
mod blocking_io_tests {
use super::*;
// Regression test for the 2026-07-24 outage: check_upnp_available,
// check_tor_connectivity, and check_dns each did blocking std::net I/O
// directly on their calling task instead of via spawn_blocking. On a
// small worker pool (4 threads in production), a handful of concurrent
// network.diagnostics calls tied up every worker thread for seconds,
// freezing every other in-flight RPC request. Proves a cheap task
// spawned alongside these checks still gets scheduled promptly, which
// only holds if the checks aren't monopolizing worker threads.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn network_checks_do_not_starve_other_tasks() {
let cheap = tokio::spawn(async {
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
std::time::Instant::now()
});
let _ = tokio::join!(
check_upnp_available(),
check_tor_connectivity(),
check_dns()
);
// The assertion is that `cheap` — spawned before the blocking checks
// and sleeping only 5ms — finishes within 1s of being spawned. If the
// checks were still blocking worker threads directly, a 2-worker
// runtime running 3 blocking checks concurrently would starve this
// task well past 1s.
tokio::time::timeout(std::time::Duration::from_secs(1), cheap)
.await
.expect("a concurrently-spawned cheap task must not be starved by blocking network checks")
.expect("cheap task panicked");
}
}

View File

@ -976,13 +976,21 @@ impl Server {
main_addr: SocketAddr, main_addr: SocketAddr,
shutdown: impl std::future::Future<Output = ()>, shutdown: impl std::future::Future<Output = ()>,
) -> Result<()> { ) -> Result<()> {
let active_connections = Arc::new(tokio::sync::Semaphore::new(1024)); // Separate pools per listener. Federation/peer connections (fips0,
// often over Tor, from nodes we don't control the behavior of) used
// to share one pool with the local web UI listener — when peer
// connections piled up in CLOSE-WAIT without ever completing, they
// starved the shared pool and took the web UI down with them
// (production outage, 2026-07-24). Peer congestion must never be
// able to block a local login.
let main_connections = Arc::new(tokio::sync::Semaphore::new(1024));
let peer_connections = Arc::new(tokio::sync::Semaphore::new(256));
let (tx, rx_main) = tokio::sync::watch::channel(false); let (tx, rx_main) = tokio::sync::watch::channel(false);
let main_task = tokio::spawn(accept_loop( let main_task = tokio::spawn(accept_loop(
self.api_handler.clone(), self.api_handler.clone(),
TcpListener::bind(main_addr).await?, TcpListener::bind(main_addr).await?,
active_connections.clone(), main_connections.clone(),
false, // main listener: no path filter false, // main listener: no path filter
rx_main, rx_main,
main_addr, main_addr,
@ -992,7 +1000,7 @@ impl Server {
// restart when fips0 comes up after onboarding. // restart when fips0 comes up after onboarding.
let peer_task = tokio::spawn(peer_late_bind_loop( let peer_task = tokio::spawn(peer_late_bind_loop(
self.api_handler.clone(), self.api_handler.clone(),
active_connections.clone(), peer_connections.clone(),
tx.subscribe(), tx.subscribe(),
)); ));
@ -1003,7 +1011,9 @@ impl Server {
// Wait up to 5s for in-flight requests. // Wait up to 5s for in-flight requests.
let drain_start = std::time::Instant::now(); let drain_start = std::time::Instant::now();
let drain_timeout = std::time::Duration::from_secs(5); let drain_timeout = std::time::Duration::from_secs(5);
while active_connections.available_permits() < 1024 { while main_connections.available_permits() < 1024
|| peer_connections.available_permits() < 256
{
if drain_start.elapsed() > drain_timeout { if drain_start.elapsed() > drain_timeout {
warn!("Drain timeout reached, forcing shutdown"); warn!("Drain timeout reached, forcing shutdown");
break; break;
@ -1096,6 +1106,11 @@ pub fn is_peer_allowed_path(path: &str) -> bool {
|| path.starts_with("/content/") || path.starts_with("/content/")
} }
/// How long a freshly-accepted connection will wait for a connection-pool
/// permit before it's dropped. Bounds worst-case fd/task growth if the pool
/// is ever genuinely saturated; under normal load this never triggers.
const PERMIT_ACQUIRE_TIMEOUT: Duration = Duration::from_secs(30);
async fn accept_loop( async fn accept_loop(
handler: Arc<ApiHandler>, handler: Arc<ApiHandler>,
listener: TcpListener, listener: TcpListener,
@ -1115,8 +1130,35 @@ async fn accept_loop(
} }
}; };
let handler = handler.clone(); let handler = handler.clone();
let permit = active_connections.clone().acquire_owned().await; let active_connections = active_connections.clone();
// Acquire the permit *inside* the spawned task, not here.
// This loop must never block on anything but accept()/shutdown:
// the main (5678) and FIPS peer (5679) listeners share one
// semaphore, and a single slow/hung connection holding the
// last permit used to freeze this whole loop — including for
// the OTHER listener — since accept() couldn't be called
// again until a permit freed up. That took down the entire
// web UI in production (2026-07-24) when federation peer
// connections piled up. Now a saturated pool just delays
// (and, past PERMIT_ACQUIRE_TIMEOUT, drops) individual
// connections instead of wedging the acceptor itself.
tokio::spawn(async move { tokio::spawn(async move {
let permit = match tokio::time::timeout(
PERMIT_ACQUIRE_TIMEOUT,
active_connections.acquire_owned(),
)
.await
{
Ok(Ok(permit)) => permit,
Ok(Err(_)) => return, // semaphore closed during shutdown
Err(_) => {
warn!(
"{} connection from {} dropped — connection pool saturated for {}s",
local_addr, peer_addr, PERMIT_ACQUIRE_TIMEOUT.as_secs()
);
return;
}
};
let _permit = permit; let _permit = permit;
let service = service_fn(move |mut req: hyper::Request<hyper::Body>| { let service = service_fn(move |mut req: hyper::Request<hyper::Body>| {
let handler = handler.clone(); let handler = handler.clone();

View File

@ -110,6 +110,61 @@ lands].
2. ❑ Mobile Home: wallet card directly under My Apps (G5 from the voice epic). 2. ❑ Mobile Home: wallet card directly under My Apps (G5 from the voice epic).
3. ❑ Mesh RF settings panel (Mesh → Device) still loads and saves. 3. ❑ Mesh RF settings panel (Mesh → Device) still loads and saves.
## H. LoRa radio firmware flashing (Heltec V3/V4, new — extends Section E)
Full v1 scope is 3 firmware families × 2 boards (6 cells); mark each cell
tested on real hardware vs. code-reviewed only as this is run.
1. ❑ From the hot-swap modal's step 1 (device already probed), press
**Flash Firmware…** → new step shows firmware-family + board pickers and
the erase-confirmation checkbox; "Erase & Flash Now" stays disabled until
family, board, AND the checkbox are all set.
2. ❑ Confirm what's currently on the test stick via the existing probe
BEFORE flashing it — don't flash the only known-good device without a
fallback board on hand.
3. ❑ Prefer a spare Heltec V3/V4 for the first destructive erase+flash run;
only exercise a primary/in-use stick once the flow is proven safe.
4. ❑ MeshCore → Heltec V3: erase + write completes, progress bar and log
tail update live, ends at "Flash complete".
5. ❑ Meshtastic → Heltec V3: same, using the extracted `*.factory.bin` from
the esp32s3 release zip.
6. ❑ Reticulum/RNode → Heltec V3: `archy-rnodeconf --autoinstall` path
completes (no raw esptool erase/write step for this family — see
`mesh/flash.rs` doc comment).
7. ❑ Repeat 4-6 against a Heltec V4. Confirmed 2026-07-23 on real hardware:
V4 uses the ESP32-S3's native-USB JTAG/serial peripheral (vid:pid
303a:1001, generic to every native-USB ESP32-S3 board, not V4-specific)
— so unlike V3's CP2102 bridge chip, V4 is permanently NOT auto-matchable
by vid:pid. Board auto-detect should fail closed for it every time
(manual board selection required, "couldn't confirm automatically"
warning shown) — this is expected steady-state behavior, not a gap to
close later.
8. ❑ After a successful flash, the modal automatically re-probes and shows
the NEW firmware's badge/details — same as unplugging and replugging
(Section E item 3), but without physically touching the cable.
9. ❑ Deliberately test a failure path once (disconnect the board mid-write,
or point at a bad cached asset) — confirm the error surfaces in the
progress log AND that `docs/troubleshooting.md`'s "LoRa radio firmware
flash failed" recovery steps (BOOT+RST bootloader entry, manual esptool/
rnodeconf command) actually get the board back to a flashable state.
10. ❑ Cancel button only appears (and only works) while still in the
"Downloading firmware…" stage — once erasing/writing starts, no cancel
affordance is offered.
11. ❑ **Boot-loop regression (2026-07-23 incident)**: after a *failed* flash
(e.g. kill network access mid-download to force a failure), confirm the
mesh listener does NOT auto-resume — `journalctl -u archipelago` should
show a single `Leaving mesh listener stopped after failed flash` line
and then go quiet for that device, not a repeating `mesh::serial:
Opened serial port... Starting Meshcore handshake` cycle every few
seconds. Reconnect manually via the hot-swap modal afterward and confirm
it connects normally (the board itself should be untouched — the
download fails before esptool/rnodeconf ever runs).
12. ❑ Separately, force a device to flap connected/disconnected a few times
in under 20s each (e.g. a marginal USB connection) and confirm
`reconnect_delay` in the logs actually escalates (5s → 10s → 20s → ...)
rather than resetting to 5s on every attempt — see
`STABLE_SESSION_THRESHOLD` in `mesh/listener/mod.rs`.
--- ---
After this passes: fold the batch + other agent's work into the next release After this passes: fold the batch + other agent's work into the next release

View File

@ -443,6 +443,86 @@ free -h
- Check Nginx WebSocket proxy config: `/etc/nginx/sites-available/archipelago` must include `proxy_set_header Upgrade $http_upgrade` - Check Nginx WebSocket proxy config: `/etc/nginx/sites-available/archipelago` must include `proxy_set_header Upgrade $http_upgrade`
- If on WiFi, try wired Ethernet for more stable connectivity - If on WiFi, try wired Ethernet for more stable connectivity
### 21. LoRa radio firmware flash failed / board unresponsive
**Symptoms**: The "Erase & Flash Now" flow in the mesh hot-swap modal reports
an error, or the radio no longer enumerates as a serial device after a flash
attempt.
**Diagnosis**:
```bash
# Poll the flash job's last-known stage/error directly
curl -s http://localhost:5678/rpc/v1 \
-H 'Content-Type: application/json' \
-d '{"method":"mesh.flash-status","params":{}}'
# Confirm the board is still enumerating at all
ls -la /dev/ttyUSB* /dev/ttyACM* /dev/mesh-radio 2>&1
# esptool/rnodeconf binaries present?
which esptool; ls -la /usr/local/bin/archy-rnodeconf
```
**Solutions**:
- A failure during `erasing`/`writing` (MeshCore/Meshtastic) or
`autoinstalling` (Reticulum) can leave the chip erased or half-written —
this is expected risk of the "always erase first" default, not a bug.
- Heltec V3/V4 boards can be forced back into bootloader mode manually: hold
**BOOT**, tap **RST**, then release **BOOT** — this puts the chip in a
state esptool can always talk to, regardless of what firmware (if any) is
currently on it.
- With the board in bootloader mode, a manual recovery flash can be run
directly over SSH without the UI:
```bash
esptool --chip esp32s3 --port /dev/ttyACM0 erase_flash
esptool --chip esp32s3 --port /dev/ttyACM0 write_flash 0x0 <known-good-image.bin>
```
- For Reticulum/RNode boards, the equivalent manual recovery is
`archy-rnodeconf /dev/ttyACM0 --autoinstall` (or `/usr/local/bin/archy-rnodeconf`
if it's not on `PATH`) — it re-runs the same fetch+erase+flash+bootstrap
sequence the UI triggers.
- If `esptool`/`archy-rnodeconf` are missing entirely, they should have been
installed by the last `self-update.sh` run — check
`sudo journalctl -u archipelago-update` for install failures, or install
`esptool` via `sudo apt-get install esptool` directly.
- Once a fresh image is confirmed written, unplug/replug the radio (or wait
for the next detection poll) — the hot-swap modal re-probes automatically
and shows whatever firmware is actually on the board now.
**Known incident (2026-07-23) — reconnect storm / device boot-loop after a
failed flash**: a real Heltec V3 got stuck cycling "connect → partial
handshake → drop" every 5-15s for 5+ minutes after a `mesh.flash-device`
attempt failed with `Reading firmware download stream`. Root cause was two
compounding issues, both now fixed:
1. `spawn_mesh_listener`'s reconnect backoff (`core/archipelago/src/mesh/listener/mod.rs`)
reset to its 5s minimum any time the prior session had been `device_connected`
at all, even for under a second — so a device that connects-then-drops
repeatedly never actually backed off. Every retry's `open()` toggles
DTR/RTS, which resets many ESP32 boards' MCU (native-USB *and*
CP2102/CH340 auto-reset-circuit boards), so the aggressive retries were
themselves *causing* the boot loop, not just observing one. Fixed by only
resetting backoff when a session ran for at least `STABLE_SESSION_THRESHOLD`
(20s) — see that constant's doc comment.
2. `mesh::flash::start_flash_job`'s post-completion handler auto-resumed the
listener unconditionally, even after a *failed* flash, immediately
re-entering the reconnect loop above with no cooldown. Fixed: on failure
the listener is now deliberately left stopped (reconnect manually via the
UI once the board is confirmed alive); on success there's a 5s settle
delay before resuming, so the board finishes booting from the flash
tool's own reset before Archipelago starts probing it again.
3. Separately, the download itself was failing because `mesh::flash`'s HTTP
client had a blanket 30s request timeout that covered the *entire*
download (including streaming a 170MB Meshtastic zip), not just
connection setup — fixed with a per-chunk stall timeout instead of a
fixed total-transfer cap.
If this symptom recurs (rapid repeating `mesh::serial: Opened serial
port... Starting Meshcore handshake` lines in `journalctl -u archipelago`
without a `LoRa firmware flash` job in progress), it's a NEW instance of the
same class of bug, not the one above — check whether backoff is actually
escalating (`Mesh session error: ... (retry in Xs)` — X should grow past 5s
within a few cycles) before assuming it's flashing-related.
--- ---
## General Maintenance ## General Maintenance

View File

@ -365,6 +365,11 @@ RUN apt-get update && apt-get -y full-upgrade && apt-get install -y --no-install
ca-certificates \ ca-certificates \
openssl \ openssl \
chrony \ chrony \
iputils-ping \
esptool \
python3-venv \
binutils \
libpython3.13 \
locales \ locales \
console-setup \ console-setup \
keyboard-configuration \ keyboard-configuration \

View File

@ -1,7 +1,7 @@
<template> <template>
<BaseModal <BaseModal
:show="show" :show="show"
:title="step === 1 ? 'Mesh Radio Detected' : 'Apply Archipelago Settings'" :title="step === 1 ? 'Mesh Radio Detected' : step === 2 ? 'Apply Archipelago Settings' : 'Flash Firmware'"
max-width="max-w-lg" max-width="max-w-lg"
content-class="max-h-[90vh] overflow-y-auto" content-class="max-h-[90vh] overflow-y-auto"
@close="dismiss" @close="dismiss"
@ -101,9 +101,111 @@
"Keep As Is" uses the radio exactly as it is nothing on it is changed, "Keep As Is" uses the radio exactly as it is nothing on it is changed,
and you can hot-swap radios any time. and you can hot-swap radios any time.
</p> </p>
<button
class="w-full text-center text-white/40 hover:text-white/70 text-[11px] mt-3 underline underline-offset-2"
:disabled="!!connecting"
@click="openFlashStep"
>
Flash Firmware
</button>
<p v-if="error" class="text-xs text-red-400 mt-2">{{ error }}</p> <p v-if="error" class="text-xs text-red-400 mt-2">{{ error }}</p>
</div> </div>
<!-- Step 3: erase + reflash destructive, opt-in only -->
<div v-else-if="step === 'flash'">
<!-- Once a job exists (started via startFlash), ALWAYS show the
progress/result view below including on failure. The old
condition (`!active && stage !== 'done'`) was also true for a
FAILED job (active:false, stage:'failed'), which silently sent
the user back to this picker instead of showing the error. -->
<template v-if="!flashJob">
<p class="text-white/60 text-xs mb-3">
Downloads the latest firmware from upstream and writes it to
<span class="font-mono text-orange-300">{{ devicePath }}</span>.
</p>
<div class="space-y-4">
<div>
<label class="block text-sm text-white/80 mb-1">Firmware family</label>
<select v-model="flashFamily" class="w-full rounded-lg bg-white/[0.06] border border-white/10 text-white px-3 py-2 text-sm focus:outline-none focus:border-orange-400/60">
<option value="">Choose</option>
<option value="meshcore">MeshCore</option>
<option value="meshtastic">Meshtastic</option>
<option value="reticulum">Reticulum RNode</option>
</select>
</div>
<div>
<label class="block text-sm text-white/80 mb-1">Board</label>
<select v-model="flashBoard" class="w-full rounded-lg bg-white/[0.06] border border-white/10 text-white px-3 py-2 text-sm focus:outline-none focus:border-orange-400/60">
<option value="">Choose</option>
<option value="heltec-v3">Heltec LoRa 32 V3</option>
<option value="heltec-v4">Heltec LoRa 32 V4</option>
</select>
<p v-if="!boardAutoDetected" class="text-[11px] text-amber-400/80 mt-1">
Couldn't confirm the board automatically double check before flashing.
Flashing the wrong board's image can brick it.
</p>
</div>
</div>
<div class="rounded-xl bg-red-500/10 border border-red-500/30 p-3 mt-4">
<label class="flex items-start gap-2 text-xs text-red-300">
<input type="checkbox" v-model="flashConfirmed" class="mt-0.5" />
<span>
This <strong>erases the entire chip</strong>, including any existing
keys, identity, and contacts. This cannot be undone.
</span>
</label>
</div>
<p v-if="error" class="text-xs text-red-400 mt-3">{{ error }}</p>
<div class="flex gap-2 mt-6">
<button class="glass-button px-4 py-2 rounded-lg text-sm" @click="step = 1">Back</button>
<button
class="flex-1 glass-button px-4 py-2 rounded-lg text-sm font-medium bg-red-500/80 hover:bg-red-500 text-white disabled:opacity-50"
:disabled="!flashFamily || !flashBoard || !flashConfirmed || starting"
@click="startFlash"
>
{{ starting ? 'Starting…' : 'Erase & Flash Now' }}
</button>
</div>
</template>
<!-- Progress -->
<template v-else>
<div class="text-center py-2">
<p class="text-white text-sm font-medium">{{ flashStageLabel }}</p>
<div class="mt-3 h-2 rounded-full bg-white/10 overflow-hidden">
<div
class="h-full bg-orange-400 transition-all"
:style="{ width: (flashJob?.percent ?? (flashJob?.stage === 'done' ? 100 : 8)) + '%' }"
></div>
</div>
<p v-if="flashJob?.error" class="text-xs text-red-400 mt-3">{{ flashJob.error }}</p>
</div>
<div class="mt-3 rounded-xl bg-black/30 border border-white/10 p-2 h-32 overflow-y-auto font-mono text-[10px] text-white/50 leading-relaxed">
<div v-for="(line, i) in (flashJob?.log_tail ?? []).slice(-40)" :key="i">{{ line }}</div>
</div>
<div class="flex gap-2 mt-4">
<button
v-if="flashJob?.stage === 'downloading' && flashJob?.active"
class="glass-button px-4 py-2 rounded-lg text-sm"
@click="cancelFlash"
>
Cancel
</button>
<button
v-if="!flashJob?.active"
class="flex-1 glass-button glass-button-warning px-4 py-2 rounded-lg text-sm font-medium"
@click="closeFlashStep"
>
Done
</button>
</div>
</template>
</div>
<!-- Step 2: our latest parameters, shown before anything is written --> <!-- Step 2: our latest parameters, shown before anything is written -->
<div v-else> <div v-else>
<p class="text-white/60 text-xs mb-3"> <p class="text-white/60 text-xs mb-3">
@ -190,7 +292,7 @@
import { ref, computed, watch } from 'vue' import { ref, computed, watch } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import BaseModal from '@/components/BaseModal.vue' import BaseModal from '@/components/BaseModal.vue'
import { useMeshStore, type MeshDeviceProbe, type MeshConfigureParams } from '@/stores/mesh' import { useMeshStore, type MeshDeviceProbe, type MeshConfigureParams, type FlashFirmwareFamily, type FlashBoard, type FlashJobStatus } from '@/stores/mesh'
import { useAppStore } from '@/stores/app' import { useAppStore } from '@/stores/app'
import { LORA_REGIONS, regionByCode, suggestRegionFromLatLon, meshcorePlanFor } from '@/utils/loraRegions' import { LORA_REGIONS, regionByCode, suggestRegionFromLatLon, meshcorePlanFor } from '@/utils/loraRegions'
import { resolveMeshDeviceImage } from '@/utils/meshDeviceImages' import { resolveMeshDeviceImage } from '@/utils/meshDeviceImages'
@ -199,7 +301,7 @@ const mesh = useMeshStore()
const appStore = useAppStore() const appStore = useAppStore()
const router = useRouter() const router = useRouter()
const step = ref<1 | 2>(1) const step = ref<1 | 2 | 'flash'>(1)
const connecting = ref<false | 'keep' | 'setup'>(false) const connecting = ref<false | 'keep' | 'setup'>(false)
const error = ref('') const error = ref('')
const probing = ref(false) const probing = ref(false)
@ -264,7 +366,10 @@ const rfPreset = computed(() => {
// (Re)probe + (re)apply presets each time a new device surfaces the modal // (Re)probe + (re)apply presets each time a new device surfaces the modal
watch([show, devicePath], async ([visible]) => { watch([show, devicePath], async ([visible]) => {
if (!visible) return if (!visible) {
stopFlashPoll()
return
}
step.value = 1 step.value = 1
error.value = '' error.value = ''
imageFailed.value = false imageFailed.value = false
@ -349,6 +454,118 @@ async function applySetup() {
connecting.value = false connecting.value = false
} }
} }
// Step 3: erase + reflash
const flashFamily = ref<FlashFirmwareFamily | ''>('')
const flashBoard = ref<FlashBoard | ''>('')
const flashConfirmed = ref(false)
const starting = ref(false)
const flashJob = ref<FlashJobStatus | null>(null)
let flashPollTimer: ReturnType<typeof setInterval> | null = null
const detectedInfo = computed(() =>
mesh.status?.detected_device_info?.find(d => d.path === devicePath.value)
)
// Mirrors mesh::flash::resolve_flash_board (core/archipelago/src/mesh/flash.rs)
// exactly matching on the display label was wrong: a Heltec V3's CP2102
// bridge chip reports "CP2102 USB to UART Bridge Controller" in its USB
// strings, not "Heltec", so meshDeviceImages.ts falls back to a generic
// "LoRa radio (CP2102 serial)" label that never matched /v3/i, showing the
// "couldn't confirm automatically" warning even though the backend CAN
// safely auto-detect V3 via vid:pid. Heltec V4 deliberately has no entry
// here, same reasoning as the backend: its vid:pid (303a:1001) is the
// ESP32-S3's generic native-USB descriptor, not V4-specific, so it can't be
// safely auto-matched and always requires manual selection.
const resolvedFlashBoard = computed<FlashBoard | ''>(() => {
const info = detectedInfo.value
if (info?.vid?.toLowerCase() === '10c4' && info?.pid?.toLowerCase() === 'ea60') return 'heltec-v3'
return ''
})
const boardAutoDetected = computed(() => !!resolvedFlashBoard.value)
const flashStageLabel = computed(() => {
switch (flashJob.value?.stage) {
case 'downloading': return 'Downloading firmware…'
case 'erasing': return 'Erasing chip…'
case 'writing': return 'Writing firmware…'
case 'autoinstalling': return 'Installing (rnodeconf)…'
case 'done': return 'Flash complete'
case 'failed': return 'Flash failed'
default: return ''
}
})
function openFlashStep() {
flashFamily.value = (probe.value?.kind as FlashFirmwareFamily) ?? ''
flashBoard.value = resolvedFlashBoard.value
flashConfirmed.value = false
flashJob.value = null
error.value = ''
step.value = 'flash'
}
function stopFlashPoll() {
if (flashPollTimer) {
clearInterval(flashPollTimer)
flashPollTimer = null
}
}
async function pollFlashStatus() {
try {
const status = await mesh.flashStatus()
flashJob.value = status
if (!status.active) {
stopFlashPoll()
if (status.done && !status.error) {
// Mirrors the unplug/replug hot-swap flow: re-probe so the details
// card reflects whatever firmware is actually on the board now.
const path = devicePath.value
probing.value = true
try {
probe.value = await mesh.probeDevice(path)
} catch {
probe.value = null
} finally {
probing.value = false
}
}
}
} catch {
stopFlashPoll()
}
}
async function startFlash() {
if (!flashFamily.value || !flashBoard.value || !flashConfirmed.value) return
starting.value = true
error.value = ''
try {
await mesh.flashDevice(devicePath.value, flashFamily.value, flashBoard.value)
flashJob.value = { active: true, stage: 'downloading', log_tail: [] }
stopFlashPoll()
flashPollTimer = setInterval(pollFlashStatus, 1500)
} catch (e) {
error.value = e instanceof Error ? e.message : 'Failed to start flashing'
} finally {
starting.value = false
}
}
async function cancelFlash() {
try {
await mesh.flashCancel()
} catch (e) {
error.value = e instanceof Error ? e.message : 'Failed to cancel'
}
}
function closeFlashStep() {
stopFlashPoll()
step.value = 1
}
</script> </script>
<style scoped> <style scoped>

View File

@ -57,6 +57,23 @@ export interface MeshDeviceProbe {
max_contacts: number | null max_contacts: number | null
} }
export type FlashFirmwareFamily = 'meshcore' | 'meshtastic' | 'reticulum'
export type FlashBoard = 'heltec-v3' | 'heltec-v4'
export type FlashStage = 'downloading' | 'erasing' | 'writing' | 'autoinstalling' | 'done' | 'failed'
/** Live progress for the one flash job that can run at a time. */
export interface FlashJobStatus {
active: boolean
board?: FlashBoard
family?: FlashFirmwareFamily
path?: string
stage?: FlashStage
percent?: number | null
log_tail?: string[]
done?: boolean
error?: string | null
}
/** Params accepted by mesh.configure (superset of the status fields). */ /** Params accepted by mesh.configure (superset of the status fields). */
export interface MeshConfigureParams { export interface MeshConfigureParams {
enabled?: boolean enabled?: boolean
@ -358,6 +375,32 @@ export const useMeshStore = defineStore('mesh', () => {
timeout: 45000, // serial probes are slow (multi-firmware handshakes) timeout: 45000, // serial probes are slow (multi-firmware handshakes)
}) })
} }
/** Available firmware version(s) for a family v1 only ever returns
* ["latest"], since firmware is always fetched from upstream at flash
* time rather than pinned/bundled. */
async function flashListFirmware(family: FlashFirmwareFamily): Promise<string[]> {
const res = await rpcClient.call<{ versions: string[] }>({
method: 'mesh.flash-list-firmware',
params: { family },
})
return res.versions
}
/** Erase + reflash a detected radio. `board` is optional omit it to let
* the backend auto-resolve from the port's USB vid:pid; if that fails
* (e.g. Heltec V4 not yet in the vid:pid table), it errors and the UI
* must ask the user to pick the board explicitly. Always erases first. */
async function flashDevice(path: string, family: FlashFirmwareFamily, board?: FlashBoard): Promise<void> {
await rpcClient.call({
method: 'mesh.flash-device',
params: board ? { path, family, board } : { path, family },
})
}
async function flashStatus(): Promise<FlashJobStatus> {
return rpcClient.call<FlashJobStatus>({ method: 'mesh.flash-status' })
}
async function flashCancel(): Promise<void> {
await rpcClient.call({ method: 'mesh.flash-cancel' })
}
let globalDetectTimer: ReturnType<typeof setInterval> | null = null let globalDetectTimer: ReturnType<typeof setInterval> | null = null
/** App-wide light poll so the detected-device modal works on every page /** App-wide light poll so the detected-device modal works on every page
* (the Mesh view's own 5s poll takes over while it is mounted). */ * (the Mesh view's own 5s poll takes over while it is mounted). */
@ -994,6 +1037,10 @@ export const useMeshStore = defineStore('mesh', () => {
undismissedDetectedDevices, undismissedDetectedDevices,
dismissDetectedDevice, dismissDetectedDevice,
probeDevice, probeDevice,
flashListFirmware,
flashDevice,
flashStatus,
flashCancel,
startGlobalDetection, startGlobalDetection,
fetchPeers, fetchPeers,
fetchMessages, fetchMessages,

View File

@ -47,11 +47,16 @@ if [ -n "$RNODECONF_SRC" ] && [ -f "$RNODECONF_SRC" ]; then
# exit()/quit() builtins, which only exist in interactive Python (site.py # exit()/quit() builtins, which only exist in interactive Python (site.py
# injects them) — a frozen app hits NameError right as it tries to quit # injects them) — a frozen app hits NameError right as it tries to quit
# cleanly, after all the real work already succeeded. See # cleanly, after all the real work already succeeded. See
# pyi_rthook_exit_builtins.py. # pyi_rthook_exit_builtins.py. A second hook fixes rnodeconf's board-flash
# step, which shells out to a bundled esptool.py via `sys.executable` —
# under a frozen binary that's the binary itself, not a real interpreter,
# so the flash subprocess call breaks. See
# pyi_rthook_fix_flasher_executable.py.
.venv/bin/pyinstaller --onefile --name archy-rnodeconf --clean --noconfirm \ .venv/bin/pyinstaller --onefile --name archy-rnodeconf --clean --noconfirm \
--collect-submodules RNS \ --collect-submodules RNS \
--collect-data RNS \ --collect-data RNS \
--runtime-hook pyi_rthook_exit_builtins.py \ --runtime-hook pyi_rthook_exit_builtins.py \
--runtime-hook pyi_rthook_fix_flasher_executable.py \
-d noarchive \ -d noarchive \
"$RNODECONF_SRC" "$RNODECONF_SRC"
echo "Built dist/archy-rnodeconf ($(du -h dist/archy-rnodeconf | cut -f1))" echo "Built dist/archy-rnodeconf ($(du -h dist/archy-rnodeconf | cut -f1))"

View File

@ -0,0 +1,34 @@
# PyInstaller runtime hook — see build.sh.
#
# rnodeconf's own board-flashing code shells out to a bundled esptool.py as
# `[sys.executable, flasher_path, "--chip", ..., "write_flash", ...]` (RNS's
# rnodeconf.py, ~line 2794 as of RNS 1.3.5). That's correct for a normal
# `python rnodeconf.py` invocation, but under a frozen PyInstaller binary
# `sys.executable` is the frozen binary itself, not a real interpreter — so
# the "subprocess" just re-invokes archy-rnodeconf's OWN argparse CLI with
# esptool-shaped flags, which it doesn't recognize, and the flash step fails
# immediately with "unrecognized arguments: --chip ...". Confirmed live
# against a real Heltec V4 (2026-07-23): device selection, band selection,
# and firmware download all worked; only the final `write_flash` subprocess
# call broke this way.
#
# Fix: point sys.executable at a real Python interpreter that has rnodeconf's
# own runtime deps available (esptool.py only needs pyserial, which RNS
# already depends on) before any of rnodeconf's code runs. Prefer the build
# venv this exact binary was frozen from — see build.sh — falling back to a
# bare `python3` on PATH if that venv isn't present on this node.
import os
import sys
if getattr(sys, "frozen", False):
_candidates = [
os.environ.get("ARCHY_RNODECONF_PYTHON", ""),
os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), "..", "reticulum-daemon", ".venv", "bin", "python3"),
os.path.expanduser("~/archy/reticulum-daemon/.venv/bin/python3"),
]
for _candidate in _candidates:
if _candidate and os.path.isfile(_candidate):
sys.executable = _candidate
break
else:
sys.executable = "python3"

View File

@ -84,6 +84,64 @@ if ! command -v nano >/dev/null 2>&1; then
fi fi
fi fi
if ! command -v ping >/dev/null 2>&1; then
log "Installing iputils-ping..."
if sudo apt-get update -qq 2>>"$LOG_FILE" && sudo apt-get install -y -qq iputils-ping 2>>"$LOG_FILE"; then
ok "ping installed"
else
warn "Unable to install ping automatically; continuing update"
fi
fi
if ! command -v esptool >/dev/null 2>&1; then
log "Installing esptool for LoRa radio firmware flashing..."
if sudo apt-get update -qq 2>>"$LOG_FILE" && sudo apt-get install -y -qq esptool 2>>"$LOG_FILE"; then
ok "esptool installed"
else
warn "Unable to install esptool automatically; radio firmware flashing will be unavailable"
fi
fi
# Debian's esptool package (4.7.0+dfsg-0.1) ships without the precompiled
# esp32s3 "stub flasher" blob (stripped for DFSG compliance — no
# buildable-from-source path Debian could verify). Without it, esptool's
# normal stub-loader mode fails outright (FileNotFoundError), and the ROM
# bootloader fallback (--no-stub) doesn't implement a full-chip erase at
# all — confirmed live 2026-07-23 flashing a real Heltec V4, both ways.
# Fetching the exact same file from the matching upstream esptool release
# tag restores full (and correct) flashing behavior — it's the same
# open-source codebase, just the one blob Debian's packaging couldn't
# include.
if command -v esptool >/dev/null 2>&1; then
STUB_DIR="/usr/lib/python3/dist-packages/esptool/targets/stub_flasher"
STUB_FILE="$STUB_DIR/stub_flasher_32s3.json"
if [ ! -f "$STUB_FILE" ]; then
log "Fetching esptool's esp32s3 stub flasher (missing from the Debian package)..."
ESPTOOL_VERSION=$(esptool version 2>/dev/null | tail -1 | tr -d ' \t')
if [ -n "$ESPTOOL_VERSION" ] && sudo curl -fsSL -o "$STUB_FILE" \
"https://raw.githubusercontent.com/espressif/esptool/v${ESPTOOL_VERSION}/esptool/targets/stub_flasher/stub_flasher_32s3.json" \
2>>"$LOG_FILE"; then
sudo chmod 644 "$STUB_FILE"
ok "esp32s3 stub flasher installed"
else
sudo rm -f "$STUB_FILE" 2>/dev/null
warn "Unable to fetch esp32s3 stub flasher; LoRa firmware flashing will be unavailable"
fi
fi
fi
# Build-time prerequisites for reticulum-daemon/build.sh's PyInstaller step
# below (discovered the hard way: ensurepip needs python3-venv, and
# PyInstaller itself needs objdump + libpython3.13.so at build time — none
# of these are pulled in by a bare `python3` package on Debian trixie).
for pkg in python3-venv binutils libpython3.13; do
if ! dpkg -s "$pkg" >/dev/null 2>&1; then
log "Installing $pkg (reticulum-daemon build prerequisite)..."
sudo apt-get update -qq 2>>"$LOG_FILE" && sudo apt-get install -y -qq "$pkg" 2>>"$LOG_FILE" \
|| warn "Unable to install $pkg automatically; reticulum-daemon tools build may fail"
fi
done
# Fetch latest # Fetch latest
log "Fetching from origin..." log "Fetching from origin..."
git fetch origin main --quiet 2>>"$LOG_FILE" git fetch origin main --quiet 2>>"$LOG_FILE"
@ -155,6 +213,30 @@ sudo cp "$BUILT_BIN" "$INSTALL_BIN"
sudo chmod +x "$INSTALL_BIN" sudo chmod +x "$INSTALL_BIN"
ok "Backend installed" ok "Backend installed"
# Build + install reticulum-daemon tools (archy-reticulum-daemon, archy-rnodeconf).
# Non-fatal: archipelago falls back to its dev venv path if the packaged
# binaries aren't present, so a missing/failed build here degrades mesh
# Reticulum support rather than breaking the update. This mirrors
# deploy-to-target.sh's existing manual-deploy step, which until now was the
# only path that ever installed these — a node that only ever received OTA
# self-updates had neither binary.
if [ -f "$REPO_DIR/reticulum-daemon/build.sh" ]; then
log "Building reticulum-daemon tools (archy-reticulum-daemon, archy-rnodeconf)..."
if (cd "$REPO_DIR/reticulum-daemon" && ./build.sh) 2>>"$LOG_FILE"; then
for tool in archy-reticulum-daemon archy-rnodeconf; do
if [ -f "$REPO_DIR/reticulum-daemon/dist/$tool" ]; then
sudo cp "$REPO_DIR/reticulum-daemon/dist/$tool" /usr/local/bin/
sudo chmod +x "/usr/local/bin/$tool"
ok "$tool installed"
else
warn "$tool not built — leaving existing /usr/local/bin/$tool (if any) in place"
fi
done
else
warn "reticulum-daemon tools build failed — continuing without updating them"
fi
fi
# Build frontend # Build frontend
log "Building Vue frontend (production)..." log "Building Vue frontend (production)..."
cd "$FRONTEND_DIR" cd "$FRONTEND_DIR"