fix(mesh): stop the flash-triggered device boot-loop, fix stuck/wedged jobs
Three real incidents from live-testing the flash feature on archy-x250-dev,
each traced through logs on the node and fixed at the root:
- A failed flash left the mesh listener auto-resuming into a reconnect loop
whose backoff reset to its 5s minimum on any momentary connection, even
mid-boot-loop — every retry's open() toggles DTR/RTS, which resets many
ESP32 boards, so the retries were themselves sustaining the loop. Backoff
now only resets after a session runs stably for 20s+, and the flash job
no longer auto-resumes the listener after a failure (only on success,
after a settle delay).
- The HTTP client's blanket 30s request timeout covered entire downloads
(Meshtastic's zip is ~170MB), killing large transfers mid-stream; fixed
with a per-chunk stall timeout instead of a fixed total-transfer cap. But
the download's initial request had no timeout at all, so a slow-to-start
server hung it forever — wedging the single-flash-job guard permanently
("already in progress" on every subsequent attempt). Fixed with a bounded
wait for the response to start, plus an absolute ceiling around the whole
job as a last-resort safety net.
- archy-rnodeconf's PyInstaller freeze broke its own internal esptool
invocation (sys.executable pointed at the frozen binary itself instead of
a real interpreter) — fixed via a new runtime hook. esptool's own
auto-reset-into-bootloader handshake is separately known to be flaky on
some CP2102/CH340 boards; it now retries once at a conservative baud
rate instead of failing outright, and failure logs show the full error
chain instead of just the outer context message.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -489,6 +489,40 @@ which esptool; ls -la /usr/local/bin/archy-rnodeconf
|
||||
for the next detection poll) — the hot-swap modal re-probes automatically
|
||||
and shows whatever firmware is actually on the board now.
|
||||
|
||||
**Known incident (2026-07-23) — reconnect storm / device boot-loop after a
|
||||
failed flash**: a real Heltec V3 got stuck cycling "connect → partial
|
||||
handshake → drop" every 5-15s for 5+ minutes after a `mesh.flash-device`
|
||||
attempt failed with `Reading firmware download stream`. Root cause was two
|
||||
compounding issues, both now fixed:
|
||||
1. `spawn_mesh_listener`'s reconnect backoff (`core/archipelago/src/mesh/listener/mod.rs`)
|
||||
reset to its 5s minimum any time the prior session had been `device_connected`
|
||||
at all, even for under a second — so a device that connects-then-drops
|
||||
repeatedly never actually backed off. Every retry's `open()` toggles
|
||||
DTR/RTS, which resets many ESP32 boards' MCU (native-USB *and*
|
||||
CP2102/CH340 auto-reset-circuit boards), so the aggressive retries were
|
||||
themselves *causing* the boot loop, not just observing one. Fixed by only
|
||||
resetting backoff when a session ran for at least `STABLE_SESSION_THRESHOLD`
|
||||
(20s) — see that constant's doc comment.
|
||||
2. `mesh::flash::start_flash_job`'s post-completion handler auto-resumed the
|
||||
listener unconditionally, even after a *failed* flash, immediately
|
||||
re-entering the reconnect loop above with no cooldown. Fixed: on failure
|
||||
the listener is now deliberately left stopped (reconnect manually via the
|
||||
UI once the board is confirmed alive); on success there's a 5s settle
|
||||
delay before resuming, so the board finishes booting from the flash
|
||||
tool's own reset before Archipelago starts probing it again.
|
||||
3. Separately, the download itself was failing because `mesh::flash`'s HTTP
|
||||
client had a blanket 30s request timeout that covered the *entire*
|
||||
download (including streaming a 170MB Meshtastic zip), not just
|
||||
connection setup — fixed with a per-chunk stall timeout instead of a
|
||||
fixed total-transfer cap.
|
||||
|
||||
If this symptom recurs (rapid repeating `mesh::serial: Opened serial
|
||||
port... Starting Meshcore handshake` lines in `journalctl -u archipelago`
|
||||
without a `LoRa firmware flash` job in progress), it's a NEW instance of the
|
||||
same class of bug, not the one above — check whether backoff is actually
|
||||
escalating (`Mesh session error: ... (retry in Xs)` — X should grow past 5s
|
||||
within a few cycles) before assuming it's flashing-related.
|
||||
|
||||
---
|
||||
|
||||
## General Maintenance
|
||||
|
||||
Reference in New Issue
Block a user