Traced a real "Operation failed" on the post-flash configure step: the
flash job's own post-completion code unconditionally called
MeshService::start() after configure(), regardless of the persisted
enabled flag. In this incident, the user had toggled mesh off shortly
before the flash (config.enabled saved as false), but the job's forced
start() ran anyway — leaving the listener actually running while
self.config.enabled stayed false. When the user then clicked "Keep As Is"
on the fresh post-flash probe, their mesh.configure call correctly saw a
false→true transition and tried to start the listener itself, hitting
"Mesh listener already running" — the listener had already been
force-started behind the config's back.
Two fixes:
1. The flash job now only restarts the listener if the freshly-loaded
config still says enabled — respecting a user's own concurrent choice
to disable mesh instead of silently overriding it.
2. MeshService::start() is now idempotent: finding the listener already
running is treated as success, not an error. Ensuring the listener is
running is what every caller actually wants; whichever caller's start()
wins a race, the other finding it already satisfied is the correct
outcome, not a failure to surface to the user.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--erase-all hit the exact same "ROM does not support function erase_flash"
error as a standalone erase_flash — confirmed live: esptool's --erase-all
is implemented as the same full-chip-erase command, not a per-sector loop,
so it needs the stub just as much. The real fix isn't finding another way
around the ROM bootloader's limitations — it's restoring the stub Debian's
package is missing.
Fetched the exact missing file (stub_flasher_32s3.json) from the matching
upstream esptool release tag and installed it directly on archy-x250-dev;
verified live with a read-only `flash_id` that stub mode now loads
("Uploading stub... Running stub..."). Removed --no-stub from flash.rs now
that normal stub-loader mode works correctly (faster and fully-featured vs.
the ROM-only fallback). scripts/self-update.sh now fetches and installs
this same file automatically whenever it's missing, so this isn't a
one-off manual fix tied to a single node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real esptool output on a live Heltec V4 exposed the next layer of the
--no-stub fix: "A fatal error occurred: ESP32-S3 ROM does not support
function erase_flash." The ESP32-S3 ROM bootloader only implements
per-sector erase (used internally as write_flash streams data), not a bulk
full-chip-erase opcode — that's a stub-firmware-only feature, and Debian's
esptool package ships without the stub (hence --no-stub in the first
place).
write_flash's own --erase-all flag gets the same "erase everything before
writing" outcome our "always erase before write" default requires, but via
the per-sector mechanism the ROM bootloader actually supports — one esptool
invocation instead of a separate erase_flash + write_flash pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced a real "Operation failed" report to its root: MeshService::stop()
ran synchronously inside start_flash_job, BEFORE the background task was
even spawned — so the RPC call itself blocked on it. The mesh listener's
reconnect/multi-candidate-probe loop doesn't check its shutdown signal
between candidates, and in this incident it was mid a Meshcore-then-
Meshtastic probe cycle when the flash request came in, so stop() never
returned. The HTTP request timed out client-side ("connection closed
before message completed" in the server log), the frontend surfaced a
generic "Operation failed", and — worse — the job had already been
inserted into the single-job-guard slot before stop() ran, so with nothing
ever spawned to complete it, every subsequent flash attempt failed with
"already in progress" until a full service restart.
The existing MAX_JOB_DURATION ceiling didn't help here: it only wraps
run_flash(), which is reached AFTER stop() returns — so a hang in stop()
itself was completely unguarded.
Fixed by moving the stop() call inside the spawned background task with
its own 20s timeout. The RPC call now always returns immediately once the
job is registered, and a slow-to-stop listener fails the job cleanly with
a clear message instead of hanging the request and wedging every future
attempt behind it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The improved error logging from the last fix immediately paid off — real
esptool output on the next attempt showed two distinct bugs:
1. Debian's esptool package (4.7.0+dfsg-0.1) ships without the precompiled
"stub flasher" blobs (stripped for DFSG compliance), so the default
stub-loader flash mode fails immediately: FileNotFoundError: ... No such
file or directory: '.../stub_flasher_32s3.json'. Fixed with --no-stub,
which talks directly to the ROM bootloader instead — the standard
workaround for this exact Debian packaging gap.
2. The fallback-baud retry appended `--baud 115200` AFTER the subcommand
token (erase_flash/write_flash), but esptool requires global flags
before the subcommand — every retry failed with "esptool: error:
unrecognized arguments: --baud 115200", so the retry logic added
earlier never actually got a chance to run. esptool_with_retry now
builds global args (--chip/--port/--baud/--no-stub) and subcommand args
separately and always concatenates them in the right order, instead of
relying on call-site ordering of a single flat arg list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run_streamed's failure path only reported "Command exited with exit status:
N" — the real stderr (already captured line-by-line into job.log_tail for
the UI's live poll) never made it into the error that gets logged to
journald. A real esptool failure just now showed only the bare exit code,
with esptool's actual error text sitting in-memory, visible in the UI but
not diagnosable from the server logs.
Now includes the last 10 log_tail lines in the returned error, so the
actual tool output shows up in journalctl too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three real incidents from live-testing the flash feature on archy-x250-dev,
each traced through logs on the node and fixed at the root:
- A failed flash left the mesh listener auto-resuming into a reconnect loop
whose backoff reset to its 5s minimum on any momentary connection, even
mid-boot-loop — every retry's open() toggles DTR/RTS, which resets many
ESP32 boards, so the retries were themselves sustaining the loop. Backoff
now only resets after a session runs stably for 20s+, and the flash job
no longer auto-resumes the listener after a failure (only on success,
after a settle delay).
- The HTTP client's blanket 30s request timeout covered entire downloads
(Meshtastic's zip is ~170MB), killing large transfers mid-stream; fixed
with a per-chunk stall timeout instead of a fixed total-transfer cap. But
the download's initial request had no timeout at all, so a slow-to-start
server hung it forever — wedging the single-flash-job guard permanently
("already in progress" on every subsequent attempt). Fixed with a bounded
wait for the response to start, plus an absolute ceiling around the whole
job as a last-resort safety net.
- archy-rnodeconf's PyInstaller freeze broke its own internal esptool
invocation (sys.executable pointed at the frozen binary itself instead of
a real interpreter) — fixed via a new runtime hook. esptool's own
auto-reset-into-bootloader handshake is separately known to be flaky on
some CP2102/CH340 boards; it now retries once at a conservative baud
rate instead of failing outright, and failure logs show the full error
chain instead of just the outer context message.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
