Slow multi-hop payments (>15s routing) surfaced as "Payment failed"
while LND settled them in the background: the shared LND REST client's
15s total timeout aborted the synchronous /v1/channels/transactions
wait, and every UI path treated that abort as a definitive failure. The
payment then succeeded anyway and only appeared in history on the next
background poll.
Backend: lnd.payinvoice now decodes the invoice up front for its payment
hash, pays on a dedicated 120s client, and answers status:"pending" with
the hash (never an error) when the wait elapses after the payment was
handed to LND — only a pre-connect failure is still a hard error. New
lnd.paymentstatus RPC reports succeeded/failed/in_flight (with humanized
failure reasons) from /v1/payments.
Frontend: new rpcClient.payLightningInvoice() pays then polls
lnd.paymentstatus to a real terminal state (3s interval, up to 2 min);
all five call sites (send modal, scan modal, web5 unified send, peer-file
purchase, app-launcher payments) migrated. Failure is only shown when LND
itself declares FAILED; a still-settling payment shows an in-flight state
and success fires the transaction refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundles all post-115 fixes into an OTA release: boot READY-before-recovery
+ Restart=always (no more 'server starting up'), and the app-install
daemon-kill fix (v6 relay only bridges live app ports; port-cleanup
excludes our own PID).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v6 app-port relay preemptively bound [::]:<port> for ALL catalog
ports, even apps not installed. Installing such an app (grafana:3000,
photoprism, uptime-kuma, jellyfin — framework-pt 2026-07-27) then hit
'address already in use' from the relay, which triggered
cleanup_stale_pasta_port -> -> killed archipelago
itself (it held the port) mid-install. The daemon crash-looped and the
half-created apps were rolled back and vanished.
Two fixes:
- relay only bridges a port that a running app already answers on over
IPv4 (probe 127.0.0.1:port first) — an uninstalled app's port is never
held, so its install sees a free port and never triggers the cleanup.
- cleanup_stale_pasta_port excludes our own PID from both the ss-based
kill and the fuser kill, so freeing a port can never terminate the
daemon even when the relay legitimately holds it (reinstall case).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.
Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
the unit reaches 'active' in seconds; recovery/reconcile/listener
continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
the daemon back. Manual is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
instead of a tokio-sleep child of the process being stopped, whose
start-half was being lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rootless podman's wildcard publish claims [::] but BLACK-HOLES inbound
v6 (accepts, forwards nothing — vaultwarden 'empty response' from the
phone, 2026-07-26), while most apps got no v6 listener at all. Two
halves: quadlets now publish unbound ports on 0.0.0.0 explicitly
(frees the v6 side; the one-time drift/recreate at upgrade is the
deploy vehicle), and the daemon runs a self-selecting V6ONLY relay on
every catalog launch port forwarding raw TCP to the v4 loopback
listener. Rescans every 60s so new installs bridge without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The companion opens catalog apps by direct port over the mesh; the
fips0 default-deny baseline blocked every one of them (apps 'stuck'
from the phone, 2026-07-26). Core now writes a second fips.d drop-in
from the generated catalog launch-port list on every install/upgrade.
Service/RPC ports (bitcoind 8332, LND 10009, Tor) stay closed.
generate-app-catalog.py emits the Rust port list alongside the TS one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fips::config::install() now writes /etc/fips/fips.d/80-web-ui.nft
(tcp 80/8443 accept) and reloads the baseline on every install/upgrade —
the hardening firewall default-denies inbound on fips0 and the UI was
unreachable over the mesh without it (root-caused live 2026-07-26).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh is IPv6-only; the main web listener bound 0.0.0.0 only, so a
phone reaching a node over its fips0 ULA got RST on :80
(ERR_CONNECTION_ABORTED) — UI over mesh was structurally impossible.
Confirmed 2026-07-26 on framework-pt: v4:80 = 200, v6:80 = refused.
Mirror an IPv4-any main listener with a V6ONLY [::] socket on the same
port; V6ONLY so it coexists with the v4 listener regardless of
net.ipv6.bindv6only.
Also: fips.yaml generator carries the fast-reconnect profile (validated
live on framework-pt since 2026-07-24; snapshot tests updated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observed live on .116 after the dedup/backoff fixes: the kiosk browser's
hot-swap auto-probe (mesh.probe-device) still interleaved its handshakes
with the listener's open sequence on the same tty every backoff window —
Linux double-opens ttys silently, both handshakes corrupted each other,
and every collision's open() DTR/RTS-reset the board again. The retry
heuristic from 5f01ec31 narrowed but couldn't close the race.
A static PORT_OPEN_LOCK now covers the listener's whole device-open
sequence and each probe attempt, so exactly one prober touches the port
at a time. With this + the settle/backoff/dedup fixes, the .116 radio
completed its first MeshCore handshake in days (node-fa161601, 8
contacts) and the session has been stable since.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/dev/mesh-radio is a udev symlink to a ttyUSB*/ttyACM* node that is also
in SERIAL_CANDIDATES, so one board was detected, probed and DTR/RTS-reset
twice per reconnect cycle (and shown twice in the UI):
- detect_serial_devices() dedupes candidates by canonical path, keeping
the stable /dev/mesh-radio name
- auto-detect fallback skips the preferred path it just probed this cycle
instead of immediately resetting the same board again
- probe_device's active-session guard compares canonical paths, so a
probe via the alias can no longer open the tty the live session holds
Together with the 2s boot-settle and stable-session backoff gate, this
takes a plugged-in CP2102/ESP32 board from up to 9 reset events per
cycle at a permanent 5s retry floor down to one probe sequence per
backoff window — enough for the radio to actually finish booting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extracted from beff5dd5 on archy-hwconfig (the rest of that commit is
flash-feature code staying on its branch): a device that connects then
drops within seconds — e.g. mid-boot-loop — kept resetting backoff to
5s forever, and every retry's open() toggles DTR/RTS which itself
resets ESP32-family boards. Backoff now only resets after a session
survives STABLE_SESSION_THRESHOLD (20s), so an unstable device gets
progressively longer quiet gaps to actually finish booting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Traced why a device that's alive and USB-enumerating correctly could still
never complete a single protocol handshake, indefinitely: journal logs
showed genuinely concurrent connection attempts on the same port
(duplicate "Opened serial port"/"Starting X handshake" lines within
microseconds of each other, from what should be sequential probe steps) —
two independent listener sessions were racing on the same tty, each
corrupting the other's reads/writes.
Root cause was in MeshService::stop() combined with mesh::flash's earlier
STOP_LISTENER_TIMEOUT fix: stop() calls `self.listener_handle.take()`
(clearing the field to None immediately) and then awaits the handle with
no bound of its own. When a caller wraps the whole stop() call in a
timeout (as the flash job does, to avoid hanging the RPC response), a slow
listener — mid multi-candidate probe when the shutdown signal arrives —
would cause that outer timeout to cancel the await. But by then
`listener_handle` was already None, so MeshService believed the listener
was stopped, while the actual task kept running, orphaned (dropping a
JoinHandle does not abort the task). A later start() then spawned a
genuinely new listener, racing the orphaned one forever on the same port.
Fixed at the source: stop() now awaits its own handle through a mutable
reference (not by value) with its own bounded timeout, so on timeout it
still owns the handle and can call .abort() on it directly — guaranteeing
the task is actually gone before stop() returns, regardless of how any
caller wraps it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit b14af20d1a)
Every one of Reticulum/Meshcore/Meshtastic's open() deasserts DTR/RTS on
every connection attempt (needed to clear stale line state, but the
transition itself resets ESP32-S3 native-USB boards and CP2102/CH340-
bridged boards wired for Arduino-style auto-reset — acknowledged in the
existing code comments). Each only waited 300ms before expecting a
handshake response — nowhere near real firmware boot time (LoRa radio
init alone routinely takes longer).
A single auto-detect cycle tries multiple protocols in sequence
(Reticulum, then Meshcore, then Meshtastic), each with its own open() and
thus its own reset. With only 300ms of settle per attempt, a board could
plausibly never finish booting from one attempt's reset before the next
attempt's open() reset it again — a self-sustaining "never finishes
booting" loop that would look identical to firmware/hardware flakiness
from the logs, regardless of which firmware family was actually flashed.
Confirmed live 2026-07-23 on both a Heltec V3 (Meshtastic) and V4
(Meshcore): continuous device-side FROM_RADIO_REBOOTED / boot-loop
symptoms with zero config-write-triggered reboots (manage_radio was false
for part of the test), pointing at the connection layer itself rather
than firmware config provisioning.
Bumped the settle delay from 300ms to 2s in Meshcore's and Meshtastic's
open() — full protocol handshakes that need real boot time. Left
reticulum.rs's probe_rnode settle at 300ms deliberately: it's a
cheap/fast KISS-detect gate designed to fail quickly for non-RNode
firmware (documented elsewhere as "~1s"), not a full handshake, and each
subsequent protocol's own open()+settle is what actually needs to cover
real boot time regardless of what probe_rnode did moments before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit 5799c37111)
- sats/BTC unit toggle on the on-chain amount field with live conversion
hint; canonical value stays sats end-to-end
- Fast / Standard / Slow fee presets (1 / 6 / 144 block targets) plus a
custom pane taking target blocks or an explicit sat/vB rate
- confirm pane shows LND's fee estimate for the chosen speed via the new
lnd.estimatefee RPC (GET /v1/transactions/fee)
- lnd.sendcoins now accepts target_conf / sat_per_vbyte with the same
mutual-exclusion + range validation as channel opens
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- lnd.closedchannels: closed-channel history via /v1/channels/closed
- channel list now includes waiting-close and force-closing pending
channels with their closing_txid
- closechannel reads the close stream's first update with a dedicated
client instead of hanging until the closing tx confirms; returns the
closing txid in display byte order
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
send_token_at split the mint swap's results with
partition(send_denoms.contains(amount)) — membership, not count. When a
change denomination equaled a send denomination (guaranteed when
spending from a proof worth exactly 2x the amount: send [n], change
[n]), the change proofs were serialized INTO the token, and the
receiver redeemed double. Both the wallet send flow and peer-files
purchases mint through this path.
The split now consumes exactly one proof per required send denomination
(everything else returns to the wallet as change) and hard-fails if the
mint returns an incomplete denomination set, so a token can never again
be silently over- or under-packed.
Note: tokens minted BEFORE this fix already contain the extra proofs
and will still redeem at their inflated value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Everything from tonight's remote field testing:
- Back-to-dashboard fixed: JS bridges on the retained WebView delegated
through live-composition callbacks (stale closures made apps refuse to
launch after remote ⇄ dashboard), and reattach forces a layout pass
(top/bottom UI was wrong until a tap).
- F*CK IPs MESH branded loader: full-screen on relaunch (startup race)
and during the first connect after scanning a node QR.
- Party 'Share this app' is now a QR of the public vps2 download link
(scan with any camera → install over any internet); direct APK
file-share kept as a secondary action.
- Mesh party pairing is MUTUAL: scanner announces itself to the scanned
phone's Flare /hello — both sides get the peer, a chat entry and a
'👋 joined the party' message; scanner auto-opens the chat.
- Party scanner asks for CAMERA permission (fresh installs/reinstalls
landed on a black preview).
- Node: device tokens mint unique companion-<id> names — pairing a
second phone no longer revokes the first phone's login (the source of
'reconnecting a lot' and the second phone's failure).
Served APK: 0.5.9 (vc29).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The companion reaches the node at its fips0 ULA, and the web UI builds app
links as http://[<ULA>]:<direct port> — but rootless-podman published ports
bind 0.0.0.0 only, so every app URL was refused over the mesh (:8334 first).
A reconcile loop in the backend mirrors public IPv4 listeners: any port
>=1024 on 0.0.0.0 without an IPv6 any-listener gets a v6-ONLY [::] forwarder
to 127.0.0.1, following /proc/net/tcp* so install/remove and hardcoded
companion ports are covered without touching a single container. Purely
additive: IPv4/LAN/Tor access paths are untouched, v6only listeners cannot
collide with or intercept v4 traffic, and foreign IPv6 listeners win.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phones on the FIPS mesh reach the node at http://[<fips0 ULA>], but every
shipped nginx config listened on IPv4 only — nothing answered [::]:80/443,
so mesh HTTP failed with the tunnel fully healthy (found live on
framework-pt during the vc24 5G session, 2026-07-23). Canonical conf gains
listen [::] lines and patch_nginx_conf self-heals deployed nodes (covers
the sites-enabled -> archipelago-http dev variant via the existing
canonicalize pass); validated by the existing nginx -t + rollback flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Paid sharing: AccessControl::Paid gains an accepted-methods list
(lightning/onchain/ecash/fedimint; empty = all, back-compat). Sellers pick
methods in ShareModal, gated on what the node can actually receive (LND
running/channel open, ecash wallet, fedimint joined) with an ⓘ that
explains exactly how to enable a missing rail. Enforced server-side (the
invoice/onchain mints refuse non-accepted methods; the serve gate only
honors tokens/hashes for accepted rails) and the buyer's pay modal only
offers what the seller accepts.
- Purchased music: the two remaining lightbox paths now use the bottom-bar
player — the immediate post-ecash-purchase viewer and the Paid Files
tab's window.open.
- Picture-in-picture buttons on the peer video player and the cloud media
lightbox (utils/pip.ts; Chromium/Safari, no-op elsewhere).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pairing QR was effectively IP-first: the phone dialed the scanned LAN
host and went dark when the LAN renumbered or it left home. FIPS peers on
npubs; IPs are only dial hints. fanchors now leads with the paired node
ITSELF (fnpub @ current host, npub-keyed so LAN contact is direct p2p over
FIPS and survives DHCP), and fips.pair-info always includes the Archipelago
vps2 public anchor (pairing hint only — the node's own anchor file is not
modified) so the phone can rendezvous through the public mesh when away.
Contract doc updated with the REQUIRED app-side changes (built on the Mac).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Name fields sit at firmware-version-dependent offsets; when an offset lands
in binary data, from_utf8_lossy handed the UI replacement-character soup in
the mesh modals and settings panel. decode_mesh_name(): strict UTF-8 to the
first NUL, no control chars, else a readable fallback (short pubkey /
node-<id>).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HA stores a voice satellite as a pinned LAN IP and never re-resolves; when
the LAN renumbers (framework-pt: entry at 192.168.1.241, LAN now
192.168.63.0/24) the speaker silently drops forever. A periodic keeper
probes IP-pinned wyoming entries, sweeps the node's local /24s for the
satellite's port when one dies, rewrites core.config_entries (HA stopped
around the edit so its shutdown flush can't clobber it) and starts HA again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every one of Reticulum/Meshcore/Meshtastic's open() deasserts DTR/RTS on
every connection attempt (needed to clear stale line state, but the
transition itself resets ESP32-S3 native-USB boards and CP2102/CH340-
bridged boards wired for Arduino-style auto-reset — acknowledged in the
existing code comments). Each only waited 300ms before expecting a
handshake response — nowhere near real firmware boot time (LoRa radio
init alone routinely takes longer).
A single auto-detect cycle tries multiple protocols in sequence
(Reticulum, then Meshcore, then Meshtastic), each with its own open() and
thus its own reset. With only 300ms of settle per attempt, a board could
plausibly never finish booting from one attempt's reset before the next
attempt's open() reset it again — a self-sustaining "never finishes
booting" loop that would look identical to firmware/hardware flakiness
from the logs, regardless of which firmware family was actually flashed.
Confirmed live 2026-07-23 on both a Heltec V3 (Meshtastic) and V4
(Meshcore): continuous device-side FROM_RADIO_REBOOTED / boot-loop
symptoms with zero config-write-triggered reboots (manage_radio was false
for part of the test), pointing at the connection layer itself rather
than firmware config provisioning.
Bumped the settle delay from 300ms to 2s in Meshcore's and Meshtastic's
open() — full protocol handshakes that need real boot time. Left
reticulum.rs's probe_rnode settle at 300ms deliberately: it's a
cheap/fast KISS-detect gate designed to fail quickly for non-RNode
firmware (documented elsewhere as "~1s"), not a full handshake, and each
subsequent protocol's own open()+settle is what actually needs to cover
real boot time regardless of what probe_rnode did moments before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced why a device that's alive and USB-enumerating correctly could still
never complete a single protocol handshake, indefinitely: journal logs
showed genuinely concurrent connection attempts on the same port
(duplicate "Opened serial port"/"Starting X handshake" lines within
microseconds of each other, from what should be sequential probe steps) —
two independent listener sessions were racing on the same tty, each
corrupting the other's reads/writes.
Root cause was in MeshService::stop() combined with mesh::flash's earlier
STOP_LISTENER_TIMEOUT fix: stop() calls `self.listener_handle.take()`
(clearing the field to None immediately) and then awaits the handle with
no bound of its own. When a caller wraps the whole stop() call in a
timeout (as the flash job does, to avoid hanging the RPC response), a slow
listener — mid multi-candidate probe when the shutdown signal arrives —
would cause that outer timeout to cancel the await. But by then
`listener_handle` was already None, so MeshService believed the listener
was stopped, while the actual task kept running, orphaned (dropping a
JoinHandle does not abort the task). A later start() then spawned a
genuinely new listener, racing the orphaned one forever on the same port.
Fixed at the source: stop() now awaits its own handle through a mutable
reference (not by value) with its own bounded timeout, so on timeout it
still owns the handle and can call .abort() on it directly — guaranteeing
the task is actually gone before stop() returns, regardless of how any
caller wraps it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced a real "Operation failed" on the post-flash configure step: the
flash job's own post-completion code unconditionally called
MeshService::start() after configure(), regardless of the persisted
enabled flag. In this incident, the user had toggled mesh off shortly
before the flash (config.enabled saved as false), but the job's forced
start() ran anyway — leaving the listener actually running while
self.config.enabled stayed false. When the user then clicked "Keep As Is"
on the fresh post-flash probe, their mesh.configure call correctly saw a
false→true transition and tried to start the listener itself, hitting
"Mesh listener already running" — the listener had already been
force-started behind the config's back.
Two fixes:
1. The flash job now only restarts the listener if the freshly-loaded
config still says enabled — respecting a user's own concurrent choice
to disable mesh instead of silently overriding it.
2. MeshService::start() is now idempotent: finding the listener already
running is treated as success, not an error. Ensuring the listener is
running is what every caller actually wants; whichever caller's start()
wins a race, the other finding it already satisfied is the correct
outcome, not a failure to surface to the user.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--erase-all hit the exact same "ROM does not support function erase_flash"
error as a standalone erase_flash — confirmed live: esptool's --erase-all
is implemented as the same full-chip-erase command, not a per-sector loop,
so it needs the stub just as much. The real fix isn't finding another way
around the ROM bootloader's limitations — it's restoring the stub Debian's
package is missing.
Fetched the exact missing file (stub_flasher_32s3.json) from the matching
upstream esptool release tag and installed it directly on archy-x250-dev;
verified live with a read-only `flash_id` that stub mode now loads
("Uploading stub... Running stub..."). Removed --no-stub from flash.rs now
that normal stub-loader mode works correctly (faster and fully-featured vs.
the ROM-only fallback). scripts/self-update.sh now fetches and installs
this same file automatically whenever it's missing, so this isn't a
one-off manual fix tied to a single node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real esptool output on a live Heltec V4 exposed the next layer of the
--no-stub fix: "A fatal error occurred: ESP32-S3 ROM does not support
function erase_flash." The ESP32-S3 ROM bootloader only implements
per-sector erase (used internally as write_flash streams data), not a bulk
full-chip-erase opcode — that's a stub-firmware-only feature, and Debian's
esptool package ships without the stub (hence --no-stub in the first
place).
write_flash's own --erase-all flag gets the same "erase everything before
writing" outcome our "always erase before write" default requires, but via
the per-sector mechanism the ROM bootloader actually supports — one esptool
invocation instead of a separate erase_flash + write_flash pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced a real "Operation failed" report to its root: MeshService::stop()
ran synchronously inside start_flash_job, BEFORE the background task was
even spawned — so the RPC call itself blocked on it. The mesh listener's
reconnect/multi-candidate-probe loop doesn't check its shutdown signal
between candidates, and in this incident it was mid a Meshcore-then-
Meshtastic probe cycle when the flash request came in, so stop() never
returned. The HTTP request timed out client-side ("connection closed
before message completed" in the server log), the frontend surfaced a
generic "Operation failed", and — worse — the job had already been
inserted into the single-job-guard slot before stop() ran, so with nothing
ever spawned to complete it, every subsequent flash attempt failed with
"already in progress" until a full service restart.
The existing MAX_JOB_DURATION ceiling didn't help here: it only wraps
run_flash(), which is reached AFTER stop() returns — so a hang in stop()
itself was completely unguarded.
Fixed by moving the stop() call inside the spawned background task with
its own 20s timeout. The RPC call now always returns immediately once the
job is registered, and a slow-to-stop listener fails the job cleanly with
a clear message instead of hanging the request and wedging every future
attempt behind it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- payments.rs: an expired invoice can never succeed on retry, so the
error now says so and tells the payer the way out (ask the recipient
for a fresh invoice) instead of echoing LND's raw reason.
- middleware.rs: the error sanitizer masked every Lightning payment
failure as 'Operation failed. Check server logs' — but LND's reasons
('invoice expired', 'unable to find a path') are written for the
payer and are all actionable. 'Payment failed', 'Invalid payment
request' and 'Missing payment_request' now pass through, with a
regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The improved error logging from the last fix immediately paid off — real
esptool output on the next attempt showed two distinct bugs:
1. Debian's esptool package (4.7.0+dfsg-0.1) ships without the precompiled
"stub flasher" blobs (stripped for DFSG compliance), so the default
stub-loader flash mode fails immediately: FileNotFoundError: ... No such
file or directory: '.../stub_flasher_32s3.json'. Fixed with --no-stub,
which talks directly to the ROM bootloader instead — the standard
workaround for this exact Debian packaging gap.
2. The fallback-baud retry appended `--baud 115200` AFTER the subcommand
token (erase_flash/write_flash), but esptool requires global flags
before the subcommand — every retry failed with "esptool: error:
unrecognized arguments: --baud 115200", so the retry logic added
earlier never actually got a chance to run. esptool_with_retry now
builds global args (--chip/--port/--baud/--no-stub) and subcommand args
separately and always concatenates them in the right order, instead of
relying on call-site ordering of a single flat arg list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run_streamed's failure path only reported "Command exited with exit status:
N" — the real stderr (already captured line-by-line into job.log_tail for
the UI's live poll) never made it into the error that gets logged to
journald. A real esptool failure just now showed only the bare exit code,
with esptool's actual error text sitting in-memory, visible in the UI but
not diagnosable from the server logs.
Now includes the last 10 log_tail lines in the returned error, so the
actual tool output shows up in journalctl too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three real incidents from live-testing the flash feature on archy-x250-dev,
each traced through logs on the node and fixed at the root:
- A failed flash left the mesh listener auto-resuming into a reconnect loop
whose backoff reset to its 5s minimum on any momentary connection, even
mid-boot-loop — every retry's open() toggles DTR/RTS, which resets many
ESP32 boards, so the retries were themselves sustaining the loop. Backoff
now only resets after a session runs stably for 20s+, and the flash job
no longer auto-resumes the listener after a failure (only on success,
after a settle delay).
- The HTTP client's blanket 30s request timeout covered entire downloads
(Meshtastic's zip is ~170MB), killing large transfers mid-stream; fixed
with a per-chunk stall timeout instead of a fixed total-transfer cap. But
the download's initial request had no timeout at all, so a slow-to-start
server hung it forever — wedging the single-flash-job guard permanently
("already in progress" on every subsequent attempt). Fixed with a bounded
wait for the response to start, plus an absolute ceiling around the whole
job as a last-resort safety net.
- archy-rnodeconf's PyInstaller freeze broke its own internal esptool
invocation (sys.executable pointed at the frozen binary itself instead of
a real interpreter) — fixed via a new runtime hook. esptool's own
auto-reset-into-bootloader handshake is separately known to be flaky on
some CP2102/CH340 boards; it now retries once at a conservative baud
rate instead of failing outright, and failure logs show the full error
chain instead of just the outer context message.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dorian
ssmithxandarchipelago