Root causes found and fixed after live debugging on archi-dev-box (all
verified on real RNode hardware, archi-dev-box <-> archy-x250-dev E2E):
- probe_rnode raced the board's own boot: opening the port pulses DTR/RTS
through USB-UART bridges (CP2102/Heltec V3), the ESP32 power-cycles and
spends ~2.5-3s in boot ROM, and the KISS DETECT written 300ms after open
landed in the void — so an RNode could NEVER connect on these boards.
Now: immediate probe (fast path), then drain-until-quiet boot settle and
a second DETECT with a fresh response window.
- MeshService::configure() only restarted the listener on enable/disable —
device_kind/device_path/advert_name/RF-param changes were silent no-ops
until a full process restart (the setup modal's apply/keep-as-is did
nothing). Material config changes now bounce the listener; the open
sequence races the shutdown signal so stop() no longer burns the full
15s timeout mid-probe; mesh.configure applies in the background instead
of stalling every status poll behind the service write-lock.
- The mesh name was write-only: config.advert_name had no reader,
server.set-name never reached the mesh service, and Reticulum's
set_advert_name was a no-op (daemon display name fixed at spawn, and the
ARCHY:2 announce blob REPLACED the LXMF display name — every archy node
was anonymous on RNS). Now: advert_name > server name precedence feeds
the session, renames restart it live, the daemon gets --display-name at
spawn plus a set_name RPC verb, and announces carry the LXMF-standard
msgpack name with the identity blob appended as an extra list element
stock clients (Sideband/NomadNet) ignore.
- Dead reticulum-daemon was invisible for up to 30min (RX-stall watchdog):
child exit / RPC-EOF now fails try_recv_frame so the session reconnects.
- Setup modal re-trigger loop: plugged_at used the tty node's mtime, which
bumps on every open — each probe invalidated the dismissal key. Use
btime/ctime (only change on real plugs).
- ARCHY:2 identity adverts (re-emitted every 60s over Reticulum) stomped
the federation twin's real name with a synthetic Archy-… placeholder and
nulled its position; blob-only announces no longer assert a name, blob
strings can never become display names, and stale blob names are healed
at peers.json load.
- mesh.broadcast on Meshtastic sent heartbeat+time only (no identity);
SendAdvert now also fires a want_response NodeInfo broadcast.
- New mesh.refresh RPC: actively re-queries the radio contact table (the
UI Refresh button previously only re-read server caches).
- Reticulum peers now track last_advert (announce time) and mark existing
peers reachable on inbound traffic.
- Boot auto-enable no longer force-enables mesh when an operator
explicitly disabled it (only fires when no config file exists).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HTTP accept loop parked on acquire_owned() when the connection
budget drained, freezing accept() for every client (the .228
session-flapping / CLOSE-WAIT signature). Permits drained because
half-open clients and hung upstreams held them indefinitely.
- try_acquire_owned + immediate 503-and-close when the budget is
exhausted; the accept loop itself never blocks
- 30s http1_header_read_timeout drops slowloris/half-open clients
- 900s watchdog bounds non-upgraded connections; websocket upgrades
are exempt (legitimately long-lived)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase A3 of docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md (RC5), measured against
the A2 dial_stats baseline:
- 25s connectivity watcher in the fips supervisor: re-applies seed anchors
immediately on an anchor-link drop, on startup-disconnected, AND on
silent data-path death (connect_fails growing with zero fips_ok — the
live .198 failure where the daemon reported "connected" while every
dial blackholed and the 300s tick never healed it). Bounded to one
re-apply per 60s.
- anchors::apply is now concurrent with a 15s per-connect cap — the old
serial loop waited unbounded on each `sudo fipsctl connect`, so one
hung subprocess stalled the whole periodic tick.
- rebindable peer listener: the accept loop returns after persistent
accept errors (was: continue forever = inbound-dead until restart) and
peer_late_bind_loop rebinds — also on fips0 ULA change.
- is_service_active gets a 10s TTL cache (was up to 2 systemctl spawns
per dial attempt and per warm-tick peer).
- the warm tick now warms the union of federation peers + configured
seed anchors (direct anchor links used to go cold between 300s ticks),
skipping the redundant per-peer service check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase A1 of docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md — the five changes that
made FIPS fall back to Tor even when a FIPS path existed:
- RC0: the fips.d drop-in now opens PEER_PORT 5679 (was 80+8443 only, so
every hardened node firewalled peers' FIPS dials; 28k drops on .198)
- RC4: /blob/ and /dwn/ added to the peer-path allowlist — mesh file
sharing and DWN sync were 404 → 100% Tor by construction
- RC2-G2: lan_fips_anchors dials PUBLISHED_UDP_PORT (2121) instead of the
dead 8668, with a drift-guard test against the rendered daemon config
- RC2-G1: direct LAN peering actually runs now — mDNS TXT advertises the
FIPS npub, discovery calls set_fips_npub, and the anchor tick hydrates
npubs from federation storage for peers on older builds
- RC3: FIPS attempt budget is a hard cap (retry no longer doubles it) and
the 12 hot call sites get explicit fips_timeout fast-fail so Tor keeps
its full budget (browse-peer, preview, /blob, DWN, node-message,
rotation notifies)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v6 app-port relay preemptively bound [::]:<port> for ALL catalog
ports, even apps not installed. Installing such an app (grafana:3000,
photoprism, uptime-kuma, jellyfin — framework-pt 2026-07-27) then hit
'address already in use' from the relay, which triggered
cleanup_stale_pasta_port -> -> killed archipelago
itself (it held the port) mid-install. The daemon crash-looped and the
half-created apps were rolled back and vanished.
Two fixes:
- relay only bridges a port that a running app already answers on over
IPv4 (probe 127.0.0.1:port first) — an uninstalled app's port is never
held, so its install sees a free port and never triggers the cleanup.
- cleanup_stale_pasta_port excludes our own PID from both the ss-based
kill and the fuser kill, so freeing a port can never terminate the
daemon even when the relay legitimately holds it (reinstall case).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rootless podman's wildcard publish claims [::] but BLACK-HOLES inbound
v6 (accepts, forwards nothing — vaultwarden 'empty response' from the
phone, 2026-07-26), while most apps got no v6 listener at all. Two
halves: quadlets now publish unbound ports on 0.0.0.0 explicitly
(frees the v6 side; the one-time drift/recreate at upgrade is the
deploy vehicle), and the daemon runs a self-selecting V6ONLY relay on
every catalog launch port forwarding raw TCP to the v4 loopback
listener. Rescans every 60s so new installs bridge without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh is IPv6-only; the main web listener bound 0.0.0.0 only, so a
phone reaching a node over its fips0 ULA got RST on :80
(ERR_CONNECTION_ABORTED) — UI over mesh was structurally impossible.
Confirmed 2026-07-26 on framework-pt: v4:80 = 200, v6:80 = refused.
Mirror an IPv4-any main listener with a V6ONLY [::] socket on the same
port; V6ONLY so it coexists with the v4 listener regardless of
net.ipv6.bindv6only.
Also: fips.yaml generator carries the fast-reconnect profile (validated
live on framework-pt since 2026-07-24; snapshot tests updated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- LND watchdog: framework-pt's LND sat wedged 14h (RPC up, server never
ready, channels inactive) with nothing noticing. 15 consecutive bad
minutes now bounce the container automatically (30min cooldown) —
silent multi-hour Lightning outages become self-healed blips.
- Send modal: Auto tab hidden; Ecash splits into Cashu and Fedi (new
wallet.fedimint-send RPC wrapping the existing spend_from_any); both
rails render the generated token as a scannable QR alongside the
copyable text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three silent failure modes killed nostr peer requests (analyzed from the
live code 2026-07-22): (1) nothing ever polled the relays except the
manual Federation Poll button — a 5-minute background poll now fetches
inbound requests and nudges the websocket when something lands (the
handler's discoverability gate still applies); (2) handshake send/poll
used only the two hardcoded config relays (one defunct) and ignored the
user-managed relay list — every nostr op now uses the merged set;
(3) publish errors were swallowed (let _ =) so 'ok' was reported even
when no relay accepted the event — now a delivery failure is an error
the UI shows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend streams LND /v1/transactions/subscribe (fires on mempool arrival
and each confirmation) and nudges the /ws/db revision per event; the Home
wallet card subscribes and refetches (debounced 800ms). An incoming
broadcast now appears while the 30s poll is still asleep. Reconnects
forever with capped backoff when LND is down/locked/absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fleet fixes that OTA can deliver (unlike the ISO-only polkit rule):
- WiFi (issue #99): nodes drive NetworkManager from a system-level service
with no logind seat, so the stock polkit rule denies them and Wi-Fi setup
fails with 'Insufficient privileges'. Fresh ISOs since 2026-05 ship the
scoped rule, but OTA'd nodes never got it (OTA replaces binary + web UI,
not host system config). Add a startup self-heal that installs the rule if
missing and best-effort ensures polkitd — both wrapped so an offline/locked
apt can never fail startup; the rule is written regardless so it activates
once polkitd lands. Idempotent on the rule's unique subject.user marker.
- FIPS: regenerating fips.yaml (this build does, once, on first boot after
the OTA) restarts the fips daemon; for a few seconds /run/fips/control.sock
is gone and every seed-anchor dial fails, islanding the node until the next
5-min tick. Detect that exact failure (all dials fail on control.sock) and
retry in 15s instead — bounded to 8 fast retries/episode so a node with no
fips daemon falls back to the steady cadence rather than busy-looping.
FIPS unit suite: 33/33 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Invites carried no trust level, and both accept_invite and the
peer-joined handler hardcoded TrustLevel::Trusted — so 'Invite a Peer'
(Observer) federated both sides as fully Trusted.
- invite codes now carry a 'trust' field (legacy codes parse as Trusted)
- federation.invite accepts trust_level: trusted|observer
- accept_invite assigns the invite's level on the acceptor side
- peer-joined resolves the granted level authoritatively by matching the
acceptor's echoed invite token against our stored outgoing invites;
the peer's own unsigned claim is honored only as a downgrade, so no
escalation is possible
- discovery/connection-request approvals mint Observer invites directly
(the post-hoc demotion in handshake.rs stays as a legacy safety net)
- asymmetry self-heal re-asserts at the locally-held level
Tests: observer threading + legacy default-trusted parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Formal gate take-2 went 4/5: iteration 5's vaultwarden stop-wait saw
'restarting' for the full 120s window. A reconcile pass had queued every
app into pending_boot_starts moments before the stop marker landed, and
the scanner's overlay painted the deliberately-stopped app Restarting
until the slow sequential pass reached it (legacy container, so no unit
state to contradict it).
Two filters: the scanner overlay skips user-stopped ids (the marker is
written before the stop runs, so it's reliably present), and the
reconciler no longer queues apps whose lifecycle op is in flight into the
pending overlay (it skips them via the same probe anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One legacy app per gate run stuck at 'stopping'/'starting' past the test
window while its container was already settled (vaultwarden:stop run C,
jellyfin:stop run D, uptime-kuma:start run E — .228, 2026-07-09). The
package scanner already sees the truth every 60s but
merge_preserving_transitional refused to report it until the RPC worker
wrote the final state, and the workers can legitimately trail the
container by minutes:
- stop workers queue behind the orchestrator per-app lock, which the
reconcile host-port repair path holds through multi-minute stability
waits (repair_manifest_host_ports_after_stability: 5s + 5s probe +
restart + 60-420s port wait + 15-90s stable-running);
- start workers hold Starting through the full readiness wait —
host_port_wait_timeout_secs is 420s for uptime-kuma (HTTP probe),
longer than any UI/test patience — and the Starting stuck-timeout is
the 20-minute INSTALLING one.
New merge rules, both truth-driven by podman's live view:
- (Stopping, Stopped) + user-stop marker → Stopped: the scanner only
normalizes exited→Stopped for user-stopped apps, so this is exactly
"the user asked for a stop and the container has exited".
- (Starting, Running) → Running: the start visibly succeeded; live
health readings are merged separately and keep reporting readiness.
Restarting is deliberately NOT resolved by a Running scan — mid-restart
readings are the pre-stop container.
Tests: merge_tests 13/13 (4 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tollgate-wrt keeps its own separate Cashu wallet on the router
(/etc/tollgate/wallet.db) — customer payments land there, never in
this node's own wallet. Its Lightning auto-payout is configured
independently in /etc/tollgate/identities.json, which is easy to
leave misconfigured or pointed at a placeholder address (found live:
both "owner" and "developer" identities on this deployment pointed at
the same unconfigured tollgate@minibits.cash default).
Rather than depend on getting that Lightning payout config right,
tollgate_sweep::sweep_once() periodically (every 5 min, via SSH)
checks the router's `tollgate wallet balance`, and if nonzero, runs
`tollgate wallet drain cashu` and receives the resulting token(s)
straight into the local wallet via the same path the "Receive ecash"
UI uses (wallet::ecash::receive_token) — bypassing Lightning payout
entirely. A no-op (Ok(0)) if no router is configured or it doesn't
have TollGate installed.
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers that opt in via a new "Share Location" toggle in Settings
(server.set-location RPC) get plotted on other trusted peers' Mesh Map
with a distinct Archy-logo marker, separate from raw LoRa radio peers.
Location is persisted locally, carried in NodeStateSnapshot, and
propagated through federation sync/delta like other node state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mesh/federation messages between co-located nodes were always falling back to
Tor because the FIPS overlay had no direct peering — every node depended on the
global anchor's spanning tree, and when that anchor link flaps a node is
isolated and all FIPS dials time out. (Diagnosed live on .116/.198: pure-FIPS
direct peering over UDP 8668 fixes it — 2.5ms vs timeout.)
Generalize the manual fix: in the existing 5-min FIPS seed-anchor apply loop,
also auto-connect every federation peer the PeerRegistry knows both a LAN
address AND a FIPS npub for, dialing its FIPS UDP transport (port 8668) at its
LAN IP via the same idempotent `fipsctl connect` path (new
anchors::lan_fips_anchors). This is FIPS's own transport over the LAN — NOT
Tailscale, NOT the HTTP/LAN messaging port. Transient (recomputed each tick from
live mDNS discovery, never persisted) so changing IPs self-correct. Remote peers
with no LAN address are untouched (still routed via the anchor).
Registry Arc hoisted out of the transport-init block so the loop can read
all_peers(). cargo check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Federated nodes failed to converge to full-mesh across the LAN<->Tailscale
boundary: nodes were invisible to peers, sync 'took ages'/timed out, and
names only updated on a manual sync. Onions were healthy in both directions
(~3-5s); the failures were app-layer.
- B: federation dials fast-fail a dead FIPS path via .fips_timeout(6s) in
sync_with_peer + notify_join, so the Tor fallback isn't stuck behind the
full 30s FIPS budget when LAN and remote peers share no FIPS path.
- A: notify_join (peer-joined) now spawns with retries+backoff instead of a
single awaited best-effort POST, so the join RPC returns instantly (no
'Request timeout') and the inviter reliably learns the joiner (was
asymmetric).
- C: new 90s periodic federation auto-sync (none existed) so renamed nodes
and roster changes propagate without a manual Sync click.
- self-heal: each auto-sync re-asserts membership to any peer that doesn't
list us back, converging the fleet to full-mesh and healing pre-existing
asymmetry with no manual re-joins.
Validated live across 7 nodes: a previously fleet-invisible node became
fully meshed automatically (logs: 'auto-sync ... reasserted=1',
'peer-joined ... delivered').
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mesh: stop broadcasting ARCHY:2 identity on the public channel (startup + every advert tick); receive path still parses inbound. No more public-channel spam.
- mesh assistant: trigger on !ai/!ask typed in 1:1 chat (was only the dead AssistQuery path + bare channel text); route the reply transport-aware via MeshService::send_message (Tor for federation peers, LoRa for radio) through a new AssistChatReply event consumed at the server layer — fixes replies never reaching federation askers.
- mesh assistant: per-contact !ai allowlist (allowed_contacts) bypassing trusted_only; config + RPC + is_sender_allowed.
- fedimint-clientd manifest: network_policy open -> bridge (invalid value made the loader skip the whole manifest, so fmcd never ran and federations never joined/listed).
- ui: AI panel — Claude model dropdown (Haiku/Sonnet/Opus presets) + allowlist contact picker.
- ui: Settings — App Updates + App Registry moved under Account.
- ui: mesh chat — overscroll-behavior: contain so chat scroll no longer bleeds to the contacts panel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A peer accepted via invite is seeded into the mesh peer table with
name=None, so it shows as "Archipelago <pubkey8>" in chat. Federation
sync later learns the real name (update_node_state writes it to
nodes.json) and discovers transitive peers (merge_transitive_peers),
but nothing pushed those into the live mesh peer table — the chat list
stayed stale until the next mesh restart, and transitive peers never
appeared as contacts at all.
Add RpcHandler::refresh_federation_mesh_peers() (re-runs the idempotent,
onion-deduped seed_federation_peers_into_mesh) and call it after every
periodic sync cycle (server.rs) and after the manual federation.sync-all
RPC. Names now correct themselves and the full roster meshes within a
sync cycle, no restart needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 wiring (task #12):
- NostrSeedDiscovery: async ProviderDiscovery that queries relays for signed
seed adverts and parses endpoint ids (swarm/iroh_provider.rs, seed_advert.rs).
- seed_and_advertise publish path; dep-free fetch/publish helpers reuse the
node's Nostr identity (build_nostr_client/load_or_create_nostr_keys made
pub(crate)).
- swarm::init builds the IrohProvider once into a OnceLock runtime; providers()
returns it; announce_held_blob() is called from update.rs after a release
component passes both hash gates.
- config swarm_enabled (ARCHIPELAGO_SWARM_ENABLED, default off); server.rs init.
Paid swarm serving (Phase 4 step F):
- swarm/paid.rs gates the iroh-blobs provider through streaming::gate,
intercepting connect + GET (peer push hard-disabled). Free by default
(content-download service disabled); denies unpaid peers when enabled;
fails open on internal error so a payment fault never blocks distribution.
Wired into IrohProvider::new.
All iroh code behind the iroh-swarm feature; the default build is inert.
Default build clean; --features iroh-swarm: 11/11 swarm tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FIPS peer content browse over the mesh was failing with "Peer returned
error: 404 Not Found" and never falling back to Tor. `is_peer_allowed_path`
only allowed `/content/<id>` (item fetches) — the catalog endpoint is
exactly `/content` (no trailing slash), so it 404'd over the FIPS peer
listener. A FIPS 404 was also treated as a successful response, so the dial
never retried Tor. Fixes: allow `/content` over the mesh; add
`fips_should_fall_back()` so a FIPS 404/5xx in Auto mode falls back to Tor
(handles version-skew peers reaching a different route). Also correct the
reconnect hint text — the public anchor is TCP/8443, not UDP/8668.
Federation: deleted nodes reappeared because transitive discovery
(`merge` of a peer's advertised trusted peers) re-added any unknown DID.
Add a tombstone store (`removed-nodes.json`): remove_node tombstones the
DID, transitive merge skips tombstoned DIDs, and a remote-triggered
peer-joined is ignored for a removed DID. Explicit local re-add (add_node)
clears the tombstone.
UI: the app credentials modal panel stretched edge-to-edge (height:100%,
max-width:none, items-stretch overlay). Constrain it to a centered card
(max-width 34rem, rounded, dimmed full-screen backdrop) matching the
AppIconGrid / wallet-receive modal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.
Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
replaces fragile post-start exec that failed under restricted-cap rootless
podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition
Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
tester, every app × every transition. Run before each release.
Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After install completes, the async-spawn wrapper wrote state=Running
but the skeletal install-time manifest (interfaces: None) persisted
until the next scheduled 60s scan. The frontend saw state=running but
hasUI=false and hid the Launch button for up to a full minute.
Add a shared Notify/watch pair between RpcHandler and the scan loop:
- scan_kick (Notify): scan loop selects! between the 60s interval
and this notify, running immediately on either.
- scan_tick (watch<u64>): scan loop bumps the counter after each
completed scan so callers can await completion.
Install and update success paths now call kick_scanner_and_wait before
flipping to Running. The scan merges via merge_preserving_transitional
(state stays Installing/Updating, manifest refreshed from live podman
with interfaces.main.ui populated from real port bindings). 2s timeout
falls back to pre-fix behavior on slow podman — no regression.
The 30s package scan loop used to blindly overwrite every package
entry from podman inspect. While a user-initiated Stop / Start /
Restart was in flight, the RPC spawn task would flip the state to
Stopping / Starting / Restarting, the next scan would see podman
still reporting "running" (for the duration of the graceful stop,
up to 600s for bitcoin-core), and clobber the transitional state
back to Running. The dashboard would then flip Running -> Stopping
-> Running -> Stopped, making it look like the stop had silently
failed until it eventually completed.
The merge loop now treats transitional variants (Stopping, Starting,
Restarting, Installing, Updating, Removing, and the three backup
variants) as owned by the RPC spawn task. For those variants,
merge_preserving_transitional keeps the existing state while still
taking live observability fields (health, exit_code, installed,
lan_address, manifest, static_files, available_update) from the
fresh scan so the UI continues to see live health readings.
Adds an escape hatch via a per-scan transitional_since side table:
if a package has been in a transitional state for more than 1200s
(2x the longest graceful stop at 600s on bitcoin-core), the scan
loop assumes the spawn task died without cleanup and overrides with
podman's live state. Prevents a crashed background task from wedging
a package in Stopping forever.
Three unit tests cover the merge rule, the observability passthrough,
and the transitional-variant classifier.
Step 6 of the rust-orchestrator migration. Construct the container
orchestrator once in main.rs, call load_manifests + adopt_existing
immediately after Config::load, log the adoption report, and spawn
BootReconciler::run_forever with the 30s default interval. Thread the
orchestrator through Server::new -> ApiHandler::new -> RpcHandler::new
so the reconciler and RPC layer share one instance.
Wire a tokio::sync::Notify through the SIGTERM/SIGINT shutdown path so
the reconciler exits cleanly alongside the server drain. Uses notify_one
so the signal stores a permit if the reconciler is mid reconcile_all
when the signal fires.
Delete the commented-out run_boot_reconciliation block in main.rs that
documented the prior bash-script approach being unsafe on unbundled
installs — the new reconciler is manifest-driven and only touches apps
present in /opt/archipelago/apps, fixing that concern.
cargo check -p archipelago clean (6 pre-existing dead-code warnings on
trait methods not yet exercised until Step 9 hot-swap). Container test
suite 43/44 pass; the one failure (container::image_versions::
test_parse_image_versions) is pre-existing and unrelated.
- core/archipelago/src/bootstrap.rs (NEW): embed scripts/container-doctor.sh
and image-recipe/configs/archipelago-doctor.{service,timer} via
include_str! and sync to disk + enable the timer on every archipelago
startup. Idempotent (content-hash compare), dev-box symlink guard keeps
the git checkout untouched, best-effort (warn-only on failure) so
bootstrap never blocks server readiness. Wired in main.rs as a
background tokio task.
- scripts/container-doctor.sh: add fix_rootless_netns_egress(). Detects
when the rootless-netns has lost its pasta tap (container-to-container
still works but outbound DNS/TCP fails) via an nsenter probe into
aardvark-dns; with a two-probe 10s debounce to rule out transients and
a host-precheck that bails out if the host itself is offline. When the
rootless-netns is truly broken, does a graceful podman stop --all /
start --all so pasta + aardvark-dns rebuild the netns from scratch.
Bitcoin-knots and every other outbound container recover in one cycle.
- core/archipelago/src/update.rs: host_sudo → pub(crate) so bootstrap.rs
can reuse the existing systemd-run escape hatch.
- apps/bitcoin-core/manifest.yml: bump app version 24.0.0 → 28.4.0 and
image bitcoin/bitcoin:24.0 → bitcoin/bitcoin:28.4. Resources aligned
with the real container-specs.sh large-disk tune (4 GiB memory cap,
cpu_limit: 0 so bitcoind can run -par=auto across every core).
- neode-ui/src/views/apps/AppCard.vue + Apps.vue: add an Update button
+ Updating spinner to every app card that has available-update set.
Wires through serverStore.updatePackage(id) — the same RPC the detail
view already calls. common.update / common.updating i18n keys added in
en.json and es.json.
- core/archipelago/src/identity_manager.rs: add create_from_signing_key()
that mirrors an existing Ed25519 key as a manager-level identity with
a deterministic id (`node-<pubkey16>`). Idempotent across restarts,
gets the hex-SVG master avatar.
- core/archipelago/src/server.rs: the auto-create path on first boot now
mirrors the node's own signing_key (seed-derived on onboarded installs)
as a "Node" identity instead of generating a random "Default" keypair.
Once this ships, the DID on the Web5 DID Status card (via node.did
RPC), the Node entry on the Identities page (via identity.list), and
the DID used for peer-to-peer connects (via server_info.pubkey) all
resolve to the same seed-derived pubkey.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-adds the TCP transport (`0.0.0.0:8443`) to the rendered fips.yaml
alongside UDP. Upstream factory default enables both; we had
inadvertently narrowed to UDP-only when the yaml rewriter was last
touched, which left nodes unable to reach fips.v0l.io (the public
anchor only answers on TCP right now) or talk across networks that
block UDP.
Backend startup now compares the installed yaml against the current
rendered schema and restarts whichever fips unit is active when they
differ — so OTA-upgrading nodes pick up the new transport without
anyone having to click Reconnect.
Dropped the earlier plan to auto-add federated peers as seed anchors:
invites don't carry a FIPS-reachable IP:port, and once TCP reconnects
the public mesh, federated peers become npub-routable without needing
a seed entry.
Seed Anchors modal cleanup: replaced malformed header icon with a
three-arc broadcast glyph, and the close button now matches the
What's New modal (embedded in the card header, same icon + hover
style) instead of the earlier floating off-design placeholder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a local seed-anchor list at <data_dir>/seed-anchors.json. Each
entry is {npub, address, transport, label}. On archipelago startup
and every 5 minutes the list is pushed into the running fips daemon
via `fipsctl connect <npub> <addr> <transport>`, so a cluster can
anchor itself independently of the global fips.v0l.io. A flaky or
unreachable public anchor no longer strands a fresh install.
New RPCs:
- fips.list-seed-anchors
- fips.add-seed-anchor (validates npub1… + host:port)
- fips.remove-seed-anchor
- fips.apply-seed-anchors (on-demand re-dial)
New standalone UI card at views/server/FipsSeedAnchorsCard.vue. Not
wired into Home.vue / Server.vue — operator places it per the
entry-point convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
