Three of five reachable fleet nodes had Tor completely dead — Home said
"Connected" on all of them — and shipping the generator fix alone would
have repaired none of them. Two additions close that loop.
heal_on_boot (wired into the bootstrap repair chain): regenerate torrc
from current config with the fixed generator, and apply-and-restart ONLY
if the live file drifted or Tor is not answering on 9050. A healthy node
is left untouched. Without this, regenerate_torrc runs only from the Tor
RPC handlers and package install, so a node carrying a poisoned torrc
keeps it until someone happens to toggle a Tor setting — and worse, the
still-running OLD binary re-poisons on any such toggle: observed live on
the dev node at 07:20, when the running daemon rewrote torrc with the
unbindable gateway line hours after it had been hand-fixed. The heal
makes the fix self-applying on every restart, i.e. the OTA itself.
ServerInfo gains tor-running, populated by a real connect to
127.0.0.1:9050 each state refresh. tor-address is read from the
hidden-service hostname file, which OUTLIVES a dead daemon — it is a
configuration artifact, and the dashboard treating it as liveness is
precisely why three dead nodes showed "Connected" for days. Liveness now
comes only from the probe; the address stays a separate fact.
The heal reports the truth: it re-probes after restarting and warns if
Tor still is not answering, rather than assuming success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught verifying the gate fixes on archi-dev-box: [fips0-ULA]:32838
answered HTTP 200 straight from nbxplorer with no credential. The
catalog declares that port auth: local — host-local by intent, pinned to
loopback, the gate deliberately keeps its hands off — but the mesh relay
bridges a STATIC port list to 127.0.0.1, so it republished it to the
whole mesh. Same bug class as the Tor onion gap: a transport that
converges on the app loopback without consulting the declaration.
PortMap now records declared-local ports and the relay withholds them
(tearing down an existing bridge if a catalog refresh newly declares
one), alongside the declared-gated withhold. Undeclared ports keep
todays behaviour — silence is not an instruction in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dev-box verification of the Tor/FIPS fixes caught a pre-existing split
brain: the orchestrator publishes containers from the signed catalog's
embedded manifests (origin-wins), but the gate classified ports from the
stale disk manifests — so it externally bound nbxplorer 32838, a port
the catalog declares auth: local and pins to loopback. Reachable behind
a login, but reachable where it deliberately was not.
- build_port_map now consults the catalog overlay first, via the same
parse/validate/image-only filter the orchestrator uses (moved to
app_catalog::catalog_manifest_overlay so the two cannot diverge again).
- GatedPort carries . The gated set still includes undeclared
Session-default ports for challenge/audit, but every action that
REDIRECTS traffic — the torrc 127.0.0.2 repoint, the FIPS relay
stand-down, the Tor-upstream bind — now keys on the declaration.
- The sweep releases held claims whose port left the gated set, so a
catalog refresh that withdraws a port (gated → local/none) takes
effect without a daemon restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh relay is a raw unauthenticated forward to the app's loopback,
and whether it or the gate owned a fips0 ULA port was decided by a bind
race — the dev box happened to be safe because the gate bound first.
The relay now skips ports declared auth: gated and tears down any
existing bridge for a port that became gated since it was bridged
(catalog refresh), releasing the bind for the gate's next sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reproduced again on this node today: with no session cookie, six app
ports answered HTTP 200 with their real UIs (18083 LND, 8334, 8175
Fedimint Guardian, 8336 FIPS Mesh, 8090, 7777), all bound 0.0.0.0 and so
served on every host address. Same bug class as the /lnd-connect-info
and /bitcoin-rpc/ leaks closed in v1.7.120, but across every app.
LAN, Tailscale, Tor and the FIPS mesh all converge on 127.0.0.1:<port>,
so this is one gate rather than four. It lives in the daemon rather than
a per-app sidecar (umbrel's app_proxy model): rootless, no extra
container per app, and it can reuse machinery that already exists.
It invents no authentication policy. verify_password, TOTP secret
decryption, verify_code with used-step replay protection, the session
store, and — importantly — the SAME LoginRateLimiter instance as the
JSON-RPC path, so an attacker cannot get a fresh budget of password
guesses by moving to an app port. Only the transport differs, an HTML
form instead of JSON-RPC, because a browser being sent to an app cannot
speak JSON-RPC.
2FA comes for free: a session still pending its TOTP step fails
validate(), so the gate rejects it without knowing what a second factor
is.
Details worth keeping:
- 401, not a redirect. A redirect to a login page is indistinguishable
from the app itself redirecting, and machine clients would follow it
and parse HTML as their API response.
- Cookie and Authorization are stripped before proxying. The app has no
use for the node session and must never be able to log or forward it.
- The challenge page names and pictures the app being opened, so the
visitor can confirm what they are authenticating to.
- device_tokens grew `apps: Option<Vec<String>>` and verify_for_app for
machine clients. None = node-wide, which every existing companion
token is; migrating them by guessing a scope would silently revoke
access nobody asked to revoke. An empty list is rejected rather than
minted, since it reads as unrestricted while authorising nothing.
The rollout is necessarily per-app and the gate is built to say so. A
container publishing 0.0.0.0:<port> claims every host address, so the
gate cannot bind that port until the app is pinned to bind: 127.0.0.1
and recreated — gate-first is impossible, and all-at-once would recreate
every container on a node simultaneously. Every port it cannot claim is
logged at warn each sweep and recorded in GateStatus::unprotected,
surfaced by security.app-gate-status. The failure mode being designed
against is a gate that binds nothing, logs at debug, and reports success
while every app stays exactly as open as before — worse than no gate,
because it stops anyone looking. Same reasoning that ruled out an
nft drop-in, whose absence is a silent no-op.
Not yet done: pinning the 39 gated ports to loopback, repointing
HiddenServicePort at the gate, and on-node verification.
Tests: 21/21 appgate, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes F-10a. Nothing here fixes a present defect: on the pinned rand
0.8.5, rand::random() and thread_rng() both resolve to a ChaCha12 CSPRNG
seeded from getrandom(2). What they lack is a STATED backend — it is
fixed by dependency and build configuration rather than by the calling
code, with no compile error if that changes. That is the structural
shape behind the 2026-07-30 COLDCARD entropy defect, and here the blast
radius includes Cashu blinded-key-exchange values, X3DH prekey material,
session bearer tokens and a ChaCha20-Poly1305 nonce.
Layer (a) — every production key, nonce and token draw now names
rand::rngs::OsRng at its own call site. The mnemonic seam is bound to
entropy::KeyGenRng, a SEALED allowlist whose supertrait lives in a
private module, so the set of RNGs that can drive the master key
hierarchy is exactly what one file says it is. This retires the false
promise at seed.rs:656: rand::CryptoRng is a marker with no
compiler-checked content, and the crate now contains zero impls of it.
Layer (d) — key material and AEAD nonces of >=12 bytes run a
degenerate-entropy predicate that refuses all-zero, all-identical and
wrapping +/-1 counter draws. Nothing heuristic: no entropy estimator, no
chi-squared. Each of the three shapes has a false-positive probability
computable in closed form (3 * 2^-88 at 12 bytes, 3 * 2^-248 at 32), and
a predicate whose false-positive rate cannot be computed cannot be
argued safe on a key-generation path. There is deliberately no retry — a
retry would paper over the broken RNG this exists to surface.
Layer (e) — the kernel-CSPRNG readiness verdict at master-seed
generation is now durable (backlog R-09). It was previously computed,
logged and thrown away, so a node could never answer after the fact
whether its keys were born from a seeded pool. The record holds a schema
version, timestamp, verdict and event name — no entropy, no key bytes.
Formats and wire shapes are proven unchanged rather than asserted:
storage_crypto and the credential store each open a HARDCODED
pre-migration ciphertext vector (a same-process round trip would pass
even if the envelope had changed), the vector was produced by an
independent RFC 8439 implementation so it pins the documented
nonce||ciphertext format rather than this implementation's output, and
the x3dh prekey bundle and bdhke values keep their field set and order.
totp.rs migrates its SOURCE only: the % charset.len() reduction and the
32-char charset are untouched. The bias there is presently zero (32
divides 256) and fixing the latent bias is R-12, which stays deferred.
Verified: cargo build clean; cargo test -p archipelago 1068 passed,
2 failed. Both failures are container::boot_reconciler timing tests
(second_pass_fires_after_interval, shutdown_terminates_loop) in a file
this change does not touch — pre-existing, not caused here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two near-identical periodic federation sync loops were running side by
side. git history shows the overlap was accidental, not load-bearing: the
30-minute loop landed first (8dd57bcb, 2026-04-19), and the 90s loop
landed later (837cc028, 2026-06-19) describing itself as "new 90s
periodic federation auto-sync (none existed)" — its author simply hadn't
seen the existing one. Running both doubled the write-race exposure
against nodes.json that plan 01-01 locked down.
The 90s loop survives; it already did strictly more (per-peer sync-result
recording, asymmetry self-heal). The deleted loop's one unique behavior —
refresh_federation_mesh_peers() after a completed pass (#42), which pushes
newly-learned names/roster into the live mesh peer table so chat contacts
refresh without a restart — is preserved at the tail of the survivor. That
call is a local, idempotent re-seed from nodes.json with no network I/O,
so running it per-pass rather than per-half-hour is cheap.
Also carried over: MissedTickBehavior::Delay, so a pass delayed by suspend
or heavy load resumes the cadence instead of firing a burst of catch-up
ticks. And node-load errors are now logged and skipped explicitly rather
than swallowed by a catch-all, so an empty roster and an unreadable one
are no longer indistinguishable.
Not carried over: the deleted loop's 5s per-peer stagger. Its stated
reason was avoiding concurrent connects against the Tor SOCKS proxy, but
both loops iterate peers sequentially and await each sync, so there were
never concurrent connects to stagger; keeping it would only push a
multi-peer pass past the 90s cadence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A failed federation sync existed only as a `debug!` line on the node, so a
peer that had not synced in days looked identical in the UI to one that
synced a minute ago. Now the failure is persisted per peer and rendered.
- `FederatedNode.last_sync_error` / `.last_sync_error_at` — the failure-side
mirror of the existing `last_transport` / `last_transport_at` pair.
- `federation::record_sync_result(data_dir, did, outcome)` — records the
message on `Err`, CLEARS both fields on `Ok` so the badge disappears when
the peer recovers. Runs under FEDERATION_STORE_LOCK via the `*_inner`
load/save convention established by plan 01-01. An unknown DID is a silent
Ok that writes nothing, so a peer removed mid-pass is never resurrected by
an in-flight sync's error write. Skips the save entirely when nothing
changed, keeping the steady state read-only rather than rewriting
nodes.json (and contending for the lock) every 90s.
- Message truncated to MAX_SYNC_ERROR_CHARS (256), counted in chars not
bytes so truncation cannot split a UTF-8 sequence (T-01-18).
- The 90s auto-sync loop calls it on both arms; the existing `debug!` line
is kept — persisting is additive, not a replacement for logs.
- `federation.list-nodes` emits both fields when set, omits them when unset.
- NodeList renders a red SYNC badge beside the transport badge on both the
trusted-node and peer rows, message + age in the `title` so the row stays
single-line.
Tests (written first, confirmed failing — 16 compile errors, E0425 on
`record_sync_result` and E0609 on `last_sync_error`):
- persists_error / success_clears_error / missing_did_is_noop /
on_empty_store_is_noop / truncates_long_error
- NodeList: badge present when set, ABSENT when unset (the guard against a
badge that always renders), and present on an observer peer row.
cargo test -p archipelago federation — 42 passed, 0 failed.
vitest NodeList.test.ts — 4 passed. npm run build — green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Root causes found and fixed after live debugging on archi-dev-box (all
verified on real RNode hardware, archi-dev-box <-> archy-x250-dev E2E):
- probe_rnode raced the board's own boot: opening the port pulses DTR/RTS
through USB-UART bridges (CP2102/Heltec V3), the ESP32 power-cycles and
spends ~2.5-3s in boot ROM, and the KISS DETECT written 300ms after open
landed in the void — so an RNode could NEVER connect on these boards.
Now: immediate probe (fast path), then drain-until-quiet boot settle and
a second DETECT with a fresh response window.
- MeshService::configure() only restarted the listener on enable/disable —
device_kind/device_path/advert_name/RF-param changes were silent no-ops
until a full process restart (the setup modal's apply/keep-as-is did
nothing). Material config changes now bounce the listener; the open
sequence races the shutdown signal so stop() no longer burns the full
15s timeout mid-probe; mesh.configure applies in the background instead
of stalling every status poll behind the service write-lock.
- The mesh name was write-only: config.advert_name had no reader,
server.set-name never reached the mesh service, and Reticulum's
set_advert_name was a no-op (daemon display name fixed at spawn, and the
ARCHY:2 announce blob REPLACED the LXMF display name — every archy node
was anonymous on RNS). Now: advert_name > server name precedence feeds
the session, renames restart it live, the daemon gets --display-name at
spawn plus a set_name RPC verb, and announces carry the LXMF-standard
msgpack name with the identity blob appended as an extra list element
stock clients (Sideband/NomadNet) ignore.
- Dead reticulum-daemon was invisible for up to 30min (RX-stall watchdog):
child exit / RPC-EOF now fails try_recv_frame so the session reconnects.
- Setup modal re-trigger loop: plugged_at used the tty node's mtime, which
bumps on every open — each probe invalidated the dismissal key. Use
btime/ctime (only change on real plugs).
- ARCHY:2 identity adverts (re-emitted every 60s over Reticulum) stomped
the federation twin's real name with a synthetic Archy-… placeholder and
nulled its position; blob-only announces no longer assert a name, blob
strings can never become display names, and stale blob names are healed
at peers.json load.
- mesh.broadcast on Meshtastic sent heartbeat+time only (no identity);
SendAdvert now also fires a want_response NodeInfo broadcast.
- New mesh.refresh RPC: actively re-queries the radio contact table (the
UI Refresh button previously only re-read server caches).
- Reticulum peers now track last_advert (announce time) and mark existing
peers reachable on inbound traffic.
- Boot auto-enable no longer force-enables mesh when an operator
explicitly disabled it (only fires when no config file exists).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HTTP accept loop parked on acquire_owned() when the connection
budget drained, freezing accept() for every client (the .228
session-flapping / CLOSE-WAIT signature). Permits drained because
half-open clients and hung upstreams held them indefinitely.
- try_acquire_owned + immediate 503-and-close when the budget is
exhausted; the accept loop itself never blocks
- 30s http1_header_read_timeout drops slowloris/half-open clients
- 900s watchdog bounds non-upgraded connections; websocket upgrades
are exempt (legitimately long-lived)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New fips/endpoints.rs: an npub-keyed store (<data_dir>/fips-endpoints.json)
of every endpoint a peer was last seen connected at (fipsctl show peers
transport_addr/transport_type — covers LAN, Tailscale, and WAN alike),
refreshed each anchor tick, 30-day retention.
The anchor tick now escalates LAN → last-known-good → anchor tree: any
federation peer with a fips npub that is neither currently connected nor
covered by a live LAN direct entry gets its last-known-good endpoint
re-dialed (idempotent fipsctl connect, bounded by apply()'s per-connect
cap). This productizes the hand-applied .116↔.198 Tailscale fix of
2026-07-20 and closes RC2's "no endpoint fallback" gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase A3 of docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md (RC5), measured against
the A2 dial_stats baseline:
- 25s connectivity watcher in the fips supervisor: re-applies seed anchors
immediately on an anchor-link drop, on startup-disconnected, AND on
silent data-path death (connect_fails growing with zero fips_ok — the
live .198 failure where the daemon reported "connected" while every
dial blackholed and the 300s tick never healed it). Bounded to one
re-apply per 60s.
- anchors::apply is now concurrent with a 15s per-connect cap — the old
serial loop waited unbounded on each `sudo fipsctl connect`, so one
hung subprocess stalled the whole periodic tick.
- rebindable peer listener: the accept loop returns after persistent
accept errors (was: continue forever = inbound-dead until restart) and
peer_late_bind_loop rebinds — also on fips0 ULA change.
- is_service_active gets a 10s TTL cache (was up to 2 systemctl spawns
per dial attempt and per warm-tick peer).
- the warm tick now warms the union of federation peers + configured
seed anchors (direct anchor links used to go cold between 300s ticks),
skipping the redundant per-peer service check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase A1 of docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md — the five changes that
made FIPS fall back to Tor even when a FIPS path existed:
- RC0: the fips.d drop-in now opens PEER_PORT 5679 (was 80+8443 only, so
every hardened node firewalled peers' FIPS dials; 28k drops on .198)
- RC4: /blob/ and /dwn/ added to the peer-path allowlist — mesh file
sharing and DWN sync were 404 → 100% Tor by construction
- RC2-G2: lan_fips_anchors dials PUBLISHED_UDP_PORT (2121) instead of the
dead 8668, with a drift-guard test against the rendered daemon config
- RC2-G1: direct LAN peering actually runs now — mDNS TXT advertises the
FIPS npub, discovery calls set_fips_npub, and the anchor tick hydrates
npubs from federation storage for peers on older builds
- RC3: FIPS attempt budget is a hard cap (retry no longer doubles it) and
the 12 hot call sites get explicit fips_timeout fast-fail so Tor keeps
its full budget (browse-peer, preview, /blob, DWN, node-message,
rotation notifies)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v6 app-port relay preemptively bound [::]:<port> for ALL catalog
ports, even apps not installed. Installing such an app (grafana:3000,
photoprism, uptime-kuma, jellyfin — framework-pt 2026-07-27) then hit
'address already in use' from the relay, which triggered
cleanup_stale_pasta_port -> -> killed archipelago
itself (it held the port) mid-install. The daemon crash-looped and the
half-created apps were rolled back and vanished.
Two fixes:
- relay only bridges a port that a running app already answers on over
IPv4 (probe 127.0.0.1:port first) — an uninstalled app's port is never
held, so its install sees a free port and never triggers the cleanup.
- cleanup_stale_pasta_port excludes our own PID from both the ss-based
kill and the fuser kill, so freeing a port can never terminate the
daemon even when the relay legitimately holds it (reinstall case).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rootless podman's wildcard publish claims [::] but BLACK-HOLES inbound
v6 (accepts, forwards nothing — vaultwarden 'empty response' from the
phone, 2026-07-26), while most apps got no v6 listener at all. Two
halves: quadlets now publish unbound ports on 0.0.0.0 explicitly
(frees the v6 side; the one-time drift/recreate at upgrade is the
deploy vehicle), and the daemon runs a self-selecting V6ONLY relay on
every catalog launch port forwarding raw TCP to the v4 loopback
listener. Rescans every 60s so new installs bridge without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh is IPv6-only; the main web listener bound 0.0.0.0 only, so a
phone reaching a node over its fips0 ULA got RST on :80
(ERR_CONNECTION_ABORTED) — UI over mesh was structurally impossible.
Confirmed 2026-07-26 on framework-pt: v4:80 = 200, v6:80 = refused.
Mirror an IPv4-any main listener with a V6ONLY [::] socket on the same
port; V6ONLY so it coexists with the v4 listener regardless of
net.ipv6.bindv6only.
Also: fips.yaml generator carries the fast-reconnect profile (validated
live on framework-pt since 2026-07-24; snapshot tests updated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- LND watchdog: framework-pt's LND sat wedged 14h (RPC up, server never
ready, channels inactive) with nothing noticing. 15 consecutive bad
minutes now bounce the container automatically (30min cooldown) —
silent multi-hour Lightning outages become self-healed blips.
- Send modal: Auto tab hidden; Ecash splits into Cashu and Fedi (new
wallet.fedimint-send RPC wrapping the existing spend_from_any); both
rails render the generated token as a scannable QR alongside the
copyable text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three silent failure modes killed nostr peer requests (analyzed from the
live code 2026-07-22): (1) nothing ever polled the relays except the
manual Federation Poll button — a 5-minute background poll now fetches
inbound requests and nudges the websocket when something lands (the
handler's discoverability gate still applies); (2) handshake send/poll
used only the two hardcoded config relays (one defunct) and ignored the
user-managed relay list — every nostr op now uses the merged set;
(3) publish errors were swallowed (let _ =) so 'ok' was reported even
when no relay accepted the event — now a delivery failure is an error
the UI shows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend streams LND /v1/transactions/subscribe (fires on mempool arrival
and each confirmation) and nudges the /ws/db revision per event; the Home
wallet card subscribes and refetches (debounced 800ms). An incoming
broadcast now appears while the 30s poll is still asleep. Reconnects
forever with capped backoff when LND is down/locked/absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fleet fixes that OTA can deliver (unlike the ISO-only polkit rule):
- WiFi (issue #99): nodes drive NetworkManager from a system-level service
with no logind seat, so the stock polkit rule denies them and Wi-Fi setup
fails with 'Insufficient privileges'. Fresh ISOs since 2026-05 ship the
scoped rule, but OTA'd nodes never got it (OTA replaces binary + web UI,
not host system config). Add a startup self-heal that installs the rule if
missing and best-effort ensures polkitd — both wrapped so an offline/locked
apt can never fail startup; the rule is written regardless so it activates
once polkitd lands. Idempotent on the rule's unique subject.user marker.
- FIPS: regenerating fips.yaml (this build does, once, on first boot after
the OTA) restarts the fips daemon; for a few seconds /run/fips/control.sock
is gone and every seed-anchor dial fails, islanding the node until the next
5-min tick. Detect that exact failure (all dials fail on control.sock) and
retry in 15s instead — bounded to 8 fast retries/episode so a node with no
fips daemon falls back to the steady cadence rather than busy-looping.
FIPS unit suite: 33/33 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Invites carried no trust level, and both accept_invite and the
peer-joined handler hardcoded TrustLevel::Trusted — so 'Invite a Peer'
(Observer) federated both sides as fully Trusted.
- invite codes now carry a 'trust' field (legacy codes parse as Trusted)
- federation.invite accepts trust_level: trusted|observer
- accept_invite assigns the invite's level on the acceptor side
- peer-joined resolves the granted level authoritatively by matching the
acceptor's echoed invite token against our stored outgoing invites;
the peer's own unsigned claim is honored only as a downgrade, so no
escalation is possible
- discovery/connection-request approvals mint Observer invites directly
(the post-hoc demotion in handshake.rs stays as a legacy safety net)
- asymmetry self-heal re-asserts at the locally-held level
Tests: observer threading + legacy default-trusted parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Formal gate take-2 went 4/5: iteration 5's vaultwarden stop-wait saw
'restarting' for the full 120s window. A reconcile pass had queued every
app into pending_boot_starts moments before the stop marker landed, and
the scanner's overlay painted the deliberately-stopped app Restarting
until the slow sequential pass reached it (legacy container, so no unit
state to contradict it).
Two filters: the scanner overlay skips user-stopped ids (the marker is
written before the stop runs, so it's reliably present), and the
reconciler no longer queues apps whose lifecycle op is in flight into the
pending overlay (it skips them via the same probe anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One legacy app per gate run stuck at 'stopping'/'starting' past the test
window while its container was already settled (vaultwarden:stop run C,
jellyfin:stop run D, uptime-kuma:start run E — .228, 2026-07-09). The
package scanner already sees the truth every 60s but
merge_preserving_transitional refused to report it until the RPC worker
wrote the final state, and the workers can legitimately trail the
container by minutes:
- stop workers queue behind the orchestrator per-app lock, which the
reconcile host-port repair path holds through multi-minute stability
waits (repair_manifest_host_ports_after_stability: 5s + 5s probe +
restart + 60-420s port wait + 15-90s stable-running);
- start workers hold Starting through the full readiness wait —
host_port_wait_timeout_secs is 420s for uptime-kuma (HTTP probe),
longer than any UI/test patience — and the Starting stuck-timeout is
the 20-minute INSTALLING one.
New merge rules, both truth-driven by podman's live view:
- (Stopping, Stopped) + user-stop marker → Stopped: the scanner only
normalizes exited→Stopped for user-stopped apps, so this is exactly
"the user asked for a stop and the container has exited".
- (Starting, Running) → Running: the start visibly succeeded; live
health readings are merged separately and keep reporting readiness.
Restarting is deliberately NOT resolved by a Running scan — mid-restart
readings are the pre-stop container.
Tests: merge_tests 13/13 (4 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tollgate-wrt keeps its own separate Cashu wallet on the router
(/etc/tollgate/wallet.db) — customer payments land there, never in
this node's own wallet. Its Lightning auto-payout is configured
independently in /etc/tollgate/identities.json, which is easy to
leave misconfigured or pointed at a placeholder address (found live:
both "owner" and "developer" identities on this deployment pointed at
the same unconfigured tollgate@minibits.cash default).
Rather than depend on getting that Lightning payout config right,
tollgate_sweep::sweep_once() periodically (every 5 min, via SSH)
checks the router's `tollgate wallet balance`, and if nonzero, runs
`tollgate wallet drain cashu` and receives the resulting token(s)
straight into the local wallet via the same path the "Receive ecash"
UI uses (wallet::ecash::receive_token) — bypassing Lightning payout
entirely. A no-op (Ok(0)) if no router is configured or it doesn't
have TollGate installed.
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers that opt in via a new "Share Location" toggle in Settings
(server.set-location RPC) get plotted on other trusted peers' Mesh Map
with a distinct Archy-logo marker, separate from raw LoRa radio peers.
Location is persisted locally, carried in NodeStateSnapshot, and
propagated through federation sync/delta like other node state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mesh/federation messages between co-located nodes were always falling back to
Tor because the FIPS overlay had no direct peering — every node depended on the
global anchor's spanning tree, and when that anchor link flaps a node is
isolated and all FIPS dials time out. (Diagnosed live on .116/.198: pure-FIPS
direct peering over UDP 8668 fixes it — 2.5ms vs timeout.)
Generalize the manual fix: in the existing 5-min FIPS seed-anchor apply loop,
also auto-connect every federation peer the PeerRegistry knows both a LAN
address AND a FIPS npub for, dialing its FIPS UDP transport (port 8668) at its
LAN IP via the same idempotent `fipsctl connect` path (new
anchors::lan_fips_anchors). This is FIPS's own transport over the LAN — NOT
Tailscale, NOT the HTTP/LAN messaging port. Transient (recomputed each tick from
live mDNS discovery, never persisted) so changing IPs self-correct. Remote peers
with no LAN address are untouched (still routed via the anchor).
Registry Arc hoisted out of the transport-init block so the loop can read
all_peers(). cargo check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Federated nodes failed to converge to full-mesh across the LAN<->Tailscale
boundary: nodes were invisible to peers, sync 'took ages'/timed out, and
names only updated on a manual sync. Onions were healthy in both directions
(~3-5s); the failures were app-layer.
- B: federation dials fast-fail a dead FIPS path via .fips_timeout(6s) in
sync_with_peer + notify_join, so the Tor fallback isn't stuck behind the
full 30s FIPS budget when LAN and remote peers share no FIPS path.
- A: notify_join (peer-joined) now spawns with retries+backoff instead of a
single awaited best-effort POST, so the join RPC returns instantly (no
'Request timeout') and the inviter reliably learns the joiner (was
asymmetric).
- C: new 90s periodic federation auto-sync (none existed) so renamed nodes
and roster changes propagate without a manual Sync click.
- self-heal: each auto-sync re-asserts membership to any peer that doesn't
list us back, converging the fleet to full-mesh and healing pre-existing
asymmetry with no manual re-joins.
Validated live across 7 nodes: a previously fleet-invisible node became
fully meshed automatically (logs: 'auto-sync ... reasserted=1',
'peer-joined ... delivered').
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mesh: stop broadcasting ARCHY:2 identity on the public channel (startup + every advert tick); receive path still parses inbound. No more public-channel spam.
- mesh assistant: trigger on !ai/!ask typed in 1:1 chat (was only the dead AssistQuery path + bare channel text); route the reply transport-aware via MeshService::send_message (Tor for federation peers, LoRa for radio) through a new AssistChatReply event consumed at the server layer — fixes replies never reaching federation askers.
- mesh assistant: per-contact !ai allowlist (allowed_contacts) bypassing trusted_only; config + RPC + is_sender_allowed.
- fedimint-clientd manifest: network_policy open -> bridge (invalid value made the loader skip the whole manifest, so fmcd never ran and federations never joined/listed).
- ui: AI panel — Claude model dropdown (Haiku/Sonnet/Opus presets) + allowlist contact picker.
- ui: Settings — App Updates + App Registry moved under Account.
- ui: mesh chat — overscroll-behavior: contain so chat scroll no longer bleeds to the contacts panel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A peer accepted via invite is seeded into the mesh peer table with
name=None, so it shows as "Archipelago <pubkey8>" in chat. Federation
sync later learns the real name (update_node_state writes it to
nodes.json) and discovers transitive peers (merge_transitive_peers),
but nothing pushed those into the live mesh peer table — the chat list
stayed stale until the next mesh restart, and transitive peers never
appeared as contacts at all.
Add RpcHandler::refresh_federation_mesh_peers() (re-runs the idempotent,
onion-deduped seed_federation_peers_into_mesh) and call it after every
periodic sync cycle (server.rs) and after the manual federation.sync-all
RPC. Names now correct themselves and the full roster meshes within a
sync cycle, no restart needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 wiring (task #12):
- NostrSeedDiscovery: async ProviderDiscovery that queries relays for signed
seed adverts and parses endpoint ids (swarm/iroh_provider.rs, seed_advert.rs).
- seed_and_advertise publish path; dep-free fetch/publish helpers reuse the
node's Nostr identity (build_nostr_client/load_or_create_nostr_keys made
pub(crate)).
- swarm::init builds the IrohProvider once into a OnceLock runtime; providers()
returns it; announce_held_blob() is called from update.rs after a release
component passes both hash gates.
- config swarm_enabled (ARCHIPELAGO_SWARM_ENABLED, default off); server.rs init.
Paid swarm serving (Phase 4 step F):
- swarm/paid.rs gates the iroh-blobs provider through streaming::gate,
intercepting connect + GET (peer push hard-disabled). Free by default
(content-download service disabled); denies unpaid peers when enabled;
fails open on internal error so a payment fault never blocks distribution.
Wired into IrohProvider::new.
All iroh code behind the iroh-swarm feature; the default build is inert.
Default build clean; --features iroh-swarm: 11/11 swarm tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FIPS peer content browse over the mesh was failing with "Peer returned
error: 404 Not Found" and never falling back to Tor. `is_peer_allowed_path`
only allowed `/content/<id>` (item fetches) — the catalog endpoint is
exactly `/content` (no trailing slash), so it 404'd over the FIPS peer
listener. A FIPS 404 was also treated as a successful response, so the dial
never retried Tor. Fixes: allow `/content` over the mesh; add
`fips_should_fall_back()` so a FIPS 404/5xx in Auto mode falls back to Tor
(handles version-skew peers reaching a different route). Also correct the
reconnect hint text — the public anchor is TCP/8443, not UDP/8668.
Federation: deleted nodes reappeared because transitive discovery
(`merge` of a peer's advertised trusted peers) re-added any unknown DID.
Add a tombstone store (`removed-nodes.json`): remove_node tombstones the
DID, transitive merge skips tombstoned DIDs, and a remote-triggered
peer-joined is ignored for a removed DID. Explicit local re-add (add_node)
clears the tombstone.
UI: the app credentials modal panel stretched edge-to-edge (height:100%,
max-width:none, items-stretch overlay). Constrain it to a centered card
(max-width 34rem, rounded, dimmed full-screen backdrop) matching the
AppIconGrid / wallet-receive modal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
