lfg2025/archy
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects Releases 61 Wiki Activity
archy/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md

18 KiB
Raw Permalink Blame History

FIPS near-100% uptime + optimistic UI state — implementation plan

Date: 2026-07-27. Status: researched + root-caused live on the fleet; ready to implement for the next release. Two workstreams: (A) make node↔node FIPS transport succeed whenever a FIPS path physically exists, (B) stop the UI reloading everything on every navigation (optimistic/cached cards, stale-while-revalidate) while keeping data fresh.

Honesty note on "100%": if a node's network blackholes every anchor (the .116 WiFi case, docs/HANDOFF-2026-07-20-fips-peer-files.md:117-133), Tor fallback is correct. The achievable target is: FIPS wins whenever a FIPS path exists, and fallback frequency is measured in-product so regressions are visible. Today several paths are 0% FIPS by construction regardless of network health — that's the bug.

Part A — why Cloud/FIPS "commonly falls back to Tor": ranked root causes

All verified live on 2026-07-27 (.116 local, .198, .228, Framework PT, x250s) plus a full code audit of core/archipelago/src/{fips,transport,federation,server.rs}.

RC0 — 🔥 The hardening firewall drops the peer-API port on every hardened node (PROVEN)

The fips0 default-deny baseline (/etc/fips/fips.nft) is opened by archipelago's drop-in 80-web-ui.nft (fips/config.rs:236-255) for 80 + 8443 + app ports only. The peer-API listener — which carries all federation sync, cloud browse/download, mesh envelopes, DWN, invoices — is PEER_PORT = 5679 (fips/dial.rs:35). 5679 is not in the allowlist. The drop-in's own comment claims "web UI + peer API" but the peer API port was never added.

Live proof (2026-07-27):

  • .116 nft chain: 5,965 dropped packets; .198: 28,670 dropped packets — that's peers' FIPS dials dying at the firewall.
  • .198 → .116 GET :5679/health: timeout (6s) before; HTTP 200 in 0.35s after nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept. Same result in reverse direction (200 in 0.64s).
  • Explains the exact fleet split in federation/nodes.json: hardened-baseline nodes (Framework PT, .198, .228, x250-dev, x250-mad2) = last_transport: tor; non-hardened nodes (Austin Sapien, X250-Beta, X250-PA) answer :5679 (404 from the path allowlist = listener reachable) = last_transport: fips.
  • Every dial to a hardened peer pays the 8s FIPS connect timeout (dial.rs:114) ×2 (retry, dial.rs:128-140) → then Tor. That's the "Cloud takes forever / shows Tor" experience.

Fix (one line + reload): add tcp dport 5679 accept to the drop-in in fips/config.rs (use a constant shared with dial.rs::PEER_PORT, not a literal). The drop-in reinstalls on every daemon config install, so it heals fleet-wide on OTA. ⚠️ Transient manual rules were inserted on .116 and .198 during diagnosis (2026-07-27) — they vanish on the next nft -f /etc/fips/fips.nft reload or reboot; the code fix makes them permanent.

RC1 — .228 (Shorty's) runs fips 0.3.0-dev; the 0.4.1 fleet can't reach it

.228's daemon: 0.3.0-dev (rev 34e00b9f6e), both anchor links "connected", but its ULA is 100% unreachable from 0.4.1 nodes (ping loss 100%). FIPS wire format is not stable across revs (docs/HANDOFF-2026-07-23-companion-apk-deploy.md:78). Everything to/from .228 rides Tor no matter what else we fix.

Fix: fleet fips-version audit + upgrade to v0.4.1 everywhere (in-product updater exists: fips/update.rs; .deb path per reference_vps2_fips_anchor). Add a version check to fips.status and surface a "peer daemon outdated" warning.

RC2 — Direct LAN/endpoint peering is dead code + wrong port + stale seed anchors

Without direct links, all peer traffic hairpins through the vps2 anchor spanning tree (observed: .116→.198 cold RTT 1.5–3.5s on the same LAN; also the wedged-anchor latency-rot incident, HANDOFF-2026-07-23:141-160).

  • G1 — lan_fips_anchors() has never run. It needs PeerRecord.fips_npub, but PeerRegistry::set_fips_npub (transport/mod.rs:302) has zero callers — mDNS TXT records only carry did/pubkey/version (transport/lan.rs:50-54). So the "co-located peers form a direct link" feature (anchors.rs:294-305, server.rs:761-766) is a fleet-wide no-op.
  • G2 — wrong UDP port. anchors.rs:293 dials 8668, but the generated fips.yaml binds UDP 2121 (fips/config.rs:187, fips/mod.rs:130). Even if G1 ran, it would dial a dead port. .116's live seed-anchors.json still carries .198@192.168.1.198:8668 — stale IP (LAN renumbered to 192.168.63.x) AND dead port; both manual entries are useless today.
  • No Tailscale/alternate endpoint fallback when LAN is unreachable (the .116↔.198 fix of 2026-07-20 was hand-applied per-node config, never productized).

Fix: (a) FIPS_UDP_PORT → crate::fips::PUBLISHED_UDP_PORT + drift-guard test; (b) hydrate fips_npub into the registry from federation storage (did-keyed join) so lan_fips_anchors goes live with no wire change; (c) advertise the npub in the mDNS TXT + set_fips_npub on resolve as the proper fix; (d) teach the LAN-anchor tick to also try a peer's Tailscale/last-known-good endpoint when LAN fails (reviewed change — this area got handoffs wrong twice, per memory).

RC3 — No fast-fail on the hottest call sites; retry silently doubles every budget

  • content.browse-peer — the Cloud page — has NO fips_timeout (api/rpc/content.rs:363-366): a cold FIPS path burns up to ~16.6s (8s connect + 600ms + 8s retry) before Tor even starts, against a UI deadline of 30s (Cloud.vue:720) — and the frontend then retries ×3. Users see errors, not fallback. 12 call sites total lack fips_timeout (browse/download/preview-peer, /blob, DWN, node_message, rotation notifies).
  • dial.rs:128-140 runs 2 full-budget attempts, so fips_timeout(6s) really means ~12.6s everywhere.

Fix: wrap send_with_retry in a single tokio::time::timeout(fips_attempt_timeout()) (call sites dial.rs:455, dial.rs:488; halve per-attempt client timeout), then add .fips_timeout(...): content.rs:366 (6s), content.rs:281 (8s), content.rs:1139 (6s), typed_messages.rs:822 (8s), dwn_sync.rs:188/213/272 (6s), node_message.rs:376 (8s), node_message.rs:412 (4s), tor/mod.rs:501 (6s), federation/handlers.rs:869 (6s). Skip the three 900s streaming downloads (content.rs:552/870/1061, proxy.rs:236) — dial.rs:311-319 documents why; the retry-budget wrap covers their connect phase.

RC4 — Two features are 100% Tor by construction (allowlist 404)

The peer listener path allowlist (server.rs:1219-1239) omits /blob/<cid> (mesh file sharing, typed_messages.rs:813-822) and /dwn/health (step 1 of DWN sync, dwn_sync.rs:186) → deterministic 404 over FIPS (dial.rs:44-46 treats 404 as fall-back) → deterministic Tor, after paying the full FIPS cost. Both endpoints are already cryptographically gated, so they meet the allowlist's stated criterion.

Fix: add || path.starts_with("/blob/") || path.starts_with("/dwn/"); extend the existing test block at server.rs:1935-1945 (assert /blob/abc + /dwn/health allowed, /blobber + /dwnx denied).

RC5 — Inbound listener can't heal; anchor flap = 5-minute Tor window; probe overhead

  • peer_late_bind_loop returns after first successful bind (server.rs:1203) and accept_loop continues on errors forever (server.rs:1249-1258): a fips0 teardown/re-key leaves the node inbound-dead until process restart → every peer falls back to Tor against it.
  • Nothing reacts to anchor-link drops: anchors re-apply only on the 300s tick (server.rs:731); worst-case 5min Tor-only after a flap (the historic "link dead timeout 30s" flapping made this chronic).
  • is_service_active() spawns up to 2 systemctl per FIPS attempt and per peer per 25s warm tick (dial.rs:284-294); warm_path skips peers without fips_npub in federation storage (fips/mod.rs:88-95); anchors::apply is serial with unbounded subprocess waits (anchors.rs:234-283).

Fix: rebindable listener; a ~25s connectivity watcher (reuse service::peer_connectivity_summary, fips/service.rs:178-207) that re-applies anchors immediately on a connected→disconnected edge with bounded backoff; 10s TTL cache for is_service_active (mirror transport/fips.rs:24-107); warm the union of federation+registry peers; make apply() concurrent with per-connect timeouts.

RC6 — Zero observability: fallbacks are invisible, so "uptime" is unfalsifiable

Fallbacks log at debug! only (dial.rs:458,491); no counters; last_transport is written by only 7 of ~20 call sites and never read to influence anything (storage.rs:120-147). The parallel TransportRouter system can't even see FIPS (FipsTransport is never constructed — server.rs:422-442 registers Tor/Mesh/LAN only).

Fix: per-reason fallback counters (F1 no-npub / F2 service-inactive / F3 DNS-fail / F4 connect-fail / F5 404 / F6 5xx) surfaced in fips.status + info! logs with a reason field; call record_peer_transport from all peer-dial sites; UI: per-peer transport badge on Cloud (the response already carries transport — content.rs:392-400 — Cloud.vue currently throws it away at :716-721).

Part A — execution phases

Phase A0 — fleet triage (no release needed; do first, validates everything)

  1. Fleet audit: fipsctl --version + nft list table inet fips + ss -tlnp | grep 5679 on every node (roster: reference_test_deploy_roster).
  2. Transient nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept on hardened nodes (already done on .116 + .198, 2026-07-27) — instant fleet-wide FIPS recovery while the code fix rides the OTA.
  3. Upgrade .228 (and any other 0.3.x) fips daemon to v0.4.1.
  4. Regenerate/clean stale seed-anchors.json on .116 (dead 192.168.1.x + :8668 entries).
  5. Baseline measurement: for each node pair, content.browse-peer time + transport.

Phase A1 — P0 code (one commit, mechanical, offline-testable)

  1. nft drop-in: open 5679 — fips/config.rs (share the constant with dial.rs::PEER_PORT). ← RC0
  2. Allowlist /blob/, /dwn/ — server.rs:1219-1239 + tests. ← RC4
  3. FIPS_UDP_PORT = PUBLISHED_UDP_PORT (2121) — anchors.rs:293 + drift-guard test against render_config_yaml(). ← RC2-G2
  4. Un-deaden lan_fips_anchors — hydrate fips_npub from federation storage in server.rs:761-766; then mDNS TXT fips key + set_fips_npub (transport/lan.rs:50-54, lan.rs:96-108, LanTransport::new 4th arg via crate::identity::fips_npub(&data_dir.join("identity"))). ← RC2-G1
  5. Retry-budget wrap + fips_timeout on 12 call sites (list in RC3). ← RC3 Verify: cd core && cargo test -p archipelago — watch test_rendered_yaml_exact_snapshot (config.rs:419) + test_render_is_deterministic (config.rs:476); item 3 must not change rendered output.

Phase A2 — telemetry BEFORE tuning (second commit)

  1. Fallback counters by reason + fips.status exposure + info! reason logs; record_peer_transport from all sites. ← RC6 (gives the baseline that makes A3 measurable and "100%" falsifiable)

Phase A3 — resilience (third commit, measured against A2 baseline)

  1. is_service_active 10s TTL cache; warm-path union + warm_path_unchecked.
  2. Link-state watcher → immediate anchor re-apply on drop (replaces waiting for the 300s tick); concurrent apply() with subprocess timeouts.
  3. Rebindable peer listener (server.rs:1203, 1249-1258).
  4. (Reviewed, separate PR) endpoint-fallback for direct peering: LAN → Tailscale → last-known-good, npub-keyed. Mesh-routing area — needs careful review per memory.

Phase A4 — verification gate (on nodes, before tag)

  • On .116/.198/framework-pt/.228: content.browse-peer to every peer must return transport: "fips" with sub-second latency (LAN pairs) / <3s (WAN), 20/20 calls.
  • Kill the fips daemon on one node → calls fall back to Tor gracefully within the fast-fail budget (<8s), UI shows partial results, no errors.
  • Restart daemon → FIPS recovers within one watcher tick (~25s), verified in fips.status counters.
  • Flap the anchor link (drop vps2 route) → direct LAN pairs keep FIPS via their direct link (G1 fix proof).
  • Add these as tests/multinode/ cases per docs/multinode-testing-plan.md; also fix the known node_rpc() missing --max-time (tracker item).

Part B — optimistic loading + state management (frontend)

Full audit: Pinia exists but pages fetch-on-mount with loading=true spinners; Dashboard.vue:89 keys the router-view by route.path, so every navigation unmounts and refetches everything; no KeepAlive/onActivated anywhere; no dedup, no abort, no SWR layer. Four hand-rolled cache implementations already exist and prove the pattern (useFleetData.ts:198-231 sessionStorage hydrate; homeStatus.ts sticky-ready loadState; Home.vue:591-621 wallet localStorage snapshot; curatedApps.ts:21-77 TTL cache). SkeletonCard.vue exists, imported by zero files.

B1 — one shared primitive: useCachedResource composable + resources Pinia store

Semantics (generalize homeStatus.ts + useFleetData.ts):

  • Keyed resource: { data, loadState: idle|loading|ready|error|refreshing, fetchedAt, error }.
  • Hydrate synchronously from memory (Pinia, survives navigation) → sessionStorage snapshot (survives reload) → then revalidate in background.
  • Sticky-ready: once ready, never regress to loading (loadState = loadState==='ready' ? 'ready' : 'loading' — the homeStatus.ts:80 idiom); keep-last-known-value on error with a stale badge (age from fetchedAt).
  • TTL per resource; revalidateOnFocus + on WS push (debounced, the Home.vue:539-542 pattern); explicit invalidate(key) for mutations.
  • Optimistic mutation helper: apply → RPC → rollback on error (generalize TransportPrefsCard.vue:112-127).

B2 — rpc-client upgrades (src/api/rpc-client.ts)

  • AbortSignal in RPCOptions (today the AbortController at :87 is timeout-only) → abort-on-unmount for fan-outs.
  • In-flight dedup keyed method+JSON(params) — collapses duplicate concurrent calls.
  • Per-call maxRetries override; set maxRetries: 1 for content.browse-peer / preview-peer (retry×3 on a 30s timeout is why one slow peer = 90s spinner).

B3 — Cloud page conversion (worst offender, the marquee win)

  • Move sectionCounts, peerNodes, myFiles, peerFiles, paidItems out of Cloud.vue component state (:403,:476,:582,:689,:427) into the cached store — instant render on revisit, background refresh.
  • Incremental per-peer fan-in: render each peer's card as its content.browse-peer resolves (today Promise.allSettled at :708-747 blocks on the slowest peer). Per-peer states: cached/fresh/loading/unreachable.
  • Surface transport per peer (already in the response, discarded at :716-721): FIPS/Tor badge + latency — this is also the fleet-wide FIPS-uptime dashboard the user asked for, for free.
  • Skeleton cards (revive SkeletonCard.vue, copy FileGrid.vue:3-19 shimmer) instead of spinners for counts/folders/peer grids.
  • Stop CloudFolder.vue:307-319 calling cloudStore.reset() on every folder entry — cache per-path listings, navigate renders cache + revalidates.
  • PeerFiles.vue: persist catalog + preview cache in the store; cap the preview-peer fan-out (:832-841, currently unbounded) with a small concurrency queue + abort-on-unmount.

B4 — roll out to remaining offenders (in audit order)

PeerFiles → Web5 wallet/ecash/LND slices → Monitoring → Lightning channels (LightningChannelsPanel.vue:650) → Federation (already has {showLoader:false} — just adopt the store) → Server → Credentials/OpenWrtGateway/ContainerApps. Apps.vue/Marketplace.vue/Fleet.vue are already good; don't touch.

B5 — freshness via the existing push channel

/ws/db firehose + sync.ts JSON-patch already exist. Wire useCachedResource revalidation to relevant WS pushes (debounced 800ms), keep the 30s staleness reconciliation as backstop. No new backend needed for v1; a per-topic subscribe can come later.

Part B verification (on nodes)

  • Navigate Cloud → Apps → Cloud: peer files render instantly from cache (0 spinner), refresh indicator while revalidating, updated data lands without layout jump.
  • One unreachable peer: its card shows stale/unreachable state; other peers render immediately (no 30s all-or-nothing).
  • Kill backend mid-view: stale data stays visible with age badge; recovery revalidates automatically.
  • Hard reload: sessionStorage hydrate paints before first RPC completes.

Sequencing for the next release

  1. A0 now (fleet triage + transient nft rules + .228 daemon upgrade + baseline).
  2. A1 + A2 land together (P0 fixes + telemetry) → deploy to .116/.198 → Phase A4 checks on the pair → framework-pt → full fleet.
  3. B1 + B2 + B3 (composable + rpc-client + Cloud) in parallel with A-testing — frontend-only, verifiable against .116 dev (reference_neode_ui_dev_testing).
  4. A3 after telemetry baseline exists; B4/B5 ride the same or next OTA.
  5. Gate: Phase A4 checklist green + Part B verification on-device + existing single-node gate stays green → tag/OTA per ship ritual.

Success criteria

  • content.browse-peer transport = fips for ≥99% of calls between healthy 0.4.1 nodes over 24h (measured by the new counters), Tor reserved for genuinely FIPS-unreachable peers (.116-WiFi-class networks).
  • Cloud revisit paints in <100ms from cache; fresh data within one revalidate.
  • Fallback counters visible in fips.status so regressions are caught on the dashboard, not by users.