From fda7feda605f7079ae708a9b294fa36397f39b6e Mon Sep 17 00:00:00 2001 From: archipelago Date: Fri, 7 Aug 2026 10:08:18 -0400 Subject: [PATCH] security: untrack operations docs; scrub infra identifiers from public docs Operations docs move out of git entirely rather than being sanitized. They stay on disk for local use and are gitignored, so the Phase 6 export (which takes HEAD) can never carry them. 15 files: the fleet runbook, hotfix process, node inventories, internal trackers, session handoffs, the key rotation/signing-posture records, and the open-source plan itself. For the docs that remain public, infra identifiers are replaced with things that are better documentation rather than placeholders: curl examples now use `archipelago.local`, the product's own mDNS name, so a reader can run them as-is instead of substituting an address that was never theirs. Deliberately NOT scrubbed, both verified as functional rather than leaked: - `tx1138.com` is the shipped default block explorer (DEFAULT_TX_EXPLORER in useTxExplorer.ts, surfaced in WalletSettingsModal). Product behavior. - `git.tx1138.com` in core/container/{image_policy,registry}.rs is a retired- registry constant the code matches on to strip stale entries from legacy node configs. Removing it would break migration for older nodes. - `192.168.1.254` in bulletproof-containers.md is the LAN gateway in a podman bug description, and `192.168.1.x` in user-walkthrough.md is already generic. Whether a personal domain should be the shipped explorer default in a public product is a separate product question, not a security one. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 21 + CHANGELOG.md | 26 +- docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md | 300 ---- docs/HANDOFF-2026-07-20-fips-peer-files.md | 238 ---- docs/OPEN-SOURCE-READINESS-PLAN.md | 322 ----- docs/PRODUCTION-MASTER-PLAN.md | 1265 ----------------- docs/RETICULUM-TRANSPORT-PROGRESS.md | 399 ------ docs/SEED-VERIFICATION.md | 4 +- docs/UNIFIED-TASK-TRACKER.md | 435 ------ docs/api-reference.md | 6 +- docs/app-developer-guide.md | 6 +- .../HANDOVER-2026-07-02-iso-feedback.md | 151 -- docs/archive/INSTALL-SCREENS-DESIGN.md | 4 +- docs/archive/SESSION-1.8.0-OTA-PROGRESS.md | 344 ----- docs/bitcoin-rpc-relay.md | 4 +- docs/bulletproof-containers.md | 2 +- docs/combined-test-plan-2026-07-22.md | 173 --- docs/companion-pairing-qr.md | 4 +- docs/hotfix-process.md | 52 - docs/operations-runbook.md | 366 ----- docs/pine-voice-release-test-plan.md | 60 - docs/security/KEY-02-FLEET-ROTATION.md | 244 ---- docs/security/KEY-03-SIGNING-POSTURE.md | 448 ------ tests/production-quality/TRACKER.md | 260 ---- 24 files changed, 49 insertions(+), 5085 deletions(-) delete mode 100644 docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md delete mode 100644 docs/HANDOFF-2026-07-20-fips-peer-files.md delete mode 100644 docs/OPEN-SOURCE-READINESS-PLAN.md delete mode 100644 docs/PRODUCTION-MASTER-PLAN.md delete mode 100644 docs/RETICULUM-TRANSPORT-PROGRESS.md delete mode 100644 docs/UNIFIED-TASK-TRACKER.md delete mode 100644 docs/archive/HANDOVER-2026-07-02-iso-feedback.md delete mode 100644 docs/archive/SESSION-1.8.0-OTA-PROGRESS.md delete mode 100644 docs/combined-test-plan-2026-07-22.md delete mode 100644 docs/hotfix-process.md delete mode 100644 docs/operations-runbook.md delete mode 100644 docs/pine-voice-release-test-plan.md delete mode 100644 docs/security/KEY-02-FLEET-ROTATION.md delete mode 100644 docs/security/KEY-03-SIGNING-POSTURE.md delete mode 100644 tests/production-quality/TRACKER.md diff --git a/.gitignore b/.gitignore index b99a37af..78a64dd4 100644 --- a/.gitignore +++ b/.gitignore @@ -92,3 +92,24 @@ scripts/resilience/reports/ # app/docs asset path with a descriptive filename. Screenshot *.png uploads/ + +# Operations docs — fleet runbooks, node inventories, internal trackers and +# session handoffs. Kept on disk for local use, deliberately NOT tracked: they +# describe live infrastructure (node addresses, hostnames, deploy procedures) +# and must never reach the public repo. See the ops-docs section of the +# open-source readiness plan. +/docs/operations-runbook.md +/docs/hotfix-process.md +/docs/PRODUCTION-MASTER-PLAN.md +/docs/UNIFIED-TASK-TRACKER.md +/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md +/docs/HANDOFF-2026-07-20-fips-peer-files.md +/docs/RETICULUM-TRANSPORT-PROGRESS.md +/docs/combined-test-plan-2026-07-22.md +/docs/pine-voice-release-test-plan.md +/docs/OPEN-SOURCE-READINESS-PLAN.md +/docs/archive/HANDOVER-2026-07-02-iso-feedback.md +/docs/archive/SESSION-1.8.0-OTA-PROGRESS.md +/docs/security/KEY-02-FLEET-ROTATION.md +/docs/security/KEY-03-SIGNING-POSTURE.md +/tests/production-quality/TRACKER.md diff --git a/CHANGELOG.md b/CHANGELOG.md index c10242dd..6124d182 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -496,7 +496,7 @@ - NetBird's browser proxy now sends API, OAuth, relay, WebSocket, and management traffic through the stable host-published server port at `169.254.1.2:8086`, avoiding stale rootless Podman DNS/IPs after `netbird-server` restarts. - Mobile App Store category chips now stay visible above the tab bar, Discover is available on mobile, and category selection updates the page route/query so the selected category is actually shown. - Apps that require a real browser tab now open directly from the app icon tap instead of first entering an in-shell app-session route, including BTCPay, Grafana, Home Assistant, Vaultwarden, Nextcloud, Portainer, OnlyOffice, Tailscale, Uptime Kuma, Gitea, and Nginx Proxy Manager. -- Validation passed with catalog JSON checks, `npm run type-check`, `cargo fmt --all --check --manifest-path core/Cargo.toml`, and `cargo check -p archipelago --manifest-path core/Cargo.toml`; live checks on `100.70.96.88` confirmed Saleor dashboard `9010`/API `8000` and NetBird API/OAuth routes survive `netbird-server` restart. +- Validation passed with catalog JSON checks, `npm run type-check`, `cargo fmt --all --check --manifest-path core/Cargo.toml`, and `cargo check -p archipelago --manifest-path core/Cargo.toml`; live checks on a fleet node confirmed Saleor dashboard `9010`/API `8000` and NetBird API/OAuth routes survive `netbird-server` restart. ## v1.7.75-alpha (2026-05-19) @@ -529,7 +529,7 @@ ## v1.7.71-alpha (2026-05-19) -- NetBird stack installs now pre-create `/var/lib/archipelago/netbird/data` before binding it into `netbird-server`, fixing the failed install/start path seen on `100.70.96.88` where Podman rejected the missing host directory. +- NetBird stack installs now pre-create `/var/lib/archipelago/netbird/data` before binding it into `netbird-server`, fixing the failed install/start path seen on a fleet node where Podman rejected the missing host directory. - NetBird start/restart ordering now starts `netbird-server` before the dashboard container so lifecycle actions bring the control plane up before the UI. - App-session invalid IDs and panel-mode fallbacks now return to `/dashboard/apps`, avoiding the stale `/apps` route that could render a 404. - Mobile launches for apps that block iframes now stay inside the Archipelago app-session fallback instead of automatically opening an external browser tab. @@ -547,7 +547,7 @@ ## v1.7.69-alpha (2026-05-19) - App installs now allow up to 10 minutes for the initial `package.install` RPC to return, matching slow container image pulls and preventing apps from disappearing from My Apps while the backend is still pulling or retrying mirrors. -- Live diagnostics on `100.70.96.88` confirmed the Gitea install did not fail; the primary registry pull timed out after 300 seconds, the fallback mirror succeeded, and Gitea came up healthy on `3001` while the frontend had already timed out at 15 seconds. +- Live diagnostics on a fleet node confirmed the Gitea install did not fail; the primary registry pull timed out after 300 seconds, the fallback mirror succeeded, and Gitea came up healthy on `3001` while the frontend had already timed out at 15 seconds. - Gitea and other Docker-image app installs now stay visible during slow registry pulls instead of being marked as failed by the browser before backend install progress can complete. - Gitea is now categorized as a known Data app in My Apps, so a running Gitea container appears with installed apps instead of being filtered into the Websites/Services split. - NetBird `0.71.2` is now available in the app catalog and fallback marketplace data as a recommended networking app using the official `docker.io/netbirdio/netbird:0.71.2` image. @@ -565,8 +565,8 @@ - App session close buttons now return to the previous dashboard screen when possible and otherwise fall back to My Apps, avoiding the 404 page after closing an app launched from an invalid or stale history entry. - System Update confirmation and mirror modals now teleport to the document body with a full-screen overlay, so they cover the whole app instead of only the right-hand dashboard panel. - Mobile app launches stay inside Archipelago's app-session webview and hide desktop-only new-tab launch affordances, including apps such as Home Assistant that previously looked like they would leave the mobile shell. -- Live recovery on `100.70.96.88` upgraded only the `btcpay-server` container to `docker.io/btcpayserver/btcpayserver:2.3.9`, preserved the existing datadir and Postgres database, and confirmed the container is healthy after a pre-upgrade backup. -- Public validation confirmed `spay.tx1138.com`/`www` redirect to BTCPay login over HTTPS and `sapien.tx1138.com`/`www` serve the L484 page over HTTPS using the issued Let's Encrypt certificates. +- Live recovery on a fleet node upgraded only the `btcpay-server` container to `docker.io/btcpayserver/btcpayserver:2.3.9`, preserved the existing datadir and Postgres database, and confirmed the container is healthy after a pre-upgrade backup. +- Public validation confirmed ``the BTCPay host`/`www` redirect to BTCPay login over HTTPS and `the L484 host`/`www` serve the L484 page over HTTPS using the issued Let's Encrypt certificates. ## v1.7.67-alpha (2026-05-18) @@ -575,18 +575,18 @@ - Settings What's New is filled through `v1.7.67-alpha`, including the missing historical `v1.7.44-alpha` through `v1.7.66-alpha` entries. - Bitcoin/Knots/Core shell lifecycle specs now match the Rust app config memory policy: 8 GiB on normal hosts, 4 GiB on low-memory hosts, and pruned Knots uses a larger dbcache on hosts with enough RAM to improve IBD throughput. - ElectrumX/electrs shell lifecycle specs now match the 4 GiB memory policy used by the Rust app config, reducing drift between first boot, reconcile, and app lifecycle paths. -- Live assessment of `100.70.96.88` identified the current IBD bottlenecks as CPU/thermal/I/O pressure rather than RAM exhaustion, with follow-up work planned for existing-node swap repair, kiosk Chromium CPU reduction, and reconcile failure cleanup. +- Live assessment of a fleet node identified the current IBD bottlenecks as CPU/thermal/I/O pressure rather than RAM exhaustion, with follow-up work planned for existing-node swap repair, kiosk Chromium CPU reduction, and reconcile failure cleanup. ## v1.7.66-alpha (2026-05-18) - Nginx Proxy Manager stale-port repair now detects stopped or `Created` Podman records by inspecting `podman ps -a` port metadata, covering records where `podman port nginx-proxy-manager` returns no mapping until start. -- Live recovery on `100.70.96.88` removed only the stale Nginx Proxy Manager container record and recreated it with `8081:81`, `8084:80`, and `8444:443`, preserving `/var/lib/archipelago/nginx-proxy-manager` data. +- Live recovery on a fleet node removed only the stale Nginx Proxy Manager container record and recreated it with `8081:81`, `8084:80`, and `8444:443`, preserving `/var/lib/archipelago/nginx-proxy-manager` data. - Validation confirmed Nginx Proxy Manager recovered as healthy and responds through direct admin port `8081`, host compatibility port `81`, and `/app/nginx-proxy-manager/`. ## v1.7.65-alpha (2026-05-18) - Orchestrator-backed app starts now run the same pre-start repairs as the legacy Podman path, so Nginx Proxy Manager stale `81:81` container metadata is removed and recreated before the orchestrator tries to start it. -- Live diagnostics on `100.70.96.88` confirmed host nginx is healthy while Nginx Proxy Manager has no listeners on `8081`, `8084`, or `8444`, causing host nginx `502` responses for NPM proxy paths. +- Live diagnostics on a fleet node confirmed host nginx is healthy while Nginx Proxy Manager has no listeners on `8081`, `8084`, or `8444`, causing host nginx `502` responses for NPM proxy paths. ## v1.7.64-alpha (2026-05-18) @@ -611,7 +611,7 @@ - Multi-container stack installs now keep their app card in the `Installing` state for up to 20 minutes while dependency containers are being pulled and prepared. - BTCPay Server installs no longer appear to vanish or fail after two minutes while Postgres and NBXplorer are still being created before the primary `btcpay-server` container exists. - The stale-transition escape hatch remains short for start, stop, restart, update, and removal operations, so genuinely wedged lifecycle actions still recover quickly. -- Live validation on `100.70.96.88` confirmed BTCPay Server completed installation and responds on port `23000` with the expected HTTP redirect. +- Live validation on a fleet node confirmed BTCPay Server completed installation and responds on port `23000` with the expected HTTP redirect. ## v1.7.60-alpha (2026-05-18) @@ -619,7 +619,7 @@ - Mesh radio auto-detection now skips known non-mesh serial devices such as Sierra Wireless LTE modems and Zooz/Z-Wave sticks, avoiding interference with production peripherals. - Meshtastic config sync now sends `want_config_id` with the correct protobuf wire type, fixing radio-side `ignore malformed toradio` errors and allowing node-info/contact ingestion. - The stable `/dev/mesh-radio` udev rule no longer claims every `ttyACM*` device; it only matches known mesh USB serial adapters and known USB CDC ACM radio vendors. -- Live validation on `100.70.96.88` confirmed Archipelago selects `/dev/ttyUSB0`, identifies the Meshtastic node, and refreshes 103 mesh contacts. +- Live validation on a fleet node confirmed Archipelago selects `/dev/ttyUSB0`, identifies the Meshtastic node, and refreshes 103 mesh contacts. ## v1.7.59-alpha (2026-05-17) @@ -643,7 +643,7 @@ - Host nginx now serves `/assets/*` hashed frontend chunks as immutable static files with a hard 404 on misses instead of falling back to `index.html`, preventing strict MIME errors when a browser has a stale pre-update HTML shell. - The SPA HTML shell and service-worker files now revalidate on every load, reducing stale frontend references after OTA updates. - OTA runtime promotion now installs the bundled `nginx-archipelago.conf` into `/etc/nginx/sites-available/archipelago` and reloads nginx after a successful config test, so frontend cache/fallback fixes reach existing nodes without a manual deploy. -- Local validation passed with `cargo check -p archipelago`; live SSH testing against `100.70.96.88` was not completed because temporary public-key authentication was rejected on the target. +- Local validation passed with `cargo check -p archipelago`; live SSH testing against a fleet node was not completed because temporary public-key authentication was rejected on the target. ## v1.7.57-alpha (2026-05-17) @@ -724,7 +724,7 @@ - Health monitor no longer pages "Auto-restart failed" for orphaned containers. After a variant switch (bitcoin-core ↔ bitcoin-knots) the previous variant's container could survive uninstall and the health monitor would try restarting it forever. Now skipped silently with a debug log. - Apps no longer disappear from My Apps when an install fails. The card stays visible with state=Stopped so the user can retry or uninstall, with the failure reason surfaced via the new install_progress.message field. - "Downloading…" progress now actually advances during multi-image stack pulls. Was sticking at 20% until all pulls finished; now interpolates 20%→70% based on which image of N has landed. -- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on OVH and tx1138. Removes a docker.io dependency from first-boot installs. +- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on the registry mirrors. Removes a docker.io dependency from first-boot installs. - Resilience harness improvements: install-fail entries no longer vanish, install/uninstall/probe cells are timing-tolerant (60s retry on ui_probe and auth_probe), dep snapshots no longer leak companion containers into the dependent app's "new containers" set. ## v1.7.45-alpha (2026-04-29) @@ -770,7 +770,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Infrastructure - CI pipeline added (.github/workflows/ci.yml) — cargo fmt, clippy, tests + frontend type-check, build -- Update system now fetches from git.tx1138.com Gitea instance (configurable via ARCHIPELAGO_UPDATE_URL) +- Update system now fetches from the release Gitea instance (configurable via ARCHIPELAGO_UPDATE_URL) - Cleaned up stale git branches (app-store, overnight/2026-03-12, overnight/2026-03-13) ## [1.3.0] - 2026-03-19 diff --git a/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md b/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md deleted file mode 100644 index 45a53f25..00000000 --- a/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md +++ /dev/null @@ -1,300 +0,0 @@ -# FIPS near-100% uptime + optimistic UI state — implementation plan - -**Date:** 2026-07-27. **Status:** researched + root-caused live on the fleet; ready to -implement for the next release. Two workstreams: (A) make node↔node FIPS transport -succeed whenever a FIPS path physically exists, (B) stop the UI reloading everything -on every navigation (optimistic/cached cards, stale-while-revalidate) while keeping -data fresh. - -**Honesty note on "100%":** if a node's network blackholes every anchor (the .116 -WiFi case, `docs/HANDOFF-2026-07-20-fips-peer-files.md:117-133`), Tor fallback is -*correct*. The achievable target is: **FIPS wins whenever a FIPS path exists, and -fallback frequency is measured in-product so regressions are visible.** Today several -paths are 0% FIPS *by construction* regardless of network health — that's the bug. - ---- - -## Part A — why Cloud/FIPS "commonly falls back to Tor": ranked root causes - -All verified live on 2026-07-27 (.116 local, .198, .228, Framework PT, x250s) plus a -full code audit of `core/archipelago/src/{fips,transport,federation,server.rs}`. - -### RC0 — 🔥 The hardening firewall drops the peer-API port on every hardened node (PROVEN) - -The fips0 default-deny baseline (`/etc/fips/fips.nft`) is opened by archipelago's -drop-in `80-web-ui.nft` (`fips/config.rs:236-255`) for **80 + 8443 + app ports only**. -The peer-API listener — which carries *all* federation sync, cloud browse/download, -mesh envelopes, DWN, invoices — is **`PEER_PORT = 5679`** (`fips/dial.rs:35`). -**5679 is not in the allowlist.** The drop-in's own comment claims "web UI + peer -API" but the peer API port was never added. - -Live proof (2026-07-27): -- .116 nft chain: 5,965 dropped packets; .198: **28,670 dropped packets** — that's - peers' FIPS dials dying at the firewall. -- .198 → .116 `GET :5679/health`: **timeout (6s)** before; **HTTP 200 in 0.35s** - after `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`. - Same result in reverse direction (200 in 0.64s). -- Explains the exact fleet split in `federation/nodes.json`: hardened-baseline nodes - (Framework PT, .198, .228, x250-dev, x250-mad2) = `last_transport: tor`; - non-hardened nodes (Austin Sapien, X250-Beta, X250-PA) answer :5679 (404 from the - path allowlist = listener reachable) = `last_transport: fips`. -- Every dial to a hardened peer pays the 8s FIPS connect timeout - (`dial.rs:114`) ×2 (retry, `dial.rs:128-140`) → then Tor. That's the "Cloud takes - forever / shows Tor" experience. - -**Fix (one line + reload):** add `tcp dport 5679 accept` to the drop-in in -`fips/config.rs` (use a constant shared with `dial.rs::PEER_PORT`, not a literal). -The drop-in reinstalls on every daemon config install, so it heals fleet-wide on OTA. -⚠️ Transient manual rules were inserted on .116 and .198 during diagnosis (2026-07-27) -— they vanish on the next `nft -f /etc/fips/fips.nft` reload or reboot; the code fix -makes them permanent. - -### RC1 — .228 (Shorty's) runs fips 0.3.0-dev; the 0.4.1 fleet can't reach it - -.228's daemon: `0.3.0-dev (rev 34e00b9f6e)`, both anchor links "connected", but its -ULA is 100% unreachable from 0.4.1 nodes (ping loss 100%). FIPS wire format is not -stable across revs (`docs/HANDOFF-2026-07-23-companion-apk-deploy.md:78`). Everything -to/from .228 rides Tor no matter what else we fix. - -**Fix:** fleet fips-version audit + upgrade to v0.4.1 everywhere (in-product updater -exists: `fips/update.rs`; .deb path per `reference_vps2_fips_anchor`). Add a version -check to `fips.status` and surface a "peer daemon outdated" warning. - -### RC2 — Direct LAN/endpoint peering is dead code + wrong port + stale seed anchors - -Without direct links, all peer traffic hairpins through the vps2 anchor spanning -tree (observed: .116→.198 cold RTT 1.5–3.5s on the same LAN; also the wedged-anchor -latency-rot incident, `HANDOFF-2026-07-23:141-160`). - -- **G1 — `lan_fips_anchors()` has never run.** It needs `PeerRecord.fips_npub`, but - `PeerRegistry::set_fips_npub` (`transport/mod.rs:302`) has **zero callers** — mDNS - TXT records only carry `did`/`pubkey`/`version` (`transport/lan.rs:50-54`). So the - "co-located peers form a direct link" feature (`anchors.rs:294-305`, - `server.rs:761-766`) is a fleet-wide no-op. -- **G2 — wrong UDP port.** `anchors.rs:293` dials `8668`, but the generated - fips.yaml binds UDP **2121** (`fips/config.rs:187`, `fips/mod.rs:130`). Even if G1 - ran, it would dial a dead port. `.116`'s live `seed-anchors.json` still carries - `.198@192.168.1.198:8668` — **stale IP (LAN renumbered to 192.168.63.x) AND dead - port**; both manual entries are useless today. -- No Tailscale/alternate endpoint fallback when LAN is unreachable (the .116↔.198 - fix of 2026-07-20 was hand-applied per-node config, never productized). - -**Fix:** (a) `FIPS_UDP_PORT` → `crate::fips::PUBLISHED_UDP_PORT` + drift-guard test; -(b) hydrate `fips_npub` into the registry from federation storage (did-keyed join) so -`lan_fips_anchors` goes live with no wire change; (c) advertise the npub in the mDNS -TXT + `set_fips_npub` on resolve as the proper fix; (d) teach the LAN-anchor tick to -also try a peer's Tailscale/last-known-good endpoint when LAN fails (reviewed change -— this area got handoffs wrong twice, per memory). - -### RC3 — No fast-fail on the hottest call sites; retry silently doubles every budget - -- `content.browse-peer` — **the Cloud page** — has NO `fips_timeout` - (`api/rpc/content.rs:363-366`): a cold FIPS path burns up to ~16.6s (8s connect + - 600ms + 8s retry) before Tor even starts, against a UI deadline of 30s - (`Cloud.vue:720`) — and the frontend then retries ×3. Users see errors, not - fallback. 12 call sites total lack `fips_timeout` (browse/download/preview-peer, - `/blob`, DWN, node_message, rotation notifies). -- `dial.rs:128-140` runs 2 full-budget attempts, so `fips_timeout(6s)` really means - ~12.6s everywhere. - -**Fix:** wrap `send_with_retry` in a single `tokio::time::timeout(fips_attempt_timeout())` -(call sites `dial.rs:455`, `dial.rs:488`; halve per-attempt client timeout), then add -`.fips_timeout(...)`: `content.rs:366` (6s), `content.rs:281` (8s), `content.rs:1139` -(6s), `typed_messages.rs:822` (8s), `dwn_sync.rs:188/213/272` (6s), -`node_message.rs:376` (8s), `node_message.rs:412` (4s), `tor/mod.rs:501` (6s), -`federation/handlers.rs:869` (6s). **Skip the three 900s streaming downloads** -(`content.rs:552/870/1061`, `proxy.rs:236`) — `dial.rs:311-319` documents why; the -retry-budget wrap covers their connect phase. - -### RC4 — Two features are 100% Tor by construction (allowlist 404) - -The peer listener path allowlist (`server.rs:1219-1239`) omits `/blob/` (mesh -file sharing, `typed_messages.rs:813-822`) and `/dwn/health` (step 1 of DWN sync, -`dwn_sync.rs:186`) → deterministic 404 over FIPS (`dial.rs:44-46` treats 404 as -fall-back) → deterministic Tor, after paying the full FIPS cost. Both endpoints are -already cryptographically gated, so they meet the allowlist's stated criterion. - -**Fix:** add `|| path.starts_with("/blob/") || path.starts_with("/dwn/")`; extend the -existing test block at `server.rs:1935-1945` (assert `/blob/abc` + `/dwn/health` -allowed, `/blobber` + `/dwnx` denied). - -### RC5 — Inbound listener can't heal; anchor flap = 5-minute Tor window; probe overhead - -- `peer_late_bind_loop` returns after first successful bind (`server.rs:1203`) and - `accept_loop` `continue`s on errors forever (`server.rs:1249-1258`): a fips0 - teardown/re-key leaves the node inbound-dead until process restart → **every peer** - falls back to Tor against it. -- Nothing reacts to anchor-link drops: anchors re-apply only on the 300s tick - (`server.rs:731`); worst-case 5min Tor-only after a flap (the historic "link dead - timeout 30s" flapping made this chronic). -- `is_service_active()` spawns up to 2 `systemctl` per FIPS attempt *and* per peer - per 25s warm tick (`dial.rs:284-294`); `warm_path` skips peers without - `fips_npub` in federation storage (`fips/mod.rs:88-95`); `anchors::apply` is - serial with unbounded subprocess waits (`anchors.rs:234-283`). - -**Fix:** rebindable listener; a ~25s connectivity watcher (reuse -`service::peer_connectivity_summary`, `fips/service.rs:178-207`) that re-applies -anchors immediately on a connected→disconnected edge with bounded backoff; 10s TTL -cache for `is_service_active` (mirror `transport/fips.rs:24-107`); warm the union of -federation+registry peers; make `apply()` concurrent with per-connect timeouts. - -### RC6 — Zero observability: fallbacks are invisible, so "uptime" is unfalsifiable - -Fallbacks log at `debug!` only (`dial.rs:458,491`); no counters; `last_transport` is -written by only 7 of ~20 call sites and **never read** to influence anything -(`storage.rs:120-147`). The parallel `TransportRouter` system can't even see FIPS -(`FipsTransport` is never constructed — `server.rs:422-442` registers Tor/Mesh/LAN -only). - -**Fix:** per-reason fallback counters (F1 no-npub / F2 service-inactive / F3 -DNS-fail / F4 connect-fail / F5 404 / F6 5xx) surfaced in `fips.status` + `info!` -logs with a `reason` field; call `record_peer_transport` from all peer-dial sites; -UI: per-peer transport badge on Cloud (the response already carries `transport` — -`content.rs:392-400` — Cloud.vue currently throws it away at `:716-721`). - ---- - -## Part A — execution phases - -### Phase A0 — fleet triage (no release needed; do first, validates everything) -1. Fleet audit: `fipsctl --version` + `nft list table inet fips` + `ss -tlnp | grep 5679` - on every node (roster: `reference_test_deploy_roster`). -2. Transient `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept` - on hardened nodes (already done on .116 + .198, 2026-07-27) — instant fleet-wide - FIPS recovery while the code fix rides the OTA. -3. Upgrade .228 (and any other 0.3.x) fips daemon to v0.4.1. -4. Regenerate/clean stale `seed-anchors.json` on .116 (dead 192.168.1.x + :8668 entries). -5. Baseline measurement: for each node pair, `content.browse-peer` time + transport. - -### Phase A1 — P0 code (one commit, mechanical, offline-testable) -1. **nft drop-in: open 5679** — `fips/config.rs` (share the constant with - `dial.rs::PEER_PORT`). ← RC0 -2. **Allowlist `/blob/`, `/dwn/`** — `server.rs:1219-1239` + tests. ← RC4 -3. **`FIPS_UDP_PORT` = `PUBLISHED_UDP_PORT` (2121)** — `anchors.rs:293` + drift-guard - test against `render_config_yaml()`. ← RC2-G2 -4. **Un-deaden `lan_fips_anchors`** — hydrate `fips_npub` from federation storage in - `server.rs:761-766`; then mDNS TXT `fips` key + `set_fips_npub` - (`transport/lan.rs:50-54`, `lan.rs:96-108`, `LanTransport::new` 4th arg via - `crate::identity::fips_npub(&data_dir.join("identity"))`). ← RC2-G1 -5. **Retry-budget wrap + `fips_timeout` on 12 call sites** (list in RC3). ← RC3 - Verify: `cd core && cargo test -p archipelago` — watch `test_rendered_yaml_exact_snapshot` - (`config.rs:419`) + `test_render_is_deterministic` (`config.rs:476`); item 3 must - not change rendered output. - -### Phase A2 — telemetry BEFORE tuning (second commit) -6. Fallback counters by reason + `fips.status` exposure + `info!` reason logs; - `record_peer_transport` from all sites. ← RC6 (gives the baseline that makes A3 - measurable and "100%" falsifiable) - -### Phase A3 — resilience (third commit, measured against A2 baseline) -7. `is_service_active` 10s TTL cache; warm-path union + `warm_path_unchecked`. -8. Link-state watcher → immediate anchor re-apply on drop (replaces waiting for the - 300s tick); concurrent `apply()` with subprocess timeouts. -9. Rebindable peer listener (`server.rs:1203`, `1249-1258`). -10. (Reviewed, separate PR) endpoint-fallback for direct peering: LAN → Tailscale → - last-known-good, npub-keyed. Mesh-routing area — needs careful review per memory. - -### Phase A4 — verification gate (on nodes, before tag) -- On .116/.198/framework-pt/.228: `content.browse-peer` to every peer must return - `transport: "fips"` with sub-second latency (LAN pairs) / <3s (WAN), 20/20 calls. -- Kill the fips daemon on one node → calls fall back to Tor gracefully within the - fast-fail budget (<8s), UI shows partial results, no errors. -- Restart daemon → FIPS recovers within one watcher tick (~25s), verified in - `fips.status` counters. -- Flap the anchor link (drop vps2 route) → direct LAN pairs keep FIPS via their - direct link (G1 fix proof). -- Add these as `tests/multinode/` cases per `docs/multinode-testing-plan.md`; also - fix the known `node_rpc()` missing `--max-time` (tracker item). - ---- - -## Part B — optimistic loading + state management (frontend) - -Full audit: Pinia exists but pages fetch-on-mount with `loading=true` spinners; -`Dashboard.vue:89` keys the router-view by `route.path`, so **every navigation -unmounts and refetches everything**; no KeepAlive/onActivated anywhere; no dedup, -no abort, no SWR layer. Four hand-rolled cache implementations already exist and -prove the pattern (`useFleetData.ts:198-231` sessionStorage hydrate; -`homeStatus.ts` sticky-ready loadState; `Home.vue:591-621` wallet localStorage -snapshot; `curatedApps.ts:21-77` TTL cache). `SkeletonCard.vue` exists, imported by -zero files. - -### B1 — one shared primitive: `useCachedResource` composable + `resources` Pinia store -Semantics (generalize `homeStatus.ts` + `useFleetData.ts`): -- Keyed resource: `{ data, loadState: idle|loading|ready|error|refreshing, fetchedAt, error }`. -- **Hydrate synchronously** from memory (Pinia, survives navigation) → sessionStorage - snapshot (survives reload) → then revalidate in background. -- Sticky-ready: once `ready`, never regress to `loading` - (`loadState = loadState==='ready' ? 'ready' : 'loading'` — the `homeStatus.ts:80` idiom); - keep-last-known-value on error with a stale badge (age from `fetchedAt`). -- TTL per resource; `revalidateOnFocus` + on WS push (debounced, the - `Home.vue:539-542` pattern); explicit `invalidate(key)` for mutations. -- Optimistic mutation helper: apply → RPC → rollback on error (generalize - `TransportPrefsCard.vue:112-127`). - -### B2 — rpc-client upgrades (`src/api/rpc-client.ts`) -- `AbortSignal` in `RPCOptions` (today the AbortController at `:87` is timeout-only) - → abort-on-unmount for fan-outs. -- In-flight dedup keyed `method+JSON(params)` — collapses duplicate concurrent calls. -- Per-call `maxRetries` override; set `maxRetries: 1` for `content.browse-peer` / - `preview-peer` (retry×3 on a 30s timeout is why one slow peer = 90s spinner). - -### B3 — Cloud page conversion (worst offender, the marquee win) -- Move `sectionCounts`, `peerNodes`, `myFiles`, `peerFiles`, `paidItems` out of - `Cloud.vue` component state (`:403,:476,:582,:689,:427`) into the cached store — - instant render on revisit, background refresh. -- **Incremental per-peer fan-in**: render each peer's card as its - `content.browse-peer` resolves (today `Promise.allSettled` at `:708-747` blocks on - the slowest peer). Per-peer states: cached/fresh/loading/unreachable. -- **Surface `transport` per peer** (already in the response, discarded at `:716-721`): - FIPS/Tor badge + latency — this is also the fleet-wide FIPS-uptime dashboard the - user asked for, for free. -- Skeleton cards (revive `SkeletonCard.vue`, copy `FileGrid.vue:3-19` shimmer) instead - of spinners for counts/folders/peer grids. -- Stop `CloudFolder.vue:307-319` calling `cloudStore.reset()` on every folder entry — - cache per-path listings, navigate renders cache + revalidates. -- `PeerFiles.vue`: persist catalog + preview cache in the store; cap the - `preview-peer` fan-out (`:832-841`, currently unbounded) with a small concurrency - queue + abort-on-unmount. - -### B4 — roll out to remaining offenders (in audit order) -PeerFiles → Web5 wallet/ecash/LND slices → Monitoring → Lightning channels -(`LightningChannelsPanel.vue:650`) → Federation (already has `{showLoader:false}` — -just adopt the store) → Server → Credentials/OpenWrtGateway/ContainerApps. -`Apps.vue`/`Marketplace.vue`/`Fleet.vue` are already good; don't touch. - -### B5 — freshness via the existing push channel -`/ws/db` firehose + `sync.ts` JSON-patch already exist. Wire `useCachedResource` -revalidation to relevant WS pushes (debounced 800ms), keep the 30s staleness -reconciliation as backstop. No new backend needed for v1; a per-topic subscribe can -come later. - -### Part B verification (on nodes) -- Navigate Cloud → Apps → Cloud: peer files render instantly from cache (0 spinner), - refresh indicator while revalidating, updated data lands without layout jump. -- One unreachable peer: its card shows stale/unreachable state; other peers render - immediately (no 30s all-or-nothing). -- Kill backend mid-view: stale data stays visible with age badge; recovery - revalidates automatically. -- Hard reload: sessionStorage hydrate paints before first RPC completes. - ---- - -## Sequencing for the next release - -1. **A0 now** (fleet triage + transient nft rules + .228 daemon upgrade + baseline). -2. **A1 + A2** land together (P0 fixes + telemetry) → deploy to .116/.198 → - Phase A4 checks on the pair → framework-pt → full fleet. -3. **B1 + B2 + B3** (composable + rpc-client + Cloud) in parallel with A-testing — - frontend-only, verifiable against .116 dev (`reference_neode_ui_dev_testing`). -4. **A3** after telemetry baseline exists; **B4/B5** ride the same or next OTA. -5. Gate: Phase A4 checklist green + Part B verification on-device + existing - single-node gate stays green → tag/OTA per ship ritual. - -## Success criteria -- `content.browse-peer` transport = fips for ≥99% of calls between healthy 0.4.1 - nodes over 24h (measured by the new counters), Tor reserved for genuinely - FIPS-unreachable peers (.116-WiFi-class networks). -- Cloud revisit paints in <100ms from cache; fresh data within one revalidate. -- Fallback counters visible in `fips.status` so regressions are caught on the - dashboard, not by users. diff --git a/docs/HANDOFF-2026-07-20-fips-peer-files.md b/docs/HANDOFF-2026-07-20-fips-peer-files.md deleted file mode 100644 index 74f18c16..00000000 --- a/docs/HANDOFF-2026-07-20-fips-peer-files.md +++ /dev/null @@ -1,238 +0,0 @@ -# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill - -Written for a fresh session that will **cut the OTA release and build the ISO**. -Everything below is already committed and pushed to `gitea-ai/main`. Last release -was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**. - ---- - -## 1. What this release carries (3 commits on top of v1.7.105-alpha) - -| Commit | What | User-visible? | -|---|---|---| -| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes | -| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) | -| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly | - -### `9e3ac9ba` — mobile transport pill -`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header -carries the name on mobile), and the transport pill was nested inside it — so it -vanished below 768px. Added a separate `md:hidden` pill next to the peer icon. -Frontend was rebuilt and the class verified present in the emitted bundle. - -Caveats worth knowing (pre-existing, not introduced here): -- On this code path the backend only ever emits `fips` or `tor`, so the `mesh` - and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead. -- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a - hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie. - The peer-files pill does not. - -### `3ab7fb52` — full error chain in logs -`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every -peer-files failure read exactly `RPC error on content.browse-peer: Failed to -connect to peer` with the real cause discarded. Now `{:#}`. The client-facing -message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so -no internal detail leaks. **This fix applies to every RPC method, not just -browse-peer.** - -### `5fd0d6c3` — typed FIPS config + mDNS -`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal. -Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key -does not degrade — **the daemon refuses to start and the node leaves the mesh**. -Now a typed serde struct tree, verified field-by-field against jmcorgan/fips -**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and -the pre-existing schema test. All pass. - -Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0) -so co-located nodes peer directly instead of depending on the public anchor. - -> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup -> drift check at `server.rs:864` compares the freshly rendered config against -> what's on disk. The render differs now, so it reinstalls the config and -> restarts the FIPS daemon **once**. This is the intended self-healing path and -> settles immediately. Do not mistake it for a regression. - -Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig` -has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it -harmlessly (verified against the v0.3.0 source). It self-activates on upgrade. - ---- - -## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled - -Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1** -(2026-07-19). Verified before touching anything: - -- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day. -- **Config forward-compatible** — every key we emit exists in 0.4.1. -- **Asset names match** what `fips/update.rs` expects (`fips__.deb` + - `checksums-linux.txt`), so the in-product updater should work. - -### Upgraded so far (2 of N) -| Node | Before | After | Result | -|---|---|---|---| -| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` | -| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) | - -The OptiPlex was still running the **old string-rendered config** and 0.4.1 -accepted it — empirical confirmation of the compat analysis, not just desk work. - -### Upgrade recipe (nodes cannot reach GitHub — sideload) -```bash -# 1. On a host with GitHub access: -curl -sL -o fips_0.4.1_amd64.deb \ - https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb -curl -sL -o checksums-linux.txt \ - https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt -sha256sum fips_0.4.1_amd64.deb # must match checksums-linux.txt -# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded - -# 2. Sideload: -scp fips_0.4.1_amd64.deb archipelago@:/tmp/ - -# 3. On the node — the same command update.rs uses: -sudo -n systemd-run --collect --wait --quiet --pipe -- \ - env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \ - /tmp/fips_0.4.1_amd64.deb - -# 4. Restart the ACTIVE unit — it is archipelago-fips.service, -# NOT fips.service (which is inactive on these nodes): -sudo -n systemctl restart archipelago-fips.service - -# 5. Verify: -fipsctl --version -sudo -n fipsctl show links # expect anchor 185.18.221.160:8443 connected -sudo -n fipsctl show tree # expect is_root: false, depth > 0 -``` - -### ISO implication (important) -`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from -**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64 -only). So a freshly built ISO will pick up whatever main is that day — probably -≥0.4.1, but it is not deterministic. Pinning is an open item in -`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1 -before building the release ISO** so the shipped version is knowable. - ---- - -## 3. The original bug — peer cloud files not loading - -**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit -because it would be easy to read this as closed. - -What is established: -- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links` - → `[]`, `show tree` → `is_root: true, depth 0`. An island. -- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the - public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on - ICMP, 443/8443/8668 all time out. `show transports` showed - `packets_sent: 760, packets_recv: 0` on both UDP and TCP. -- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock - Tailscale anti-spoof DROPs). -- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's - WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`). -- With no FIPS tree, everything falls back to Tor. Every peer in - `federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`. -- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×, - HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`. - -What is **not** established: why three specific `content.browse-peer` calls -failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never -reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic -is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and -the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is -cold-circuit timeouts on first fetch after idle — **a guess, not a finding.** -`3ab7fb52` means the next occurrence will log the actual cause. - -### Corrections to earlier claims in this session -- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the - `ip:port` in `fipsctl connect` is only an underlay endpoint hint. -- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy - (`delivery_ratio` 1.0 both directions, bloom filter syncing). The - `bytes_recv: 0` link counters are simply uninstrumented in 0.3.0. - ---- - -## 4. Open items — decisions NOT taken - -1. **Second FIPS anchor (user asked for this; not built).** Needs a host running - FIPS that is reachable from the restricted WiFi. Candidate found: OVH - **`146.59.87.168`** — pings fine from the thinkpad and general egress works - (github 200), while the upstream anchor fails even ICMP there. But it does not - run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** — - a production change, deliberately not made unprompted. Code side is easy after: - `fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list - (`default_public_anchor()` → `default_public_anchors() -> Vec`). -2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228` - (`100.64.204.114`) has been **offline ~20h** and could not be included. -3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet, - so mDNS is not actually live anywhere. That is what this OTA is for. -4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so - mDNS may not rescue that particular node even after the OTA. It will help - co-located nodes on sane networks. -5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO. - ---- - -## 5. Release ritual (from prior sessions — follow exactly) - -Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`, -`release-manifest.json`, `releases/manifest.json` modified, and an untracked -`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another -agent may share this tree; never `git add -A`. - -```bash -V=1.7.106-alpha - -# Frontend build — MUST verify dist actually changed (build can silently no-op) -cd neode-ui && npm run build # → web/dist/neode-ui/ -grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js # sanity - -# Backend -cd core && cargo build --release -p archipelago -# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache -# corruption — rebuild with CARGO_INCREMENTAL=0 - -# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s -tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui . -tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3 # ./ then ./index.html -# Exclude the ~17MB companion APK from tarballs. - -# Ship -scripts/create-release.sh $V -scripts/publish-release-assets.sh $V gitea-vps2 -git push origin main && git push origin --tags # tag or the Releases page stays empty -git push gitea-ai main # main is protected; use the `ai` account - -# Verify the live manifest -curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json -``` - -Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is -done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first -(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with -user benefit. - -### ISO -```bash -UNBUNDLED=1 bash image-recipe/build-debian-iso.sh -``` -ISO builds are **always unbundled** — the default env silently builds the wrong -full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output -filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is -`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO. - ---- - -## 6. Node access quick reference - -- **thinkpad (`.116`) is the local machine** — do not SSH to it; read - `journalctl -u archipelago` and `/var/lib/archipelago/**` directly. -- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user - `archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale. -- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and - it is in real use; don't touch uninvited. -- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex. -- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores. -- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660). -- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74). diff --git a/docs/OPEN-SOURCE-READINESS-PLAN.md b/docs/OPEN-SOURCE-READINESS-PLAN.md deleted file mode 100644 index 75e55f2d..00000000 --- a/docs/OPEN-SOURCE-READINESS-PLAN.md +++ /dev/null @@ -1,322 +0,0 @@ -# Open-Source Readiness Plan — Archipelago public launch - -> Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup. -> A second agent is working the same goal concurrently — before executing any phase, -> diff against `git log` since `7e8d3314` and skip/merge what's already done. -> (Session plan file: `~/.claude/plans/resilient-moseying-reef.md`.) - -## Context - -The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep -exploration passes (docs/structure, code health, secrets sweep) found the repo is -fundamentally strong — README, `apps/` manifest examples, ADRs, the bats lifecycle gate, -1,104 Rust tests — but has hard blockers: **two live Anthropic API keys committed in -tracked files**, node passwords in 7 tracked files, no LICENSE (README links a 404), -5.5 GB `.git` (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea -registry `146.59.87.168:3000` that make every app image unpullable for outsiders, and -~28 internal AI-session/tracker docs mixed into `docs/`. - -**Decisions made by the user:** -1. **Fresh-history publish** — new public repo with a clean initial commit; private repo keeps full history. -2. **Registry: domain + parameterize** — real domain in front of the existing registry; host configurable everywhere. -3. **Deep code cleanup** — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly). -4. **Internal docs: sanitize and keep public** — scrub creds/IPs/hostnames but publish plans/trackers for transparency. - -**Invariant throughout:** the single-node production gate (`tests/lifecycle/run-gate.sh`) -is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E -especially). All cargo verification uses `--all-features` to match CI. Stage by explicit -path, never `git add -A` (shared tree). - -## Current local pass status - -This branch is replayed on top of `origin/main` as `public-prelaunch`. - -Completed locally in this pass: - -- Redacted the two tracked Anthropic API key literals from - `scripts/setup-aiui-server.sh` and - `image-recipe/_archived/build-auto-installer-iso.sh`. -- Removed `Android/app/debug.keystore` and `core/.env.production` from the - source tree; copies were preserved in - `~/Desktop/archipelago-sensitive-backup-2026-07-27/`. -- Reworked `scripts/audit-secrets.sh` to scan tracked source more aggressively - and to catch non-example env files and credential file patterns. -- Reworked `scripts/validate-app-manifest.sh` so the current `app:` manifest - schema can be audited without a Python `PyYAML` dependency. -- Updated root/community docs, CI, PR template, app developer notes, and - container/deployment docs toward public contributor expectations. -- Fixed native FIPS activation fallback: nodes that have the packaged - `fips.service` but not `archipelago-fips.service` now start the available - unit instead of repeatedly failing activation against a missing unit. This - now covers startup, supervisor self-heal, manual dashboard start/reconnect, - and post-onboarding activation. The UI now labels the action as `Start` - instead of making native FIPS look like an installable app. -- Fixed the FIPS app-port relay design so it binds relays to the node's FIPS - ULA instead of wildcard `[::]`, avoiding collisions with Podman-published app - ports such as FileBrowser `8083` and Botfights `9100`. -- Added `docs/nostr-git-source-hosting.md`, a NIP-34/ngit/GRASP source hosting - plan using a Bitcoin Core-style maintainer model: public review and easy - forks, with canonical merge rights held by a small signed maintainer set. - -Verified locally: - -- `./scripts/audit-secrets.sh` passes. -- Full `apps/*/manifest.yml` repository audit passes with warnings only. -- `bash -n` passes for the edited shell scripts. -- Targeted FIPS dashboard vitest passes. -- Targeted Rust tests for FIPS service unit detection and FIPS app relay - address selection pass. - -Verified on a Linux Archipelago verification node: - -- Native FIPS was restored by starting the already-installed packaged - `fips.service`; the daemon became active and joined the FIPS tree. -- Correct local lifecycle API endpoint is HTTP, not HTTPS - (`ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http`). -- Read-only lifecycle run progressed past login and confirmed required - containers, Bitcoin RPC, ElectrumX TCP, and manifest port-drift checks, but - did not complete cleanly: `botfights` and `filebrowser` remained in - `restarting` longer than the matrix window, and the LND `lncli getinfo` - probe hung. Do not run the destructive gate until those live-node issues are - understood. -- After the node updated to `1.7.116-alpha`, `botfights`, `filebrowser`, and - `lnd` were active/running and ports `8083`/`9100` were held by Podman's - `rootlessport` as expected. The packaged `fips.service` remained installed - and enabled but inactive, so the native FIPS service fallback should still - ship before the public launch. - -Still required before public publish: - -- Rotate/revoke compromised credentials listed in Phase 0. -- Finish Phase 1 password/node/token sanitization beyond the two API keys. -- Publish from fresh history after the sanitized tree is final. -- Run full Rust, frontend, Android, and lifecycle gate verification. -- Resolve the live-node lifecycle blockers above, then rerun the read-only - suite followed by the destructive gate only on an approved verification node. -- Decide the canonical Archipelago maintainer npub and merge-maintainer npub - list before publishing the Nostr Git source-hosting workflow. - ---- - -## Phase 0 — Credential rotation (DEFERRED to the pre-publish gate, 2026-08-07) - -> **Sequencing decision (user, 2026-08-07):** rotation/revocation moved from first to -> last. This is safe *only* because the publish is fresh-history — the scrub commits -> never become public, so scrubbing before rotating leaks nothing to outsiders. -> -> **Hard gate: Phase 6 MUST NOT run until every item below is done.** The export is the -> point where a missed literal becomes public and a live key becomes an incident. -> Everything here is still live as of this writing. Phase 6 step 3 now includes an -> explicit rotation sign-off. - -Treat all of these as already compromised; rotate even though we're doing fresh-history: - -- **Anthropic API key #1**: `image-recipe/_archived/build-auto-installer-iso.sh:2837` (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (`ISO_ANTHROPIC_API_KEY`), keep the alpha-baking behavior if desired but never the literal in git. -- **Anthropic API key #2**: `scripts/setup-aiui-server.sh:28` — a *different* live key, not covered by the documented alpha exception. Revoke; parameterize the script. -- **The shared node SSH/sudo/UI password** (two variants) — was in **8** tracked files (see - Phase 1 status) + 24+ commits. Now scrubbed from the tree; still live on the fleet. - Rotate fleet-wide (user task). -- **Gitea `ai` account password + 2 Gitea tokens** — embedded in `.git/config` remote URLs - (not tracked, but leaks in any directory copy/tarball). **Verified 2026-08-07: both tokens - are already dead** — `localhost:3000` and `146.59.87.168:3000` both return 401. Only the - `ai` password on `source.archipelago-foundation.org` is live. Rotate it; switch remotes to - credential-helper storage instead of URL-embedded creds. -- **Framework node SSH** — its password was rotated out-of-band and is not recorded anywhere; - key auth is also rejected. Whoever holds it should capture it before the fleet rotation, or - that node becomes unreachable for the rotation itself. - -## Phase 1 — Secrets & sanitization of tracked files - -**Status 2026-08-07: items 1, 2 and 4 DONE** (`e3b98ed1`, `19082a44`). The password was in -**8 files, not 7** — the reworked audit found three in `.planning/` that this list missed. -`scripts/audit-secrets.sh` is 5/5 green and canary-tested. Items 3 (infra identifiers) and -5 (`.gitignore`) remain. - -1. Strip the password/credential lines from the 7 files: - `docs/PRODUCTION-MASTER-PLAN.md` (lines ~428–429, 454–457, 483, 521–528, 886 — the fleet cred table), - `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`, `docs/archive/HANDOVER-2026-07-02-iso-feedback.md`, - `docs/bitcoin-version-bulletproof-rollout.md`, `tests/production-quality/TRACKER.md`, - `tests/multinode/meshtastic.sh:26`, `neode-ui/test-openwrt.mjs:4` (→ env var). -2. `.gitea/workflows/post-install-tests.yml` — remove `sshpass -p '…'` + default target IP; use secrets/vars. -3. Sanitize infra identifiers repo-wide (in the *sanitize-and-keep* docs and scripts): - replace Tailscale IPs (17 unique, 14 files), LAN IPs (`192.168.1.x`, 93 files), hostnames - (`tx1138`, `shorty-s`, `archy-x250`, `archy-dev-pa`) with placeholders like `` / - `NODE_IP`. Key script targets: `scripts/deploy-config-defaults.sh`, `scripts/deploy-tailscale.sh`, - `docs/operations-runbook.md` (opens with real node IPs), `docs/developer-guide.md`, `docs/api-reference.md`, `docs/hotfix-process.md`. -4. Fix the audit tool that let this happen: `scripts/audit-secrets.sh:28` — remove `\.md$` and - bare `test` from ALLOW_PATTERNS; add `sk-ant-` and password-table patterns; scan all - tracked files not just `*.env`. Run it clean as a Phase-1 exit check. -5. `.gitignore` additions: `.claude/`, `*.key`, `*.pem`, `id_rsa*`, `*.sqlite`, `*.db` - (`.claude/settings.local.json` with creds is currently only ignored by a machine-global rule). -6. Product-security note to raise (not fix now): `password123` is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced. - -## Phase 2 — Repo restructure: deletions, binaries, layout - -Delete (each its own commit): -- **`.planning/` — 199 tracked files, not in the original plan.** GSD phase/session material - (RESUME notes, phase SUMMARYs, `.continue-here.md`); three of them held the fleet password. - Same class as `loop/` and `.agents/`: internal agent working state, not product. Decide - delete-vs-`docs/history/` explicitly — it is the largest un-triaged block of internal - material still tracked. -- `loop/` (AI overnight harness w/ node SSH lines), `.agents/`, `.codex`, `.githooks/pre-push` - (the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history). -- `indeedhub/` submodule + `.gitmodules` entry (points at private HTTP Gitea, breaks `--recursive` - clones); `indeedhub-demo/` (single Dockerfile — merge or drop). -- `RELEASE-NOTES-v1.0.0.md` (superseded by CHANGELOG), `neode-ui/docs/GAMEPAD-NAV-MAP.md` (duplicate of `docs/GAMEPAD-NAV.md`). -- Stray generated HTML: `docs/container-architecture.html` (311 KB), `docs/archive/architecture-review.html`, `docs/archive/lora-functionality.html`. -- `Android/local.properties` from tracking (local absolute path); remove `Android/app/debug.keystore` (standard practice). - -Move out of git (→ release assets on the Releases page, referenced by URL): -- `neode-ui/public/packages/archipelago-companion.apk` (27 MB), `wireguard.apk` (17 MB), `atob.s9pk` (23 MB). -- `Android/archipelago-0.3.0-debug.apk.zip` (16 MB, stale). -- `demo/content/music/*` + heavy `demo/aiui/assets` (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size). -- `neode-ui/dev-dist/` (generated Workbox output) → gitignore. - -Rename/fix the naming lie: `image-recipe/_archived/` contains the *production* ISO builder -(`build-auto-installer-iso.sh`, referenced by `.gitea/workflows/build-iso.yml`). Move live -files up into `image-recipe/`, delete the genuinely archived rest. - -## Phase 3 — Registry domain + parameterization (functional blocker) - -Infra (user assists: DNS + TLS): -- Put a domain (e.g. `registry.archipelago-os.org` / `git.archipelago-os.org`) with HTTPS in - front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS. - -Repo changes: -- Introduce a single source of truth for the registry host (e.g. `REGISTRY_HOST` in - `scripts/lib/` + a default in the orchestrator config). Replace `146.59.87.168:3000` in: - all 56 `apps/*/manifest.yml`, `app-catalog/catalog.json`, `releases/manifest.json`, - `release-manifest.json`, the 11 scripts (`self-update.sh`, `create-release.sh`, - `generate-app-catalog.sh`, `validate-app-manifest.sh`, `first-boot-containers.sh`, …), - both `demo-images.yml` workflows, `demo-deploy/.env.example`, and the Android sources - (`FipsPreferences.kt`, `PartyScreen.kt`). -- Because the catalog is signed: regenerate + re-sign + republish the app catalog after the - manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise). - Signing needs the user's mnemonic → schedule one ceremony after manifests are final. -- Verify: fresh machine with no LAN/tailnet access can `podman pull` one app image via the - domain and the gate node still installs apps after the re-signed catalog lands. - -## Phase 4 — Documentation overhaul - -### 4a. Community/legal files (missing today) -- `LICENSE` — MIT (matches existing README badge). Add `[workspace.package] license` + - `license.workspace = true` in the 5 member Cargo.tomls (also see Phase A4). -- `SECURITY.md` — disclosure address, PGP key, supported-versions; cite the March 2026 audit (`docs/archive/security-code-audit-2026-03.md`). -- `CODE_OF_CONDUCT.md` — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today). -- `CONTRIBUTING.md` edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb - the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model, - non-destructive migrations), versioning policy note for the `-alpha` scheme. -- `CLAUDE.md` — rewrite: keep invariants/build-verify (public-worthy), remove status banner, - node numbers, `gitea-ai` push mechanics, MEMORY references (those move to private notes). - -### 4b. New developer docs (the three real gaps for app developers) -1. **`docs/quadlet-compilation.md`** — how a manifest becomes a Quadlet/systemd unit: naming, - `systemctl --user` lifecycle, where units land, how to inspect/debug one. (Source: - `core/archipelago/src/container/quadlet*.rs`, prod_orchestrator.) -2. **`docs/container-lifecycle.md`** — the 30 s level-triggered reconciler, install/adopt/ - restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped - `docs/bulletproof-containers.md` as the current description; salvage its content.) -3. **`docs/secrets.md`** — `generated_secrets` declaration → materialisation by - `container::secrets` (0600, rootless) → injection; what developers must never do. -- Also: make every example in `docs/app-developer-guide.md` + `apps/*/manifest.yml` copy-paste - work against the new public registry host; add an end-to-end "write your first app" walkthrough - that a stranger can follow with only the public repo + an Archipelago node. - -### 4c. Sanitize-and-keep internal docs (user's transparency choice) -- Keep, after Phase-1 scrubbing: `docs/PRODUCTION-MASTER-PLAN.md`, `docs/UNIFIED-TASK-TRACKER.md`, - `docs/1.8.0-RELEASE-HARDENING-PLAN.md`, `docs/RETICULUM-TRANSPORT-PROGRESS.md`, HANDOFF-*, test - plans, `docs/archive/*` — but **move all session/handoff/tracker material under - `docs/history/`** (extending the existing honest `docs/archive/README.md` pattern) so the - top-level `docs/` reads as current reference only. Add a banner to each: "historical working - document, sanitized; not maintained." -- Remove dangling agent-memory references in tracked docs (`docs/bulletproof-containers.md`, - `docs/RETICULUM-TRANSPORT-PROGRESS.md`, `docs/registry-manifest-design.md`, - `docs/bitcoin-multi-version-design.md` progress block). -- De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status - field; e.g. `docs/APP-PACKAGING-MIGRATION-PLAN.md` → public app-platform design doc). -- Extract North-Star narrative from PRODUCTION-MASTER-PLAN into `docs/ROADMAP.md`; extract - the "run the gate ON the node" philosophy from `docs/multinode-testing-plan.md` into - `tests/lifecycle/TESTING.md`. -- Add `docs/README.md` index (bitcoin/bitcoin `doc/` style): Getting started / Architecture / - App development / Operations / Design docs (ADRs) / History. -- README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized - docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at - CONTRIBUTING.md only. - -## Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit) - -### A. Zero-risk deletions & metadata (S each, own commits) -- **A1** Delete orphan non-compiling StartOS crates: `core/models`, `core/helpers`, - `core/js-engine` (incl. 2 committed `JS_SNAPSHOT.*.bin`), `core/container-init` (~4,100 LOC, - zero references). Verify: `cargo build --workspace && cargo test --all-features`. -- **A2** Delete unreferenced Vue components: `neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue`. Verify: `npm run type-check && npm run build`. -- **A3** Fix `.gitignore` lockfile lines (7: `Cargo.lock`, 15: `package-lock.json`) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles. -- **A4** LICENSE + Cargo license fields (see 4a). Verify with `cargo metadata`. -- **A5** `core/rust-toolchain.toml` pinning `1.95.0`; align `.github/workflows/ci.yml` (remove explicit `toolchain: stable` input so the file wins). Upgrades become deliberate PRs. -- **A6** `core/rustfmt.toml` codifying **defaults only** (`edition = "2021"` + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify `cargo fmt --all -- --check` yields no diff. - -### B. CI guards (zero runtime risk) -- **B1** Enable vitest in CI: run `cd neode-ui && npm run test` locally; fix trivial failures, `.skip`+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration). -- **B2** Raw podman/systemctl **ratchet, not migration**: the 132 raw `Command::new("podman"/"systemctl")` sites use subcommands the `core/container/src/podman_client.rs` wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical `install.rs`, and the prod path intentionally uses Quadlet+systemctl. Add `scripts/ci/raw-podman-ratchet.sh` (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design. - -### C. Clippy suppression trim (`core/archipelago/src/main.rs:8-18`, per-lint commits) -- Remove cheaply: `assertions_on_constants`, `drop_non_drop`, `wildcard_in_or_patterns`, `doc_lazy_continuation`, `enum_variant_names` (targeted allows on serde enums — never rename wire variants). -- Own careful commit: `unused_io_amount` — a **correctness** lint; fix sites with `read_exact`/`write_all` or documented targeted allows (`mesh/serial.rs:456,496` has raw partial reads; serial framing may be intentional). Full test suite + gate after. -- Keep crate-wide with justifying comment: `too_many_arguments`, `type_complexity`; attempt `ptr_arg` (`&Vec`→`&[T]`, mechanical) if time allows — first to cut. -- Verify each: `cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features`. - -### D. dead_code lift — Tiers 1–2 pre-launch, Tier 3 → commented allows + issues -Per-module procedure (one file per commit): remove `#![allow(dead_code)]` → `cargo check ---all-targets --all-features` → triage each warning: (a) genuinely dead → delete; -(b) future-feature/protocol-mandated → targeted `#[allow(dead_code)] // TODO(#NNN): …`; -(c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) → -clippy `-D warnings` + tests → commit. -- **Tier 1 (small/leaf, S each):** `swarm/seed_advert.rs`, `transport/{mesh_transport,lan,chunking,delta}.rs`, `mesh/{crypto,alerts,types,outbox}.rs`, `streaming/mod.rs`, `wallet/mod.rs`. -- **Tier 2 (M each):** `fips/{mod,iface,dial}.rs` (41 external refs → little residual deadness), `mesh/{x3dh,ratchet,steganography,message_types}.rs` — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why). -- **Tier 3 (defer, riskiest):** `mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs`, `transport/mod.rs` — change each blanket allow to `#![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN`. -- Optional S/M win: move `prod_orchestrator.rs`'s 5,034-line `#[cfg(test)]` module to a sibling file via `#[path]` (pure move, halves the 6,291-line file). - -### E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated) -Legacy installers for immich/btcpay/mempool/indeedhub (`core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498`, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging `INSTALL ORCH SKIP` (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920). -1. Run the full gate on the node; grep install logs for `INSTALL ORCH SKIP`. -2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + `adopt_stack_if_exists`; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue. -3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers". - -### Explicitly deferred → public tracked issues at launch -PodmanClient API extension + call-site migration; god-module splits (`install.rs`, `update.rs`, `mesh/mod.rs`); Playwright in CI; Tier-3 dead_code triage; `password123` default hardening. - -## Phase 6 — Fresh-history publish - -1. Freeze: all phases merged on internal `main`, gate green, catalog re-signed. -2. Build the public tree: `git archive`-style export of HEAD (never copy `.git/` — it holds - credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"), - optionally preserving CHANGELOG.md as the human-readable history. -3. **Rotation sign-off (blocking):** confirm every Phase 0 item is rotated/revoked — both - Anthropic keys dead, fleet SSH password changed fleet-wide, Gitea `ai` password rotated, - remotes moved off URL-embedded creds. Do not proceed on "it's scrubbed" — scrubbed and - rotated are different things, and only rotation covers the private history. -4. Pre-publish gate on the export: `scripts/audit-secrets.sh` (fixed version) clean; grep-zero for - `sk-ant-`, rotated-password strings, `146.59.87.168`, tailnet `100.` IPs, `192.168.1.`, - internal hostnames; `du -sh .git` sanity (< ~100 MB); fresh `git clone` + `cd core && cargo build` - + `cd neode-ui && npm ci && npm run build` on a clean machine/container; one app image pull - from the public domain. -5. Publish to GitHub; enable issue templates (already present in `.github/`); file the deferred-work - issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points. -6. Internal repo remains the private full-history remote; decide sync direction post-launch - (recommend: public repo becomes canonical, private keeps only ops/infra notes). - -## Verification (end-to-end) - -- `tests/lifecycle/run-gate.sh` green on the node after Phases 3 + 5E (and after any lifecycle-touching commit). -- CI green on every phase commit: `cargo fmt --check`, `clippy -D warnings`, `cargo test --all-features`, frontend type-check + build + (new) vitest. -- Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer. -- Docs acceptance: a reader following `docs/app-developer-guide.md` + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra. - -## Sequencing / cut-line - -Order (revised 2026-08-07): 1 → 2 → (3 ∥ 4) → 5 (A→E) → **0** → 6. Phase 0 rotation now sits -immediately before publish as a blocking gate rather than first; see the Phase 0 banner for why -that is safe under fresh-history. Phases 1–2 are non-negotiable security; Phase 3 is the -functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any -commit (minimum viable: A1–A6, B1–B2, unused_io_amount fix); Phase 6 last. If the timeline -compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds. diff --git a/docs/PRODUCTION-MASTER-PLAN.md b/docs/PRODUCTION-MASTER-PLAN.md deleted file mode 100644 index cde97ca0..00000000 --- a/docs/PRODUCTION-MASTER-PLAN.md +++ /dev/null @@ -1,1265 +0,0 @@ -# PRODUCTION MASTER PLAN — Archipelago App Platform & Registry - -> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc remains -> the authoritative north-star narrative and detailed workstream history, but for -> "what's left, in priority order" work off the unified tracker instead of hunting -> through §6/§8b here. -> -> **✅ SINGLE-NODE PRODUCTION GATE IS GREEN (2026-06-23): `run-gate.sh` 5/5 on .228, 0 failures.** -> This remains the authoritative plan for the broader north star (manifest-driven -> platform, registry-distributed manifests, external marketplace), but it is no -> longer a hard priority banner blocking all other work. Remaining workstreams are -> in §6 / §8b. Next exit-criteria: multinode (`docs/multinode-testing-plan.md`) + -> workstreams B/C/D. -> -> Last updated: 2026-06-26 · zombie-container guard + gitea launch-port fix shipped, binary `040df5ce` rolled to the fleet (see §8b SESSION h). Prior: orchestrator Fix A+B (`a721532f`/`e0343137`) deployed + proven. - ---- - -## 1. The North Star - -Make Archipelago a **world-class, developer-ready app platform** where: - -1. **Every app is manifest-driven** — install/run/update/uninstall needs only the - app's manifest (+ catalog entry). **Zero OS-level code reliance**: no per-app - Rust installers, no `sudo mkdir/chown`, no host provisioning. -2. **Manifests are distributed via the (signed) registry**, not baked into the - binary OTA as disk files. Bumping/adding an app = a signed catalog change. -3. **Third-party developers can build and ship apps via an external registry** — - a decentralized marketplace (DID-signed manifests, Nostr discovery, reputation), - not a gatekept central store. `archy app validate/render/install/test` tooling. -4. The platform stays **rootless, secure-by-default, elegant, robust, and - 100%-uptime-capable** (reboot-survivable, self-healing, no data loss on migrate). - -**Definition of done:** the production test gate (§5) is green for the app set on -real nodes. Until then, this plan is the priority. - -## 2. Invariants (never violate) - -- **Rootless Podman only.** No rootful, no Docker-socket mounts, no privileged - containers unless explicitly approved. (ADR-001, ADR-009.) -- **No app-specific business logic in the Rust backend.** The orchestrator owns - the lifecycle state machine; apps are declarative. Legacy `install_immich_stack` - (hardcoded `podman run` + `sudo chown`) is the anti-pattern being deleted. -- **Secrets are manifest-declared** (`generated_secrets`, materialised by - `container::secrets` 0600/rootless, idempotent + self-healing) — never hardcoded, - per-app, or logged. Replaces the deleted `ensure_fmcd_password`. -- **Migrations never destroy data.** Preserve `/var/lib/archipelago/`, - generated secrets, displayed credentials, public ports, and adoption container - names. Always provide a rollback path. Stop/recreate only when necessary. -- **Verify on the real node .228 before any tag.** (Fleet/multinode verification is - a separate pass → `docs/multinode-testing-plan.md`.) - -## 3. Current state (2026-06-21) - -- **~40 apps are manifest-based and Quadlet-migrated** (survive - `archipelago.service` restart + reboot). Exhaustive per-app table: - `docs/archive/app-registry-status-2026-06-21.md`. -- **Legacy holdout: immich** — the one app with **no manifest** and a hardcoded - Rust stack installer (in-cgroup, not Quadlet). 3 containers, healthy, live data. - The migration proof case. -- **Manifests still travel by OTA disk rsync** (`apps/ → /opt/archipelago/apps`). - The signed catalog (`app-catalog.json`) currently distributes **only image - overrides** — not full manifests. Gap closed by workstream B. -- **The 4 companions** (`archy-bitcoin-ui`, `-lnd-ui`, `-electrs-ui`, - `-fedimint-ui`) build from `docker/` contexts via `companion.rs`, not the - manifest registry — a later phase folds them in. -- **No app has passed the formal production gate.** That is the blocker. - -## 4. Workstreams (each links its authoritative detail doc) - -| # | Workstream | Detail doc | Status | -|---|-----------|-----------|--------| -| A | **Manifest-driven app platform** — packaging contract, single/multi-container runtime, routing, controlled hooks, dev tooling (6 phases, security model, migration rules) | `APP-PACKAGING-MIGRATION-PLAN.md` | mostly done; immich + multi-container polish remain | -| B | **Registry-distributed manifests** — catalog carries full signed manifest; orchestrator installs from registry; disk = migration fallback | `registry-manifest-design.md` | **phases 1+2 done** (node consume + opt-in publisher embed); not yet flipped on for the fleet | -| C | **Developer-ready external registry** — 3rd-party DID-signed manifests, decentralized Nostr discovery (NIP-78 kind 30078) + trust score, `archy app …` tooling | `marketplace-protocol.md`, `app-developer-guide.md` | design exists; tooling + trust UX pending | -| D | **Distribution backbone** — signed catalog, BLAKE3 content-addressing, iroh swarm (origin-always-wins) | `dht-distribution-design.md` | phases 0–2 code-complete (worktree) | -| E | **Production test gate** — 5× lifecycle on **.228**, per-app L1/L2 matrix; multinode is split out → `multinode-testing-plan.md` | `tests/lifecycle/TESTING.md`, `bulletproof-containers.md` | **✅ .228 5×-GREEN (110/110 ×5, 0 not-ok, 2026-06-23)** — but this is DESTRUCTIVE-tier / ~8 core apps only; see §6c for the coverage gaps | -| F | **Lifecycle perfection — cascade + progress + ALL apps** — extend the gate to uninstall/reinstall (cascade), real install/uninstall progress UI, and EVERY installed app (not just the 8 core). The "insanely-perfect OS/container environment" bar. | §6c (below), `tests/lifecycle/TESTING.md` | **IN PROGRESS (2026-06-26)** — root bug FIXED: uninstall could hang → ghost/stuck-bar/reinstall-block (`71cc9ac4`, unbounded systemctl/podman in `quadlet::disable_remove`); `cascade-uninstall.bats` **7/7 green on .228** w/ binary `ae349a75`. Remaining: wire CASCADE into the canonical gate run, progress-UI truthfulness, all-apps matrix, guardian/IBD state. | - -**Orchestrator architecture** (foundation for A/B): `archive/rust-orchestrator-migration.md` -(ProdContainerOrchestrator, BootReconciler 30s level-triggered reconcile, adoption -scan, Quadlet rendering) and `bulletproof-containers.md` (the six container failure -modes FM1–FM6 + the desired-state-first reconciler that fixes them). - -## 5. Production test gate (exit criterion) - -An app is **production-ready** only when `tests/lifecycle/run-gate.sh` is green -across the full matrix — install / UI-reachable / stop / start / restart / -reinstall / **reboot-survive** / **archipelago-restart-survive** / uninstall — -**5× on .228** (`ARCHY_ITERATIONS=5`). **The gate runs ON the node** (it uses local -podman/systemctl/bitcoin probes; running it via RPC from another host silently -tests the runner). **Multinode / fleet verification (.198 + others) is a SEPARATE -plan — `docs/multinode-testing-plan.md` — NOT part of this single-node criterion.** -Coverage today: L0 unit (631 ●), L1 RPC ● for 6 core apps, L2 UI ● dashboard + -proxies; L3 survival ◐; ~30 apps have zero automated coverage. - -> ⚠️ **The 2026-06-23 5×-green is NOT the full bar.** `run-gate.sh` runs only the -> **DESTRUCTIVE tier** (stop/start/restart/survive) over ~8 core apps; it **skips -> uninstall/reinstall** (CASCADE is gated behind `ARCHY_ALLOW_CASCADE_DESTRUCTIVE`, -> never set by the gate) and tests no install/uninstall **progress UI**. Real -> uninstall/reinstall/progress bugs (immich + grafana) were found in manual testing -> right after — see **§6c (workstream F)** for the gap and the expanded-gate plan. -> The true "every app, fully" criterion is F's definition-of-done, not this run. - -## 6. Immediate sequence (live workstream) - -1. ✅ **B-phase 1** — `manifest` field on `AppCatalogEntry`; `load_manifests` - catalog-wins merge; `manifest_dir` kept (build-source catalog manifests skipped - in phase 1); unit tests. *(commit 220666d3)* -2. ✅ **B-phase 2** — `EMBED_MANIFESTS` publisher generator + round-trip guard. - *(7bfbe8fe; signing via existing ceremony — not yet flipped on for the fleet.)* -3. ✅ **C immich proof** — immich is a manifest-driven stack (immich + immich-postgres - + immich-redis) installed via `install_stack_via_orchestrator`; legacy installer - is now fallback-only. Live-migrated + verified on .228. Found+fixed: container_name - duplicate-on-shared-PGDATA, version-digit validation, partial-fallback hardening, - data_uid 100998. Canonical app_id `immich` (title+icon). *(9e6c5370, d5ef4573)* -4. ✅ **Reboot-survival** — podman-restart.service enabled (startup, fleet-wide) - for the podman-`--restart` path. *(f160e0c4)* -5. ✅ **E** — 5× gate on **.228** (`ARCHY_ITERATIONS=5`) is **GREEN: 5/5, 0 not-ok** - (2026-06-23). Two real orchestrator bugs were found + fixed en route (package.stop - per-app grace; package.restart phantom stack-member injection → `order_present_containers`, - commit 92d7f52d) plus two single-shot-read probes hardened (bitcoin-knots state, immich - lan_address). The single-node criterion is met. -6. ✅ Banner demoted (this doc, 2026-06-23). Next: multinode pass + workstreams B/C/D. - -**Multinode / fleet verification (.198 and the rest) is split into its own plan:** -`docs/multinode-testing-plan.md`. Do it AFTER the .228 single-node gate is green. - -**Not yet done / deliberate follow-ups:** flip `EMBED_MANIFESTS` on for the -published catalog (then sign) to actually distribute manifests via the registry; -Phase-3 `use_quadlet_backends` rollout so orchestrator backends are Quadlet (not -just podman-`--restart`). - -## 6b. Post-deploy task order (agreed 2026-06-23) - -After the 2026-06-23 multinode test deploy (latest backend + UX frontend to .116/.198/.228 -+ Tailscale testers), do these IN ORDER: -1. **netbird #20 ph4** — the last real manifest migration (workstream A). -2. **Phase-3 `use_quadlet_backends`** — orchestrator backends become Quadlet units. -3. **§6c Lifecycle perfection** (workstream F) — the comprehensive uninstall/reinstall + - progress-UI + all-apps gate expansion below. - -## 6b-bis. Bitcoin multi-version bulletproofing (2026-06-29) — READY TO MERGE + DEPLOY - -Branch `bitcoin-version-bulletproof` (base `095a76cd`). Fixes the "switch version silently -fails / crash-loops" class + a data-access mismatch that can corrupt a node's index. All -code + images + catalog + frontend DONE; **.228** carries it (Knots chainstate mid-reindex -recovery). The **coordinated fleet rollout** (OTA binary+frontend, mirror catalog publish, -`:latest` repoint sequencing, full switch-matrix test) is the remaining work — fold it into -the next release. **Authoritative detail + exact remaining steps + test matrix → -`docs/bitcoin-version-bulletproof-rollout.md`.** Pairs with `docs/bitcoin-multi-version-design.md`. - -## 6c. Lifecycle perfection — what "green" MISSED (workstream F, the perfection bar) - -**Why this exists:** the 2026-06-23 single-node gate went 5×-green but is **NOT** the -"every app fully lifecycle-tested" guarantee a user reasonably assumes. The canonical gate -(`run-gate.sh`) only runs the **DESTRUCTIVE tier** (stop / start / restart / survive) over -**~8 core apps** (bitcoin-knots, btcpay, electrumx, lnd, mempool, immich, fedimint, -filebrowser). It explicitly **SKIPS uninstall/reinstall** (the CASCADE tier is gated behind -`ARCHY_ALLOW_CASCADE_DESTRUCTIVE`, which `run-gate.sh` never sets) and has **zero coverage** -for the other ~30 apps (grafana, jellyfin, vaultwarden, penpot, nextcloud, photoprism, -uptime-kuma, homeassistant, … — see `archive/app-registry-status-2026-06-21.md`). So uninstall, -reinstall, install-progress UI, and most apps were never under test. - -**Real bugs found in manual multinode testing on .198 (2026-06-23) — the motivating evidence:** -- **Uninstall is broken for immich + grafana:** takes very long, the progress bar sits at a - **solid full-red with no real progression**, and the app **does not actually uninstall** — - it still appears in **My Apps** afterward (ghost entry / state not cleared). -- **grafana reinstall just stops** partway (no completion, no clear error). -- **fedimint guardian** suddenly showed **"starting up — Guardian opens a wait page until - Bitcoin finishes initial sync" / "starting"** on that node — verify this is correct - wait-for-IBD behavior vs a stuck/false state (it's a backend that depends on bitcoin sync). - -**✅ 2026-06-26 — root cause of the immich/grafana uninstall trio FOUND + FIXED (`71cc9ac4`).** -Single cause: `quadlet::disable_remove()` (first op in uninstall teardown, via companion + -orchestrator) ran `systemctl --user stop` / `daemon-reload` / `podman rm -f` with **no timeout**. -On rootless podman a generated unit can wedge "deactivating" while podman hangs → `systemctl stop` -blocks forever → the spawned uninstall task returns neither Ok nor Err, so (a) `set_uninstall_stage` -never fires → **frozen full-red bar**, (b) `remove_package_state_entry` never runs → **ghost stuck in -`Removing`**, (c) the install guard rejects reinstall (`already Removing`). The spawn wrapper already -reverts state on Err/removes on Ok — only a *hang* stranded it. Fix bounds all three calls -(stop→`QUADLET_STOP_TIMEOUT` + SIGKILL/reset-failed escalation; daemon-reload→30s; podman rm→timeout). -**Validated live: `cascade-uninstall.bats` 7/7 on .228** (binary `ae349a75`) — grafana install → -uninstall (no ghost, data dir gone) → reinstall → running → cleanup. NOTE: proves the happy path + -no-regression; the original hang was load/timing-induced and not separately reproduced. - -**Workstream F scope — the gate must grow to (in priority order):** -1. **CASCADE tier in the canonical gate:** uninstall → verify the app is GONE from My Apps / - `container-list` / package state (no ghost), data preserved per policy, then reinstall → - verify it returns healthy. Catch the immich/grafana ghost + reinstall-stops bugs. - *(✅ DONE `b7d92107`: `run-gate.sh` now runs ONE cascade pass after the 5× loop when - `ARCHY_GATE_CASCADE=1` (+`ARCHY_ALLOW_DESTRUCTIVE=1`), counted into the tally — opt-in so default - behavior is unchanged, and deliberately NOT folded into all 5 iterations. `cascade-uninstall.bats` - 7/7 on .228. Next: extend cascade coverage beyond the single throwaway app to the multi-container - stacks, e.g. an immich/btcpay cascade variant.)* -2. **Progress-UI assertions:** install AND uninstall must report monotonic, truthful progress - (not a stuck full-red bar); a long op must surface a real stage/percentage and a terminal - success/failure — no silent hang. (Likely both a backend progress-event fix AND a UI fix.) - *(✅ 2026-06-26 `9f17ba68`: the "stuck full-red bar" was `AppCard.vue` hardcoding the uninstall - bar to `w-full bg-red-400/60 animate-pulse` — solid, full, red, fake-pulse. Now derives a real - percentage from the backend's existing `uninstall-stage` label ("Stopping containers (X/N)"→10–50%, - "Cleaning up volumes"→70%, "Removing app data"→90%) and renders like install (neutral fill, real - width+%, shimmer). FE built `index-DtZyZomC.js`, rolled to .228/.116/.198/.89 (+.88/.5/.120). - STILL TODO: a bats/UI assertion that the bar is monotonic + lands on a terminal state; possibly a - backend numeric-progress field so the UI doesn't parse stage strings.)* -3. **ALL-apps coverage:** a generic per-app lifecycle matrix (install / UI-reach / stop / start / - restart / uninstall / reinstall / reboot-survive) driven by the manifest set, so grafana and - the ~30 uncovered apps are gated too — not just the 8 core. Manifest-driven, so new apps are - covered automatically. - *(✅ 2026-06-26 `43934eef`: `bats/all-apps-lifecycle.bats` — DESTRUCTIVE counterpart to the - read-only `all-apps-matrix.bats`. Discovers the app set from My Apps ∩ the node `catalog.json`; - drives stop/start/restart for every app and, under `ARCHY_ALLOW_CASCADE_DESTRUCTIVE`, a FULL - teardown (uninstall→no-ghost→reinstall) with the catalog `{dockerImage, containerConfig}` as the - reinstall spec. PROTECTED (never touched): bitcoin*/electrum* (resync cost) + lnd/btcpay*/fedimint* - (irreversible wallet loss — user asked to protect only bitcoin+electrum; wallet apps added for - safety, override via `ARCHY_MATRIX_PROTECT`). Validated on .228 (discovery + 1-app lifecycle - green). HEAVY/destructive → a supervised pass on LAN nodes (.116/.198/.228), NOT folded into - run-gate. Invoke: `ARCHY_ALLOW_DESTRUCTIVE=1 ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1 ARCHY_PASSWORD=… - ARCHY_SCHEME=https bats bats/all-apps-lifecycle.bats`.)* - **✅ FIRST FULL DESTRUCTIVE RUN on .228 (2026-06-26):** lifecycle **11/11 clean**; teardown - **8/11** (immich 3-container stack incl.) — and it surfaced **3 real reinstall bugs** (the payoff): - 1. **fresh-install bind-dir ownership = root:root** → EACCES on reinstall (jellyfin `/config` - denied exit 139; netbird-server can't open its SQLite store). Fix B's chown-to-parent only - runs on the reconcile path, **not** `package.install`. The important orchestrator fix. - 2. **netbird reinstall adopts leftover containers → skips the manifest cert/file render** - (tls.crt/key/nginx.conf never written → proxy can't start → app reads absent). Only a fully - clean reinstall renders them. - 3. **portainer image pin `lfg2025/portainer:2.19.4` is `manifest unknown`** (never pushed to the - registry) and the pin OVERRIDES the RPC dockerImage → portainer is un(re)installable - fleet-wide. Registry/catalog data bug (push the image or change the pin). - .228 restored (jellyfin+netbird via manual chown / clean reinstall; all installed apps running, - 28 ctrs; portainer left uninstalled — uninstallable until #3 fixed). TODO: fix #1 (extend chown - to install path) + #2 + #3; add reboot-survive + UI-reach per app to the matrix. -4. **Guardian/IBD-dependent states:** assert that "waiting for bitcoin sync"-style states are a - legitimate, surfaced wait (with a path to ready) and never a permanent stuck state. - -**Definition of done for F:** the expanded gate (CASCADE + progress + all-apps) is 5×-green on -.228, then re-verified across the multinode fleet — i.e. an *insanely-perfect* OS/container -environment where every app installs, runs, updates, uninstalls, and reinstalls cleanly with -honest progress, no ghosts, no data loss, reboot-survivable. - -## 7. Release blockers & operational gotchas (durable) - -Carried forward from prior handoffs (deduped against persistent memory): - -- **Rootless control-plane responsiveness** — slow `podman ps`/store cleanup at - startup must not surface a false "no apps installed" UI. **My Apps must preserve - last-known apps during scanner backoff**, never show empty during a transient. -- **Reboot survival** — gate on ≥3 (prefer 5) consecutive clean post-reboot - lifecycle passes. Quadlet units under `user.slice` survive `archipelago.service` - restart; legacy in-cgroup containers get SIGKILLed and reconciled back. -- **Startup patterns** — wait on a socket/health, never `sleep`. Tailscale waits - for its socket; Fedimint Guardian waits for Bitcoin RPC `initialblockdownload:false` - before launching fedimintd (proxy/wait companion on :8175 during IBD). -- **Bitcoin must run full** (`txindex=1`, non-pruned) for ElectrumX/mempool. -- **Adoption** — match existing containers by name and adopt without recreate; - record a migration version in app state; preserve Nostr signer bridges - (IndeeHub needs `/nostr-provider.js` served, not just port reachability). -- **Image presence** — use bounded targeted `podman image inspect`, not - `podman image exists` (avoids store-walk stalls). -- **Companion rebuilds** — `companion.rs` must rebuild `:latest` when the build - context changes (staleness check), else baked-in fixes (e.g. guardian CSS) never - reach nodes. `:local` is a manual override, never auto-rebuilt. - -## 8. Roadmap - -**Pipeline:** Feature Testing (internal) → User Testing (controlled hardware) → -Beta Live (public). Hardening priorities feeding the gate: - -- **P0** Container app reliability — bulletproof install/health/restart/uninstall - across all apps, dependency chains, multi-container stacks. -- **P0** Networking stack first-install → reboot-proof (WireGuard/NetBird, Tor - hidden services, LND Connect). -- **P1** LUKS2 full-partition encryption for `/var/lib/archipelago/` - (AES-256-XTS, Argon2id, key from setup password + hardware salt). -- **P1** Meshtastic plug-and-play parity with MeshCore. -- **P1 ✅ CODE-COMPLETE** (branch `companion-mobile-ux`, 2026-06-23; needs - on-device + mobile-web verification before merge to `main`) — Mobile app-launch - UX — drop the "this app opens in a tab" interstitial. - Two surfaces (both: no interstitial screen, launch the app directly): - - **Companion app (Android):** open **every** app in the **in-app WebView** - (not just non-iframeable ones) — *and* carry the current mobile-iframe footer - controls into the WebView (back/forward/reload/close — good, useful UX). - - **Mobile web browser (PWA):** open tab-apps directly in a **new browser tab**. - Touch points: `neode-ui/src/stores/appLauncher.ts`, `AppLauncherOverlay.vue`, - the Android in-app WebView bridge, and the mesh-mobile iframe footer controls. - (Reference prior work: `b5a9deb8` in-app webview for non-iframeable apps, - `d1fbcd9b` "open in browser" via native bridge.) - - **✅ Done (branch `companion-mobile-ux`):** mobile launches now use the - store-driven panel (no route push) so the background tab no longer changes and - closing returns you where you launched; tab-only apps open directly (in-app - WebView on companion via `openInApp`, new browser tab on PWA) with **no - interstitial**; the Android `InAppBrowser` (`WebViewScreen.kt`) gained a bottom - footer bar (back/forward/reload/open-in-browser/close) + a centered loading - screen (favicon + progress); a shared `AppLoadingScreen` (icon + progress) - replaced the black/spinner loaders on the app session **and** legacy iframe - overlay; the dashboard is pinned to `100dvh` on mobile so the mesh chat/tools - panes stop sliding under the tab bar in mobile browsers (no-op in companion); - ElectrumX shows its real icon in My Apps. Companion APK bumped to **v0.4.7** - (versionCode 11) with a committed shared debug keystore so updates install - without an uninstall. **Not yet:** merge to `main`; publish the 0.4.7 companion - download (deferred until the gate work lands so they ship together). - -**Post-beta (deferred — do not start until gate is green):** P2P encrypted -voice/video (WebRTC over federation via Tor); watch-only wallet + mesh BTC -hardening; paid swarm streaming + IndeeHub source (`phase4-streaming-ecash-plan.md`); -Meshroller Rust-native mesh AI (`meshroller-integration-design.md`); dual-ecash -phases 2–6 (`dual-ecash-design.md`). - -## 8b. SESSION STATE + RESUME (updated 2026-06-26) — READ §8b "CURRENT STATE + RESUME" FIRST - -### ▶ SESSION i (2026-06-30) — CURRENT HANDOFF / 1.8.0 OTA RESUME - -**Branch/worktree:** currently on `bitcoin-version-bulletproof`, not `main`. Worktree is dirty. -Do **not** discard mesh changes: they include E2E/transport indicator plumbing and the Meshtastic -receive-path fixes below. Separate recovery note: `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`. - -**What was done this session:** -1. ✅ **Local Rust release gate fixed and green.** `cargo test -p archipelago --bin archipelago` is - green: **849/849** after fixing stale tests and the invalid `fedimint-clientd` manifest - (`cpu_limit` was `0.25`, invalid for the current schema; now integer). `cargo check -p archipelago` - also green after mesh edits. -2. ✅ **Catalog/release static gates green.** `python3 scripts/check-app-catalog-drift.py --release - --strict` is green. `scripts/check-release-manifest.sh` is green for the currently staged - `1.7.99-alpha` manifest/artifacts. `npm run build` and `npm run type-check` are green. -3. ✅ **Frontend unit gate fixed.** `npx vitest run --silent` now green: **81 files / 668 tests**. Fixes - were test-only: add `router.onError` to the login test router mock and update the `AppIconGrid` - mobile unresolved-new-tab expectation to match current app-launcher behavior. -4. ✅ **Workstream F harness gap closed.** `tests/lifecycle/bats/cascade-uninstall.bats` now asserts - uninstall progress truthfulness via backend `uninstall-stage`: stage must be parseable, monotonic, - below 100 before terminal absence, and present before the app disappears. Non-destructive skip-mode - parse check is green: `ARCHY_PASSWORD=dummy bats tests/lifecycle/bats/cascade-uninstall.bats` → 7 skip-ok. -5. ✅ **3ccc → .116 Meshtastic receive bug taken over and partially live-validated.** Context: `3ccc` - is the stock/non-Archy Meshtastic peer. The bug was LoRa text from `3ccc` not surfacing in - `.116` `mesh.messages`. Root causes/fixes: - - The prior attempted fix dropped any packet older than 10 minutes by `rx_time`; live `.116` logs - showed `FromRadio.packet` from `!433e3ccc` being dropped as stale (`rx_time` about an hour old). - The window is now **24h**, so recent radio FIFO/store-forward backlog surfaces instead of vanishing. - - Radios with unset clocks can report tiny nonzero epoch values; those are now treated as unknown, - not stale. - - Serial prevalidation was rejecting valid `FromRadio.queueStatus` frames (`field 11`, live bytes like - `5a04100e1810`) as corrupt payloads; field 11 and other modern non-message `FromRadio` variants - are now accepted/ignored instead of poisoning the stream. - - Focused Meshtastic tests green: **8/8**, including `packet_to_inbound_frame_accepts_recent_meshtastic_backlog` - and `packet_to_inbound_frame_accepts_stock_peer_with_unset_clock`. - - Deployed patched binary to **.116**: sha256 - `028ec6ff9a60ca8970c081987457d78ed1c517cd81f7089f51b9a01745b5c3c4` at `/usr/local/bin/archipelago`. - Service active. Post-deploy checked window showed `FromRadio field=11` accepted and no new - `Dropping stale ... !433e3ccc` entries. - - There are stale other-agent `RXDIAG` shell watcher processes on `.116`; leave them unless they - actively interfere. -6. ✅ **Phase-3 Quadlet read-only check on .116 skip-clean.** Copied lifecycle tests to `.116` and ran - `bats bats/use-quadlet-backends-install.bats`: **6/6 skip-clean** because no backend `.container` - units exist. This confirms `use_quadlet_backends` is not active on `.116`; Phase-3 remains a rollout gate. - -**Commands/results worth trusting:** -- `cargo test -p archipelago --bin archipelago` → 849/849 green. -- `npx vitest run --silent` from `neode-ui/` → 81 files / 668 tests green. -- `npm run build` from `neode-ui/` → green, bundle `index-CYaDgfX3.js`. -- `python3 scripts/check-app-catalog-drift.py --release --strict` → green. -- `scripts/check-release-manifest.sh` → green for **v1.7.99-alpha** staged artifacts. -- `tests/release/run.sh --manifest` was rerun after `cargo fmt`; it previously reached frontend tests, - which are now fixed. Re-run it from scratch as the next static gate. - -**Remaining blockers / decisions before 1.8.0 OTA:** -1. **Release version metadata is not 1.8.0 yet.** `releases/manifest.json`, Cargo, and npm still say - `1.7.99-alpha`; `CHANGELOG.md` top says `v1.8.00-alpha` (note double zero). Do not silently publish - until the release version naming is decided (`1.8.0-alpha` vs `1.8.00-alpha` vs `1.8.0`). -2. **Workstream B signing is blocked on the offline release-root mnemonic.** `docs/workstream-b-signing-runbook.md` - says catalog distribution/embedded manifests are live, but authenticity requires the publisher to pin - `RELEASE_ROOT_PUBKEY_HEX` and sign `releases/app-catalog.json` with `RELEASE_MASTER_MNEMONIC`. - This cannot be automated by an agent without the offline mnemonic. -3. **Phase-3 `use_quadlet_backends` is implemented but default-off.** Completing this requires explicit - node/fleet flag rollout plus backend reinstall/migration verification. `.116` currently skip-clean only. -4. **Bitcoin multi-version coordinated rollout is still separately owned/blocked by its runbook.** See - `docs/bitcoin-version-bulletproof-rollout.md`; do not repoint `bitcoin-knots:latest` before fixed binary - is fleet-wide. -5. **True RF validation of 3ccc requires either a live 3ccc send or waiting for another FIFO/backlog packet.** - Parser/unit coverage and `.116` logs strongly validate the drop-path fix, but no human was available to - send a fresh 3ccc message during this session. - -**Immediate next steps for the next agent:** -1. Run `tests/release/run.sh --manifest` from repo root again; frontend unit failures are fixed, so expect - it to pass or continue from the next failing stage. -2. If `.116` is still the canary, monitor logs after any 3ccc activity: - `journalctl -u archipelago --since "