diff --git a/docs/HANDOFF-2026-07-20-fips-peer-files.md b/docs/HANDOFF-2026-07-20-fips-peer-files.md new file mode 100644 index 00000000..74f18c16 --- /dev/null +++ b/docs/HANDOFF-2026-07-20-fips-peer-files.md @@ -0,0 +1,238 @@ +# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill + +Written for a fresh session that will **cut the OTA release and build the ISO**. +Everything below is already committed and pushed to `gitea-ai/main`. Last release +was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**. + +--- + +## 1. What this release carries (3 commits on top of v1.7.105-alpha) + +| Commit | What | User-visible? | +|---|---|---| +| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes | +| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) | +| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly | + +### `9e3ac9ba` — mobile transport pill +`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header +carries the name on mobile), and the transport pill was nested inside it — so it +vanished below 768px. Added a separate `md:hidden` pill next to the peer icon. +Frontend was rebuilt and the class verified present in the emitted bundle. + +Caveats worth knowing (pre-existing, not introduced here): +- On this code path the backend only ever emits `fips` or `tor`, so the `mesh` + and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead. +- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a + hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie. + The peer-files pill does not. + +### `3ab7fb52` — full error chain in logs +`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every +peer-files failure read exactly `RPC error on content.browse-peer: Failed to +connect to peer` with the real cause discarded. Now `{:#}`. The client-facing +message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so +no internal detail leaks. **This fix applies to every RPC method, not just +browse-peer.** + +### `5fd0d6c3` — typed FIPS config + mDNS +`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal. +Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key +does not degrade — **the daemon refuses to start and the node leaves the mesh**. +Now a typed serde struct tree, verified field-by-field against jmcorgan/fips +**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and +the pre-existing schema test. All pass. + +Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0) +so co-located nodes peer directly instead of depending on the public anchor. + +> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup +> drift check at `server.rs:864` compares the freshly rendered config against +> what's on disk. The render differs now, so it reinstalls the config and +> restarts the FIPS daemon **once**. This is the intended self-healing path and +> settles immediately. Do not mistake it for a regression. + +Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig` +has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it +harmlessly (verified against the v0.3.0 source). It self-activates on upgrade. + +--- + +## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled + +Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1** +(2026-07-19). Verified before touching anything: + +- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day. +- **Config forward-compatible** — every key we emit exists in 0.4.1. +- **Asset names match** what `fips/update.rs` expects (`fips__.deb` + + `checksums-linux.txt`), so the in-product updater should work. + +### Upgraded so far (2 of N) +| Node | Before | After | Result | +|---|---|---|---| +| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` | +| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) | + +The OptiPlex was still running the **old string-rendered config** and 0.4.1 +accepted it — empirical confirmation of the compat analysis, not just desk work. + +### Upgrade recipe (nodes cannot reach GitHub — sideload) +```bash +# 1. On a host with GitHub access: +curl -sL -o fips_0.4.1_amd64.deb \ + https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb +curl -sL -o checksums-linux.txt \ + https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt +sha256sum fips_0.4.1_amd64.deb # must match checksums-linux.txt +# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded + +# 2. Sideload: +scp fips_0.4.1_amd64.deb archipelago@:/tmp/ + +# 3. On the node — the same command update.rs uses: +sudo -n systemd-run --collect --wait --quiet --pipe -- \ + env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \ + /tmp/fips_0.4.1_amd64.deb + +# 4. Restart the ACTIVE unit — it is archipelago-fips.service, +# NOT fips.service (which is inactive on these nodes): +sudo -n systemctl restart archipelago-fips.service + +# 5. Verify: +fipsctl --version +sudo -n fipsctl show links # expect anchor 185.18.221.160:8443 connected +sudo -n fipsctl show tree # expect is_root: false, depth > 0 +``` + +### ISO implication (important) +`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from +**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64 +only). So a freshly built ISO will pick up whatever main is that day — probably +≥0.4.1, but it is not deterministic. Pinning is an open item in +`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1 +before building the release ISO** so the shipped version is knowable. + +--- + +## 3. The original bug — peer cloud files not loading + +**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit +because it would be easy to read this as closed. + +What is established: +- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links` + → `[]`, `show tree` → `is_root: true, depth 0`. An island. +- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the + public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on + ICMP, 443/8443/8668 all time out. `show transports` showed + `packets_sent: 760, packets_recv: 0` on both UDP and TCP. +- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock + Tailscale anti-spoof DROPs). +- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's + WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`). +- With no FIPS tree, everything falls back to Tor. Every peer in + `federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`. +- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×, + HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`. + +What is **not** established: why three specific `content.browse-peer` calls +failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never +reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic +is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and +the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is +cold-circuit timeouts on first fetch after idle — **a guess, not a finding.** +`3ab7fb52` means the next occurrence will log the actual cause. + +### Corrections to earlier claims in this session +- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the + `ip:port` in `fipsctl connect` is only an underlay endpoint hint. +- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy + (`delivery_ratio` 1.0 both directions, bloom filter syncing). The + `bytes_recv: 0` link counters are simply uninstrumented in 0.3.0. + +--- + +## 4. Open items — decisions NOT taken + +1. **Second FIPS anchor (user asked for this; not built).** Needs a host running + FIPS that is reachable from the restricted WiFi. Candidate found: OVH + **`146.59.87.168`** — pings fine from the thinkpad and general egress works + (github 200), while the upstream anchor fails even ICMP there. But it does not + run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** — + a production change, deliberately not made unprompted. Code side is easy after: + `fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list + (`default_public_anchor()` → `default_public_anchors() -> Vec`). +2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228` + (`100.64.204.114`) has been **offline ~20h** and could not be included. +3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet, + so mDNS is not actually live anywhere. That is what this OTA is for. +4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so + mDNS may not rescue that particular node even after the OTA. It will help + co-located nodes on sane networks. +5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO. + +--- + +## 5. Release ritual (from prior sessions — follow exactly) + +Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`, +`release-manifest.json`, `releases/manifest.json` modified, and an untracked +`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another +agent may share this tree; never `git add -A`. + +```bash +V=1.7.106-alpha + +# Frontend build — MUST verify dist actually changed (build can silently no-op) +cd neode-ui && npm run build # → web/dist/neode-ui/ +grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js # sanity + +# Backend +cd core && cargo build --release -p archipelago +# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache +# corruption — rebuild with CARGO_INCREMENTAL=0 + +# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s +tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui . +tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3 # ./ then ./index.html +# Exclude the ~17MB companion APK from tarballs. + +# Ship +scripts/create-release.sh $V +scripts/publish-release-assets.sh $V gitea-vps2 +git push origin main && git push origin --tags # tag or the Releases page stays empty +git push gitea-ai main # main is protected; use the `ai` account + +# Verify the live manifest +curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json +``` + +Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is +done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first +(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with +user benefit. + +### ISO +```bash +UNBUNDLED=1 bash image-recipe/build-debian-iso.sh +``` +ISO builds are **always unbundled** — the default env silently builds the wrong +full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output +filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is +`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO. + +--- + +## 6. Node access quick reference + +- **thinkpad (`.116`) is the local machine** — do not SSH to it; read + `journalctl -u archipelago` and `/var/lib/archipelago/**` directly. +- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user + `archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale. +- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and + it is in real use; don't touch uninvited. +- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex. +- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores. +- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660). +- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74).