archy/docs/HANDOFF-2026-07-20-fips-peer-files.md

Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill

Written for a fresh session that will cut the OTA release and build the ISO. Everything below is already committed and pushed to gitea-ai/main. Last release was v1.7.105-alpha (e2f83c01); the next one should be v1.7.106-alpha.

---

1. What this release carries (3 commits on top of v1.7.105-alpha)

Commit	What	User-visible?
9e3ac9ba	Show the FIPS/Tor transport pill on mobile peer files	Yes
3ab7fb52	Log the full anyhow error chain on RPC failures	No (diagnostics)
5fd0d6c3	Generate fips.yaml from typed structs + enable mDNS LAN discovery	Indirectly

9e3ac9ba — mobile transport pill

PeerFiles.vue:15 wraps the peer title in hidden md:block (the global header carries the name on mobile), and the transport pill was nested inside it — so it vanished below 768px. Added a separate md:hidden pill next to the peer icon. Frontend was rebuilt and the class verified present in the emitted bundle.

Caveats worth knowing (pre-existing, not introduced here):

• On this code path the backend only ever emits fips or tor, so the mesh and lan branches in transportPill (PeerFiles.vue:609-627) are dead.
• For received mesh messages, mesh/mod.rs:1519-1533 falls back to a hardcoded "tor" when the transport is unknown — that pill can genuinely lie. The peer-files pill does not.

3ab7fb52 — full error chain in logs

api/rpc/mod.rs:441 logged only the outermost anyhow context, so every peer-files failure read exactly RPC error on content.browse-peer: Failed to connect to peer with the real cause discarded. Now {:#}. The client-facing message still goes through sanitize_error_message(&e.to_string()) ({}), so no internal detail leaks. This fix applies to every RPC method, not just browse-peer.

5fd0d6c3 — typed FIPS config + mDNS

fips/config.rs built /etc/fips/fips.yaml by format!-ing a string literal. Upstream's config structs are #[serde(deny_unknown_fields)], so a wrong key does not degrade — the daemon refuses to start and the node leaves the mesh. Now a typed serde struct tree, verified field-by-field against jmcorgan/fips v0.4.1, with 4 tests: exact-output snapshot, determinism, mDNS key path, and the pre-existing schema test. All pass.

Also enables node.discovery.lan.enabled (mDNS/DNS-SD, new upstream in v0.4.0) so co-located nodes peer directly instead of depending on the public anchor.

⚠️ Expected one-time behaviour on first boot after this lands: the startup drift check at server.rs:864 compares the freshly rendered config against what's on disk. The render differs now, so it reinstalls the config and restarts the FIPS daemon once. This is the intended self-healing path and settles immediately. Do not mistake it for a regression.

Emitted unconditionally rather than version-gated: v0.3.0's DiscoveryConfig has no lan field and no deny_unknown_fields, so v0.3.0 daemons ignore it harmlessly (verified against the v0.3.0 source). It self-activates on upgrade.

---

2. FIPS 0.4.1 — validated, but the fleet is NOT rolled

Fleet was on FIPS 0.3.0 / 0.3.0-dev (2026-05-11). Upstream is v0.4.1 (2026-07-19). Verified before touching anything:

• Wire-compatible 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day.
• Config forward-compatible — every key we emit exists in 0.4.1.
• Asset names match what fips/update.rs expects (fips_<ver>_<arch>.deb + checksums-linux.txt), so the in-product updater should work.

Upgraded so far (2 of N)

Node	Before	After	Result
OptiPlex .198 / 100.114.134.21	0.3.0-dev-1	0.4.1	✅ anchor connected, is_parent: true, tree depth: 4
thinkpad (this machine)	0.3.0	0.4.1	✅ service active, but still islanded (see §4)

The OptiPlex was still running the old string-rendered config and 0.4.1 accepted it — empirical confirmation of the compat analysis, not just desk work.

Upgrade recipe (nodes cannot reach GitHub — sideload)

# 1. On a host with GitHub access:
curl -sL -o fips_0.4.1_amd64.deb \
  https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb
curl -sL -o checksums-linux.txt \
  https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt
sha256sum fips_0.4.1_amd64.deb   # must match checksums-linux.txt
# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded

# 2. Sideload:
scp fips_0.4.1_amd64.deb archipelago@<node>:/tmp/

# 3. On the node — the same command update.rs uses:
sudo -n systemd-run --collect --wait --quiet --pipe -- \
  env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \
  /tmp/fips_0.4.1_amd64.deb

# 4. Restart the ACTIVE unit — it is archipelago-fips.service,
#    NOT fips.service (which is inactive on these nodes):
sudo -n systemctl restart archipelago-fips.service

# 5. Verify:
fipsctl --version
sudo -n fipsctl show links   # expect anchor 185.18.221.160:8443 connected
sudo -n fipsctl show tree    # expect is_root: false, depth > 0

ISO implication (important)

image-recipe/build/auto-installer/Dockerfile.rootfs:23 builds FIPS from unpinned upstream main (git clone --depth 1, no rev/tag/checksum, amd64 only). So a freshly built ISO will pick up whatever main is that day — probably ≥0.4.1, but it is not deterministic. Pinning is an open item in docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322. Consider pinning to v0.4.1 before building the release ISO so the shipped version is knowable.

---

3. The original bug — peer cloud files not loading

Status: root-caused for the thinkpad; NOT fully explained. Being explicit because it would be easy to read this as closed.

What is established:

• FIPS was fully down on the thinkpad: fipsctl show peers → [], show links → [], show tree → is_root: true, depth 0. An island.
• Cause is network egress, not FIPS config: the thinkpad cannot reach the public anchor 185.18.221.160 (fips.v0l.io) at all — 100% packet loss on ICMP, 443/8443/8668 all time out. show transports showed packets_sent: 760, packets_recv: 0 on both UDP and TCP.
• Local firewall is not the cause (nft/iptables policy accept; only stock Tailscale anti-spoof DROPs).
• The OptiPlex, on the same /24, reaches the anchor fine → it's the thinkpad's WiFi segment (wlp3s0), which also blocks L2 to .198 (ip neigh → FAILED).
• With no FIPS tree, everything falls back to Tor. Every peer in federation/nodes.json reads last_transport: "tor", never "fips".
• Tor itself is healthy: fetched the OptiPlex's /content over Tor 3×, HTTP 200 in 4.1–8.5s — well inside the 30s budget at content.rs:349.

What is not established: why three specific content.browse-peer calls failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never reproduced. Two hypotheses were tested and disproved: the Tor fallback logic is correct (FIPS-unreachable returns None and falls through in Auto mode), and the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is cold-circuit timeouts on first fetch after idle — a guess, not a finding. 3ab7fb52 means the next occurrence will log the actual cause.

Corrections to earlier claims in this session

• "Point FIPS at the Tailscale IP" was wrong. FIPS routes by npub; the ip:port in fipsctl connect is only an underlay endpoint hint.
• "The public anchor may be dead fleet-wide" was wrong. Its peer is healthy (delivery_ratio 1.0 both directions, bloom filter syncing). The bytes_recv: 0 link counters are simply uninstrumented in 0.3.0.

---

4. Open items — decisions NOT taken

1. Second FIPS anchor (user asked for this; not built). Needs a host running FIPS that is reachable from the restricted WiFi. Candidate found: OVH 146.59.87.168 — pings fine from the thinkpad and general egress works (github 200), while the upstream anchor fails even ICMP there. But it does not run FIPS yet, so this means installing FIPS on the box that hosts Gitea — a production change, deliberately not made unprompted. Code side is easy after: fips/anchors.rs:47-50 is a single hardcoded anchor that should become a list (default_public_anchor() → default_public_anchors() -> Vec<SeedAnchor>).
2. Fleet rollout of FIPS 0.4.1 — only 2 nodes done. .228 (100.64.204.114) has been offline ~20h and could not be included.
3. Deploying the archipelago binary carrying 5fd0d6c3 — no node has it yet, so mDNS is not actually live anywhere. That is what this OTA is for.
4. mDNS caveat: on the thinkpad's WiFi, multicast may also be blocked, so mDNS may not rescue that particular node even after the OTA. It will help co-located nodes on sane networks.
5. Pin FIPS in the ISO build (see §2) — recommended before the release ISO.

---

5. Release ritual (from prior sessions — follow exactly)

Working tree at handoff had pre-existing unrelated dirt: core/Cargo.lock, release-manifest.json, releases/manifest.json modified, and an untracked neode-ui/vite.preview.config.mts. Stage explicitly by path — another agent may share this tree; never git add -A.

V=1.7.106-alpha

# Frontend build — MUST verify dist actually changed (build can silently no-op)
cd neode-ui && npm run build          # → web/dist/neode-ui/
grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js   # sanity

# Backend
cd core && cargo build --release -p archipelago
# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache
# corruption — rebuild with CARGO_INCREMENTAL=0

# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s
tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui .
tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3   # ./ then ./index.html
# Exclude the ~17MB companion APK from tarballs.

# Ship
scripts/create-release.sh $V
scripts/publish-release-assets.sh $V gitea-vps2
git push origin main && git push origin --tags     # tag or the Releases page stays empty
git push gitea-ai main                              # main is protected; use the `ai` account

# Verify the live manifest
curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json

Notes: vps2 (146.59.87.168) is the primary OTA manifest host. Signing is done at the user's TTY — do not attempt it unattended. Clean /tmp first (past releases hit ENOSPC). Changelogs must be layman-readable, leading with user benefit.

ISO

UNBUNDLED=1 bash image-recipe/build-debian-iso.sh

ISO builds are always unbundled — the default env silently builds the wrong full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output filename contains unbundled and is ≈2.4G. The ISO's frontend source is /opt/archipelago/web-ui — rsync dist there first and verify inside the ISO.

---

6. Node access quick reference

• thinkpad (.116) is the local machine — do not SSH to it; read journalctl -u archipelago and /var/lib/archipelago/** directly.
• OptiPlex .198 = Tailscale archipelago-5 / 100.114.134.21, user archipelago. Its LAN IP is unreachable from the thinkpad — use Tailscale.
• .228 = archipelago-2 / 100.64.204.114 — offline as of 2026-07-20, and it is in real use; don't touch uninvited.
• archipelago-1 (100.82.34.38) is a Ryzen AI Max desktop, not the OptiPlex.
• Nodes have no sqlite3 — use sudo -n python3 to read the JSON stores.
• fipsctl needs sudo -n (socket is root:fips 0660).
• Never run archipelago --version on fleet nodes (deployed binaries predate #74).