archy/docs/HANDOFF-2026-07-20-fips-peer-files.md

# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill

Written for a fresh session that will **cut the OTA release and build the ISO**.
Everything below is already committed and pushed to `gitea-ai/main`. Last release
was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**.

---

## 1. What this release carries (3 commits on top of v1.7.105-alpha)

| Commit | What | User-visible? |
|---|---|---|
| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes |
| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) |
| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly |

### `9e3ac9ba` — mobile transport pill
`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header
carries the name on mobile), and the transport pill was nested inside it — so it
vanished below 768px. Added a separate `md:hidden` pill next to the peer icon.
Frontend was rebuilt and the class verified present in the emitted bundle.

Caveats worth knowing (pre-existing, not introduced here):
- On this code path the backend only ever emits `fips` or `tor`, so the `mesh`
  and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead.
- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a
  hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie.
  The peer-files pill does not.

### `3ab7fb52` — full error chain in logs
`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every
peer-files failure read exactly `RPC error on content.browse-peer: Failed to
connect to peer` with the real cause discarded. Now `{:#}`. The client-facing
message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so
no internal detail leaks. **This fix applies to every RPC method, not just
browse-peer.**

### `5fd0d6c3` — typed FIPS config + mDNS
`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal.
Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key
does not degrade — **the daemon refuses to start and the node leaves the mesh**.
Now a typed serde struct tree, verified field-by-field against jmcorgan/fips
**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and
the pre-existing schema test. All pass.

Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0)
so co-located nodes peer directly instead of depending on the public anchor.

> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup
> drift check at `server.rs:864` compares the freshly rendered config against
> what's on disk. The render differs now, so it reinstalls the config and
> restarts the FIPS daemon **once**. This is the intended self-healing path and
> settles immediately. Do not mistake it for a regression.

Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig`
has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it
harmlessly (verified against the v0.3.0 source). It self-activates on upgrade.

---

## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled

Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1**
(2026-07-19). Verified before touching anything:

- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day.
- **Config forward-compatible** — every key we emit exists in 0.4.1.
- **Asset names match** what `fips/update.rs` expects (`fips_<ver>_<arch>.deb` +
  `checksums-linux.txt`), so the in-product updater should work.

### Upgraded so far (2 of N)
| Node | Before | After | Result |
|---|---|---|---|
| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` |
| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) |

The OptiPlex was still running the **old string-rendered config** and 0.4.1
accepted it — empirical confirmation of the compat analysis, not just desk work.

### Upgrade recipe (nodes cannot reach GitHub — sideload)
```bash
# 1. On a host with GitHub access:
curl -sL -o fips_0.4.1_amd64.deb \
  https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb
curl -sL -o checksums-linux.txt \
  https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt
sha256sum fips_0.4.1_amd64.deb   # must match checksums-linux.txt
# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded

# 2. Sideload:
scp fips_0.4.1_amd64.deb archipelago@<node>:/tmp/

# 3. On the node — the same command update.rs uses:
sudo -n systemd-run --collect --wait --quiet --pipe -- \
  env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \
  /tmp/fips_0.4.1_amd64.deb

# 4. Restart the ACTIVE unit — it is archipelago-fips.service,
#    NOT fips.service (which is inactive on these nodes):
sudo -n systemctl restart archipelago-fips.service

# 5. Verify:
fipsctl --version
sudo -n fipsctl show links   # expect anchor 185.18.221.160:8443 connected
sudo -n fipsctl show tree    # expect is_root: false, depth > 0
```

### ISO implication (important)
`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from
**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64
only). So a freshly built ISO will pick up whatever main is that day — probably
≥0.4.1, but it is not deterministic. Pinning is an open item in
`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1
before building the release ISO** so the shipped version is knowable.

---

## 3. The original bug — peer cloud files not loading

**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit
because it would be easy to read this as closed.

What is established:
- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links`
  → `[]`, `show tree` → `is_root: true, depth 0`. An island.
- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the
  public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on
  ICMP, 443/8443/8668 all time out. `show transports` showed
  `packets_sent: 760, packets_recv: 0` on both UDP and TCP.
- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock
  Tailscale anti-spoof DROPs).
- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's
  WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`).
- With no FIPS tree, everything falls back to Tor. Every peer in
  `federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`.
- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×,
  HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`.

What is **not** established: why three specific `content.browse-peer` calls
failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never
reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic
is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and
the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is
cold-circuit timeouts on first fetch after idle — **a guess, not a finding.**
`3ab7fb52` means the next occurrence will log the actual cause.

### Corrections to earlier claims in this session
- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the
  `ip:port` in `fipsctl connect` is only an underlay endpoint hint.
- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy
  (`delivery_ratio` 1.0 both directions, bloom filter syncing). The
  `bytes_recv: 0` link counters are simply uninstrumented in 0.3.0.

---

## 4. Open items — decisions NOT taken

1. **Second FIPS anchor (user asked for this; not built).** Needs a host running
   FIPS that is reachable from the restricted WiFi. Candidate found: OVH
   **`146.59.87.168`** — pings fine from the thinkpad and general egress works
   (github 200), while the upstream anchor fails even ICMP there. But it does not
   run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** —
   a production change, deliberately not made unprompted. Code side is easy after:
   `fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list
   (`default_public_anchor()` → `default_public_anchors() -> Vec<SeedAnchor>`).
2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228`
   (`100.64.204.114`) has been **offline ~20h** and could not be included.
3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet,
   so mDNS is not actually live anywhere. That is what this OTA is for.
4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so
   mDNS may not rescue that particular node even after the OTA. It will help
   co-located nodes on sane networks.
5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO.

---

## 5. Release ritual (from prior sessions — follow exactly)

Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`,
`release-manifest.json`, `releases/manifest.json` modified, and an untracked
`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another
agent may share this tree; never `git add -A`.

```bash
V=1.7.106-alpha

# Frontend build — MUST verify dist actually changed (build can silently no-op)
cd neode-ui && npm run build          # → web/dist/neode-ui/
grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js   # sanity

# Backend
cd core && cargo build --release -p archipelago
# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache
# corruption — rebuild with CARGO_INCREMENTAL=0

# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s
tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui .
tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3   # ./ then ./index.html
# Exclude the ~17MB companion APK from tarballs.

# Ship
scripts/create-release.sh $V
scripts/publish-release-assets.sh $V gitea-vps2
git push origin main && git push origin --tags     # tag or the Releases page stays empty
git push gitea-ai main                              # main is protected; use the `ai` account

# Verify the live manifest
curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json
```

Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is
done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first
(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with
user benefit.

### ISO
```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh
```
ISO builds are **always unbundled** — the default env silently builds the wrong
full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output
filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is
`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO.

---

## 6. Node access quick reference

- **thinkpad (`.116`) is the local machine** — do not SSH to it; read
  `journalctl -u archipelago` and `/var/lib/archipelago/**` directly.
- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user
  `archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale.
- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and
  it is in real use; don't touch uninvited.
- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex.
- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores.
- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660).
- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74).