docs: handoff for the v1.7.106-alpha OTA + ISO release
Captures the three commits this release carries, the FIPS 0.4.1 upgrade state (2 nodes done, fleet not rolled), the peer-files diagnosis including what is explicitly NOT explained, the open decisions not taken, and the exact release + ISO ritual. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
parent
5fd0d6c3c8
commit
4fb200e938
238
docs/HANDOFF-2026-07-20-fips-peer-files.md
Normal file
238
docs/HANDOFF-2026-07-20-fips-peer-files.md
Normal file
@ -0,0 +1,238 @@
|
||||
# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill
|
||||
|
||||
Written for a fresh session that will **cut the OTA release and build the ISO**.
|
||||
Everything below is already committed and pushed to `gitea-ai/main`. Last release
|
||||
was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this release carries (3 commits on top of v1.7.105-alpha)
|
||||
|
||||
| Commit | What | User-visible? |
|
||||
|---|---|---|
|
||||
| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes |
|
||||
| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) |
|
||||
| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly |
|
||||
|
||||
### `9e3ac9ba` — mobile transport pill
|
||||
`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header
|
||||
carries the name on mobile), and the transport pill was nested inside it — so it
|
||||
vanished below 768px. Added a separate `md:hidden` pill next to the peer icon.
|
||||
Frontend was rebuilt and the class verified present in the emitted bundle.
|
||||
|
||||
Caveats worth knowing (pre-existing, not introduced here):
|
||||
- On this code path the backend only ever emits `fips` or `tor`, so the `mesh`
|
||||
and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead.
|
||||
- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a
|
||||
hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie.
|
||||
The peer-files pill does not.
|
||||
|
||||
### `3ab7fb52` — full error chain in logs
|
||||
`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every
|
||||
peer-files failure read exactly `RPC error on content.browse-peer: Failed to
|
||||
connect to peer` with the real cause discarded. Now `{:#}`. The client-facing
|
||||
message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so
|
||||
no internal detail leaks. **This fix applies to every RPC method, not just
|
||||
browse-peer.**
|
||||
|
||||
### `5fd0d6c3` — typed FIPS config + mDNS
|
||||
`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal.
|
||||
Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key
|
||||
does not degrade — **the daemon refuses to start and the node leaves the mesh**.
|
||||
Now a typed serde struct tree, verified field-by-field against jmcorgan/fips
|
||||
**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and
|
||||
the pre-existing schema test. All pass.
|
||||
|
||||
Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0)
|
||||
so co-located nodes peer directly instead of depending on the public anchor.
|
||||
|
||||
> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup
|
||||
> drift check at `server.rs:864` compares the freshly rendered config against
|
||||
> what's on disk. The render differs now, so it reinstalls the config and
|
||||
> restarts the FIPS daemon **once**. This is the intended self-healing path and
|
||||
> settles immediately. Do not mistake it for a regression.
|
||||
|
||||
Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig`
|
||||
has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it
|
||||
harmlessly (verified against the v0.3.0 source). It self-activates on upgrade.
|
||||
|
||||
---
|
||||
|
||||
## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled
|
||||
|
||||
Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1**
|
||||
(2026-07-19). Verified before touching anything:
|
||||
|
||||
- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day.
|
||||
- **Config forward-compatible** — every key we emit exists in 0.4.1.
|
||||
- **Asset names match** what `fips/update.rs` expects (`fips_<ver>_<arch>.deb` +
|
||||
`checksums-linux.txt`), so the in-product updater should work.
|
||||
|
||||
### Upgraded so far (2 of N)
|
||||
| Node | Before | After | Result |
|
||||
|---|---|---|---|
|
||||
| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` |
|
||||
| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) |
|
||||
|
||||
The OptiPlex was still running the **old string-rendered config** and 0.4.1
|
||||
accepted it — empirical confirmation of the compat analysis, not just desk work.
|
||||
|
||||
### Upgrade recipe (nodes cannot reach GitHub — sideload)
|
||||
```bash
|
||||
# 1. On a host with GitHub access:
|
||||
curl -sL -o fips_0.4.1_amd64.deb \
|
||||
https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb
|
||||
curl -sL -o checksums-linux.txt \
|
||||
https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt
|
||||
sha256sum fips_0.4.1_amd64.deb # must match checksums-linux.txt
|
||||
# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded
|
||||
|
||||
# 2. Sideload:
|
||||
scp fips_0.4.1_amd64.deb archipelago@<node>:/tmp/
|
||||
|
||||
# 3. On the node — the same command update.rs uses:
|
||||
sudo -n systemd-run --collect --wait --quiet --pipe -- \
|
||||
env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \
|
||||
/tmp/fips_0.4.1_amd64.deb
|
||||
|
||||
# 4. Restart the ACTIVE unit — it is archipelago-fips.service,
|
||||
# NOT fips.service (which is inactive on these nodes):
|
||||
sudo -n systemctl restart archipelago-fips.service
|
||||
|
||||
# 5. Verify:
|
||||
fipsctl --version
|
||||
sudo -n fipsctl show links # expect anchor 185.18.221.160:8443 connected
|
||||
sudo -n fipsctl show tree # expect is_root: false, depth > 0
|
||||
```
|
||||
|
||||
### ISO implication (important)
|
||||
`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from
|
||||
**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64
|
||||
only). So a freshly built ISO will pick up whatever main is that day — probably
|
||||
≥0.4.1, but it is not deterministic. Pinning is an open item in
|
||||
`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1
|
||||
before building the release ISO** so the shipped version is knowable.
|
||||
|
||||
---
|
||||
|
||||
## 3. The original bug — peer cloud files not loading
|
||||
|
||||
**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit
|
||||
because it would be easy to read this as closed.
|
||||
|
||||
What is established:
|
||||
- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links`
|
||||
→ `[]`, `show tree` → `is_root: true, depth 0`. An island.
|
||||
- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the
|
||||
public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on
|
||||
ICMP, 443/8443/8668 all time out. `show transports` showed
|
||||
`packets_sent: 760, packets_recv: 0` on both UDP and TCP.
|
||||
- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock
|
||||
Tailscale anti-spoof DROPs).
|
||||
- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's
|
||||
WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`).
|
||||
- With no FIPS tree, everything falls back to Tor. Every peer in
|
||||
`federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`.
|
||||
- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×,
|
||||
HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`.
|
||||
|
||||
What is **not** established: why three specific `content.browse-peer` calls
|
||||
failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never
|
||||
reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic
|
||||
is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and
|
||||
the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is
|
||||
cold-circuit timeouts on first fetch after idle — **a guess, not a finding.**
|
||||
`3ab7fb52` means the next occurrence will log the actual cause.
|
||||
|
||||
### Corrections to earlier claims in this session
|
||||
- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the
|
||||
`ip:port` in `fipsctl connect` is only an underlay endpoint hint.
|
||||
- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy
|
||||
(`delivery_ratio` 1.0 both directions, bloom filter syncing). The
|
||||
`bytes_recv: 0` link counters are simply uninstrumented in 0.3.0.
|
||||
|
||||
---
|
||||
|
||||
## 4. Open items — decisions NOT taken
|
||||
|
||||
1. **Second FIPS anchor (user asked for this; not built).** Needs a host running
|
||||
FIPS that is reachable from the restricted WiFi. Candidate found: OVH
|
||||
**`146.59.87.168`** — pings fine from the thinkpad and general egress works
|
||||
(github 200), while the upstream anchor fails even ICMP there. But it does not
|
||||
run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** —
|
||||
a production change, deliberately not made unprompted. Code side is easy after:
|
||||
`fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list
|
||||
(`default_public_anchor()` → `default_public_anchors() -> Vec<SeedAnchor>`).
|
||||
2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228`
|
||||
(`100.64.204.114`) has been **offline ~20h** and could not be included.
|
||||
3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet,
|
||||
so mDNS is not actually live anywhere. That is what this OTA is for.
|
||||
4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so
|
||||
mDNS may not rescue that particular node even after the OTA. It will help
|
||||
co-located nodes on sane networks.
|
||||
5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO.
|
||||
|
||||
---
|
||||
|
||||
## 5. Release ritual (from prior sessions — follow exactly)
|
||||
|
||||
Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`,
|
||||
`release-manifest.json`, `releases/manifest.json` modified, and an untracked
|
||||
`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another
|
||||
agent may share this tree; never `git add -A`.
|
||||
|
||||
```bash
|
||||
V=1.7.106-alpha
|
||||
|
||||
# Frontend build — MUST verify dist actually changed (build can silently no-op)
|
||||
cd neode-ui && npm run build # → web/dist/neode-ui/
|
||||
grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js # sanity
|
||||
|
||||
# Backend
|
||||
cd core && cargo build --release -p archipelago
|
||||
# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache
|
||||
# corruption — rebuild with CARGO_INCREMENTAL=0
|
||||
|
||||
# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s
|
||||
tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui .
|
||||
tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3 # ./ then ./index.html
|
||||
# Exclude the ~17MB companion APK from tarballs.
|
||||
|
||||
# Ship
|
||||
scripts/create-release.sh $V
|
||||
scripts/publish-release-assets.sh $V gitea-vps2
|
||||
git push origin main && git push origin --tags # tag or the Releases page stays empty
|
||||
git push gitea-ai main # main is protected; use the `ai` account
|
||||
|
||||
# Verify the live manifest
|
||||
curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json
|
||||
```
|
||||
|
||||
Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is
|
||||
done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first
|
||||
(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with
|
||||
user benefit.
|
||||
|
||||
### ISO
|
||||
```bash
|
||||
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh
|
||||
```
|
||||
ISO builds are **always unbundled** — the default env silently builds the wrong
|
||||
full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output
|
||||
filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is
|
||||
`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO.
|
||||
|
||||
---
|
||||
|
||||
## 6. Node access quick reference
|
||||
|
||||
- **thinkpad (`.116`) is the local machine** — do not SSH to it; read
|
||||
`journalctl -u archipelago` and `/var/lib/archipelago/**` directly.
|
||||
- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user
|
||||
`archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale.
|
||||
- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and
|
||||
it is in real use; don't touch uninvited.
|
||||
- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex.
|
||||
- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores.
|
||||
- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660).
|
||||
- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74).
|
||||
Loading…
x
Reference in New Issue
Block a user