# HANDOFF — deploy companion APK 0.5.1 (vc21) to nodes **For: the agent on archi-dev-box.** User-reported failure this evening: pairing flow on Framework PT — downloaded the companion from the node's QR, then the pairing scan didn't work. The APK the node serves predates today's scanner fixes; the pipeline below gets the fixed build into the user's hands. ## What changed on main today (all merged) - **Pairing-scanner fix** (`QrScannerOverlay.kt`): ZXing decode attempts are frame-gated (~7/s, was every frame — the CPU contention made the preview stutter badly enough to never decode) and PreviewView uses TextureView (no more black flash on open). This is the likely fix for "doesn't scan". - Three-finger menu gesture (was two-finger, collided with scroll) + one-time teaching overlay ~2 min after login. - Native wallet QR scanner behind `window.ArchipelagoQr` + WebView file-chooser support; web scan modal hands live scanning to it. - npub-keyed saved servers (pairing contract item 1, PR #106). - Served APK refreshed: `neode-ui/public/packages/archipelago-companion.apk` is now **0.5.3 / versionCode 23**. On top of the 0.5.1 scanner fixes it guarantees dual-path peering — the node's LAN endpoint (direct p2p, npub- keyed dial hints) AND the Archipelago public anchor (vps2, baked into the app so even an old node's QR can't leave the phone LAN-only) — and fixes the two field failures from the user's 5G test (screenshots, 21:54): - **Mesh VPN no longer kills the phone's internet** — the IPv6-only TUN never called `allowFamily(AF_INET)`, so Android blocked all IPv4 while the mesh was up. Now allowed (+ `allowBypass`). - **Off-LAN connect works** — `connect()` no longer hard-fails when the scanned LAN IP doesn't answer; it brings the mesh up and probes the node's ULA (`meshIp`) with retries before reporting failure. ## What to do 1. Redeploy the web-ui bundle from current main to the active nodes — web root `/opt/archipelago/web-ui/` (NOT a neode-ui/ subfolder), all nodes the user pairs against, at minimum the one Framework PT scans. 2. Verify the served artifact really updated: `curl -sI http:///packages/archipelago-companion.apk` — size should change (~27 MB build of 2026-07-23), or pull it and check `aapt dump badging` shows `versionCode='21' versionName='0.5.1'`. 3. The demo stack gets its images from CI (run 100 pushed today with the new web bundle) — confirm the Portainer stack re-pulled, or trigger its redeploy, so the demo QR also serves vc21. 4. **Node side is half the 5G story**: away-from-home reachability needs the NODE connected to the public anchor too. On Framework PT (and any test node): deploy current main (node-side npub-first `fips.pair-info`), then verify `sudo -n fipsctl show status` reports the anchor connected — `fips.reconnect` RPC if not. A phone can dial the anchor perfectly and still fail if the node never enrolled with it. 5. Re-test the user's exact flows with **vc24** (updates any older install in place): (a) pair ON the LAN, then switch the phone to 5G — the UI must come up via the mesh ULA; (b) pair while ALREADY on 5G (never on the node's LAN) — scan, VPN consent, and the connect must succeed through the anchor. ## Live diagnosis update (22:30–22:50, phone on adb — Mac agent) vc23 on-device testing found and fixed the phone-side blocker, and narrowed what remains to the node side. State as of vc24: - **Fixed: TUN reader died at startup.** Android hands the VpnService fd over non-blocking; the fips fork's blocking reader thread treats EAGAIN as fatal ("TUN read error … Try again (os error 11)") — so mesh sessions came up but NO packet ever entered the tunnel. archy-fips-core now forces the fd blocking before `start_with_tun_fd`. Verified on-device: reader survives, and the 30s anchor-link flap disappeared with it (stable 8+ min on 5G). - **Fixed: VPN marked not-metered** (`setMetered(false)`) — Android 10+ defaults VPNs to metered, putting the phone into data-restricted behaviour while the mesh is up. Note the user's phone also has system **always-on VPN** enabled for the app (`always_on_vpn_app`), a Settings-side toggle. - **Verified good on-device**: peer store has node (LAN udp/tcp hints) + vps2 anchor; saved server is npub-keyed with ULA; anchor session establishes from 5G in ~6s; VPN is bypassable, VALIDATED, only fd00::/8 routed. - **REMAINING BLOCKER (node side)**: from the phone (app uid), ping6 and HTTP to the node's ULA `fd79:1aa:b9e9:4c9f:1f80:5376:9385:1824` get no reply — packets enter the mesh, nothing returns. Phone↔anchor works, so suspect phone-fork ↔ node-daemon session/routing mismatch (FIPS wire format is not stable across revs; phone pins fips-native fork 46494a74). From Framework PT please capture: - `fipsctl show status` (daemon version + anchor state) - `fipsctl show sessions` and `show bloom` while the phone pings - `ping6 ` from the node (tests the reverse path) - `ip addr show fips0` + confirm the web server listens on `[::]:80` Report whether the node ever sees a session attempt from `npub132c5whrsa6ccs0eylcpzaejq9uxul5ldvczz0axq78dh7fxkqj9st4uvzu`. ## Node-side diagnosis complete (23:00–23:30, archi-dev-box agent) Chain of findings, each verified live: 1. **FIXED: nginx had no IPv6 listener anywhere** — every shipped config listened on 0.0.0.0 only, so `http://[]` could NEVER connect on any node, ever. Live-fixed on framework-pt + .116, canonical conf + bootstrap self-heal shipped (`1e89362e`), heal binary deployed. ULA HTTP verified answering on both nodes (local + over-mesh). 2. Firewall clean, fd00::/8 routes correct both ends, wire compat proven (node + vps2 anchor both run fips 0.4.1 rev 15db6471db — the latest upstream stable; nothing newer exists). 3. **THE REMAINING PROBLEM IS MESH SESSION PATH QUALITY.** From the vps2 anchor — a DIRECT connected peer — `GET /health` on the node's ULA takes **15–17 s per request and intermittently fails outright** (nginx logs show 499 client-gave-up then 200; TCP SYN-retransmit backoff signature). Session MMP is wildly asymmetric: node→vps2 srtt 204 ms, **vps2→node srtt 4270 ms** — on a direct link whose raw RTT is 4 ms. Session traffic is not riding the direct link; it appears to route through the ~1271-node public tree (node's tree root is the public 00001a8c, depth 8; the node's log also shows chronic "Discovery lookup timed out" for other targets). 4. The phone's npub never appears in the node's sessions — consistent with discovery/handshake dying on the same degraded tree path, and the app's ~8 s probe window being far smaller than the observed 15 s+ first-request latency even on the GOOD path. ### Recommendations - **App side (Mac agent):** widen the ULA probe/connect window to ≥30 s with retransmit-friendly pacing, and PRE-WARM the mesh session (start pinging the node ULA as soon as the VPN is up, decoupled from the UI probe) so the WebView hits a warm session. - **Infra decision (user):** consider detaching the fleet from the public v0l mesh — private tree rooted at the vps2 anchor (drop the legacy 185.18.221.160 seed anchor fleet-wide AND vps2's public peering). A 2-hop private tree would make session paths ride the direct links and should collapse latency to ms. Trade-off: no reachability to/from the broader public mesh. - **Upstream:** report the direct-peer session-path asymmetry to jmcorgan/fips (0.4.1). ## App-side recommendations implemented (23:30–23:50, Mac agent — 0.5.5/vc25) - Connect probe: mesh ULA now probed inside a **60s budget** with 15s per-phase timeouts (rides out TCP retransmit backoff), replacing the old ~8s window. - **Session pre-warm**: the VPN service starts probing every saved node ULA the moment the tunnel is up (5s cadence for the first minute, then a 60s keep-warm tick) — discovery/handshake cost is paid in the background, and the session never idles out while the mesh is connected. - (A phone-side ping test in this window still showed zero replies — that measurement predated the vps2 daemon restart below and is superseded.) ## RESOLVED — root cause was vps2's degraded daemon, NOT the public tree (23:45) The privatize-the-mesh recommendation above is WITHDRAWN. Final diagnosis: vps2's fips daemon (3 days uptime, 0.2% CPU, idle box) had internally degraded — EVERY link it carried showed ~4.5 s RTT (even to peers 30 ms away), and since the anchor sits on the phone↔node path, everything through it inherited that. `systemctl restart fips` on vps2 restored link RTTs to 40–340 ms, and anchor→node mesh HTTP went from 14–17 s (intermittent hard fails) to a steady **165–275 ms**. Node↔node direct sessions were always fine (.116→framework-pt ULA HTTP: 894 ms cold, sub-second warm) — the user's read was correct. Actions taken: dead legacy anchor (185.18.221.160) removed from framework-pt + .116 seed files (fleet keeps vps2 + public-mesh membership via vps2 — we stay in the open mesh); fresh daemons on both nodes; **vps2 fips now has RuntimeMaxSec=1d + Restart=always** so a wedged anchor daemon can never rot for days again. Report the slow-degradation behaviour upstream (jmcorgan/fips, 0.4.1): long-running daemon in a ~1400-node mesh accumulates multi-second link latency at idle CPU, cleared by restart. Phone side: vc25's 60 s probe + pre-warm now has a millisecond-latency mesh to work with. Ready for the user's 5G test. ## NEXT (00:05, Mac agent → dev-box agent): app direct ports are IPv4-only over the mesh The kiosk loads over the ULA now — but opening any APP dies with `ERR_CONNECTION_REFUSED` at `http://[]:/`. User-hit first on **Bitcoin Knots (:8334)**, and it will be every catalog app: the web UI builds app URLs from the current host + the app's DIRECT port (Direct Port Rule), and container-published ports only bind 0.0.0.0. Verified: `192.168.63.249:8334` → HTTP 200 (nginx), ULA:8334 → refused. Same disease as your :80 nginx fix, one layer down. Fix must cover EVERY catalog app port and survive app install/remove. Two shapes; pick what fits the container layer best: 1. **IPv6 publish at the container layer** — publish on `[::]` too (pasta/rootless podman support address-specific `-p`), wired into the container manager so new apps inherit it; or 2. **Host-side v6→v4 forwarders** — generated nginx `stream {}` (or systemd-socket) units: `listen [::]:` → `127.0.0.1:`, one per catalog app port, regenerated on app install/remove, boot-time self-healed like the :80 fix. Keeps the Direct Port Rule URL contract without touching containers. Either way: extend the bootstrap self-heal, and verify from the MESH side (curl the ULA on 2–3 app ports incl. :8334 from vps2 or .116) — not just from the LAN. ## DONE (00:30, dev-box agent): app direct ports live over the mesh Shape 2-variant implemented INSIDE the backend (`mesh_ports.rs`, `2ad57c63`): a reconcile loop mirrors every public IPv4 listener (>=1024, bound 0.0.0.0, no existing IPv6 any-listener) as a v6-ONLY `[::]:` forwarder to `127.0.0.1:`, following `/proc/net/tcp*` every 15s — so app install/remove and hardcoded companion ports (bitcoin-ui :8334) are covered with zero container changes and no generated units; self-healing because it lives in the binary. Strictly ADDITIVE: IPv4/LAN/Tor paths untouched, v6only cannot intercept v4, foreign IPv6 listeners win. Verified FROM THE MESH (vps2 → node ULA): :8334 HTTP 200 (466ms), :18083 200 (306ms), :50002 200 (239ms); LAN :8334 still 200. Deployed to framework-pt + .116 (binary sha 52ac0d8a…). Direct-port apps should now open in the companion over 5G.