Merge remote-tracking branch 'origin/main' into archy-hwconfig
This commit is contained in:
@@ -48,9 +48,158 @@ user's hands.
|
||||
verify `sudo -n fipsctl show status` reports the anchor connected —
|
||||
`fips.reconnect` RPC if not. A phone can dial the anchor perfectly and
|
||||
still fail if the node never enrolled with it.
|
||||
5. Re-test the user's exact flows with vc23 (updates any older install in
|
||||
5. Re-test the user's exact flows with **vc24** (updates any older install in
|
||||
place): (a) pair ON the LAN, then switch the phone to 5G — the UI must
|
||||
come up via the mesh ULA; (b) pair while ALREADY on 5G (never on the
|
||||
node's LAN) — scan, VPN consent, and the connect must succeed through
|
||||
the anchor. If either fails, capture `adb logcat` around the attempt and
|
||||
`fipsctl show sessions` on the node, and report back.
|
||||
the anchor.
|
||||
|
||||
## Live diagnosis update (22:30–22:50, phone on adb — Mac agent)
|
||||
|
||||
vc23 on-device testing found and fixed the phone-side blocker, and narrowed
|
||||
what remains to the node side. State as of vc24:
|
||||
|
||||
- **Fixed: TUN reader died at startup.** Android hands the VpnService fd over
|
||||
non-blocking; the fips fork's blocking reader thread treats EAGAIN as fatal
|
||||
("TUN read error … Try again (os error 11)") — so mesh sessions came up but
|
||||
NO packet ever entered the tunnel. archy-fips-core now forces the fd
|
||||
blocking before `start_with_tun_fd`. Verified on-device: reader survives,
|
||||
and the 30s anchor-link flap disappeared with it (stable 8+ min on 5G).
|
||||
- **Fixed: VPN marked not-metered** (`setMetered(false)`) — Android 10+
|
||||
defaults VPNs to metered, putting the phone into data-restricted behaviour
|
||||
while the mesh is up. Note the user's phone also has system **always-on
|
||||
VPN** enabled for the app (`always_on_vpn_app`), a Settings-side toggle.
|
||||
- **Verified good on-device**: peer store has node (LAN udp/tcp hints) + vps2
|
||||
anchor; saved server is npub-keyed with ULA; anchor session establishes
|
||||
from 5G in ~6s; VPN is bypassable, VALIDATED, only fd00::/8 routed.
|
||||
- **REMAINING BLOCKER (node side)**: from the phone (app uid), ping6 and
|
||||
HTTP to the node's ULA `fd79:1aa:b9e9:4c9f:1f80:5376:9385:1824` get no
|
||||
reply — packets enter the mesh, nothing returns. Phone↔anchor works, so
|
||||
suspect phone-fork ↔ node-daemon session/routing mismatch (FIPS wire
|
||||
format is not stable across revs; phone pins fips-native fork 46494a74).
|
||||
From Framework PT please capture:
|
||||
- `fipsctl show status` (daemon version + anchor state)
|
||||
- `fipsctl show sessions` and `show bloom` while the phone pings
|
||||
- `ping6 <phone ULA fd68:496d:fe34:a06d:cf1:6e4:b6a4:3586>` from the
|
||||
node (tests the reverse path)
|
||||
- `ip addr show fips0` + confirm the web server listens on `[::]:80`
|
||||
Report whether the node ever sees a session attempt from
|
||||
`npub132c5whrsa6ccs0eylcpzaejq9uxul5ldvczz0axq78dh7fxkqj9st4uvzu`.
|
||||
|
||||
## Node-side diagnosis complete (23:00–23:30, archi-dev-box agent)
|
||||
|
||||
Chain of findings, each verified live:
|
||||
|
||||
1. **FIXED: nginx had no IPv6 listener anywhere** — every shipped config
|
||||
listened on 0.0.0.0 only, so `http://[<ULA>]` could NEVER connect on any
|
||||
node, ever. Live-fixed on framework-pt + .116, canonical conf + bootstrap
|
||||
self-heal shipped (`1e89362e`), heal binary deployed. ULA HTTP verified
|
||||
answering on both nodes (local + over-mesh).
|
||||
2. Firewall clean, fd00::/8 routes correct both ends, wire compat proven
|
||||
(node + vps2 anchor both run fips 0.4.1 rev 15db6471db — the latest
|
||||
upstream stable; nothing newer exists).
|
||||
3. **THE REMAINING PROBLEM IS MESH SESSION PATH QUALITY.** From the vps2
|
||||
anchor — a DIRECT connected peer — `GET /health` on the node's ULA takes
|
||||
**15–17 s per request and intermittently fails outright** (nginx logs
|
||||
show 499 client-gave-up then 200; TCP SYN-retransmit backoff signature).
|
||||
Session MMP is wildly asymmetric: node→vps2 srtt 204 ms, **vps2→node
|
||||
srtt 4270 ms** — on a direct link whose raw RTT is 4 ms. Session traffic
|
||||
is not riding the direct link; it appears to route through the ~1271-node
|
||||
public tree (node's tree root is the public 00001a8c, depth 8; the node's
|
||||
log also shows chronic "Discovery lookup timed out" for other targets).
|
||||
4. The phone's npub never appears in the node's sessions — consistent with
|
||||
discovery/handshake dying on the same degraded tree path, and the app's
|
||||
~8 s probe window being far smaller than the observed 15 s+ first-request
|
||||
latency even on the GOOD path.
|
||||
|
||||
### Recommendations
|
||||
|
||||
- **App side (Mac agent):** widen the ULA probe/connect window to ≥30 s
|
||||
with retransmit-friendly pacing, and PRE-WARM the mesh session (start
|
||||
pinging the node ULA as soon as the VPN is up, decoupled from the UI
|
||||
probe) so the WebView hits a warm session.
|
||||
- **Infra decision (user):** consider detaching the fleet from the public
|
||||
v0l mesh — private tree rooted at the vps2 anchor (drop the legacy
|
||||
185.18.221.160 seed anchor fleet-wide AND vps2's public peering). A
|
||||
2-hop private tree would make session paths ride the direct links and
|
||||
should collapse latency to ms. Trade-off: no reachability to/from the
|
||||
broader public mesh.
|
||||
- **Upstream:** report the direct-peer session-path asymmetry to
|
||||
jmcorgan/fips (0.4.1).
|
||||
|
||||
## App-side recommendations implemented (23:30–23:50, Mac agent — 0.5.5/vc25)
|
||||
|
||||
- Connect probe: mesh ULA now probed inside a **60s budget** with 15s
|
||||
per-phase timeouts (rides out TCP retransmit backoff), replacing the old
|
||||
~8s window.
|
||||
- **Session pre-warm**: the VPN service starts probing every saved node ULA
|
||||
the moment the tunnel is up (5s cadence for the first minute, then a 60s
|
||||
keep-warm tick) — discovery/handshake cost is paid in the background, and
|
||||
the session never idles out while the mesh is connected.
|
||||
- (A phone-side ping test in this window still showed zero replies — that
|
||||
measurement predated the vps2 daemon restart below and is superseded.)
|
||||
|
||||
## RESOLVED — root cause was vps2's degraded daemon, NOT the public tree (23:45)
|
||||
|
||||
The privatize-the-mesh recommendation above is WITHDRAWN. Final diagnosis:
|
||||
vps2's fips daemon (3 days uptime, 0.2% CPU, idle box) had internally
|
||||
degraded — EVERY link it carried showed ~4.5 s RTT (even to peers 30 ms
|
||||
away), and since the anchor sits on the phone↔node path, everything through
|
||||
it inherited that. `systemctl restart fips` on vps2 restored link RTTs to
|
||||
40–340 ms, and anchor→node mesh HTTP went from 14–17 s (intermittent hard
|
||||
fails) to a steady **165–275 ms**. Node↔node direct sessions were always
|
||||
fine (.116→framework-pt ULA HTTP: 894 ms cold, sub-second warm) — the
|
||||
user's read was correct.
|
||||
|
||||
Actions taken: dead legacy anchor (185.18.221.160) removed from
|
||||
framework-pt + .116 seed files (fleet keeps vps2 + public-mesh membership
|
||||
via vps2 — we stay in the open mesh); fresh daemons on both nodes;
|
||||
**vps2 fips now has RuntimeMaxSec=1d + Restart=always** so a wedged anchor
|
||||
daemon can never rot for days again. Report the slow-degradation behaviour
|
||||
upstream (jmcorgan/fips, 0.4.1): long-running daemon in a ~1400-node mesh
|
||||
accumulates multi-second link latency at idle CPU, cleared by restart.
|
||||
|
||||
Phone side: vc25's 60 s probe + pre-warm now has a millisecond-latency mesh
|
||||
to work with. Ready for the user's 5G test.
|
||||
|
||||
## NEXT (00:05, Mac agent → dev-box agent): app direct ports are IPv4-only over the mesh
|
||||
|
||||
The kiosk loads over the ULA now — but opening any APP dies with
|
||||
`ERR_CONNECTION_REFUSED` at `http://[<ULA>]:<port>/`. User-hit first on
|
||||
**Bitcoin Knots (:8334)**, and it will be every catalog app: the web UI
|
||||
builds app URLs from the current host + the app's DIRECT port (Direct Port
|
||||
Rule), and container-published ports only bind 0.0.0.0. Verified:
|
||||
`192.168.63.249:8334` → HTTP 200 (nginx), ULA:8334 → refused. Same disease
|
||||
as your :80 nginx fix, one layer down.
|
||||
|
||||
Fix must cover EVERY catalog app port and survive app install/remove. Two
|
||||
shapes; pick what fits the container layer best:
|
||||
|
||||
1. **IPv6 publish at the container layer** — publish on `[::]` too
|
||||
(pasta/rootless podman support address-specific `-p`), wired into the
|
||||
container manager so new apps inherit it; or
|
||||
2. **Host-side v6→v4 forwarders** — generated nginx `stream {}` (or
|
||||
systemd-socket) units: `listen [::]:<port>` → `127.0.0.1:<port>`, one per
|
||||
catalog app port, regenerated on app install/remove, boot-time
|
||||
self-healed like the :80 fix. Keeps the Direct Port Rule URL contract
|
||||
without touching containers.
|
||||
|
||||
Either way: extend the bootstrap self-heal, and verify from the MESH side
|
||||
(curl the ULA on 2–3 app ports incl. :8334 from vps2 or .116) — not just
|
||||
from the LAN.
|
||||
|
||||
## DONE (00:30, dev-box agent): app direct ports live over the mesh
|
||||
|
||||
Shape 2-variant implemented INSIDE the backend (`mesh_ports.rs`, `2ad57c63`):
|
||||
a reconcile loop mirrors every public IPv4 listener (>=1024, bound 0.0.0.0,
|
||||
no existing IPv6 any-listener) as a v6-ONLY `[::]:<port>` forwarder to
|
||||
`127.0.0.1:<port>`, following `/proc/net/tcp*` every 15s — so app
|
||||
install/remove and hardcoded companion ports (bitcoin-ui :8334) are covered
|
||||
with zero container changes and no generated units; self-healing because it
|
||||
lives in the binary. Strictly ADDITIVE: IPv4/LAN/Tor paths untouched, v6only
|
||||
cannot intercept v4, foreign IPv6 listeners win.
|
||||
|
||||
Verified FROM THE MESH (vps2 → node ULA): :8334 HTTP 200 (466ms),
|
||||
:18083 200 (306ms), :50002 200 (239ms); LAN :8334 still 200. Deployed to
|
||||
framework-pt + .116 (binary sha 52ac0d8a…). Direct-port apps should now
|
||||
open in the companion over 5G.
|
||||
|
||||
Reference in New Issue
Block a user