diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 63d06e39..43b15b44 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -19,6 +19,7 @@ declared exit criteria (multinode pass + workstreams B/C/F), `.planning/codebase - [ ] **FED-05**: Inter-node Lightning channel opening UX — the UI shows the node's shareable Lightning URI; lists trusted (federated) nodes by hostname for one-click channel opening; and lets the user browse/request channels with public nodes — using the existing design system and components, verified on the :8100 dev preview against archi-dev before deploy - [ ] **FED-06**: On-brand payment success animation — the invoice "paid" tick's circle uses the screensaver-style ring with outer EQ-segment lines (reuse `ScreensaverRing.vue`'s compact size) in place of the current success burst, applied consistently everywhere the paid tick shows - [ ] **FED-08**: Lightning invoices created by the wallet embed route hints (LND `private` flag) so nodes whose channels are unannounced can actually receive payments — diagnosed on archy-x250-mad2 2026-07-31, where every wallet-UI invoice had `route_hints: []` and was unroutable; the bug is unconditional and affects any node without a public channel +- [ ] **FED-09**: The container doctor does not restart Tor on every run — it recognises Tor's own setgid `2700` hidden-service directory mode as correct rather than "fixing" it to `700` and restarting, a loop that reset Tor every ~5 minutes, starved it of its consensus/HSDir cache (`No more HSDir available to query`), and broke the mesh's Tor fallback entirely; genuinely permissive modes are still corrected, and a restart backoff makes the failure class non-recurring - [ ] **FED-07**: Fedimint gateway never installs with a pre-set password — gateway credentials are generated per-install via manifest-declared `generated_secrets` (or explicitly set by the user), never baked into the image/manifest; existing installs with the default password get a migration path (BLOCKER — default credentials are a security hole) ### UI Fixes (UIFIX) — user-reported blockers, added 2026-07-30 @@ -113,6 +114,7 @@ Which phases cover which requirements. Updated during roadmap creation. | FED-06 | Phase 1 | Pending | | FED-07 | Phase 1 | Pending | | FED-08 | Phase 1 | Pending | +| FED-09 | Phase 1 | Pending | | UIFIX-01 | Phase 1 | Pending | | UIFIX-02 | Phase 1 | Pending | | UIFIX-03 | Phase 1 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 6d769e44..c91befe7 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -35,7 +35,7 @@ signed/decentralized registry and a user installs it on their node. **Goal**: Federation and mesh are tight — a structured review of the fleet/federation and mesh code feeds fixes so node removal sticks, sync converges, and mesh messaging (including attachments) behaves identically everywhere it runs **Depends on**: Nothing (first phase) -**Requirements**: FED-01, FED-02, FED-03, FED-04, FED-05, FED-06, FED-07, UIFIX-01, UIFIX-02, UIFIX-03, UIFIX-04, UIFIX-05, UIFIX-06, FED-08 +**Requirements**: FED-01, FED-02, FED-03, FED-04, FED-05, FED-06, FED-07, UIFIX-01, UIFIX-02, UIFIX-03, UIFIX-04, UIFIX-05, UIFIX-06, FED-08, FED-09 **Success Criteria** (what must be TRUE): 1. A structured code review of the federation/fleet area (`core/archipelago/src/federation`, node sync, FIPS/transport dial layer) and the mesh area (`core/archipelago/src/mesh`, mesh RPC surface) produces a findings list, and every finding is fixed or explicitly deferred with a reason @@ -57,6 +57,7 @@ signed/decentralized registry and a user installs it on their node. Plans: +- [ ] 01-20-PLAN.md — URGENT wave 1: doctor stops restarting Tor every 5min (mesh Tor fallback) (FED-09) - [ ] 01-19-PLAN.md — URGENT wave 1: wallet invoices embed route hints so private-channel nodes can receive (FED-08) - [ ] 01-01-PLAN.md — Serialize the federation node store and make removal stick (FED-01) - [ ] 01-02-PLAN.md — Demo mesh/federation RPC parity + automated parity harness (FED-04) @@ -100,7 +101,7 @@ Plans: 3. Secondary screens open without a blocking full reload; repeat visits are instant 4. The fixes are verified on real node hardware (not just the dev box) — the sluggishness the user reported is gone on-device -**Plans**: 10/10 plans executed +**Plans**: 11/11 plans executed Plans: **Wave 1** @@ -134,6 +135,10 @@ Plans: - [x] 02-10-PLAN.md — Timing-regression verdict: three-way re-measure, clear or name each surface (PERF-02/03) +**Wave 8** *(gap closure — blocked on Wave 7 completion)* + +- [x] 02-11-PLAN.md — Profile the real cause of the six confirmed regressions, fix what's fixable, re-measure (PERF-02/03) + **UI hint**: yes ### Phase 3: Multinode Verification Pass diff --git a/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md b/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md new file mode 100644 index 00000000..781cf7ec --- /dev/null +++ b/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md @@ -0,0 +1,160 @@ +--- +phase: 01-federation-mesh-hardening +plan: 20 +type: execute +wave: 1 +depends_on: [] +files_modified: + - scripts/container-doctor.sh +autonomous: false +requirements: [FED-09] + +must_haves: + truths: + - "The container doctor no longer restarts Tor on every run — a node left alone shows Tor uptime growing past the doctor's 5-minute timer interval" + - "A hidden-service directory at mode 2700 (Tor's own setting) is recognised as correct and triggers no chmod and no restart" + - "A genuinely insecure hidden-service directory (group- or other-readable, e.g. 750 or 707) is still corrected" + - "Even when a real permission fix IS applied, Tor cannot be restarted more than once per backoff window, so no future defect can reproduce a restart storm" + - "Tor retains its consensus/HSDir cache long enough to resolve .onion addresses, so the mesh Tor fallback works" + prohibitions: + - "MUST NOT loosen hidden-service directory permissions — group and other access must remain denied" + - "MUST NOT disable the doctor's other fixes or the timer itself" +--- + + +Mesh sends fail entirely on affected nodes because both transports are down, and +the second failure is self-inflicted. + +Diagnosed 2026-07-31 on a live node: +1. The FIPS direct transport (Yggdrasil-style `fd..` IPv6) times out with + `connect_fail` for peers other than the currently-connected tree peers, so + every send falls back to Tor. +2. The Tor fallback then fails with `No more HSDir available to query` — Tor + cannot resolve any peer `.onion` address. + +Root cause of (2): **the container doctor restarts Tor every ~5 minutes, +forever.** `fix_tor_permissions()` in `scripts/container-doctor.sh` treats any +mode other than the literal string `700` as broken: + +```sh +perms=$(stat -c '%a' "$dir") +if [ "$perms" != "700" ]; then + chmod 700 "$dir"; fixed=true +fi +... +if $fixed; then systemctl restart tor@default; fi +``` + +Tor sets its own `HiddenServiceDir` to **2700** (setgid). So every run the doctor +sees `2700 != 700`, "fixes" it, and restarts Tor; Tor comes back up and sets 2700 +again; the timer fires 5 minutes later and the cycle repeats. Observed restarts +on the node: 13:07:56 → 13:13:14 → 13:18:39 → 13:23:57, each within a second of +an `archipelago-doctor.timer` firing. Tor never survives long enough to build a +usable consensus/HSDir cache, so onion lookups fail and the mesh's only remaining +transport dies with it. + +`2700` is not a defect — the setgid bit is harmless here and group/other access +is still fully denied, which is the property that actually matters. + +**Scope note:** this plan fixes the restart loop only. The FIPS direct-transport +`connect_fail` (problem 1) is a separate concern and belongs with FED-03's +structured review of the transport/dial layer — record it there, do not attempt +both here. + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + + + + Task 1: Stop the doctor from fighting Tor over the setgid bit + Contained to one shell function; revert is a single-file change. + scripts/container-doctor.sh + + - `scripts/container-doctor.sh` — `fix_tor_permissions()` (~lines 139-164) and how other `fix_*` functions signal "changed" vs "no drift" + - `image-recipe/configs/archipelago-doctor.timer` — `OnUnitActiveSec=5min`, `RandomizedDelaySec=60`: this is the loop's clock + - `core/archipelago/src/bootstrap.rs:24-31` — the script is embedded via `include_str!` and written to `/home/archipelago/archy/scripts/container-doctor.sh` on every boot, so nodes pick the fix up through the normal binary release + + + Correct the permission predicate so it tests the property that matters — + group and other have no access — instead of exact-matching one octal string. + Compare the low three digits of `stat -c '%a'` (which omits leading zeros, so + handle both `700` and `2700` forms), and treat the directory as correct when + those are `700`. Only a genuinely permissive mode (any group or other bit + set, e.g. `750`, `707`, `2755`) is a real defect worth fixing. + + Keep correcting real defects, and keep restarting Tor when a real fix is + applied — but add a **restart backoff** so a restart storm is impossible even + if some future condition makes the fix fire repeatedly: record the last + restart time (e.g. a timestamp file under `/var/lib/archipelago/`) and skip + the restart if one happened within the last 30 minutes, logging that it was + skipped. The current defect is being fixed at the predicate, but the backoff + is what makes the class of failure non-recurring. + + Log clearly in both directions — when a directory is accepted as already + correct (at debug level, so a healthy node stays quiet) and when a real fix + is applied. The original bug was invisible precisely because "Fixed + permissions on ... (2700 -> 700)" looked like the doctor working correctly. + + + bash -n scripts/container-doctor.sh && sudo bash -c 'set -e; d=$(mktemp -d); mkdir -p "$d/hidden_service_test"; chmod 2700 "$d/hidden_service_test"; stat -c "%a" "$d/hidden_service_test"' + + + - `bash -n scripts/container-doctor.sh` passes + - A directory at mode `2700` is accepted: no chmod, no restart, `fixed` stays false + - A directory at mode `750` or `707` is still corrected to deny group/other + - A second real fix within the backoff window logs a skip instead of restarting Tor + - The doctor's other fixes and the timer are untouched + + The doctor recognises Tor's own 2700 as correct, so it stops restarting Tor every five minutes, and a backoff prevents any future restart storm. + + + + Task 2: Confirm Tor stays up and mesh sends recover + + The doctor no longer mistakes Tor's setgid `2700` hidden-service directory for + a permission defect, so it stops chmod-ing it and restarting Tor every ~5 + minutes. A restart backoff makes a restart storm impossible even if some other + condition triggers the fix repeatedly. + + + This ships to affected nodes via the **normal OTA release only**. Never deploy + directly to a user's device (`archy-x250-mad2` or any node that is not ours). + + + On an affected node, after the release lands: + 1. `systemctl status tor@default` — note the uptime. Wait 15 minutes (three + doctor intervals) and check again: uptime should keep growing, with no + restart. Before the fix it reset roughly every 5 minutes. + 2. `journalctl -u archipelago-doctor -n 50` — no recurring + "Fixed permissions on ... hidden_service_* (2700 -> 700)" lines. + 3. Once Tor has been up ~20-30 minutes, confirm onion resolution works: a + mesh send to a peer reachable only via Tor should succeed, and the logs + should no longer show `No more HSDir available to query`. + 4. Confirm the doctor still does its job: temporarily `chmod 750` a + hidden-service directory, wait for the next doctor run, and check it is + corrected back to deny group/other access. + + Type "approved", or describe what you saw — which step, what happened instead. + + + + + +- `bash -n` passes; the 2700 case is accepted and the 750/707 cases are still fixed +- On an affected node, Tor uptime exceeds the doctor's interval and onion resolution recovers + + + +- Tor is no longer restarted every ~5 minutes by the doctor +- Tor keeps its consensus/HSDir cache, so `.onion` peers resolve and the mesh Tor fallback works again +- Genuinely insecure hidden-service directory permissions are still corrected +- A restart backoff makes this class of failure non-recurring + + + +Create `.planning/phases/01-federation-mesh-hardening/01-20-SUMMARY.md`. It MUST record the corrected predicate, the backoff mechanism and window, and a note handing the FIPS direct-transport `connect_fail` (problem 1 of the original diagnosis) to FED-03's transport/dial-layer review. +