docs(01): add FED-09 — doctor's Tor restart loop breaks mesh Tor fallback

container-doctor.sh's fix_tor_permissions() exact-matches mode '700', but Tor sets its hidden-service dirs to 2700 (setgid). Every 5-minute doctor run 'fixes' 2700->700 and restarts tor@default; Tor resets it and the cycle repeats, so Tor never builds a usable consensus/HSDir cache. Result on a live node: 'No more HSDir available to query', onion peers unresolvable, and with the FIPS direct path also timing out, mesh sends failed entirely. Planned as 01-20 in wave 1 so it ships in the same release. FIPS direct connect_fail is a separate concern, handed to FED-03's transport review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 09:37:41 -04:00 · 2026-07-31 09:37:41 -04:00 · 68d63b1140
commit 68d63b1140
parent 9ad31a7495
3 changed files with 169 additions and 2 deletions
--- a/.planning/REQUIREMENTS.md
+++ b/.planning/REQUIREMENTS.md
@ -19,6 +19,7 @@ declared exit criteria (multinode pass + workstreams B/C/F), `.planning/codebase
 - [ ] **FED-05**: Inter-node Lightning channel opening UX — the UI shows the node's shareable Lightning URI; lists trusted (federated) nodes by hostname for one-click channel opening; and lets the user browse/request channels with public nodes — using the existing design system and components, verified on the :8100 dev preview against archi-dev before deploy
 - [ ] **FED-06**: On-brand payment success animation — the invoice "paid" tick's circle uses the screensaver-style ring with outer EQ-segment lines (reuse `ScreensaverRing.vue`'s compact size) in place of the current success burst, applied consistently everywhere the paid tick shows
 - [ ] **FED-08**: Lightning invoices created by the wallet embed route hints (LND `private` flag) so nodes whose channels are unannounced can actually receive payments — diagnosed on archy-x250-mad2 2026-07-31, where every wallet-UI invoice had `route_hints: []` and was unroutable; the bug is unconditional and affects any node without a public channel
+- [ ] **FED-09**: The container doctor does not restart Tor on every run — it recognises Tor's own setgid `2700` hidden-service directory mode as correct rather than "fixing" it to `700` and restarting, a loop that reset Tor every ~5 minutes, starved it of its consensus/HSDir cache (`No more HSDir available to query`), and broke the mesh's Tor fallback entirely; genuinely permissive modes are still corrected, and a restart backoff makes the failure class non-recurring
 - [ ] **FED-07**: Fedimint gateway never installs with a pre-set password — gateway credentials are generated per-install via manifest-declared `generated_secrets` (or explicitly set by the user), never baked into the image/manifest; existing installs with the default password get a migration path (BLOCKER — default credentials are a security hole)

 ### UI Fixes (UIFIX) — user-reported blockers, added 2026-07-30
@ -113,6 +114,7 @@ Which phases cover which requirements. Updated during roadmap creation.
 | FED-06 | Phase 1 | Pending |
 | FED-07 | Phase 1 | Pending |
 | FED-08 | Phase 1 | Pending |
+| FED-09 | Phase 1 | Pending |
 | UIFIX-01 | Phase 1 | Pending |
 | UIFIX-02 | Phase 1 | Pending |
 | UIFIX-03 | Phase 1 | Pending |
--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@ -35,7 +35,7 @@ signed/decentralized registry and a user installs it on their node.

 **Goal**: Federation and mesh are tight — a structured review of the fleet/federation and mesh code feeds fixes so node removal sticks, sync converges, and mesh messaging (including attachments) behaves identically everywhere it runs
 **Depends on**: Nothing (first phase)
-**Requirements**: FED-01, FED-02, FED-03, FED-04, FED-05, FED-06, FED-07, UIFIX-01, UIFIX-02, UIFIX-03, UIFIX-04, UIFIX-05, UIFIX-06, FED-08
+**Requirements**: FED-01, FED-02, FED-03, FED-04, FED-05, FED-06, FED-07, UIFIX-01, UIFIX-02, UIFIX-03, UIFIX-04, UIFIX-05, UIFIX-06, FED-08, FED-09
 **Success Criteria** (what must be TRUE):

  1. A structured code review of the federation/fleet area (`core/archipelago/src/federation`, node sync, FIPS/transport dial layer) and the mesh area (`core/archipelago/src/mesh`, mesh RPC surface) produces a findings list, and every finding is fixed or explicitly deferred with a reason
@ -57,6 +57,7 @@ signed/decentralized registry and a user installs it on their node.

 Plans:

+- [ ] 01-20-PLAN.md — URGENT wave 1: doctor stops restarting Tor every 5min (mesh Tor fallback) (FED-09)
 - [ ] 01-19-PLAN.md — URGENT wave 1: wallet invoices embed route hints so private-channel nodes can receive (FED-08)
 - [ ] 01-01-PLAN.md — Serialize the federation node store and make removal stick (FED-01)
 - [ ] 01-02-PLAN.md — Demo mesh/federation RPC parity + automated parity harness (FED-04)
@ -100,7 +101,7 @@ Plans:
  3. Secondary screens open without a blocking full reload; repeat visits are instant
  4. The fixes are verified on real node hardware (not just the dev box) — the sluggishness the user reported is gone on-device

-**Plans**: 10/10 plans executed
+**Plans**: 11/11 plans executed

 Plans:
 **Wave 1**
@ -134,6 +135,10 @@ Plans:

 - [x] 02-10-PLAN.md — Timing-regression verdict: three-way re-measure, clear or name each surface (PERF-02/03)

+**Wave 8** *(gap closure — blocked on Wave 7 completion)*
+
+- [x] 02-11-PLAN.md — Profile the real cause of the six confirmed regressions, fix what's fixable, re-measure (PERF-02/03)
+
 **UI hint**: yes

 ### Phase 3: Multinode Verification Pass
--- a/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md
+++ b/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md
@ -0,0 +1,160 @@
+---
+phase: 01-federation-mesh-hardening
+plan: 20
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - scripts/container-doctor.sh
+autonomous: false
+requirements: [FED-09]
+
+must_haves:
+  truths:
+    - "The container doctor no longer restarts Tor on every run — a node left alone shows Tor uptime growing past the doctor's 5-minute timer interval"
+    - "A hidden-service directory at mode 2700 (Tor's own setting) is recognised as correct and triggers no chmod and no restart"
+    - "A genuinely insecure hidden-service directory (group- or other-readable, e.g. 750 or 707) is still corrected"
+    - "Even when a real permission fix IS applied, Tor cannot be restarted more than once per backoff window, so no future defect can reproduce a restart storm"
+    - "Tor retains its consensus/HSDir cache long enough to resolve .onion addresses, so the mesh Tor fallback works"
+  prohibitions:
+    - "MUST NOT loosen hidden-service directory permissions — group and other access must remain denied"
+    - "MUST NOT disable the doctor's other fixes or the timer itself"
+---
+
+<objective>
+Mesh sends fail entirely on affected nodes because both transports are down, and
+the second failure is self-inflicted.
+
+Diagnosed 2026-07-31 on a live node:
+1. The FIPS direct transport (Yggdrasil-style `fd..` IPv6) times out with
+   `connect_fail` for peers other than the currently-connected tree peers, so
+   every send falls back to Tor.
+2. The Tor fallback then fails with `No more HSDir available to query` — Tor
+   cannot resolve any peer `.onion` address.
+
+Root cause of (2): **the container doctor restarts Tor every ~5 minutes,
+forever.** `fix_tor_permissions()` in `scripts/container-doctor.sh` treats any
+mode other than the literal string `700` as broken:
+
+```sh
+perms=$(stat -c '%a' "$dir")
+if [ "$perms" != "700" ]; then
+    chmod 700 "$dir"; fixed=true
+fi
+...
+if $fixed; then systemctl restart tor@default; fi
+```
+
+Tor sets its own `HiddenServiceDir` to **2700** (setgid). So every run the doctor
+sees `2700 != 700`, "fixes" it, and restarts Tor; Tor comes back up and sets 2700
+again; the timer fires 5 minutes later and the cycle repeats. Observed restarts
+on the node: 13:07:56 → 13:13:14 → 13:18:39 → 13:23:57, each within a second of
+an `archipelago-doctor.timer` firing. Tor never survives long enough to build a
+usable consensus/HSDir cache, so onion lookups fail and the mesh's only remaining
+transport dies with it.
+
+`2700` is not a defect — the setgid bit is harmless here and group/other access
+is still fully denied, which is the property that actually matters.
+
+**Scope note:** this plan fixes the restart loop only. The FIPS direct-transport
+`connect_fail` (problem 1) is a separate concern and belongs with FED-03's
+structured review of the transport/dial layer — record it there, do not attempt
+both here.
+</objective>
+
+<execution_context>
+@$HOME/.claude/gsd-core/workflows/execute-plan.md
+@$HOME/.claude/gsd-core/templates/summary.md
+</execution_context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Stop the doctor from fighting Tor over the setgid bit</name>
+  <reversibility rating="reversible">Contained to one shell function; revert is a single-file change.</reversibility>
+  <files>scripts/container-doctor.sh</files>
+  <read_first>
+    - `scripts/container-doctor.sh` — `fix_tor_permissions()` (~lines 139-164) and how other `fix_*` functions signal "changed" vs "no drift"
+    - `image-recipe/configs/archipelago-doctor.timer` — `OnUnitActiveSec=5min`, `RandomizedDelaySec=60`: this is the loop's clock
+    - `core/archipelago/src/bootstrap.rs:24-31` — the script is embedded via `include_str!` and written to `/home/archipelago/archy/scripts/container-doctor.sh` on every boot, so nodes pick the fix up through the normal binary release
+  </read_first>
+  <action>
+    Correct the permission predicate so it tests the property that matters —
+    group and other have no access — instead of exact-matching one octal string.
+    Compare the low three digits of `stat -c '%a'` (which omits leading zeros, so
+    handle both `700` and `2700` forms), and treat the directory as correct when
+    those are `700`. Only a genuinely permissive mode (any group or other bit
+    set, e.g. `750`, `707`, `2755`) is a real defect worth fixing.
+
+    Keep correcting real defects, and keep restarting Tor when a real fix is
+    applied — but add a **restart backoff** so a restart storm is impossible even
+    if some future condition makes the fix fire repeatedly: record the last
+    restart time (e.g. a timestamp file under `/var/lib/archipelago/`) and skip
+    the restart if one happened within the last 30 minutes, logging that it was
+    skipped. The current defect is being fixed at the predicate, but the backoff
+    is what makes the class of failure non-recurring.
+
+    Log clearly in both directions — when a directory is accepted as already
+    correct (at debug level, so a healthy node stays quiet) and when a real fix
+    is applied. The original bug was invisible precisely because "Fixed
+    permissions on ... (2700 -> 700)" looked like the doctor working correctly.
+  </action>
+  <verify>
+    <automated>bash -n scripts/container-doctor.sh && sudo bash -c 'set -e; d=$(mktemp -d); mkdir -p "$d/hidden_service_test"; chmod 2700 "$d/hidden_service_test"; stat -c "%a" "$d/hidden_service_test"' </automated>
+  </verify>
+  <acceptance_criteria>
+    - `bash -n scripts/container-doctor.sh` passes
+    - A directory at mode `2700` is accepted: no chmod, no restart, `fixed` stays false
+    - A directory at mode `750` or `707` is still corrected to deny group/other
+    - A second real fix within the backoff window logs a skip instead of restarting Tor
+    - The doctor's other fixes and the timer are untouched
+  </acceptance_criteria>
+  <done>The doctor recognises Tor's own 2700 as correct, so it stops restarting Tor every five minutes, and a backoff prevents any future restart storm.</done>
+</task>
+
+<task type="checkpoint:human-verify" gate="blocking">
+  <name>Task 2: Confirm Tor stays up and mesh sends recover</name>
+  <what-built>
+    The doctor no longer mistakes Tor's setgid `2700` hidden-service directory for
+    a permission defect, so it stops chmod-ing it and restarting Tor every ~5
+    minutes. A restart backoff makes a restart storm impossible even if some other
+    condition triggers the fix repeatedly.
+  </what-built>
+  <constraint priority="highest">
+    This ships to affected nodes via the **normal OTA release only**. Never deploy
+    directly to a user's device (`archy-x250-mad2` or any node that is not ours).
+  </constraint>
+  <how-to-verify>
+    On an affected node, after the release lands:
+    1. `systemctl status tor@default` — note the uptime. Wait 15 minutes (three
+       doctor intervals) and check again: uptime should keep growing, with no
+       restart. Before the fix it reset roughly every 5 minutes.
+    2. `journalctl -u archipelago-doctor -n 50` — no recurring
+       "Fixed permissions on ... hidden_service_* (2700 -> 700)" lines.
+    3. Once Tor has been up ~20-30 minutes, confirm onion resolution works: a
+       mesh send to a peer reachable only via Tor should succeed, and the logs
+       should no longer show `No more HSDir available to query`.
+    4. Confirm the doctor still does its job: temporarily `chmod 750` a
+       hidden-service directory, wait for the next doctor run, and check it is
+       corrected back to deny group/other access.
+  </how-to-verify>
+  <resume-signal>Type "approved", or describe what you saw — which step, what happened instead.</resume-signal>
+</task>
+
+</tasks>
+
+<verification>
+- `bash -n` passes; the 2700 case is accepted and the 750/707 cases are still fixed
+- On an affected node, Tor uptime exceeds the doctor's interval and onion resolution recovers
+</verification>
+
+<success_criteria>
+- Tor is no longer restarted every ~5 minutes by the doctor
+- Tor keeps its consensus/HSDir cache, so `.onion` peers resolve and the mesh Tor fallback works again
+- Genuinely insecure hidden-service directory permissions are still corrected
+- A restart backoff makes this class of failure non-recurring
+</success_criteria>
+
+<output>
+Create `.planning/phases/01-federation-mesh-hardening/01-20-SUMMARY.md`. It MUST record the corrected predicate, the backoff mechanism and window, and a note handing the FIPS direct-transport `connect_fail` (problem 1 of the original diagnosis) to FED-03's transport/dial-layer review.
+</output>