diff --git a/.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md b/.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md index 304f39d8..3873eb1c 100644 --- a/.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md +++ b/.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md @@ -41,8 +41,8 @@ coverage: ref: "core/archipelago/src/container/secrets.rs#rotates_a_denylisted_gateway_credential" status: pass - kind: manual_procedural - ref: "Task 2 blocking checkpoint on archi-dev-box — NOT RUN" - status: deferred + ref: "Task 2 checkpoint, archi-dev-box 2026-08-01 — the credential FILE rotates correctly (~15s after restart, fresh unique value, 0600), but the RUNNING gateway keeps the pre-rotation credential: Quadlet rewrites the unit without restarting it, and the gateway is classified restart-sensitive so drift is detected and deliberately ignored on every tick" + status: fail human_judgment: true - id: D2 description: "A node already carrying a unique credential is left completely alone; detection never fires on merely-unrecognised values" @@ -79,8 +79,8 @@ coverage: ref: "The info! line interpolates self.secrets_dir and the secret NAME only; no value is in scope at the call site (rotate returns bool, not the credential)" status: pass - kind: manual_procedural - ref: "Task 2 step 3 — confirming no value appears in a real node's log — NOT RUN" - status: deferred + ref: "Task 2 step 3, archi-dev-box 2026-08-01 — one info line fired, naming /var/lib/archipelago/secrets/fedimint-gateway-hash.pw; no credential value anywhere in the log" + status: pass human_judgment: true - id: D6 description: "Generation where no credential exists stays ensure_gateway_credential's job" @@ -91,19 +91,20 @@ coverage: status: pass human_judgment: false -duration: 45min +duration: 140min completed: 2026-08-01 -status: task-1-complete-checkpoint-pending +status: task-1-complete-checkpoint-FAILED-recreate-does-not-fire --- # Phase 1 Plan 16: Rotate Existing Installs Off the Shipped Gateway Credential (FED-07) Summary -**Task 1 is complete: an affected node now detects the published default on its next reconcile tick and rotates itself onto a unique credential, with the container rebuilt through the platform's own drift-recreate path rather than any hand-rolled teardown. Task 2 — the blocking on-node checkpoint — has NOT been run.** +**Task 1 rotates the credential correctly and was proven to do so on a real node. But the checkpoint DISPROVED the assumption it rests on: the rotated credential never reaches the running container, because the Quadlet path does not restart units and the gateway is classified restart-sensitive. FED-07 remains open.** ## Status -**FED-07 is not yet closed.** This plan's requirement stays open until the Task 2 checkpoint runs on a -real node. The code half is done and verified by unit tests; the on-node half is untouched. +**FED-07 is NOT closed, and this plan alone cannot close it.** The checkpoint ran on archi-dev-box on +2026-08-01 and found that rotation does not propagate to the running gateway. A follow-up plan is +required — see the checkpoint result below. ## Accomplishments @@ -128,7 +129,71 @@ real node. The code half is done and verified by unit tests; the on-node half is So boot and reconcile funnel through one chokepoint and **no boot-specific wiring was added**; `boot_reconciler.rs` is not in `files_modified`. -### The recreate fires through `secret_env_hash` — mechanism confirmed by reading, not yet observed running +### ⛔ CHECKPOINT RESULT: the recreate does NOT fire. FED-07 is not closed by this plan. + +**Run on archi-dev-box, 2026-08-01.** The code-reading conclusion below was **wrong in practice**, +which is exactly why the plan made this a blocking checkpoint. The rotation works; the propagation +to the running container does not. + +Observed: rotation fired ~15s after restart, wrote a fresh unique credential, and logged correctly. +But **25 minutes later the running gatewayd process was still using the pre-rotation credential.** +`/proc//environ` for PID 1 held the old value while the file and the podman secret held the new +one. The orchestrator says why, in its own log lines: + +``` +Quadlet unit drift-synced — file rewritten, .service NOT restarted + (operator restart picks up new config) app_id=fedimint-gateway +container drift detected during boot reconcile; + leaving running restart-sensitive app untouched app_id=fedimint-gateway +``` + +Two independent guards, both deliberate: + +1. **The Quadlet path rewrites the `.container` unit but never restarts the `.service`.** The unit + file was rewritten at 15:51:36 (same second as the rotation) carrying the new + `secret-env-hash=de9870c642a515f7` label — so the definition updated correctly. Systemd does not + apply a changed unit to a running container without a restart. +2. **`fedimint-gateway` is classified restart-sensitive**, so the drift check *detects* the change on + every reconcile tick and then deliberately leaves the container alone. That line repeated at + 15:51, 15:53, 15:54, 15:56 — it will repeat forever. + +**Consequence: on a real affected node, rotation makes the credential file unique but the gateway +keeps answering to the compromised one indefinitely** — until an unrelated reboot or a manual +restart. Worse, the operator reading `fedimint-gateway-hash.pw` gets a password the running gateway +does not accept, which is the lockout risk T-01-77 inverted. + +Confirmed the fix works when applied: `systemctl --user restart fedimint-gateway.service` produced a +new PID whose `FEDI_HASH` is the rotated value, container healthy, same name, same ports, marker file +and `gatewayd.db` intact. + +**Deliberately NOT hand-rolled.** The plan's action says: "If that path does not fire for this app for +some reason you discover, do not hand-roll a remove-and-run; stop and record what you found." So this +is recorded, not patched. The fix belongs in a follow-up and has to answer a real design question: +a compromised credential is arguably the one case that should override restart-sensitivity — or, +failing that, the rotation must raise an operator-facing "restart required" alert rather than logging +into the void. + +### What the checkpoint DID prove + +| Step | Result | +|---|---| +| 1. State recorded | Node was CLEAN; affected state seeded deliberately (plan's Planner Assumption) | +| 2. Deploy | `install -m0755` to `/usr/local/bin/archipelago` + `systemctl restart archipelago`; rollback kept at `archipelago.bak-pre-fed07` | +| 3. Rotation announced | ✅ One info line, fired once, names the `.pw` path, **no credential value in the log** | +| 4. Credential unique | ✅ Third distinct value (not the default, not the pre-test original), `0600 archipelago:archipelago` | +| 5. Data preserved | ✅ Marker file and all 13 `gatewayd.db` files incl. `IDENTITY` intact; same container name; same ports 8176/9737 | +| 5. Container recreated | ❌ **FAILED — see above** | +| 6. Auth proof | ⚠️ Not obtainable: `gateway-cli` in this image returns the same "Invalid request" for a correct and an incorrect password, so it cannot distinguish them. Substituted PID-1 `environ` comparison, which is stronger evidence of *which* credential is in force. | +| 7. Fresh install differs | ⏸ Not run | +| 8. `run-gate.sh` | ⏸ Not run | + +Also proved incidentally: **restarting `archipelago` does not kill containers on this box** — 29/29 +and later 31/31 survived, and the orchestrator logged "Adopted 31 existing container(s)". The +CLAUDE.md "restart SIGKILLs containers" rule does not apply under `ARCHIPELAGO_USE_QUADLET_BACKENDS=true` +with podman in the user slice (the service is `system.slice`/`KillMode=control-group`; the containers +live in `user-1000.slice/…/libpod-*`, a different cgroup entirely). + +### Original code-reading conclusion (retained — it is what the checkpoint disproved) `resolve_dynamic_env` computes `secret_env_content_hash(&secret_bearing)` over the resolved secret-bearing env and stores it as `manifest.app.container.secret_env_hash`