docs(01-16): checkpoint FAILED on archi-dev-box — rotation never reaches the container
Ran Task 2's blocking checkpoint on a real node. The rotation works and was
proven end to end: it fired ~15s after restart, wrote a fresh unique
credential (0600, service-owned), logged exactly one line naming the .pw path
with no value in it, left every other secret and the gateway's data untouched,
and kept the container's name and ports.
But the assumption the plan rests on is WRONG, and the checkpoint is what
caught it. 25 minutes after rotating, /proc/<pid>/environ showed the running
gatewayd still using the PRE-ROTATION credential while the file and podman
secret held the new one. The orchestrator explains itself in its own logs:
Quadlet unit drift-synced — file rewritten, .service NOT restarted
(operator restart picks up new config)
container drift detected during boot reconcile;
leaving running restart-sensitive app untouched
Two deliberate guards: the Quadlet path never restarts a unit it rewrites, and
fedimint-gateway is classified restart-sensitive so drift is detected on every
tick and then ignored — logged at 15:51, 15:53, 15:54, 15:56 and counting.
So on a real affected node the credential file becomes unique while the
gateway keeps answering to the compromised one until an unrelated reboot, and
the operator reading the .pw gets a password the gateway rejects — T-01-77
inverted. FED-07 is NOT closed and this plan alone cannot close it.
Not hand-rolled around, per the plan's own instruction. The fix needs a design
decision: whether a compromised credential is the case that should override
restart-sensitivity, or whether rotation must raise an operator-facing
"restart required" alert instead of logging into the void.
Incidentally disproved: restarting archipelago does NOT kill containers here
(29/29 then 31/31 survived; "Adopted 31 existing container(s)"). The service
is system.slice/KillMode=control-group while containers live in
user-1000.slice/…/libpod-*. The CLAUDE.md SIGKILL rule predates Quadlet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
095664a8a7
commit
56f7b367ff
@@ -41,8 +41,8 @@ coverage:
|
||||
ref: "core/archipelago/src/container/secrets.rs#rotates_a_denylisted_gateway_credential"
|
||||
status: pass
|
||||
- kind: manual_procedural
|
||||
ref: "Task 2 blocking checkpoint on archi-dev-box — NOT RUN"
|
||||
status: deferred
|
||||
ref: "Task 2 checkpoint, archi-dev-box 2026-08-01 — the credential FILE rotates correctly (~15s after restart, fresh unique value, 0600), but the RUNNING gateway keeps the pre-rotation credential: Quadlet rewrites the unit without restarting it, and the gateway is classified restart-sensitive so drift is detected and deliberately ignored on every tick"
|
||||
status: fail
|
||||
human_judgment: true
|
||||
- id: D2
|
||||
description: "A node already carrying a unique credential is left completely alone; detection never fires on merely-unrecognised values"
|
||||
@@ -79,8 +79,8 @@ coverage:
|
||||
ref: "The info! line interpolates self.secrets_dir and the secret NAME only; no value is in scope at the call site (rotate returns bool, not the credential)"
|
||||
status: pass
|
||||
- kind: manual_procedural
|
||||
ref: "Task 2 step 3 — confirming no value appears in a real node's log — NOT RUN"
|
||||
status: deferred
|
||||
ref: "Task 2 step 3, archi-dev-box 2026-08-01 — one info line fired, naming /var/lib/archipelago/secrets/fedimint-gateway-hash.pw; no credential value anywhere in the log"
|
||||
status: pass
|
||||
human_judgment: true
|
||||
- id: D6
|
||||
description: "Generation where no credential exists stays ensure_gateway_credential's job"
|
||||
@@ -91,19 +91,20 @@ coverage:
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
duration: 45min
|
||||
duration: 140min
|
||||
completed: 2026-08-01
|
||||
status: task-1-complete-checkpoint-pending
|
||||
status: task-1-complete-checkpoint-FAILED-recreate-does-not-fire
|
||||
---
|
||||
|
||||
# Phase 1 Plan 16: Rotate Existing Installs Off the Shipped Gateway Credential (FED-07) Summary
|
||||
|
||||
**Task 1 is complete: an affected node now detects the published default on its next reconcile tick and rotates itself onto a unique credential, with the container rebuilt through the platform's own drift-recreate path rather than any hand-rolled teardown. Task 2 — the blocking on-node checkpoint — has NOT been run.**
|
||||
**Task 1 rotates the credential correctly and was proven to do so on a real node. But the checkpoint DISPROVED the assumption it rests on: the rotated credential never reaches the running container, because the Quadlet path does not restart units and the gateway is classified restart-sensitive. FED-07 remains open.**
|
||||
|
||||
## Status
|
||||
|
||||
**FED-07 is not yet closed.** This plan's requirement stays open until the Task 2 checkpoint runs on a
|
||||
real node. The code half is done and verified by unit tests; the on-node half is untouched.
|
||||
**FED-07 is NOT closed, and this plan alone cannot close it.** The checkpoint ran on archi-dev-box on
|
||||
2026-08-01 and found that rotation does not propagate to the running gateway. A follow-up plan is
|
||||
required — see the checkpoint result below.
|
||||
|
||||
## Accomplishments
|
||||
|
||||
@@ -128,7 +129,71 @@ real node. The code half is done and verified by unit tests; the on-node half is
|
||||
So boot and reconcile funnel through one chokepoint and **no boot-specific wiring was added**;
|
||||
`boot_reconciler.rs` is not in `files_modified`.
|
||||
|
||||
### The recreate fires through `secret_env_hash` — mechanism confirmed by reading, not yet observed running
|
||||
### ⛔ CHECKPOINT RESULT: the recreate does NOT fire. FED-07 is not closed by this plan.
|
||||
|
||||
**Run on archi-dev-box, 2026-08-01.** The code-reading conclusion below was **wrong in practice**,
|
||||
which is exactly why the plan made this a blocking checkpoint. The rotation works; the propagation
|
||||
to the running container does not.
|
||||
|
||||
Observed: rotation fired ~15s after restart, wrote a fresh unique credential, and logged correctly.
|
||||
But **25 minutes later the running gatewayd process was still using the pre-rotation credential.**
|
||||
`/proc/<pid>/environ` for PID 1 held the old value while the file and the podman secret held the new
|
||||
one. The orchestrator says why, in its own log lines:
|
||||
|
||||
```
|
||||
Quadlet unit drift-synced — file rewritten, .service NOT restarted
|
||||
(operator restart picks up new config) app_id=fedimint-gateway
|
||||
container drift detected during boot reconcile;
|
||||
leaving running restart-sensitive app untouched app_id=fedimint-gateway
|
||||
```
|
||||
|
||||
Two independent guards, both deliberate:
|
||||
|
||||
1. **The Quadlet path rewrites the `.container` unit but never restarts the `.service`.** The unit
|
||||
file was rewritten at 15:51:36 (same second as the rotation) carrying the new
|
||||
`secret-env-hash=de9870c642a515f7` label — so the definition updated correctly. Systemd does not
|
||||
apply a changed unit to a running container without a restart.
|
||||
2. **`fedimint-gateway` is classified restart-sensitive**, so the drift check *detects* the change on
|
||||
every reconcile tick and then deliberately leaves the container alone. That line repeated at
|
||||
15:51, 15:53, 15:54, 15:56 — it will repeat forever.
|
||||
|
||||
**Consequence: on a real affected node, rotation makes the credential file unique but the gateway
|
||||
keeps answering to the compromised one indefinitely** — until an unrelated reboot or a manual
|
||||
restart. Worse, the operator reading `fedimint-gateway-hash.pw` gets a password the running gateway
|
||||
does not accept, which is the lockout risk T-01-77 inverted.
|
||||
|
||||
Confirmed the fix works when applied: `systemctl --user restart fedimint-gateway.service` produced a
|
||||
new PID whose `FEDI_HASH` is the rotated value, container healthy, same name, same ports, marker file
|
||||
and `gatewayd.db` intact.
|
||||
|
||||
**Deliberately NOT hand-rolled.** The plan's action says: "If that path does not fire for this app for
|
||||
some reason you discover, do not hand-roll a remove-and-run; stop and record what you found." So this
|
||||
is recorded, not patched. The fix belongs in a follow-up and has to answer a real design question:
|
||||
a compromised credential is arguably the one case that should override restart-sensitivity — or,
|
||||
failing that, the rotation must raise an operator-facing "restart required" alert rather than logging
|
||||
into the void.
|
||||
|
||||
### What the checkpoint DID prove
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| 1. State recorded | Node was CLEAN; affected state seeded deliberately (plan's Planner Assumption) |
|
||||
| 2. Deploy | `install -m0755` to `/usr/local/bin/archipelago` + `systemctl restart archipelago`; rollback kept at `archipelago.bak-pre-fed07` |
|
||||
| 3. Rotation announced | ✅ One info line, fired once, names the `.pw` path, **no credential value in the log** |
|
||||
| 4. Credential unique | ✅ Third distinct value (not the default, not the pre-test original), `0600 archipelago:archipelago` |
|
||||
| 5. Data preserved | ✅ Marker file and all 13 `gatewayd.db` files incl. `IDENTITY` intact; same container name; same ports 8176/9737 |
|
||||
| 5. Container recreated | ❌ **FAILED — see above** |
|
||||
| 6. Auth proof | ⚠️ Not obtainable: `gateway-cli` in this image returns the same "Invalid request" for a correct and an incorrect password, so it cannot distinguish them. Substituted PID-1 `environ` comparison, which is stronger evidence of *which* credential is in force. |
|
||||
| 7. Fresh install differs | ⏸ Not run |
|
||||
| 8. `run-gate.sh` | ⏸ Not run |
|
||||
|
||||
Also proved incidentally: **restarting `archipelago` does not kill containers on this box** — 29/29
|
||||
and later 31/31 survived, and the orchestrator logged "Adopted 31 existing container(s)". The
|
||||
CLAUDE.md "restart SIGKILLs containers" rule does not apply under `ARCHIPELAGO_USE_QUADLET_BACKENDS=true`
|
||||
with podman in the user slice (the service is `system.slice`/`KillMode=control-group`; the containers
|
||||
live in `user-1000.slice/…/libpod-*`, a different cgroup entirely).
|
||||
|
||||
### Original code-reading conclusion (retained — it is what the checkpoint disproved)
|
||||
|
||||
`resolve_dynamic_env` computes `secret_env_content_hash(&secret_bearing)` over the resolved
|
||||
secret-bearing env and stores it as `manifest.app.container.secret_env_hash`
|
||||
|
||||
Reference in New Issue
Block a user