lfg2025/archy
1
0
Fork 0
Code Issues 27 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

feat(01-16): rotate existing installs off the shipped gateway credential (FED-07)

Browse Source 
01-11 stopped new installs from ever taking a shipped credential, but did
nothing for the nodes that already did — those gateways still answer to a
password published in this repository.

rotate_compromised_gateway_credential() detects an EXACT match against the
denylist and replaces the pair; absent, unique, or merely unrecognised values
are left alone and return false. That distinction is the point: an operator
who deliberately set their own credential also has an "unrecognised" one, and
rotating it would be the same class of harm as leaving the default in place.

It hangs off resolve_dynamic_env beside ensure_generated_secrets, gated on the
gateway's app id, so an affected node heals on its next reconcile tick. There
is deliberately no teardown here: the new hash changes the resolved secret env,
which changes secret_env_hash, which the drift check reads as a container-label
mismatch — so the platform's own recreate path rebuilds the gateway around its
unchanged data directory, ports, volumes and name.

Rotation is self-terminating (the value written is not on the denylist, so the
next tick is a no-op) and errors propagate rather than being swallowed, because
the atomic write leaves the previous credential intact on failure.

Bcrypt generation was factored out of ensure_one into write_bcrypt_pair, which
both generation and rotation call — 01-11's SUMMARY claimed such a helper
existed but the arm was still inline, and rotation cannot reuse
ensure_gateway_credential because its idempotent fast path returns early
exactly when the file is present, which is the case rotation acts on.

Also fixes cargo fmt drift left by 42652547 in install.rs.

Verified: 6 new tests, secrets suite 16/16; full suite 1008 passed with one
known wall-clock flake (green 4/4 in isolation). NOT verified on a node —
Task 2's blocking checkpoint has not been run, so FED-07 stays open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-01 13:17:56 -04:00
co-authored by Claude Opus 5
parent d238bad012
commit 9e2d2ef236
4 changed files with 456 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md
+242
View File
@@ -0,0 +1,242 @@
---
phase: 01-federation-mesh-hardening
plan: 16
subsystem: security
tags: [secrets, bcrypt, fedimint, migration, rotation, reconcile]
requires:
- phase: 01-federation-mesh-hardening
provides: "01-11's KNOWN_DEFAULT_GATEWAY_HASHES denylist, ensure_gateway_credential, gateway_bcrypt_hash and the atomic 0600 write_secret — rotation reuses all of it and adds no new generation or file-writing code"
provides:
- "rotate_compromised_gateway_credential(secrets_dir) -> Result<bool>: denylist-exact detection plus rotation of a shipped gateway credential"
- "Self-healing on the existing reconcile tick, so an affected node rotates without operator action and without a hand-rolled container teardown"
affects: [fedimint-gateway, container-secrets, reconcile]
tech-stack:
added: []
patterns:
- "Rotate by changing the secret, not by touching the container: writing the new credential changes the resolved secret env, which changes secret_env_hash, which the drift check reads as a container-label mismatch — so the platform's own recreate path rebuilds the container around unchanged data, ports, volumes and name."
- "Denylist-exact detection: rotate only on an exact match against known-compromised values, never on 'unrecognised'. An operator's deliberately-set credential is unrecognised too."
key-files:
created: []
modified:
- core/archipelago/src/container/secrets.rs
- core/archipelago/src/container/prod_orchestrator.rs
key-decisions:
- "Bcrypt generation was factored out of ensure_one's Bcrypt arm into write_bcrypt_pair(dir, name), which both ensure_one and rotation call. 01-11 had left that arm inline, and rotation cannot reuse ensure_gateway_credential directly because ensure_one's idempotent fast path returns early when the file is present and non-empty — which is exactly the case rotation must act on."
- "The rotation call is gated on `manifest.app.id == \"fedimint-gateway\"` rather than running for every app on every tick. It hangs off resolve_dynamic_env, immediately after ensure_generated_secrets, as the plan specified."
- "Errors propagate (`?`) rather than being logged-and-continued: write_secret's atomic temp-file-plus-rename leaves the previous credential intact on failure, so surfacing the error is strictly safer than proceeding with a half-rotated gateway."
- "No boot-specific wiring was added — see the boot-reconciler finding below."
requirements-completed: []
coverage:
- id: D1
description: "A node carrying the shipped default rotates itself onto a unique credential without operator action"
requirement: "FED-07"
verification:
- kind: unit
ref: "core/archipelago/src/container/secrets.rs#rotates_a_denylisted_gateway_credential"
status: pass
- kind: manual_procedural
ref: "Task 2 blocking checkpoint on archi-dev-box — NOT RUN"
status: deferred
human_judgment: true
- id: D2
description: "A node already carrying a unique credential is left completely alone; detection never fires on merely-unrecognised values"
requirement: "FED-07"
verification:
- kind: unit
ref: "…#leaves_a_unique_gateway_credential_alone, …#leaves_an_unrecognised_credential_alone"
status: pass
human_judgment: false
- id: D3
description: "Rotation runs at most once per affected node; later ticks detect nothing and change nothing"
requirement: "FED-07"
verification:
- kind: unit
ref: "…#rotation_is_idempotent"
status: pass
human_judgment: false
- id: D4
description: "Rotation replaces one credential pair and nothing else — no other secret, and no app data, is touched"
requirement: "FED-07"
verification:
- kind: unit
ref: "…#rotation_touches_no_other_secret (four bystander secrets asserted byte-identical)"
status: pass
- kind: other
ref: "git diff of prod_orchestrator.rs contains zero added rm -f / remove_dir_all / podman rm / chown"
status: pass
human_judgment: false
- id: D5
description: "The rotation is announced in the node's logs without ever printing the credential"
requirement: "FED-07"
verification:
- kind: other
ref: "The info! line interpolates self.secrets_dir and the secret NAME only; no value is in scope at the call site (rotate returns bool, not the credential)"
status: pass
- kind: manual_procedural
ref: "Task 2 step 3 — confirming no value appears in a real node's log — NOT RUN"
status: deferred
human_judgment: true
- id: D6
description: "Generation where no credential exists stays ensure_gateway_credential's job"
requirement: "FED-07"
verification:
- kind: unit
ref: "…#no_op_when_no_gateway_credential_exists"
status: pass
human_judgment: false
duration: 45min
completed: 2026-08-01
status: task-1-complete-checkpoint-pending
---
# Phase 1 Plan 16: Rotate Existing Installs Off the Shipped Gateway Credential (FED-07) Summary
**Task 1 is complete: an affected node now detects the published default on its next reconcile tick and rotates itself onto a unique credential, with the container rebuilt through the platform's own drift-recreate path rather than any hand-rolled teardown. Task 2 — the blocking on-node checkpoint — has NOT been run.**
## Status
**FED-07 is not yet closed.** This plan's requirement stays open until the Task 2 checkpoint runs on a
real node. The code half is done and verified by unit tests; the on-node half is untouched.
## Accomplishments
- `rotate_compromised_gateway_credential(secrets_dir) -> Result<bool>` in `container::secrets`:
reads the canonical hash file, returns `Ok(false)` for absent/unreadable/unique/unrecognised, and
only on an **exact** denylist match writes a fresh pair and returns `Ok(true)`.
- `write_bcrypt_pair(dir, name)` factored out of `ensure_one`'s `Bcrypt` arm so there is exactly one
bcrypt-generation implementation, called by both generation and rotation.
- Wired into `resolve_dynamic_env` beside `ensure_generated_secrets`, gated on the gateway's app id,
with an info-level announcement that names the *path* to the new plaintext and never the value.
- Six new tests covering rotate-on-denylisted (including 0600 modes and that the `.pw` sibling
verifies against the new hash), no-op-on-unique, no-op-on-unrecognised, no-op-on-absent,
idempotence, and four bystander secrets left byte-identical.
## Findings the plan asked for
### Boot reconciler needs no separate call
`boot_reconciler` calls `reconcile_all()` → `reconcile_all_with_mode()` → per-manifest
`ensure_running_with_mode()` (prod_orchestrator.rs:1714) → `resolve_dynamic_env()`
(prod_orchestrator.rs:1914) → the rotation call. `install_fresh` reaches it by the same route.
So boot and reconcile funnel through one chokepoint and **no boot-specific wiring was added**;
`boot_reconciler.rs` is not in `files_modified`.
### The recreate fires through `secret_env_hash` — mechanism confirmed by reading, not yet observed running
`resolve_dynamic_env` computes `secret_env_content_hash(&secret_bearing)` over the resolved
secret-bearing env and stores it as `manifest.app.container.secret_env_hash`
(prod_orchestrator.rs:3309). The drift check (prod_orchestrator.rs:3374) inspects the running
container's `SECRET_ENV_HASH_LABEL` and returns "drifted" when it differs from the expected hash,
which drives the existing recreate. The gateway's `FEDI_HASH` comes from the rotated file, so a
rotation necessarily changes that hash and therefore the label comparison.
**This is a code-reading conclusion. It has not been observed firing on a node** — that is Task 2
step 5, and it is the single most important thing the checkpoint proves.
### Operator recovery: the surface exists but does NOT cover this app — a real gap
- The UI path is live: `Apps.vue` calls `package.credentials` with an `app_id` before launching an
app and renders a credentials modal from the response.
- The backend, `handle_package_credentials` in
`core/archipelago/src/api/rpc/package/install.rs:2093`, is a hardcoded per-app if-chain covering
**only `filebrowser` and `photoprism`**. Every other app, including `fedimint-gateway`, falls
through to `Ok(json!({ "credentials": [] }))`.
- **Consequence:** after rotation the operator has no in-UI way to obtain the new gateway password.
The recovery path is the file the log line names: `/var/lib/archipelago/secrets/fedimint-gateway-hash.pw`
(0600, service user), readable over SSH.
- **Gap owner:** `handle_package_credentials` in `core/archipelago/src/api/rpc/package/install.rs`.
Adding a `fedimint-gateway` arm that reads the `.pw` sibling would close it; the UI needs no change.
Deliberately not done here — this plan's `files_modified` is scoped to two files, and that handler
belongs to the app-credentials surface, not to FED-07's rotation.
## Adjacent finding — NOT part of this plan, raised deliberately
`apps/photoprism/manifest.yml:35` sets `PHOTOPRISM_ADMIN_PASSWORD=archipelago`, and
`handle_package_credentials` hands that same literal back to the UI. That is a shipped default
credential in a manifest — the same class of defect as FED-07, on a different app. Every node running
PhotoPrism answers to `admin` / `archipelago`.
It is out of scope here (this plan is the gateway migration) and was not touched. It wants its own
requirement and plan, and probably the same treatment: a `generated_secrets` entry plus a denylist
entry for the shipped value.
## Deviations from Plan
### Bcrypt generation had to be factored out first
**Found during:** Task 1
**Issue:** The plan says rotation should "generate a replacement pair through the same helper
`ensure_gateway_credential` uses". 01-11 never actually created such a helper — it left the bcrypt
arm inline in `ensure_one` and had `ensure_gateway_credential` call `ensure_one`. Rotation cannot
call `ensure_gateway_credential`, because `ensure_one`'s idempotent fast path returns early when the
target files are present and non-empty, which is precisely the state rotation acts on.
**Resolution:** Extracted `write_bcrypt_pair(dir, name)` from the `Bcrypt` arm; `ensure_one` and
rotation both call it. Still exactly one generation implementation, which is what the instruction was
protecting.
**Files modified:** `core/archipelago/src/container/secrets.rs`
## Known Stubs
None.
## Threat Flags
- **T-01-72 (critical, EoP)** — mitigated in code, **not yet proven on a node**. Task 2 step 6 (old
credential rejected, new one accepted) is the proof and has not been run.
- **T-01-73 (DoS, rotation loop)** — mitigated and unit-tested: the rotated value is not on the
denylist, so the next tick is a no-op (`rotation_is_idempotent`).
- **T-01-74 (info disclosure)** — mitigated structurally: `rotate_compromised_gateway_credential`
returns `bool`, so the credential is not even in scope at the logging call site.
- **T-01-75 (tampering / data loss)** — mitigated: no teardown primitives added (grep-verified), the
recreate goes through `secret_env_hash`. On-node data-survival check is Task 2 step 5, not run.
- **T-01-76 (repudiation — signing off without exercising rotation)** — **live risk, unresolved.**
Whether archi-dev-box is affected or already clean is still unknown; the plan requires declaring
which case it is and deliberately seeding the old value if the node is clean.
- **T-01-77 (operator lockout)** — partially mitigated: the plaintext exists at a named 0600 path and
the log line points at it, but there is no UI retrieval path (see the gap above).
- **T-01-SC** — no crates added.
## Self-Check
- CONFIRMED: `cargo test -p archipelago secrets` → **16 passed, 0 failed** (the `container::secrets`
module holds 14 `#[test]` fns, all six new rotation cases among them:
`rotates_a_denylisted_gateway_credential`, `leaves_a_unique_gateway_credential_alone`,
`leaves_an_unrecognised_credential_alone`, `no_op_when_no_gateway_credential_exists`,
`rotation_is_idempotent`, `rotation_touches_no_other_secret`)
- FOUND: `rotate_compromised_gateway_credential` in `secrets.rs` (definition + 5 test uses)
- FOUND: exactly 1 non-comment reference in `prod_orchestrator.rs`
- CONFIRMED: 0 added teardown primitives (`rm -f` / `remove_dir_all` / `podman rm` / `chown`) in the
`prod_orchestrator.rs` diff
- CONFIRMED: `cargo fmt --check -p archipelago` clean. It was **not** clean before this plan —
`install.rs` carried drift introduced by 01-11's commit (`42652547`), fixed here. That check has
blocked the release gate before (`37d293be`), so it is worth keeping green rather than discovering
at ship time.
- CONFIRMED: `cargo test -p archipelago` (after `cargo clean -p archipelago`) — **1008 passed, 1
failed**. The failure is `container::boot_reconciler::tests::second_pass_fires_after_interval`, the
same wall-clock-timed test (50ms tick) that was flaky during 01-11; re-run in isolation it is
**4 passed / 0 failed in 0.46s**. `boot_reconciler.rs` is untouched by this plan.
- **NOT RUN:** Task 2's eight-step on-node checkpoint, and `tests/lifecycle/run-gate.sh`
### A false alarm worth recording, because it cost an hour
An intermediate full-suite run reported `credentials::operations::tests::test_list_credentials_filter_by_did`
failing with "invalid utf-8 sequence of 1 bytes from index 2" — an identity-credentials test in a
module this plan does not touch, which had passed in the 01-11 run two hours earlier.
Cause: **corrupted build artifacts, not a regression.** Two duplicate `cargo test` runs had been
started against the same workspace lock and one was `SIGTERM`ed to free it. The next compile surfaced
`rust-lld: error: undefined hidden symbol` — precisely the incremental-cache corruption CLAUDE.md
documents. After `cargo clean -p archipelago` the credentials test passes and the only failure is the
known timing flake above.
Lesson for the next executor on this box: do not kill an in-flight `cargo` to free the build lock —
let it finish. A corrupted target dir produces failures in modules you never touched, which reads
exactly like a real regression and is not one.
</content>