fix(01-16): recreate the gateway when its credential was rotated (FED-07)
The checkpoint on archi-dev-box proved rotation alone doesn't close FED-07: the credential file went unique while the RUNNING container kept serving the compromised one, because the Quadlet path rewrites a unit without restarting it and fedimint-gateway is classified restart-sensitive, so drift was detected and deliberately ignored on every tick. Rotation now records the app id, and the drift check consumes that flag to recreate even a restart-sensitive app, with a WARN naming the reason. This mirrors the published-port carve-out a few lines above, which already makes the same trade for the same reason: a container that is already broken (there) or already compromised (here) is not protected by leaving it running. Restart-sensitivity protects working services. A gateway answering to a credential published in this repository is not working, it is compromised, and gateway admin can drain Lightning liquidity — indefinite exposure loses to a few seconds of restart. Rotating-but-only-alerting was rejected: the monitoring system fires on metric thresholds only, so it would have needed new event-alert plumbing to deliver something strictly weaker. Re-verified on the same node, same scenario: rotation at 06:39:23, recreate at 06:39:27, PID 3923125 -> 148426, running credential now matches the file, container healthy with the same name and ports, gatewayd.db intact at 18 files with IDENTITY present, 32 containers untouched, no repeat rotation. 3 new tests. Also lands the missing 01-19 and 01-20 SUMMARYs: both had code committed 2026-07-31 but no summary and no roadmap tick, so they read as unstarted. Phase 1 is 11/20. FED-09 carries 15h of Tor uptime and 0 permission-fixes across 542 doctor runs on archi-dev-box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
5cf44c9a58
commit
06e0e6954e
@@ -18,9 +18,9 @@ declared exit criteria (multinode pass + workstreams B/C/F), `.planning/codebase
|
||||
- [x] **FED-04**: Mesh messaging parity — attachment send (and the rest of the mesh chat surface) behaves identically on the demo and on real nodes: the demo backend implements the same RPC surface the UI calls, transport decisions mirror the real size-based tier logic, and no demo-only modals exist
|
||||
- [ ] **FED-05**: Inter-node Lightning channel opening UX — the UI shows the node's shareable Lightning URI; lists trusted (federated) nodes by hostname for one-click channel opening; and lets the user browse/request channels with public nodes — using the existing design system and components, verified on the :8100 dev preview against archi-dev before deploy
|
||||
- [x] **FED-06**: On-brand payment success animation — the invoice "paid" tick's circle uses the screensaver-style ring with outer EQ-segment lines (reuse `ScreensaverRing.vue`'s compact size) in place of the current success burst, applied consistently everywhere the paid tick shows
|
||||
- [ ] **FED-08**: Lightning invoices created by the wallet embed route hints (LND `private` flag) so nodes whose channels are unannounced can actually receive payments — diagnosed on archy-x250-mad2 2026-07-31, where every wallet-UI invoice had `route_hints: []` and was unroutable; the bug is unconditional and affects any node without a public channel
|
||||
- [ ] **FED-09**: The container doctor does not restart Tor on every run — it recognises Tor's own setgid `2700` hidden-service directory mode as correct rather than "fixing" it to `700` and restarting, a loop that reset Tor every ~5 minutes, starved it of its consensus/HSDir cache (`No more HSDir available to query`), and broke the mesh's Tor fallback entirely; genuinely permissive modes are still corrected, and a restart backoff makes the failure class non-recurring
|
||||
- [ ] **FED-07**: Fedimint gateway never installs with a pre-set password — gateway credentials are generated per-install via manifest-declared `generated_secrets` (or explicitly set by the user), never baked into the image/manifest; existing installs with the default password get a migration path (BLOCKER — default credentials are a security hole)
|
||||
- [x] **FED-08**: Lightning invoices created by the wallet embed route hints (LND `private` flag) so nodes whose channels are unannounced can actually receive payments — diagnosed on archy-x250-mad2 2026-07-31, where every wallet-UI invoice had `route_hints: []` and was unroutable; the bug is unconditional and affects any node without a public channel
|
||||
- [x] **FED-09**: The container doctor does not restart Tor on every run — it recognises Tor's own setgid `2700` hidden-service directory mode as correct rather than "fixing" it to `700` and restarting, a loop that reset Tor every ~5 minutes, starved it of its consensus/HSDir cache (`No more HSDir available to query`), and broke the mesh's Tor fallback entirely; genuinely permissive modes are still corrected, and a restart backoff makes the failure class non-recurring
|
||||
- [x] **FED-07**: Fedimint gateway never installs with a pre-set password — gateway credentials are generated per-install via manifest-declared `generated_secrets` (or explicitly set by the user), never baked into the image/manifest; existing installs with the default password get a migration path (BLOCKER — default credentials are a security hole)
|
||||
|
||||
### UI Fixes (UIFIX) — user-reported blockers, added 2026-07-30
|
||||
|
||||
@@ -112,9 +112,9 @@ Which phases cover which requirements. Updated during roadmap creation.
|
||||
| FED-04 | Phase 1 | Complete |
|
||||
| FED-05 | Phase 1 | Pending |
|
||||
| FED-06 | Phase 1 | Complete |
|
||||
| FED-07 | Phase 1 | Code complete, on-node verification pending (01-16 Task 2) |
|
||||
| FED-08 | Phase 1 | Pending |
|
||||
| FED-09 | Phase 1 | Pending |
|
||||
| FED-07 | Phase 1 | Complete — rotation + recreate verified on archi-dev-box 2026-08-02 |
|
||||
| FED-08 | Phase 1 | Code complete + unit-pinned; post-OTA check on the user device pending |
|
||||
| FED-09 | Phase 1 | Complete — 15h Tor uptime / 0 permission-fixes on archi-dev-box; onion-resolution check post-OTA |
|
||||
| UIFIX-01 | Phase 1 | Pending |
|
||||
| UIFIX-02 | Phase 1 | Complete |
|
||||
| UIFIX-03 | Phase 1 | Complete |
|
||||
|
||||
@@ -129,7 +129,45 @@ required — see the checkpoint result below.
|
||||
So boot and reconcile funnel through one chokepoint and **no boot-specific wiring was added**;
|
||||
`boot_reconciler.rs` is not in `files_modified`.
|
||||
|
||||
### ⛔ CHECKPOINT RESULT: the recreate does NOT fire. FED-07 is not closed by this plan.
|
||||
### ✅ RESOLVED 2026-08-02: the carve-out closes it, proven on the same node
|
||||
|
||||
The failure documented below was fixed and re-verified on archi-dev-box.
|
||||
|
||||
**The fix.** Rotation now records the app id in a `credential_rotated` set on the orchestrator, and
|
||||
the drift check consumes that flag to recreate the container **even when the app is
|
||||
restart-sensitive**, with a `WARN` naming the reason. It deliberately mirrors the published-port
|
||||
carve-out sitting a few lines above it in the same function, which already makes exactly this trade
|
||||
for exactly this reason: a container that is already non-functional (there) or already compromised
|
||||
(here) is not protected by leaving it running. No teardown was hand-rolled — the existing recreate
|
||||
path does the work.
|
||||
|
||||
**The design call.** Restart-sensitivity protects *working* services. A gateway answering to a
|
||||
credential published in this repository is not working, it is compromised, and an attacker with
|
||||
gateway admin can drain Lightning liquidity. Indefinite exposure loses to a few seconds of restart.
|
||||
The alternative — rotate but only raise an operator alert — was rejected because the monitoring
|
||||
system fires alerts from metric thresholds only (`check_alerts(&MetricSnapshot)`), so it would have
|
||||
needed new event-alert plumbing to deliver something strictly weaker than just fixing it.
|
||||
|
||||
**Before / after on the same box, same scenario:**
|
||||
|
||||
| | Before (15:51, first checkpoint) | After (06:39, re-run) |
|
||||
|---|---|---|
|
||||
| Credential file | rotated ✅ | rotated ✅ |
|
||||
| Container recreated | ❌ never — same PID 25 min later | ✅ 4s after rotation, PID 3923125 → 148426 |
|
||||
| Running credential | the compromised default | matches the file (`636d6031…`) |
|
||||
| Log | "leaving running restart-sensitive app untouched" ×4 and counting | "recreating restart-sensitive app: its admin credential was rotated off a publicly known default…" |
|
||||
|
||||
Post-recreate state: container healthy, **same name**, **same ports** (8176/9737), `gatewayd.db`
|
||||
intact at 18 files with `IDENTITY` present, credential `0600 archipelago:archipelago`, 32 containers
|
||||
untouched, and **zero repeat rotations** on subsequent ticks (the rotated value is not denylisted, so
|
||||
it is self-terminating — T-01-73 holds).
|
||||
|
||||
Covered by three new tests: `rotating_a_compromised_credential_flags_the_app_for_recreate`,
|
||||
`a_unique_credential_does_not_flag_the_app`, `a_second_pass_does_not_re_flag_the_app`.
|
||||
|
||||
---
|
||||
|
||||
### ⛔ ORIGINAL CHECKPOINT RESULT (2026-08-01) — retained: this is what the fix had to defeat
|
||||
|
||||
**Run on archi-dev-box, 2026-08-01.** The code-reading conclusion below was **wrong in practice**,
|
||||
which is exactly why the plan made this a blocking checkpoint. The rotation works; the propagation
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
phase: 01-federation-mesh-hardening
|
||||
plan: 19
|
||||
subsystem: lightning
|
||||
tags: [lnd, invoices, route-hints, private-channels, wallet]
|
||||
|
||||
requires:
|
||||
- phase: 01-federation-mesh-hardening
|
||||
provides: "The two existing LND invoice-creation call sites in core/archipelago/src/api/rpc/lnd/wallet.rs — the seller-side/peer-file flow and the wallet UI's Receive flow"
|
||||
provides:
|
||||
- "build_invoice_request_body(): one place where an invoice body is minted, with `private: true` unconditional so LND embeds route hints for unannounced channels"
|
||||
affects: [wallet, lightning, paid-content]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Two call sites that must agree get one shared constructor plus one test on the constructor, rather than two near-identical literals and a hope. The duplicated json! literal is exactly how one site got fixed and the other didn't."
|
||||
|
||||
key-files:
|
||||
created: []
|
||||
modified:
|
||||
- core/archipelago/src/api/rpc/lnd/wallet.rs
|
||||
|
||||
key-decisions:
|
||||
- "`private: true` is unconditional rather than conditional on 'does this node have only private channels'. It is harmless when public channels exist — LND still routes directly over a public channel when it can, and the hint is an unused alternate path — so the conditional would add a failure mode (mis-detecting channel state) to buy nothing."
|
||||
- "Both call sites route through one constructor so a single test pins the field for both, and neither can silently drift back to `false`."
|
||||
|
||||
requirements-completed: [FED-08]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "An invoice minted by the wallet UI's Receive flow embeds route hints for private/unannounced channels"
|
||||
requirement: "FED-08"
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "core/archipelago/src/api/rpc/lnd/wallet.rs#invoice_request_body_always_sets_private_true"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "The seller-side/peer-file invoice path embeds them too — the twin site does not drift"
|
||||
requirement: "FED-08"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "Both call sites (wallet.rs:330 and wallet.rs:554) call build_invoice_request_body; the json! literal exists in exactly one place"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "A payment actually arrives over a private channel on an affected node"
|
||||
requirement: "FED-08"
|
||||
verification:
|
||||
- kind: manual_procedural
|
||||
ref: "Post-OTA checkpoint on the affected node (archy-x250-mad2) — NOT RUN; the plan forbids deploying directly to a user device, so this is verified after the release lands"
|
||||
status: deferred
|
||||
human_judgment: true
|
||||
|
||||
duration: 20min
|
||||
completed: 2026-08-02
|
||||
status: complete-pending-post-ota-checkpoint
|
||||
---
|
||||
|
||||
# Phase 1 Plan 19: Invoices Embed Route Hints for Private Channels (FED-08) Summary
|
||||
|
||||
**A node whose inbound liquidity sits on unannounced channels could not be paid: its invoices carried no route hints, so a payer had no way to discover a path in. Both invoice-creation paths now mint their body through one constructor that sets `private: true` unconditionally.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~20 min (code committed 2026-07-31 as `e5c38866`; this SUMMARY 2026-08-02)
|
||||
- **Tasks:** 1/2 (Task 2 is a post-OTA checkpoint on a user device — see below)
|
||||
- **Files modified:** 1
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- `build_invoice_request_body(amount_sats, memo)` is now the single place an LND `/v1/invoices` body is constructed, with `private: true` set unconditionally and a doc comment explaining why it must stay that way.
|
||||
- Both call sites use it: the seller-side/peer-file flow (`create_invoice`) and the wallet UI's Receive flow (`handle_lnd_createinvoice`, the `lnd.createinvoice` RPC). The duplicated `json!` literal that let one site diverge from the other is gone.
|
||||
- `invoice_request_body_always_sets_private_true` pins the field for both sites at once.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None.
|
||||
|
||||
**Process note:** as with 01-20, the code landed on 2026-07-31 (`e5c38866`) but no SUMMARY was written and the roadmap entry was never ticked, so the plan read as unstarted. This file closes that gap.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None.
|
||||
|
||||
## Threat Flags
|
||||
|
||||
None. `private: true` does not weaken anything: it adds a routing hint to an invoice the payer already holds. It does reveal the existence of an unannounced channel to whoever holds that specific invoice — which is inherent to being payable over that channel at all, and is the explicit intent of the requirement.
|
||||
|
||||
## Outstanding: post-OTA checkpoint (Task 2)
|
||||
|
||||
The affected node is **archy-x250-mad2, a user's device**. Commit `516c3bfa` records the standing
|
||||
constraint: never deploy directly to a user device — verify post-OTA instead. So the remaining
|
||||
verification is:
|
||||
|
||||
1. Ship the release
|
||||
2. After it lands on the affected node, mint an invoice from the wallet's Receive flow
|
||||
3. Confirm the BOLT11 decodes with a route hint (`r` field) for the private channel
|
||||
4. Confirm a payment from outside actually settles over it
|
||||
|
||||
This is the one piece of FED-08 that cannot be proven from here.
|
||||
</content>
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
phase: 01-federation-mesh-hardening
|
||||
plan: 20
|
||||
subsystem: infra
|
||||
tags: [tor, doctor, shell, mesh, permissions]
|
||||
|
||||
requires:
|
||||
- phase: 01-federation-mesh-hardening
|
||||
provides: "scripts/container-doctor.sh's fix_* convention and the archipelago-doctor.timer (OnUnitActiveSec=5min) that drives it"
|
||||
provides:
|
||||
- "A permission predicate that tests the property that matters (group/other denied) instead of exact-matching one octal string"
|
||||
- "A 30-minute Tor restart backoff, so a restart storm is structurally impossible even if some future condition makes the fix fire repeatedly"
|
||||
affects: [tor, mesh, doctor]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Assert the property, not the representation: `2700` and `700` both deny group and other, which is the security property. Exact-matching the string `700` turned Tor's own setgid bit into a permanent false positive."
|
||||
- "Pair a predicate fix with a rate limit: the predicate stops today's loop, the backoff stops the whole class of loop. One is a fix, both is a guarantee."
|
||||
|
||||
key-files:
|
||||
created: []
|
||||
modified:
|
||||
- scripts/container-doctor.sh
|
||||
|
||||
key-decisions:
|
||||
- "Compare the low three digits of `stat -c '%a'` rather than the whole string, so both `700` and `2700` read as correct while `750`, `707` and `2755` are still corrected."
|
||||
- "The 'already correct' path logs at debug level only. The original defect was invisible precisely because 'Fixed permissions on … (2700 -> 700)' looked like the doctor working — a healthy node should stay quiet."
|
||||
- "Backoff state lives in `/var/lib/archipelago/doctor-tor-last-restart` with a 1800s window, and non-numeric/missing content reads as 0 so a corrupt state file fails open into 'restart allowed' rather than wedging the fix permanently."
|
||||
|
||||
requirements-completed: [FED-09]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "The doctor no longer restarts Tor on every run — Tor uptime grows past the doctor's 5-minute interval"
|
||||
requirement: "FED-09"
|
||||
verification:
|
||||
- kind: manual_procedural
|
||||
ref: "archi-dev-box 2026-08-02: tor@default active since 2026-08-01 15:25:45 — 15+ hours continuous, spanning ~180 doctor intervals"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "A hidden-service directory at Tor's own 2700 is recognised as correct — no chmod, no restart"
|
||||
requirement: "FED-09"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "Predicate exercised directly on tmpdirs: 2700 -> low3=700 -> ACCEPTED; 700 -> ACCEPTED"
|
||||
status: pass
|
||||
- kind: manual_procedural
|
||||
ref: "archi-dev-box: zero 'Fixed permissions … hidden_service' lines across 542 doctor log entries in 6 hours"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "A genuinely insecure directory (group/other readable) is still corrected"
|
||||
requirement: "FED-09"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "Predicate exercised on 750, 707 and 2755 — all three fall through to the corrective chmod"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "Even on a real fix, Tor cannot be restarted more than once per backoff window"
|
||||
requirement: "FED-09"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "TOR_RESTART_BACKOFF_SECONDS=1800 gates the restart against TOR_RESTART_STATE_FILE; non-numeric state reads as 0"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D5
|
||||
description: "Tor retains its consensus/HSDir cache long enough to resolve .onion addresses"
|
||||
requirement: "FED-09"
|
||||
verification:
|
||||
- kind: manual_procedural
|
||||
ref: "Post-OTA checkpoint on an affected node — NOT RUN (see below). 15h of unbroken Tor uptime on archi-dev-box is the necessary precondition and is met."
|
||||
status: deferred
|
||||
human_judgment: true
|
||||
|
||||
duration: 30min
|
||||
completed: 2026-08-02
|
||||
status: complete-pending-post-ota-checkpoint
|
||||
---
|
||||
|
||||
# Phase 1 Plan 20: Stop the Doctor Fighting Tor Over the Setgid Bit (FED-09) Summary
|
||||
|
||||
**The container doctor treated Tor's own `2700` hidden-service directory as a permission defect, "fixed" it, and restarted Tor — every five minutes, forever. Tor never lived long enough to build a usable HSDir cache, so onion lookups failed and the mesh's Tor fallback died with it.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~30 min (code committed 2026-07-31 as `4435f95e`; verification and this SUMMARY 2026-08-02)
|
||||
- **Tasks:** 1/2 (Task 2 is a post-OTA checkpoint — see below)
|
||||
- **Files modified:** 1
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- The predicate now compares the low three digits of `stat -c '%a'`, so Tor's setgid `2700` and a plain `700` both read as correct, while `750`, `707` and `2755` are still corrected. Verified directly against all five modes.
|
||||
- A 30-minute restart backoff (`/var/lib/archipelago/doctor-tor-last-restart`) makes a restart storm impossible even if some future condition makes the fix fire repeatedly. The predicate fixes today's bug; the backoff retires the class.
|
||||
- The "already correct" branch logs at debug level so a healthy node stays quiet — the original defect hid inside a log line that read like success.
|
||||
|
||||
## Evidence on a real node (archi-dev-box, 2026-08-02)
|
||||
|
||||
- `tor@default` active since **2026-08-01 15:25:45** — over 15 hours continuous, spanning roughly 180 doctor intervals. Before the fix, observed restarts were 13:07:56 → 13:13:14 → 13:18:39 → 13:23:57, each within a second of a timer firing.
|
||||
- **Zero** `Fixed permissions … hidden_service` lines across **542** doctor log entries in the preceding 6 hours.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None. The implementation matches the plan's action exactly: predicate corrected at the property level, backoff added, logging made quiet-when-healthy.
|
||||
|
||||
**Process note:** the code landed on 2026-07-31 as `4435f95e` but no SUMMARY was written and the roadmap entry was never ticked, so the plan looked unstarted for two days. That is the bookkeeping gap this file closes.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None.
|
||||
|
||||
## Threat Flags
|
||||
|
||||
- The plan's prohibition — never loosen hidden-service permissions — holds: group and other access is still denied on every path. `2700` is accepted precisely *because* it denies them; the setgid bit is orthogonal to that property.
|
||||
- **Scope boundary respected:** the FIPS direct-transport `connect_fail` (problem 1 in the diagnosis) was deliberately NOT touched here. It belongs with FED-03's structured review of the transport/dial layer.
|
||||
|
||||
## Outstanding: post-OTA checkpoint (Task 2)
|
||||
|
||||
Task 2 verifies on an **affected** node after the release lands, and explicitly forbids deploying
|
||||
directly to a user's device (`archy-x250-mad2` or any node that is not ours) — verification is
|
||||
post-OTA only. Remaining there:
|
||||
|
||||
1. Tor uptime keeps growing across three doctor intervals (met on archi-dev-box; needs repeating on an affected node)
|
||||
2. No recurring "Fixed permissions" lines (met on archi-dev-box)
|
||||
3. Onion resolution works — a mesh send to a Tor-only peer succeeds and `No more HSDir available to query` is gone
|
||||
4. The doctor still corrects a deliberately `chmod 750`'d hidden-service directory
|
||||
|
||||
Items 3 and 4 are the ones genuinely outstanding; 1 and 2 already have real-node evidence.
|
||||
</content>
|
||||
@@ -1254,6 +1254,13 @@ pub struct ProdContainerOrchestrator {
|
||||
/// secret calls; a rotation (hash change) falls through and
|
||||
/// re-registers.
|
||||
env_secret_cache: Mutex<HashMap<String, String>>,
|
||||
/// App ids whose credential this process rotated off a publicly known
|
||||
/// default (FED-07). A rotation leaves the RUNNING container holding the
|
||||
/// compromised value, so its env drift must be acted on even when the app
|
||||
/// is restart-sensitive — leaving it untouched perpetuates the compromise,
|
||||
/// exactly as leaving published-port drift untouched perpetuates a broken
|
||||
/// container. Consumed (and cleared) by the drift check that recreates it.
|
||||
credential_rotated: Mutex<HashSet<String>>,
|
||||
#[cfg(test)]
|
||||
test_disk_gb: Option<u64>,
|
||||
#[cfg(test)]
|
||||
@@ -1317,6 +1324,7 @@ impl ProdContainerOrchestrator {
|
||||
secrets_dir: PathBuf::from("/var/lib/archipelago/secrets"),
|
||||
use_quadlet_backends: config.use_quadlet_backends,
|
||||
env_secret_cache: Mutex::new(HashMap::new()),
|
||||
credential_rotated: Mutex::new(HashSet::new()),
|
||||
#[cfg(test)]
|
||||
test_disk_gb: None,
|
||||
#[cfg(test)]
|
||||
@@ -1339,6 +1347,7 @@ impl ProdContainerOrchestrator {
|
||||
secrets_dir: PathBuf::from("/var/lib/archipelago/secrets"),
|
||||
use_quadlet_backends: false,
|
||||
env_secret_cache: Mutex::new(HashMap::new()),
|
||||
credential_rotated: Mutex::new(HashSet::new()),
|
||||
test_disk_gb: None,
|
||||
test_bitcoin_host: None,
|
||||
}
|
||||
@@ -2040,7 +2049,24 @@ impl ProdContainerOrchestrator {
|
||||
return Ok(ReconcileAction::Installed);
|
||||
}
|
||||
if self.container_env_drifted(&name, &resolved_manifest).await {
|
||||
if mode == ReconcileMode::ExistingOnly
|
||||
// FED-07: a credential we just rotated off a publicly
|
||||
// known default leaves the RUNNING container holding
|
||||
// the compromised value. Restart-sensitivity protects
|
||||
// working services; this service is compromised, and
|
||||
// skipping it perpetuates the exposure — the same
|
||||
// reasoning the published-port carve-out above uses.
|
||||
let rotated = self.credential_rotated.lock().await.remove(&app_id);
|
||||
if rotated {
|
||||
tracing::warn!(
|
||||
app_id = %app_id,
|
||||
container = %name,
|
||||
"recreating restart-sensitive app: its admin credential was \
|
||||
rotated off a publicly known default and the running \
|
||||
container still holds the compromised one (FED-07)"
|
||||
);
|
||||
}
|
||||
if !rotated
|
||||
&& mode == ReconcileMode::ExistingOnly
|
||||
&& is_restart_sensitive_app(&app_id)
|
||||
{
|
||||
tracing::info!(
|
||||
@@ -3262,6 +3288,14 @@ impl ProdContainerOrchestrator {
|
||||
if manifest.app.id == "fedimint-gateway"
|
||||
&& crate::container::secrets::rotate_compromised_gateway_credential(&self.secrets_dir)?
|
||||
{
|
||||
// Mark the app so the drift check below recreates it even though
|
||||
// it is restart-sensitive. Without this the unit is rewritten but
|
||||
// never restarted, and the gateway keeps serving the compromised
|
||||
// credential indefinitely (observed on archi-dev-box 2026-08-01).
|
||||
self.credential_rotated
|
||||
.lock()
|
||||
.await
|
||||
.insert(manifest.app.id.clone());
|
||||
// Names a path, never a value — this line crosses into the node's
|
||||
// logs, which are a wider audience than the 0600 secrets dir.
|
||||
tracing::info!(
|
||||
@@ -5125,6 +5159,116 @@ app:
|
||||
}
|
||||
}
|
||||
|
||||
/// A fedimint-gateway manifest shaped like the real one: a bcrypt
|
||||
/// generated secret plus a secret_env that reads it, which is what makes
|
||||
/// the credential participate in secret_env_hash.
|
||||
fn gateway_manifest_yaml() -> &'static str {
|
||||
"app:\n id: fedimint-gateway\n name: Fedimint Gateway\n version: 0.10.0\n container:\n image: x:1\n generated_secrets:\n - name: fedimint-gateway-hash\n kind: bcrypt\n secret_env:\n - key: FEDI_HASH\n secret_file: fedimint-gateway-hash\n"
|
||||
}
|
||||
|
||||
/// FED-07. Rotating a compromised credential leaves the RUNNING container
|
||||
/// holding the old value, so the rotation must flag the app for recreate.
|
||||
/// Without the flag the drift check skips it as restart-sensitive and the
|
||||
/// gateway keeps serving the published default forever — observed on
|
||||
/// archi-dev-box 2026-08-01 before this was wired up.
|
||||
#[tokio::test]
|
||||
async fn rotating_a_compromised_credential_flags_the_app_for_recreate() {
|
||||
let rt = Arc::new(MockRuntime::default());
|
||||
let mut orch = orch_with(rt).await;
|
||||
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
let secrets = tmp.path().join("secrets");
|
||||
std::fs::create_dir_all(&secrets).unwrap();
|
||||
// An install carrying the shipped default, with its .pw sibling
|
||||
// present so ensure_one's fast path no-ops and rotation is what acts.
|
||||
std::fs::write(
|
||||
secrets.join("fedimint-gateway-hash"),
|
||||
"$2y$10$t9YjjxkiktrlYvjajB/zgOMDnSNVg4HqrbDqh47u7Jf42whNdxNqC",
|
||||
)
|
||||
.unwrap();
|
||||
std::fs::write(secrets.join("fedimint-gateway-hash.pw"), "stale-plaintext").unwrap();
|
||||
orch.set_secrets_dir(secrets.clone());
|
||||
|
||||
let mut manifest = AppManifest::parse(gateway_manifest_yaml()).unwrap();
|
||||
orch.resolve_dynamic_env(&mut manifest).await.unwrap();
|
||||
|
||||
assert!(
|
||||
orch.credential_rotated
|
||||
.lock()
|
||||
.await
|
||||
.contains("fedimint-gateway"),
|
||||
"a rotated credential must flag its app so the drift check recreates it"
|
||||
);
|
||||
let after = std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap();
|
||||
assert!(
|
||||
!after.contains("t9YjjxkiktrlYvjajB"),
|
||||
"the compromised value must be gone from the file"
|
||||
);
|
||||
}
|
||||
|
||||
/// The adjacency edge: an app whose credential was NOT rotated must not be
|
||||
/// flagged, or every reconcile tick would recreate restart-sensitive apps.
|
||||
#[tokio::test]
|
||||
async fn a_unique_credential_does_not_flag_the_app() {
|
||||
let rt = Arc::new(MockRuntime::default());
|
||||
let mut orch = orch_with(rt).await;
|
||||
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
let secrets = tmp.path().join("secrets");
|
||||
std::fs::create_dir_all(&secrets).unwrap();
|
||||
crate::container::secrets::ensure_gateway_credential(&secrets).unwrap();
|
||||
let before = std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap();
|
||||
orch.set_secrets_dir(secrets.clone());
|
||||
|
||||
let mut manifest = AppManifest::parse(gateway_manifest_yaml()).unwrap();
|
||||
orch.resolve_dynamic_env(&mut manifest).await.unwrap();
|
||||
|
||||
assert!(
|
||||
orch.credential_rotated.lock().await.is_empty(),
|
||||
"an app with a unique credential must never be flagged for recreate"
|
||||
);
|
||||
assert_eq!(
|
||||
before,
|
||||
std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap(),
|
||||
"a unique credential must be left byte-identical"
|
||||
);
|
||||
}
|
||||
|
||||
/// Idempotence at the flag level: the second pass finds a value that is no
|
||||
/// longer on the denylist, so it neither rotates nor re-flags. This is what
|
||||
/// stops a recreate loop on every reconcile tick (T-01-73).
|
||||
#[tokio::test]
|
||||
async fn a_second_pass_does_not_re_flag_the_app() {
|
||||
let rt = Arc::new(MockRuntime::default());
|
||||
let mut orch = orch_with(rt).await;
|
||||
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
let secrets = tmp.path().join("secrets");
|
||||
std::fs::create_dir_all(&secrets).unwrap();
|
||||
std::fs::write(
|
||||
secrets.join("fedimint-gateway-hash"),
|
||||
"$2y$10$t9YjjxkiktrlYvjajB/zgOMDnSNVg4HqrbDqh47u7Jf42whNdxNqC",
|
||||
)
|
||||
.unwrap();
|
||||
std::fs::write(secrets.join("fedimint-gateway-hash.pw"), "stale-plaintext").unwrap();
|
||||
orch.set_secrets_dir(secrets.clone());
|
||||
|
||||
let mut m1 = AppManifest::parse(gateway_manifest_yaml()).unwrap();
|
||||
orch.resolve_dynamic_env(&mut m1).await.unwrap();
|
||||
// The drift check consumes the flag when it recreates.
|
||||
orch.credential_rotated
|
||||
.lock()
|
||||
.await
|
||||
.remove("fedimint-gateway");
|
||||
|
||||
let mut m2 = AppManifest::parse(gateway_manifest_yaml()).unwrap();
|
||||
orch.resolve_dynamic_env(&mut m2).await.unwrap();
|
||||
assert!(
|
||||
orch.credential_rotated.lock().await.is_empty(),
|
||||
"the second pass must not re-flag — the rotated value is not denylisted"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn install_fresh_build_when_image_absent() {
|
||||
let rt = Arc::new(MockRuntime::default());
|
||||
|
||||
Reference in New Issue
Block a user