lfg2025/archy
1
0
Fork 0
Code Issues 27 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

fix(01-16): recreate the gateway when its credential was rotated (FED-07)

Browse Source 
The checkpoint on archi-dev-box proved rotation alone doesn't close FED-07:
the credential file went unique while the RUNNING container kept serving the
compromised one, because the Quadlet path rewrites a unit without restarting
it and fedimint-gateway is classified restart-sensitive, so drift was detected
and deliberately ignored on every tick.

Rotation now records the app id, and the drift check consumes that flag to
recreate even a restart-sensitive app, with a WARN naming the reason. This
mirrors the published-port carve-out a few lines above, which already makes
the same trade for the same reason: a container that is already broken (there)
or already compromised (here) is not protected by leaving it running.

Restart-sensitivity protects working services. A gateway answering to a
credential published in this repository is not working, it is compromised, and
gateway admin can drain Lightning liquidity — indefinite exposure loses to a
few seconds of restart. Rotating-but-only-alerting was rejected: the
monitoring system fires on metric thresholds only, so it would have needed new
event-alert plumbing to deliver something strictly weaker.

Re-verified on the same node, same scenario: rotation at 06:39:23, recreate at
06:39:27, PID 3923125 -> 148426, running credential now matches the file,
container healthy with the same name and ports, gatewayd.db intact at 18 files
with IDENTITY present, 32 containers untouched, no repeat rotation.

3 new tests. Also lands the missing 01-19 and 01-20 SUMMARYs: both had code
committed 2026-07-31 but no summary and no roadmap tick, so they read as
unstarted. Phase 1 is 11/20. FED-09 carries 15h of Tor uptime and 0
permission-fixes across 542 doctor runs on archi-dev-box.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-02 07:20:20 -04:00
co-authored by Claude Opus 5
parent 5cf44c9a58
commit 06e0e6954e
5 changed files with 424 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/container/prod_orchestrator.rs
+145 -1
View File
@@ -1254,6 +1254,13 @@ pub struct ProdContainerOrchestrator {
/// secret calls; a rotation (hash change) falls through and
/// re-registers.
env_secret_cache: Mutex<HashMap<String, String>>,
/// App ids whose credential this process rotated off a publicly known
/// default (FED-07). A rotation leaves the RUNNING container holding the
/// compromised value, so its env drift must be acted on even when the app
/// is restart-sensitive — leaving it untouched perpetuates the compromise,
/// exactly as leaving published-port drift untouched perpetuates a broken
/// container. Consumed (and cleared) by the drift check that recreates it.
credential_rotated: Mutex<HashSet<String>>,
#[cfg(test)]
test_disk_gb: Option<u64>,
#[cfg(test)]
@@ -1317,6 +1324,7 @@ impl ProdContainerOrchestrator {
secrets_dir: PathBuf::from("/var/lib/archipelago/secrets"),
use_quadlet_backends: config.use_quadlet_backends,
env_secret_cache: Mutex::new(HashMap::new()),
credential_rotated: Mutex::new(HashSet::new()),
#[cfg(test)]
test_disk_gb: None,
#[cfg(test)]
@@ -1339,6 +1347,7 @@ impl ProdContainerOrchestrator {
secrets_dir: PathBuf::from("/var/lib/archipelago/secrets"),
use_quadlet_backends: false,
env_secret_cache: Mutex::new(HashMap::new()),
credential_rotated: Mutex::new(HashSet::new()),
test_disk_gb: None,
test_bitcoin_host: None,
}
@@ -2040,7 +2049,24 @@ impl ProdContainerOrchestrator {
return Ok(ReconcileAction::Installed);
}
if self.container_env_drifted(&name, &resolved_manifest).await {
if mode == ReconcileMode::ExistingOnly
// FED-07: a credential we just rotated off a publicly
// known default leaves the RUNNING container holding
// the compromised value. Restart-sensitivity protects
// working services; this service is compromised, and
// skipping it perpetuates the exposure — the same
// reasoning the published-port carve-out above uses.
let rotated = self.credential_rotated.lock().await.remove(&app_id);
if rotated {
tracing::warn!(
app_id = %app_id,
container = %name,
"recreating restart-sensitive app: its admin credential was \
rotated off a publicly known default and the running \
container still holds the compromised one (FED-07)"
);
}
if !rotated
&& mode == ReconcileMode::ExistingOnly
&& is_restart_sensitive_app(&app_id)
{
tracing::info!(
@@ -3262,6 +3288,14 @@ impl ProdContainerOrchestrator {
if manifest.app.id == "fedimint-gateway"
&& crate::container::secrets::rotate_compromised_gateway_credential(&self.secrets_dir)?
{
// Mark the app so the drift check below recreates it even though
// it is restart-sensitive. Without this the unit is rewritten but
// never restarted, and the gateway keeps serving the compromised
// credential indefinitely (observed on archi-dev-box 2026-08-01).
self.credential_rotated
.lock()
.await
.insert(manifest.app.id.clone());
// Names a path, never a value — this line crosses into the node's
// logs, which are a wider audience than the 0600 secrets dir.
tracing::info!(
@@ -5125,6 +5159,116 @@ app:
}
}
/// A fedimint-gateway manifest shaped like the real one: a bcrypt
/// generated secret plus a secret_env that reads it, which is what makes
/// the credential participate in secret_env_hash.
fn gateway_manifest_yaml() -> &'static str {
"app:\n id: fedimint-gateway\n name: Fedimint Gateway\n version: 0.10.0\n container:\n image: x:1\n generated_secrets:\n - name: fedimint-gateway-hash\n kind: bcrypt\n secret_env:\n - key: FEDI_HASH\n secret_file: fedimint-gateway-hash\n"
}
/// FED-07. Rotating a compromised credential leaves the RUNNING container
/// holding the old value, so the rotation must flag the app for recreate.
/// Without the flag the drift check skips it as restart-sensitive and the
/// gateway keeps serving the published default forever — observed on
/// archi-dev-box 2026-08-01 before this was wired up.
#[tokio::test]
async fn rotating_a_compromised_credential_flags_the_app_for_recreate() {
let rt = Arc::new(MockRuntime::default());
let mut orch = orch_with(rt).await;
let tmp = tempfile::TempDir::new().unwrap();
let secrets = tmp.path().join("secrets");
std::fs::create_dir_all(&secrets).unwrap();
// An install carrying the shipped default, with its .pw sibling
// present so ensure_one's fast path no-ops and rotation is what acts.
std::fs::write(
secrets.join("fedimint-gateway-hash"),
"$2y$10$t9YjjxkiktrlYvjajB/zgOMDnSNVg4HqrbDqh47u7Jf42whNdxNqC",
)
.unwrap();
std::fs::write(secrets.join("fedimint-gateway-hash.pw"), "stale-plaintext").unwrap();
orch.set_secrets_dir(secrets.clone());
let mut manifest = AppManifest::parse(gateway_manifest_yaml()).unwrap();
orch.resolve_dynamic_env(&mut manifest).await.unwrap();
assert!(
orch.credential_rotated
.lock()
.await
.contains("fedimint-gateway"),
"a rotated credential must flag its app so the drift check recreates it"
);
let after = std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap();
assert!(
!after.contains("t9YjjxkiktrlYvjajB"),
"the compromised value must be gone from the file"
);
}
/// The adjacency edge: an app whose credential was NOT rotated must not be
/// flagged, or every reconcile tick would recreate restart-sensitive apps.
#[tokio::test]
async fn a_unique_credential_does_not_flag_the_app() {
let rt = Arc::new(MockRuntime::default());
let mut orch = orch_with(rt).await;
let tmp = tempfile::TempDir::new().unwrap();
let secrets = tmp.path().join("secrets");
std::fs::create_dir_all(&secrets).unwrap();
crate::container::secrets::ensure_gateway_credential(&secrets).unwrap();
let before = std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap();
orch.set_secrets_dir(secrets.clone());
let mut manifest = AppManifest::parse(gateway_manifest_yaml()).unwrap();
orch.resolve_dynamic_env(&mut manifest).await.unwrap();
assert!(
orch.credential_rotated.lock().await.is_empty(),
"an app with a unique credential must never be flagged for recreate"
);
assert_eq!(
before,
std::fs::read_to_string(secrets.join("fedimint-gateway-hash")).unwrap(),
"a unique credential must be left byte-identical"
);
}
/// Idempotence at the flag level: the second pass finds a value that is no
/// longer on the denylist, so it neither rotates nor re-flags. This is what
/// stops a recreate loop on every reconcile tick (T-01-73).
#[tokio::test]
async fn a_second_pass_does_not_re_flag_the_app() {
let rt = Arc::new(MockRuntime::default());
let mut orch = orch_with(rt).await;
let tmp = tempfile::TempDir::new().unwrap();
let secrets = tmp.path().join("secrets");
std::fs::create_dir_all(&secrets).unwrap();
std::fs::write(
secrets.join("fedimint-gateway-hash"),
"$2y$10$t9YjjxkiktrlYvjajB/zgOMDnSNVg4HqrbDqh47u7Jf42whNdxNqC",
)
.unwrap();
std::fs::write(secrets.join("fedimint-gateway-hash.pw"), "stale-plaintext").unwrap();
orch.set_secrets_dir(secrets.clone());
let mut m1 = AppManifest::parse(gateway_manifest_yaml()).unwrap();
orch.resolve_dynamic_env(&mut m1).await.unwrap();
// The drift check consumes the flag when it recreates.
orch.credential_rotated
.lock()
.await
.remove("fedimint-gateway");
let mut m2 = AppManifest::parse(gateway_manifest_yaml()).unwrap();
orch.resolve_dynamic_env(&mut m2).await.unwrap();
assert!(
orch.credential_rotated.lock().await.is_empty(),
"the second pass must not re-flag — the rotated value is not denylisted"
);
}
#[tokio::test]
async fn install_fresh_build_when_image_absent() {
let rt = Arc::new(MockRuntime::default());