Demo images / Build & push demo images (push) Successful in 3m34s
Rotating LND's macaroons was an SSH-only script, which in practice meant it did not happen — while a macaroon is a bearer token with no revocation and no expiry, so anything that ever read one keeps the ability to spend until they are replaced. Settings → Lightning credentials now does it behind the node password, shows a step checklist, and refuses to report success unless it has confirmed the node identity and channel census are unchanged. Three findings from performing a real rotation on a dev node, each fixed here: 1. BTCPay was left holding a dead credential, silently. Its connection string carries the macaroon INLINE (LND's datadir is owned by its container subuid, so btcpay cannot bind-mount the file), and the daemon only regenerates that secret when LND's TLS cert thumbprint changes — which macaroon rotation does not touch. Result: btcpay up, LND up, both healthy, every Lightning payment failing, nothing anywhere saying why. 2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the change visible as env drift, but the reconcile loop runs `ExistingOnly` at boot AND periodically, and there it deliberately leaves running restart-sensitive apps untouched — observed once per tick for half an hour on the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same reasoning: restart sensitivity protects apps that are working, and this one is working only in appearance. The shell script cannot reach an in-process flag, so it removes the container and lets desired-state recovery rebuild it. 3. LND stayed locked forever on a loaded node. The unlocker is only served after channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30 containers; the unlock helper gave up at ~60s. That is not a harmless retry — reconcile records the post-start hook as failed, restarts LND, and the slow open begins again, so the wallet never opens and every LND-dependent app stays broken. The not-ready budget is now ~10 minutes; a genuinely wrong password still exits on the first pass via `all_rejected`. Safety properties worth not regressing: - No macaroon content in any response, error, log line or the polled progress feed — digests and byte counts only. - Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no code path from "rotate my credentials" to `recreate_wallet_destructively`. A wallet whose password this node lacks fails the rotation with the wallet intact. - Channels are compared as active+inactive totals, not `num_active_channels`, which legitimately dips after any restart while peers reconnect. - Backup verified by file count before anything is deleted. Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing container::lnd tests pass, vue-tsc clean, and the built bundle contains the three new RPC method names (the frontend build can silently no-op). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
94 lines
4.5 KiB
Rust
94 lines
4.5 KiB
Rust
//! Orchestrator trait — the shared surface the RPC layer talks to.
|
|
//!
|
|
//! Step 4 of the rust-orchestrator migration. Unifies the container lifecycle
|
|
//! surface of `DevContainerOrchestrator` and `ProdContainerOrchestrator` so
|
|
//! `RpcHandler` can hold `Arc<dyn ContainerOrchestrator>` and stop caring
|
|
//! which mode it is in.
|
|
//!
|
|
//! The trait takes `app_id: &str` everywhere (never a manifest path). Dev and
|
|
//! Prod both resolve app_id → manifest internally. The legacy
|
|
//! `container-install { manifest_path }` RPC shape is preserved as a concrete
|
|
//! `install_container_from_path` method on `DevContainerOrchestrator` only,
|
|
//! since that ad-hoc workflow is a dev convenience and has no prod meaning.
|
|
//!
|
|
//! See `docs/rust-orchestrator-migration.md`.
|
|
|
|
use anyhow::Result;
|
|
use archipelago_container::ContainerStatus;
|
|
use async_trait::async_trait;
|
|
|
|
/// Lifecycle + query operations every orchestrator exposes to the RPC layer.
|
|
#[async_trait]
|
|
pub trait ContainerOrchestrator: Send + Sync {
|
|
/// Build-or-pull the image, create the container, and start it. Returns the
|
|
/// podman container name that was created. Assumes the app_id corresponds
|
|
/// to a manifest the orchestrator already knows about.
|
|
async fn install(&self, app_id: &str) -> Result<String>;
|
|
|
|
/// True when this orchestrator holds a manifest for `app_id` (disk or
|
|
/// signed-catalog overlay) — i.e. `install(app_id)` would not fail with
|
|
/// "unknown app_id". Lets the RPC layer route any manifest-driven app
|
|
/// through the orchestrator without a per-app allowlist. Defaults to
|
|
/// `false` so orchestrators without a manifest registry keep routing
|
|
/// through the legacy install flow.
|
|
async fn knows_app(&self, _app_id: &str) -> bool {
|
|
false
|
|
}
|
|
|
|
/// Rebuild the in-memory manifest map (disk + signed-catalog overlay).
|
|
/// Called after a runtime catalog refresh detects changed bytes so catalog
|
|
/// manifest changes take effect without a service restart — without this,
|
|
/// `load_manifests` only runs at startup and a freshly published manifest
|
|
/// sits dormant until the next restart. Returns the merged manifest count.
|
|
/// Defaults to a no-op for orchestrators without a manifest registry.
|
|
async fn reload_manifests(&self) -> Result<usize> {
|
|
Ok(0)
|
|
}
|
|
|
|
/// Start an already-created container.
|
|
async fn start(&self, app_id: &str) -> Result<()>;
|
|
|
|
/// Stop a running container. No-op on Prod if already stopped.
|
|
async fn stop(&self, app_id: &str) -> Result<()>;
|
|
|
|
/// Stop-then-start. Best-effort: ignores stop failure.
|
|
async fn restart(&self, app_id: &str) -> Result<()>;
|
|
|
|
/// Remove the container. `preserve_data = true` keeps the volumes; `false`
|
|
/// is honored on a best-effort basis (Dev cleans, Prod leaves the volume
|
|
/// management to the data layer).
|
|
async fn remove(&self, app_id: &str, preserve_data: bool) -> Result<()>;
|
|
|
|
/// Pull/rebuild the image and recreate the container from scratch.
|
|
async fn upgrade(&self, app_id: &str) -> Result<()>;
|
|
|
|
/// Current state of a single container.
|
|
async fn status(&self, app_id: &str) -> Result<ContainerStatus>;
|
|
|
|
/// All containers this orchestrator knows about.
|
|
async fn list(&self) -> Result<Vec<ContainerStatus>>;
|
|
|
|
/// Tail the container's stdout+stderr.
|
|
async fn logs(&self, app_id: &str, lines: u32) -> Result<Vec<String>>;
|
|
|
|
/// Coarse health summary: "healthy", "unhealthy", "starting", "paused", "unknown".
|
|
async fn health(&self, app_id: &str) -> Result<String>;
|
|
|
|
/// Declare that a credential this app consumes has just been rotated, so
|
|
/// the running container is now holding an invalid one.
|
|
///
|
|
/// Restart-sensitivity normally protects apps like `btcpay-server` from
|
|
/// being recreated on drift — correct when the running container is
|
|
/// working, and exactly wrong when it is working only in appearance. After
|
|
/// an LND macaroon rotation, BTCPay is up and healthy while every Lightning
|
|
/// operation it attempts fails against a credential LND no longer honours;
|
|
/// leaving it untouched perpetuates the breakage rather than protecting
|
|
/// anything. This is the same carve-out FED-07 uses for the Fedimint
|
|
/// gateway, reached from the RPC layer instead of from inside a reconcile.
|
|
///
|
|
/// Consumed by the next drift check, which recreates the container around
|
|
/// its unchanged data directory, ports and volumes. Default no-op: an
|
|
/// orchestrator without restart-sensitivity has nothing to override.
|
|
async fn mark_credential_rotated(&self, _app_id: &str) {}
|
|
}
|