Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.install routed to the manifest-driven orchestrator only for a
hardcoded per-app allowlist; every other app fell through to the legacy
flow, which ignores manifests entirely and creates a bare container with
no ports or volumes (strfry crash-looped this way on .228, 2026-07-09).
New ContainerOrchestrator::knows_app(app_id) (default false; prod checks
its loaded-manifest map, disk + signed-catalog overlay). The install gate
is now allowlist OR knows_app — no per-app Rust for manifest-driven apps,
matching the packaging invariant. Apps without a manifest keep the legacy
flow including the container-exists adopt block; the unknown-app_id →
legacy fallback inside the orchestrator branch is unchanged.
Tests: knows_app_reflects_loaded_manifests + install suite 37/37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 4 of the rust-orchestrator migration. Unifies the container lifecycle
surface behind a single trait so the RPC layer stops caring whether it is
talking to the dev or prod orchestrator.
* New trait core/archipelago/src/container/traits.rs: ContainerOrchestrator
with install / start / stop / restart / remove / upgrade / status / list /
logs / health, all keyed by app_id. Every method is async_trait-based.
* ProdContainerOrchestrator: the lifecycle methods are moved from inherent
impl into the trait impl (avoids name-shadowing recursion). Adoption and
reconcile remain inherent since only main.rs / BootReconciler call them.
* DevContainerOrchestrator: new trait impl that forwards to the existing
Dev-named methods, applying the dev container-name + port-offset rules
internally. New load_manifest_for() helper resolves app_id to
<data_dir>/apps/<app_id>/manifest.yml so trait-level install(app_id)
works in dev too. install_container(manifest, path) stays inherent for
the manifest-path RPC shape.
* RpcHandler now holds Option<Arc<dyn ContainerOrchestrator>> and, when in
dev mode, a separate Option<Arc<DevContainerOrchestrator>> for the
manifest_path install RPC. In prod mode RpcHandler::new() constructs a
ProdContainerOrchestrator and calls load_manifests() at startup.
* All seven container-* RPC guards no longer say dev mode required.
container-install still requires dev mode because its manifest_path
argument has no prod meaning; every other container RPC now works in both
modes via the trait.
BOOT STILL DOES NOT USE THIS. main.rs wire-up (Step 6) and BootReconciler
(Step 5) come next. Until then the prod orchestrator is constructed but nothing
populates /opt/archipelago/apps so it has zero manifests to manage, matching
the pre-Step-4 behaviour.
Verification: cargo build -p archipelago clean (11 expected unused method
warnings for methods not yet wired from main.rs). cargo test -p archipelago:
all 21 container::* tests pass (16 prod_orchestrator + 5 others). 24 other
test failures are pre-existing and unrelated (identity_manager / session /
wallet / mesh / credentials — all independently flaky on file-backed state).
