The 2026-08-08 unwire set the bar: a DURABLE record of 'this app is
installed' must drive reaping, never runtime inference. installed-apps.json
is that record (written on install, cleared on uninstall, backfilled from
live containers, held through container absence). The reconciler's
companion loop now reaps against it — and only when the registry file
actually exists and parses: 'I could not look' and 'nothing is installed'
both surface as an empty set from the lossy loader, so a new
load_installed_apps_if_recorded keeps the distinction alive. The
runtime-derived set is deliberately not a fallback; it is the input class
that caused the incident. ORPHAN_GRACE still applies on top.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Caught on archi-dev-box within minutes of deploying 3ac59a73: the reaper
removed archy-bitcoin-ui and archy-lnd-ui, whose backends ARE installed.
archy-bitcoin-ui was gone for 36 minutes, until the operator reinstalled
bitcoin-knots and `reconcile` put the companion back.
Not a logic error — the arithmetic did what it was told. The inputs were
false. Both backends' containers were missing because of the clean-exit
vanishing bug (8908fb4f), and both had already aged out of
running-containers.json, which only ever records what is CURRENTLY RUNNING.
So the two signals `installed_app_ids` combines are not independent: one root
cause falsifies both simultaneously. ORPHAN_GRACE could not help either — the
condition was persistent, not transient, which is exactly the case the grace
period cannot distinguish.
The asymmetry decides it. An un-reaped orphan costs a stale UI tile. A
wrongly-reaped companion costs a working screen and turns one lost app into
two — the reaper amplifies the very failure it was meant to tidy up after.
`reap_orphans` and its tests stay, documented as NOT TO BE WIRED until a
durable record of "this app is installed" exists to drive it. Inferring
installation from runtime state cannot answer that question, however many
runtime signals are combined.
The provisioning half is untouched and is the actual fix for "fedimint
installs but does not work": driving `reconcile` from installed_app_ids means
a companion is never stood up for an app nobody installed, so no NEW orphans
appear. The one genuine orphan on this node (archy-fedimint-ui, for an app
never installed) was correctly removed before this change landed.
Unit tests passed the reaper because they verify the set arithmetic, not
whether the "installed" signal is truthful. Only the device could show that.
Container suite 221/221.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box was running archy-fedimint-ui and archy-lnd-ui with no
fedimint and no lnd container anywhere on the box. The Fedimint Guardian
UI sat on :8175 serving its "waiting for Bitcoin" page forever with
nothing behind it, which is what the operator reported as "fedimint
guardian installs but does not work" — there was nothing to install, the
UI was already up.
The boot reconciler drove companion provisioning from manifest_ids(),
which is every manifest the node can SEE: the whole apps/ directory plus
the signed-catalog overlay, 56 of them. The app reconciler has drawn this
line since phase 3 (ReconcileMode::ExistingOnly, "merely listing a
catalog manifest never installs an unqualified app"); the companion stage
never got the equivalent guard, so it stood up a UI for every app that
merely had a manifest and then self-healed it forever.
The other half is that reconcile() could only ever ADD. remove_for fires
only on the explicit uninstall RPC, so nothing ever subtracted: an
install that failed after its companion landed, or a container removed
by any other route, left a Restart=always unit alive permanently.
- installed_app_ids() replaces manifest_ids(): app ids whose container
actually exists. Returns Option, because a caller that removes things
on absence must not read "I could not look" as "nothing is installed".
Container presence in ANY state is the whole test — it deliberately
does not inherit the user_stopped/disabled filters, since a stopped app
is still an installed app and treating it otherwise would tear its
companion down and rebuild it on the next start.
- manifest_ids() is deleted rather than left unused. Its contract reads
as "installed" to anyone skimming, which is the whole bug.
- reap_orphans() removes companions whose backend is not installed, after
ORPHAN_GRACE (300s). The grace period is required, not defensive: this
node runs ARCHIPELAGO_USE_QUADLET_BACKENDS=true and a Quadlet app is
briefly containerless while restarting, so reaping on the first absent
tick would cost a healthy companion a teardown plus a possible 900s
image rebuild. A backend that reappears clears its clock.
- Reap failures are logged but kept out of the backoff input. Repair
keeps a companion available; reaping only tidies one away, and a wedged
reap must not back the repair path off to its 1h ceiling.
Every uncertain signal resolves toward not removing: no unit file and a
hung is-active reads as leave-it-alone.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fleet log sweep (2026-07-22): (1) recovery paths podman-start'ed
containers that were already running, producing hundreds of benign but
scary conmon 'Failed to create container' + cgroup Permission-denied
pairs per node per day — both paths now check state first; (2) the 30s
companion-reconcile loop retried registry pulls/builds forever against
unreachable registries (~174×/image/day on an offline node) — failing
rounds now back off exponentially to a 1h cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The companion-unit repair stage ran at the END of each boot-reconciler tick, after
reconcile_existing(). On a heavily loaded node that per-app pass takes >60-90s, so a
deleted/lost companion unit (electrs-ui, bitcoin-ui, …) wasn't repaired within any
reasonable window (gate test 31 'deleted unit recreated within one reconcile tick'
timed out at 90s on the 45-app .228 node). Detecting + rewriting a companion unit is
cheap, so spawn it as its own ~interval(30s) loop, independent of the slow app pass.
Handle is aborted when the main loop exits (shutdown uses notify_one, so a second
waiter would steal the wake permit). tick() is now app-reconcile only.
All 4 boot_reconciler cadence tests still green (companion_stage=false in tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3a of the install path consolidation. Two coupled changes:
1. install.rs handle_package_install: gate the legacy "container exists →
adopt + return" probe on !orchestrator_managed. Apps the orchestrator
knows about (bitcoin-knots, bitcoin-core, lnd, electrumx, fedimint,
filebrowser, btcpay-server stack apps, mempool stack apps, plus the
companion UIs that just moved to Quadlet) skip the legacy probe and
fall straight into the orchestrator branch.
The legacy adopt block was returning success on a bare `podman start`
exit-0 — even when the process inside the container crashed seconds
later. That's the .228 "running but unreachable" failure mode. The
orchestrator's ensure_running honors the manifest's health check and
pre-start hooks (e.g. re-renders bitcoin-ui's nginx.conf if the RPC
password rotated), so this is a behavioral upgrade, not just a
refactor.
2. ProdContainerOrchestrator::install: make idempotent. Previously it
blindly called install_fresh which would fail on `podman create` if
the container name already existed. Now it delegates to ensure_running:
- Container Running + healthy → no-op (refresh hooks, restart if
config rewritten)
- Container Stopped/Exited → start (with hook refresh)
- Container missing → install_fresh
- Container in wedged state (Created/Paused/Unknown) → force-recreate
Without this, change #1 would regress every "container already exists"
case for the 18 orchestrator-managed app IDs. With it, install becomes
the single source of truth for "make app X be in the desired state."
Tests: 654 passed across the workspace (614 unit + 37 orchestration + 3
rpc), 0 failures. The 20 prod_orchestrator tests cover the install /
ensure_running / reconcile paths the new install delegates through.
Net delta: install.rs grows by ~30 lines (gating wrapper + comments),
prod_orchestrator.rs grows by ~30 lines (idempotent install body). Both
are temporary — the larger deletions (~1700 lines) come once every app
has been verified through the orchestrator path in subsequent phases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion UI containers (archy-bitcoin-ui, archy-lnd-ui,
archy-electrs-ui) used to be launched as fire-and-forget tokio::spawn
blocks from install.rs. If archipelago crashed mid-spawn or the
container's cgroup was reaped, companions vanished from podman ps -a
and only a manual rm/run could bring them back (the .228 incident).
Now each companion is rendered as a Quadlet .container unit under
~/.config/containers/systemd/, daemon-reloaded, and started via
systemctl --user. systemd owns supervision from that point on:
- archipelago can crash, restart, or be uninstalled without touching
any companion.
- Quadlet's Restart=always + RestartSec=10 handles container exits.
- A 30s reconcile tick in boot_reconciler enumerates expected
companion units and re-installs any whose unit file or service
vanished — defense-in-depth against external tampering.
New module layout:
- container/quadlet.rs: pure unit renderer + atomic write_if_changed
+ systemctl helpers (daemon_reload_user / enable_now / disable_remove
/ is_active). 6 unit tests, no I/O in the renderer.
- container/companion.rs: per-app companion specs, install/remove/
reconcile, image presence (build local first, fall back to insecure
registry only via image_uses_insecure_registry whitelist). 2 tests.
install.rs handle_package_install now ends with a single call to
companion::install_for(package_id), replacing 287 lines of spawn-and-
hope shellouts plus a ~120-line nginx auth-injector helper that worked
around per-node RPC password baking. The helper is gone too — the
pre-start hook renders the per-node nginx.conf to /var/lib/archipelago/
bitcoin-ui/nginx.conf and the Quadlet unit bind-mounts it read-only.
runtime.rs handle_package_uninstall now disables companions before
the container rm loop. Otherwise systemd's Restart=always would
respawn each companion within ~10s of removal.
Tests: 53 container tests pass, including 6 quadlet renderer tests
(host network, bridge network, capability set, atomic write idempotence)
and 2 companion specs (per-app companion lookup, build_unit shape).
boot_reconciler tests gain a #[cfg(test)] without_companion_stage()
flag so the paused-clock fixtures don't race the real systemctl I/O.
A bats regression test (companion-survives-archipelago-restart.bats,
gated on ARCHY_ALLOW_DESTRUCTIVE=1) asserts the .228 failure mode
cannot recur: every installed companion has a unit file, services
stay active across systemctl --user restart archipelago, and a
deleted unit file is recreated within one reconcile tick.
Net delta: +941 / -363, but the +941 is mostly tests (~440 lines)
and the new declarative layer; the imperative tokio::spawn block and
its nginx-auth helper are gone, removing two failure classes
(orphan companions on archipelago crash, and post-start exec races
under tightly-confined cgroups) that previously needed manual SSH
recovery.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 5 of the rust-orchestrator migration. New file boot_reconciler.rs holds a
small Tokio task that calls ProdContainerOrchestrator::reconcile_all() on a
30-second cadence (answered design Q3).
* BootReconciler::new(orch, interval, shutdown) — shutdown is an Arc<Notify>
so callers can trigger a graceful exit without pulling in tokio-util.
* run_forever(self) — does one reconcile immediately, then loops on
tokio::select! { sleep_until | shutdown.notified() }. Shutdown interrupts
the sleep but never an in-flight reconcile_all call.
* Per-pass outcomes are logged at debug/warn; failures never propagate out
because reconcile_all already absorbs per-app errors into ReconcileReport.
Four tokio::test(start_paused = true) tests verify the loop cadence against a
CountingRuntime test double:
* initial_pass_fires_immediately — first reconcile runs with no delay
* second_pass_fires_after_interval — second pass fires after exactly
interval elapses in paused-clock time
* shutdown_terminates_loop — notify_one() lets run_forever return
* failure_in_one_pass_does_not_stop_loop — the loop keeps ticking even when
the first pass had to install a missing container
Not wired into main.rs yet — that is Step 6. Re-exported from container::mod
as BootReconciler + RECONCILER_DEFAULT_INTERVAL for the wire-up step.
