fix(container): remember that an app is installed, so a vanished one comes back

An app whose container disappeared could be lost permanently. Desired-state
recovery decided whether to recreate it from running-containers.json — "what
was running at the last snapshot" — which is a different question and a
perishable answer: it records only what is running NOW, so an app that stays
down long enough simply ages out. Once out, boot's ExistingOnly mode will not
recreate it, because it cannot tell "installed and lost" from "merely
available in the catalog". Manifest still on disk, nothing to bring it back.

This is the second occurrence of one root cause. indeedhub-minio/-postgres
went permanently absent on one node (2026-08-06); the fix then was
`absent_stack_member_with_live_sibling`, which only rescues a stack member
that still has a living sibling. bitcoin-knots is standalone, so on
archi-dev-box (2026-08-08) it vanished, aged out, and stayed gone — LND
crash-looping on `lookup bitcoin-knots: no such host` for hours, electrumx
unable to reach its daemon, and an orphaned fedimint container waiting 30
hours for a host that no longer resolved. Recovering it took a manual
reinstall. This is the general fix the narrow one implied.

Installation is a DECISION, not a runtime observation, so it gets a record
that no amount of downtime erodes: installed-apps.json, written when an
install succeeds and cleared on uninstall, in the same breath as
mark_user_uninstalled — leaving a stale claim would let recovery recreate the
app that was just removed. It is the durable counterpart to the
user-uninstalled marker that already existed.

Safety, in order of how badly each could go wrong:
- Cannot resurrect a deliberate uninstall: user_uninstalled is checked
  earlier in ensure_running_with_mode and returns before anything is created,
  and uninstall clears this record too.
- Cannot install an app nobody asked for: only names in the record qualify,
  and ExistingOnly's other guards are untouched.
- Cannot mislead a node upgrading into the feature: backfill seeds from
  ADOPTED containers only — evidence that something is really there — skips
  anything user-uninstalled, is additive so a momentarily-down app is never
  dropped, and no-ops on an empty adoption list (podman unreachable must not
  read as "nothing is installed").

Four tests, including the one that states the point: the record must outlive
a running-snapshot that has gone empty.

Container suite 221/221, crash_recovery 15/15.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

core/archipelago/src/container/prod_orchestrator.rs

@@ -1836,6 +1836,9 @@ impl ProdContainerOrchestrator {
         // app whose container vanished (e.g. a wedged teardown cleared by a
         // reboot) instead of leaving it down. See the immich .198 incident.
         let was_running = crate::crash_recovery::load_last_running_names(&self.data_dir).await;
+        // Durable installation record, consulted alongside the perishable
+        // `was_running` snapshot for desired-state recovery below.
+        let installed_apps = crate::crash_recovery::load_installed_apps(&self.data_dir).await;
         let (manifests, container_name_by_app_id): (
             Vec<LoadedManifest>,
             std::collections::HashMap<String, String>,
@@ -1941,6 +1944,22 @@ impl ProdContainerOrchestrator {
                     if mode == ReconcileMode::ExistingOnly
                         && reason == "absent"
                         && (was_running.contains(&compute_container_name(&lm.manifest))
+                            // The durable answer, and the one that does not
+                            // erode. `was_running` only records what was
+                            // running at the last snapshot, so an app that
+                            // stays down long enough ages out of it and can
+                            // never be recovered — bitcoin-knots on
+                            // archi-dev-box (2026-08-08), and
+                            // indeedhub-minio/-postgres before it. The
+                            // stack-member clause below was the narrow patch
+                            // for that second case; this is the general one.
+                            // Safe against resurrecting something removed on
+                            // purpose: `user_uninstalled` is checked earlier in
+                            // ensure_running_with_mode and returns before
+                            // anything is created, and uninstall clears this
+                            // record in the same breath as setting that marker.
+                            || installed_apps.contains(&app_id)
+                            || installed_apps.contains(&compute_container_name(&lm.manifest))
                             // Absent STACK MEMBER whose siblings have live
                             // containers: the stack is installed, so the
                             // missing member is a hole, not a choice. The
@@ -4244,6 +4263,10 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
         // baseline app the user had previously uninstalled would hit the
         // same reconcile guard and silently no-op.
         crate::crash_recovery::clear_user_uninstalled(&self.data_dir, app_id).await;
+        // Durable record that this app is installed, so a container that later
+        // vanishes is recovered however long it has been gone — the
+        // `was_running` snapshot alone forgets after a few daemon restarts.
+        crate::crash_recovery::mark_installed(&self.data_dir, app_id).await;
         // Idempotent: if the container is already up and healthy, just
         // refresh hooks and return. If it's stopped, start it. If it's
         // missing or in a wedged state, install fresh.
@@ -4307,6 +4330,10 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
         // install; the start/restart RPC handlers also clear it).
         crate::crash_recovery::clear_user_stopped(&self.data_dir, app_id).await;
         crate::crash_recovery::clear_user_uninstalled(&self.data_dir, app_id).await;
+        // Durable record that this app is installed, so a container that later
+        // vanishes is recovered however long it has been gone — the
+        // `was_running` snapshot alone forgets after a few daemon restarts.
+        crate::crash_recovery::mark_installed(&self.data_dir, app_id).await;
         let lm = self.loaded(app_id).await?;
         let action = self.ensure_running(&lm).await?;
         match action {
@@ -4525,6 +4552,10 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
         // survives to the next restart. Only mark on the success path above
         // — a failed removal means the app isn't actually gone.
         crate::crash_recovery::mark_user_uninstalled(&self.data_dir, app_id).await;
+        // …and drop the installation record in the same breath. Leaving a
+        // stale claim behind would let desired-state recovery recreate the very
+        // app that was just uninstalled.
+        crate::crash_recovery::clear_installed(&self.data_dir, app_id).await;
         Ok(())
     }