Promote runtime manifests before starting app reconciliation

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Finished runtime app-file promotion before manifest loading, preventing startup catalog refresh from forgetting disk-only apps.
+
 - Named the app in compact readiness messages and kept app-card actions aligned at the bottom.
 - Removed duplicate Mempool cards caused by frontend container aliases in restored inventory.
 

core/archipelago/src/bootstrap.rs

@@ -138,6 +138,24 @@ const NGINX_FEDIMINT_NEW: &str = "        sub_filter_types text/css application/
 const NGINX_FEDIMINT_SNIPPET_ANCHOR: &str = "proxy_pass http://127.0.0.1:8175/;";
 const NGINX_FEDIMINT_SNIPPET_INSERT: &str = "proxy_pass http://127.0.0.1:8175/;\n    proxy_set_header Accept-Encoding \"\";\n    sub_filter_types text/css application/javascript application/json;\n    sub_filter_once off;\n    sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n    sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n    sub_filter \"href='/\" \"href='/app/fedimint/\";\n    sub_filter \"src='/\" \"src='/app/fedimint/\";\n    sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n    sub_filter \"url('/\" \"url('/app/fedimint/\";\n    sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';";
 
+/// Finish manifest promotion before constructing the orchestrator or starting
+/// catalog refresh/reconciliation. Replacing the app tree in the background
+/// could let a reload observe its temporary empty state and forget disk-only apps.
+pub async fn ensure_runtime_assets_ready() {
+    match run_runtime_assets().await {
+        Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
+        Ok(_) => debug!("No OTA runtime payload to synchronize"),
+        Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
+    }
+    match run_apps_dir_repair().await {
+        Ok(true) => {
+            info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
+        }
+        Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
+        Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
+    }
+}
+
 /// Entry point called from main startup. Never returns an error to the caller —
 /// failing to bootstrap host artifacts must not prevent the backend from serving.
 pub async fn ensure_doctor_installed() {
@@ -146,11 +164,6 @@ pub async fn ensure_doctor_installed() {
         Ok(false) => debug!("No stale Archipelago dev-mode service override found"),
         Err(e) => warn!("Service override repair failed (non-fatal): {:#}", e),
     }
-    match run_runtime_assets().await {
-        Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
-        Ok(_) => debug!("No OTA runtime payload to synchronize"),
-        Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
-    }
     match run().await {
         Ok(changed) if changed => info!("Doctor artifacts synchronized with binary"),
         Ok(_) => debug!("Doctor artifacts already in sync"),
@@ -168,13 +181,6 @@ pub async fn ensure_doctor_installed() {
         Ok(false) => debug!("No stale bitcoin.conf found"),
         Err(e) => warn!("Bitcoin RPC repair failed (non-fatal): {:#}", e),
     }
-    match run_apps_dir_repair().await {
-        Ok(true) => {
-            info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
-        }
-        Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
-        Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
-    }
     match run_tor_helper_sync().await {
         Ok(true) => info!("tor-helper.sh synchronized with binary"),
         Ok(false) => debug!("tor-helper.sh already current"),

core/archipelago/src/main.rs

@@ -256,6 +256,10 @@ async fn main() -> Result<()> {
             boot_report.recovered, boot_report.total, boot_report.failed
         );
     }
+    // Disk manifests must be stable before the initial load and all later
+    // catalog reloads. Do not move this into the background doctor bootstrap.
+    bootstrap::ensure_runtime_assets_ready().await;
+
     // Construct the container orchestrator once. In prod mode we load the
     // on-disk app manifests, do an initial adoption pass, and spawn the
     // BootReconciler loop (Step 5/6 of the rust-orchestrator migration).

docs/next-release-20260930.md

@@ -142,3 +142,25 @@ across reboot; it does not substitute for final new-runtime delivery checks.
   prerequisite refusal, management restart and cleanup all passed. Both temporary
   fixtures and their network were removed. Actual dev API verification remains
   pending after removing an incomplete legacy-created adapter.
+
+## Startup manifest reload race
+
+Live Angor acceptance exposed a separate startup race: runtime asset bootstrap
+cleared and copied `/opt/archipelago/apps` in the background while the startup
+catalog refresh reloaded it. The daemon logged 62 loaded manifests followed by
+54 and then rejected the new disk-only app as unknown. A stable manifest snapshot
+confirmed the diagnosis: supported uninstall/reinstall produced the correct
+rootless Quadlet service with its declared port and network.
+
+Runtime promotion and the legacy installer-directory repair now finish before
+orchestrator construction. The background doctor no longer changes that tree.
+The final source backend suite passed 1,608 tests (four existing opt-in tests
+ignored). Optimized build and normal-path live restart verification remain pending.
+
+Actual dev Angor acceptance passed managed service identity, no capabilities,
+UID 101:101, archy-net, public block height, CORS and both fee URL forms. During
+Bitcoin initial sync, the real Mempool fee API returns 503; the adapter faithfully
+returns the same status and body. Full-sync fee availability remains unverified;
+ready-backend API and failure/recovery behavior passed the isolated live fixture.
+The temporary `/run/archy-candidate-manifests` snapshot override must be removed
+when deploying the startup-order fix, then normal startup/reload must be checked.