chore: release v1.7.48-alpha

Hotfix: archipelago.service ExecStartPre now mkdirs /run/containers and
/var/lib/containers before the unit's mount-namespace setup tries to bind
them. Without this, fresh nodes that don't have /run/containers (e.g.
nodes provisioned without a prior podman session) fail at the namespace
step with:

  Failed to set up mount namespacing: /run/containers: No such file or directory
  Failed at step NAMESPACE spawning /bin/bash: No such file or directory

Existing nodes don't pick up systemd unit changes via OTA — they need a
one-time `systemctl edit archipelago` adding the same mkdir. ISO installs
from this version forward have the fix baked in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## v1.7.48-alpha (2026-04-29)
+
+- archipelago.service no longer fails to start with "Failed to set up mount namespacing: /run/containers: No such file or directory" on nodes where /run/containers wasn't pre-created. ExecStartPre now creates it. Existing nodes need a one-time `systemctl edit archipelago` to add the mkdir; ISO installs from this version forward have the fix baked in.
+
+## v1.7.47-alpha (2026-04-29)
+
+- Bitcoin Knots/Core sync is now significantly faster. The container now uses every available core for script verification (was capped at 2) and has 8GB of memory instead of 4GB so its 4GB UTXO cache has headroom for the mempool and peer connections. Existing nodes pick up the new limits on next install/update; freshly-installed nodes start at full speed.
+- ElectrumX initial indexing is faster too. Its container memory bumped from 1GB to 2GB and its internal cache is now 2GB (default was 1.2GB).
+
+## v1.7.46-alpha (2026-04-29)
+
+- Health monitor no longer pages "Auto-restart failed" for orphaned containers. After a variant switch (bitcoin-core ↔ bitcoin-knots) the previous variant's container could survive uninstall and the health monitor would try restarting it forever. Now skipped silently with a debug log.
+- Apps no longer disappear from My Apps when an install fails. The card stays visible with state=Stopped so the user can retry or uninstall, with the failure reason surfaced via the new install_progress.message field.
+- "Downloading…" progress now actually advances during multi-image stack pulls. Was sticking at 20% until all pulls finished; now interpolates 20%→70% based on which image of N has landed.
+- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on OVH and tx1138. Removes a docker.io dependency from first-boot installs.
+- Resilience harness improvements: install-fail entries no longer vanish, install/uninstall/probe cells are timing-tolerant (60s retry on ui_probe and auth_probe), dep snapshots no longer leak companion containers into the dependent app's "new containers" set.
+
 ## v1.7.45-alpha (2026-04-29)
 
 - Bitcoin RPC auth is durable. The dashboard reliably connects across container restart, image update, and reboot. Was failing on registry-pulled images that shipped a stale baked-in password.

app-catalog/catalog.json

@@ -31,7 +31,7 @@
       "author": "Bitcoin Core contributors",
       "category": "money",
       "tier": "optional",
-      "dockerImage": "docker.io/bitcoin/bitcoin:28.4",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin:28.4",
       "repoUrl": "https://github.com/bitcoin/bitcoin"
     },
     {
@@ -125,7 +125,7 @@
       "icon": "/assets/img/app-icons/gitea.svg",
       "author": "Gitea",
       "category": "development",
-      "dockerImage": "docker.io/gitea/gitea:1.23",
+      "dockerImage": "146.59.87.168:3000/lfg2025/gitea:1.23",
       "repoUrl": "https://gitea.com"
     },
     {
@@ -263,7 +263,7 @@
       "icon": "/assets/img/app-icons/nextcloud.webp",
       "author": "Nextcloud",
       "category": "data",
-      "dockerImage": "docker.io/nextcloud:28",
+      "dockerImage": "146.59.87.168:3000/lfg2025/nextcloud:28",
       "repoUrl": "https://github.com/nextcloud/server"
     }
   ]

apps/bitcoin-core/manifest.yml

@@ -12,11 +12,16 @@ app:
     network: archy-net
     entrypoint: ["sh", "-lc"]
     custom_args:
+      # Sync-speed flags: -par=0 uses every core (was capped at 2 by
+      # --cpus=2, now removed for bitcoin/electrumx). -dbcache sized to
+      # the IBD sweet spot — 4GB on full nodes, 1GB on pruned. Container
+      # --memory=8g (config.rs::get_memory_limit) leaves headroom for
+      # mempool + connections.
       - >-
         if [ "${DISK_GB:-0}" -lt 1000 ]; then
-          exec bitcoind -server=1 -prune=550 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=512 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+          exec bitcoind -server=1 -prune=550 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=1024 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
         else
-          exec bitcoind -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+          exec bitcoind -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
         fi
     derived_env:
       - key: DISK_GB

core/Cargo.lock

@@ -80,7 +80,7 @@ checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
 
 [[package]]
 name = "archipelago"
-version = "1.7.45-alpha"
+version = "1.7.48-alpha"
 dependencies = [
  "anyhow",
  "archipelago-container",

core/archipelago/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "archipelago"
-version = "1.7.45-alpha"
+version = "1.7.48-alpha"
 edition = "2021"
 description = "Archipelago Bitcoin Node OS - Native backend"
 authors = ["Archipelago Team"]

core/archipelago/src/api/rpc/package/async_lifecycle.rs

@@ -113,11 +113,26 @@ impl RpcHandler {
                 Err(e) => {
                     error!("package.install {} failed: {:#}", package_id_spawn, e);
                     install_log(&format!("INSTALL FAIL: {} — {:#}", package_id_spawn, e)).await;
-                    // No pre-state to revert to — remove the entry entirely so
-                    // the UI shows the app as not installed. The next package
-                    // scan will re-create it only if podman actually has a
-                    // container for it (partial install recovery).
-                    remove_package_entry(&handler.state_manager, &package_id_spawn).await;
+                    // Don't remove the entry — that's what made the card
+                    // vanish from My Apps mid-install / between retry-loop
+                    // attempts (e.g. tailscale's entrypoint failure). Leave
+                    // the entry visible with state=Stopped + the install
+                    // error in install_progress.message so the user can see
+                    // what went wrong and decide whether to retry or
+                    // uninstall. clear_install_progress would erase the
+                    // message, so we set it explicitly here instead.
+                    let err_msg = format!("Install failed: {:#}", e);
+                    let (mut data, _) = handler.state_manager.get_snapshot().await;
+                    if let Some(entry) = data.package_data.get_mut(&package_id_spawn) {
+                        entry.state = PackageState::Stopped;
+                        entry.install_progress = Some(crate::data_model::InstallProgress {
+                            size: 0,
+                            downloaded: 0,
+                            phase: None,
+                            message: Some(err_msg),
+                        });
+                        handler.state_manager.update_data(data).await;
+                    }
                 }
             }
         });

core/archipelago/src/api/rpc/package/config.rs

@@ -244,13 +244,19 @@ pub(super) fn get_health_check_args(app_id: &str, _rpc_pass: &str) -> Vec<String
 /// Get per-app memory limit.
 pub(super) fn get_memory_limit(app_id: &str) -> &'static str {
     match app_id {
-        // Heavy apps
-        "bitcoin" | "bitcoin-core" | "bitcoin-knots" => "4g",
+        // Heavy apps. Bitcoin: dbcache uses ~4GB; the daemon also needs
+        // headroom for mempool + connection buffers + script-verifier
+        // memory + I/O. 4g caused OOM-cascades during IBD. 8g is the
+        // floor; ideally this would be host-RAM aware (next pass).
+        "bitcoin" | "bitcoin-core" | "bitcoin-knots" => "8g",
+        // ElectrumX: bumped from 1g to 2g so its CACHE_MB has somewhere
+        // to live during initial blockchain indexing. CACHE_MB=2048 in
+        // env vars below requires this much.
+        "electrumx" | "mempool-electrs" | "electrs" => "2g",
         "cryptpad" => "512m",
         "ollama" => "4g",
         // Medium apps
         "lnd" => "512m",
-        "electrumx" | "mempool-electrs" | "electrs" => "1g",
         "nextcloud" => "1g",
         "immich_server" | "immich" => "1g",
         "btcpay-server" | "btcpayserver" => "1g",
@@ -497,6 +503,16 @@ pub(super) async fn get_app_config(
             // only what's in bitcoin.conf + argv. The shared bitcoin.conf
             // carries rpcauth; we inject the networking flags as CLI args so
             // RPC is reachable from the bitcoin-ui companion container.
+            //
+            // Sync-speed flags:
+            //   -dbcache=4096 — UTXO set cache; 4GB is the sweet spot before
+            //     diminishing returns. Container has --memory=8g now so
+            //     there's headroom for mempool + connections.
+            //   -par=0       — use all available cores for script
+            //     verification (defaults to NCPU-1 capped at 16). Was
+            //     effectively pinned at 2 by --cpus=2 (now removed).
+            //   -maxconnections=125 — default but explicit, so ops can
+            //     tune downward on bandwidth-constrained nodes.
             Some(vec![
                 "-server=1".to_string(),
                 "-rpcbind=0.0.0.0".to_string(),
@@ -504,6 +520,9 @@ pub(super) async fn get_app_config(
                 "-rpcport=8332".to_string(),
                 "-printtoconsole=1".to_string(),
                 "-datadir=/home/bitcoin/.bitcoin".to_string(),
+                "-dbcache=4096".to_string(),
+                "-par=0".to_string(),
+                "-maxconnections=125".to_string(),
             ]),
         ),
         "bitcoin" | "bitcoin-knots" => (
@@ -597,6 +616,13 @@ pub(super) async fn get_app_config(
                     "COIN=Bitcoin".to_string(),
                     "DB_DIRECTORY=/data".to_string(),
                     "SERVICES=tcp://:50001,rpc://0.0.0.0:8000".to_string(),
+                    // Sync-speed: bigger LRU/write cache during initial
+                    // history index. Default is 1200MB, container now
+                    // gets 2g (config.rs::get_memory_limit) so 2048 fits.
+                    "CACHE_MB=2048".to_string(),
+                    // Block-fetcher concurrency — defaults are conservative
+                    // for shared hosts; 4 is plenty for one bitcoind backend.
+                    "MAX_SEND=10000000".to_string(),
                 ],
                 None,
                 None,

core/archipelago/src/api/rpc/package/install.rs

@@ -567,7 +567,18 @@ impl RpcHandler {
         let memory_limit = get_memory_limit(package_id);
         let mem_arg = format!("--memory={}", memory_limit);
         run_args.push(&mem_arg);
-        run_args.push("--cpus=2");
+        // Bitcoin (and friends) need every core they can get during initial
+        // blockchain download — script verification is parallelizable and
+        // the limiting factor on most home boxes. --cpus=2 was halving sync
+        // speed for 4-8 core machines. ElectrumX likewise scales with cores
+        // during its initial reorg/indexing phase.
+        let cpu_capped = !matches!(
+            package_id,
+            "bitcoin" | "bitcoin-core" | "bitcoin-knots" | "electrumx" | "electrs" | "mempool-electrs"
+        );
+        if cpu_capped {
+            run_args.push("--cpus=2");
+        }
 
         // Uptime Kuma image entrypoint (`extra/entrypoint.sh`) attempts
         // `setpriv --clear-groups` and fails under our rootless + cap-drop

core/archipelago/src/api/rpc/package/progress.rs

@@ -25,6 +25,7 @@ impl RpcHandler {
             size,
             downloaded,
             phase: existing_phase,
+            message: None,
         });
         self.state_manager.update_data(data).await;
     }
@@ -55,6 +56,7 @@ impl RpcHandler {
             size,
             downloaded,
             phase: Some(phase),
+            message: None,
         });
         self.state_manager.update_data(data).await;
     }
@@ -97,6 +99,7 @@ impl RpcHandler {
             size: total,
             downloaded,
             phase: existing_phase,
+            message: None,
         });
         state_manager.update_data(data).await;
     }

core/archipelago/src/api/rpc/package/stacks.rs

@@ -201,7 +201,7 @@ impl RpcHandler {
 
         let images = [
             "146.59.87.168:3000/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
-            "docker.io/valkey/valkey:7-alpine",
+            "146.59.87.168:3000/lfg2025/valkey:7-alpine",
             "146.59.87.168:3000/lfg2025/immich-server:release",
         ];
         self.set_install_phase("immich", InstallPhase::PullingImage)
@@ -300,7 +300,7 @@ impl RpcHandler {
                 "--health-cmd=valkey-cli ping || exit 1",
                 "--health-interval=30s",
                 "--health-retries=3",
-                "docker.io/valkey/valkey:7-alpine",
+                "146.59.87.168:3000/lfg2025/valkey:7-alpine",
             ])
             .output()
             .await;

core/archipelago/src/data_model.rs

@@ -255,6 +255,12 @@ pub struct InstallProgress {
     /// a fixed UI percentage and a descriptive label.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub phase: Option<InstallPhase>,
+    /// Optional explicit message — used to surface install failures so
+    /// the UI can keep the app card visible with an error description
+    /// instead of silently removing the entry on fail. UI's PHASE_INFO
+    /// label takes precedence when phase is set.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub message: Option<String>,
 }
 
 /// Phases of the install / update pipeline, surfaced to the UI so users

core/archipelago/src/health_monitor.rs

@@ -539,6 +539,20 @@ pub fn spawn_health_monitor(state: Arc<StateManager>, data_dir: PathBuf) {
                         debug!("Skipping uninstalled container: {}", container.name);
                         continue;
                     }
+                } else {
+                    // Orphan: container exists in podman but archipelago has
+                    // no package_data entry for it. Common after a variant
+                    // switch (bitcoin-core ↔ bitcoin-knots) where the
+                    // uninstall removed the package entry but the prior
+                    // variant's container survived in stopped state. Without
+                    // this guard the health monitor pages every minute with
+                    // "Auto-restart failed (attempt N/10)" for an app the
+                    // user can no longer see in the dashboard.
+                    debug!(
+                        "Skipping orphan container (not in package_data): {}",
+                        container.name
+                    );
+                    continue;
                 }
 
                 if container.healthy {

image-recipe/configs/archipelago.service

@@ -10,7 +10,7 @@ Environment="ARCHIPELAGO_BIND=127.0.0.1:5678"
 # DEV_MODE disabled in production — enabled via override.conf on dev servers
 Environment="XDG_RUNTIME_DIR=/run/user/1000"
 # + prefix runs these as root (needed for chown/mkdir outside ReadWritePaths)
-ExecStartPre=+/bin/bash -c 'mkdir -p /run/user/1000 && chown archipelago:archipelago /run/user/1000 && chmod 700 /run/user/1000'
+ExecStartPre=+/bin/bash -c 'mkdir -p /run/user/1000 /run/containers /var/lib/containers && chown archipelago:archipelago /run/user/1000 && chmod 700 /run/user/1000'
 ExecStartPre=+/bin/bash -c 'mkdir -p /var/lib/archipelago && chown archipelago:archipelago /var/lib/archipelago && echo "ARCHIPELAGO_HOST_IP=$(hostname -I 2>/dev/null | awk "{print $$1}")" > /var/lib/archipelago/host-ip.env && chown archipelago:archipelago /var/lib/archipelago/host-ip.env'
 ExecStart=/usr/local/bin/archipelago
 Restart=on-failure

neode-ui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "neode-ui",
   "private": true,
-  "version": "1.7.45-alpha",
+  "version": "1.7.48-alpha",
   "type": "module",
   "scripts": {
     "start": "./start-dev.sh",

neode-ui/public/catalog.json

@@ -31,7 +31,7 @@
       "author": "Bitcoin Core contributors",
       "category": "money",
       "tier": "optional",
-      "dockerImage": "docker.io/bitcoin/bitcoin:28.4",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin:28.4",
       "repoUrl": "https://github.com/bitcoin/bitcoin"
     },
     {
@@ -125,7 +125,7 @@
       "icon": "/assets/img/app-icons/gitea.svg",
       "author": "Gitea",
       "category": "development",
-      "dockerImage": "docker.io/gitea/gitea:1.23",
+      "dockerImage": "146.59.87.168:3000/lfg2025/gitea:1.23",
       "repoUrl": "https://gitea.com"
     },
     {
@@ -263,7 +263,7 @@
       "icon": "/assets/img/app-icons/nextcloud.webp",
       "author": "Nextcloud",
       "category": "data",
-      "dockerImage": "docker.io/nextcloud:28",
+      "dockerImage": "146.59.87.168:3000/lfg2025/nextcloud:28",
       "repoUrl": "https://github.com/nextcloud/server"
     }
   ]

neode-ui/src/stores/server.ts

@@ -63,18 +63,44 @@ export const useServerStore = defineStore('server', () => {
           if (progress.phase) {
             const info = PHASE_INFO[progress.phase]
             if (info) {
+              // Within the PullingImage band (20→70%), interpolate the
+              // bar based on how many images / bytes have landed so far.
+              // Without this, multi-container stacks (indeedhub: 7,
+              // mempool: 3, btcpay: 4) just sit at 20% for the entire
+              // pull duration — exactly what the user reported as
+              // "Downloading sticks at 20% mostly". X-of-N progress
+              // comes from set_install_progress(i, n) in stacks.rs.
+              let bandProgress = info.progress
+              if (progress.phase === 'pulling-image' && progress.size > 0) {
+                const fraction = Math.min(progress.downloaded / progress.size, 1)
+                // PullingImage band: 20% → 70%, so 50pp to interpolate over.
+                bandProgress = 20 + Math.round(fraction * 50)
+              }
               // Only advance forward — never let the bar step backward
               // between patches (can happen briefly during scan merges).
-              const nextProgress = Math.max(current.progress, info.progress)
+              const nextProgress = Math.max(current.progress, bandProgress)
+              // Show explicit message when set (e.g. install-fail descriptions
+              // surfaced via install_progress.message) — otherwise PHASE_INFO label.
+              const label = progress.message || info.message
               installingApps.value.set(appId, {
                 ...current,
                 status: info.status,
                 progress: nextProgress,
-                message: info.message,
+                message: label,
               })
               continue
             }
           }
+          // No phase but message is set (install-fail path) — show the message
+          // even if PHASE_INFO doesn't apply. Status stays whatever the watcher
+          // currently has.
+          if (progress.message) {
+            installingApps.value.set(appId, {
+              ...current,
+              message: progress.message,
+            })
+            continue
+          }
           // Fallback: byte counters (rare — podman usually doesn't
           // emit parseable progress on a piped stderr).
           const pct = progress.size > 0 ? Math.round((progress.downloaded / progress.size) * 100) : 0

neode-ui/src/types/api.ts

@@ -166,6 +166,9 @@ export interface InstallProgress {
    *  counters — podman pull doesn't emit parseable progress when
    *  stderr is piped, so byte counters are usually (0,0). */
   phase?: InstallPhase
+  /** Optional explicit message — surfaced on install failures so the
+   *  UI can show what went wrong instead of silently removing the card. */
+  message?: string
 }
 
 // RPC Request/Response types

scripts/resilience/resilience.sh

@@ -129,19 +129,34 @@ snapshot_containers() {
     ssh_run "podman ps -a --format '{{.Names}}' | sort"
 }
 
-# Whether $app currently has any of its expected containers running. Uses
+# Whether $app currently has ALL of its expected containers running. Uses
 # the per-app metadata table in lib.sh (expected_containers_for) so variant
 # apps (bitcoin-knots/bitcoin-core sharing slots) and stacks are detected
 # correctly. Falls back to name-prefix match for apps the table doesn't know.
+#
+# Returns true only when every expected container is present. Earlier
+# versions returned true on ANY match — that caused dep installs (e.g.
+# bitcoin-knots required by btcpay) to be declared "installed" as soon as
+# the backend container appeared, before the UI companion (archy-bitcoin-ui)
+# was up. The before-snapshot then missed the companion, the after-snapshot
+# caught it, and it leaked into the dependent app's "new containers" set,
+# false-positive-FAILing stop/uninstall when the companion (correctly) did
+# not respond to the dependent app's package.stop.
 app_already_installed() {
     local app="$1"
     local snap; snap=$(snapshot_containers)
     local expected
     expected=$(expected_containers_for "$app")
-    local c
-    for c in $expected; do
-        echo "$snap" | grep -qxF "$c" && return 0
-    done
+    if [ -n "$expected" ] && [ "$expected" != "$app" ]; then
+        local c missing=0
+        for c in $expected; do
+            echo "$snap" | grep -qxF "$c" || missing=1
+        done
+        [ "$missing" -eq 0 ] && return 0
+        # Fall through to prefix match if the expected_containers list has
+        # gaps; a partial install still counts as "installed enough" for
+        # preclean purposes.
+    fi
     # Generic prefix fallback for apps not in the expected_containers_for table.
     echo "$snap" | grep -qE "^(${app}|${app}-|archy-${app}|archy-${app}-)"
 }
@@ -291,8 +306,18 @@ run_app_matrix() {
     fi
 
     # ── 02 ui_probe ──────────────────────────────────────────────
+    # Retry with backoff — install just finished, but the app's backend
+    # (fedimint, immich, mempool stack) may take 30+s to be ready to serve
+    # HTTP. Probing immediately false-positive-FAILed those apps; pass on
+    # first 2xx/3xx within 60s.
     local code
-    code=$(probe_app_proxy "$app")
+    local ui_deadline=$(($(date +%s) + 60))
+    while :; do
+        code=$(probe_app_proxy "$app")
+        [[ "$code" =~ ^(2[0-9][0-9]|3[0-9][0-9])$ ]] && break
+        [ "$(date +%s)" -ge "$ui_deadline" ] && break
+        sleep 5
+    done
     # Accept all 2xx/3xx — proxy reaches backend, app may redirect to login,
     # serve OAuth flow (307), or use 308 permanent. 401/403 still fail because
     # those mean "backend reached, app rejected request" which is the
@@ -300,17 +325,27 @@ run_app_matrix() {
     if [[ "$code" =~ ^(2[0-9][0-9]|3[0-9][0-9])$ ]]; then
         record "$app" ui_probe PASS "HTTP $code"
     else
-        record "$app" ui_probe FAIL "HTTP $code (expected 2xx/3xx)"
+        record "$app" ui_probe FAIL "HTTP $code (expected 2xx/3xx, retried 60s)"
     fi
 
     # ── 03 auth_probe (only for apps with a credentialed/data endpoint) ──
+    # Same backoff treatment: bitcoin-ui's nginx config bind-mount is
+    # picked up at start, but the bitcoin-core backend may not have
+    # accepted RPC connections yet on a fresh install.
     local probe_code; local pass_codes
+    pass_codes=$(auth_probe_pass_codes "$app")
     if probe_code=$(auth_probe_for "$app" 2>/dev/null) && [ -n "$probe_code" ]; then
-        pass_codes=$(auth_probe_pass_codes "$app")
+        local auth_deadline=$(($(date +%s) + 60))
+        while :; do
+            echo " $pass_codes " | grep -qF " $probe_code " && break
+            [ "$(date +%s)" -ge "$auth_deadline" ] && break
+            sleep 5
+            probe_code=$(auth_probe_for "$app" 2>/dev/null) || break
+        done
         if echo " $pass_codes " | grep -qF " $probe_code "; then
             record "$app" auth_probe PASS "HTTP $probe_code"
         else
-            record "$app" auth_probe FAIL "HTTP $probe_code (expected one of: $pass_codes — credential plumbing broken)"
+            record "$app" auth_probe FAIL "HTTP $probe_code (expected one of: $pass_codes; retried 60s — credential plumbing broken)"
         fi
     else
         record "$app" auth_probe SKIP "no authenticated probe defined"