fix: release v1.7.51-alpha install hardening

Hotfix: archipelago.service ExecStartPre now mkdirs /run/containers and
/var/lib/containers before the unit's mount-namespace setup tries to bind
them. Without this, fresh nodes that don't have /run/containers (e.g.
nodes provisioned without a prior podman session) fail at the namespace
step with:

  Failed to set up mount namespacing: /run/containers: No such file or directory
  Failed at step NAMESPACE spawning /bin/bash: No such file or directory

Existing nodes don't pick up systemd unit changes via OTA — they need a
one-time `systemctl edit archipelago` adding the same mkdir. ISO installs
from this version forward have the fix baked in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.gitignore

@@ -74,3 +74,6 @@ loop/loop.log.bak
 web/
 
 ._*
+
+# Resilience harness reports (generated, contains session cookies)
+scripts/resilience/reports/

CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## v1.7.49-alpha (2026-04-30)
+
+- Bitcoin Knots/Core UI now reports connection, reconnecting, syncing, and error states from a backend status bridge instead of showing a stale "Unable to connect" message while the node is warming up.
+- ElectrumX UI now exposes indexed height, local Bitcoin height, known headers, status, and progress source so indexing/waiting states are readable during long initial sync.
+- Added container doctor timer and smoke/lifecycle test coverage for Bitcoin Knots/Core, ElectrumX, Mempool, BTCPay/NBXplorer, and UI surface availability.
+- Bitcoin Core and Bitcoin Knots are mutually exclusive variants, with a real Bitcoin Core manifest and corrected install conflict handling.
+- IndeeHub now launches only on direct web UI port `7778`; the broken `/app/indeedhub/` path proxy was removed, and port `7777` remains the Nostr relay.
+- BTCPay/NBXplorer Postgres environment formatting fixed so installs do not carry malformed connection strings.
+
+## v1.7.48-alpha (2026-04-29)
+
+- archipelago.service no longer fails to start with "Failed to set up mount namespacing: /run/containers: No such file or directory" on nodes where /run/containers wasn't pre-created. ExecStartPre now creates it. Existing nodes need a one-time `systemctl edit archipelago` to add the mkdir; ISO installs from this version forward have the fix baked in.
+
+## v1.7.47-alpha (2026-04-29)
+
+- Bitcoin Knots/Core sync is now significantly faster. The container now uses every available core for script verification (was capped at 2) and has 8GB of memory instead of 4GB so its 4GB UTXO cache has headroom for the mempool and peer connections. Existing nodes pick up the new limits on next install/update; freshly-installed nodes start at full speed.
+- ElectrumX initial indexing is faster too. Its container memory bumped from 1GB to 2GB and its internal cache is now 2GB (default was 1.2GB).
+
+## v1.7.46-alpha (2026-04-29)
+
+- Health monitor no longer pages "Auto-restart failed" for orphaned containers. After a variant switch (bitcoin-core ↔ bitcoin-knots) the previous variant's container could survive uninstall and the health monitor would try restarting it forever. Now skipped silently with a debug log.
+- Apps no longer disappear from My Apps when an install fails. The card stays visible with state=Stopped so the user can retry or uninstall, with the failure reason surfaced via the new install_progress.message field.
+- "Downloading…" progress now actually advances during multi-image stack pulls. Was sticking at 20% until all pulls finished; now interpolates 20%→70% based on which image of N has landed.
+- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on OVH and tx1138. Removes a docker.io dependency from first-boot installs.
+- Resilience harness improvements: install-fail entries no longer vanish, install/uninstall/probe cells are timing-tolerant (60s retry on ui_probe and auth_probe), dep snapshots no longer leak companion containers into the dependent app's "new containers" set.
+
+## v1.7.45-alpha (2026-04-29)
+
+- Bitcoin RPC auth is durable. The dashboard reliably connects across container restart, image update, and reboot. Was failing on registry-pulled images that shipped a stale baked-in password.
+- Multi-container apps show real install progress. IndeedHub (7), BTCPay (4), Mempool (3), Immich (3) — bar advances through Preparing → Pulling → Creating → Done instead of sitting at 0% until the very end.
+- Apps no longer disappear from the dashboard mid-install. The container scanner now respects in-flight installs and updates instead of evicting an entry while its containers are still being created.
+- IndeedHub installs cleanly on a fresh node. Five missing environment variables fixed; Nostr sign-in works on first install.
+- Tailscale install no longer fails with "executable not found". Container command was a malformed shell string; now a proper command array.
+- Removed three catalog entries that hung installs for ten minutes (dwn, endurain, ollama — no source images in our registries). Restored Nextcloud, sourced from docker.io.
+- Bitcoin Core update path uses the correct image name (was pulling from a non-existent path).
+- New ISO installs now allocate swap (sized to RAM, capped at 8GB, on the encrypted data partition). Without swap, container image builds and memory spikes were hitting OOM under load.
+
 ## v1.7.44-alpha (2026-04-28)
 
 43de3b73 feat(orchestrator): complete container migration and release hardening

app-catalog/catalog.json

@@ -1,7 +1,7 @@
 {
   "version": 2,
   "updated": "2026-04-22T00:00:00Z",
-  "registry": "git.tx1138.com/lfg2025",
+  "registry": "146.59.87.168:3000/lfg2025",
   "featured": {
     "id": "indeedhub",
     "banner": "/assets/img/featured/indeedhub-banner.jpg",
@@ -11,200 +11,260 @@
   },
   "apps": [
     {
-      "id": "bitcoin-knots", "title": "Bitcoin Knots", "version": "28.1.0",
+      "id": "bitcoin-knots",
+      "title": "Bitcoin Knots",
+      "version": "28.1.0",
       "description": "Run a full Bitcoin node. Validate and relay blocks and transactions.",
       "icon": "/assets/img/app-icons/bitcoin-knots.webp",
-      "author": "Bitcoin Knots", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/bitcoin-knots:latest",
+      "author": "Bitcoin Knots",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin-knots:latest",
       "repoUrl": "https://github.com/bitcoinknots/bitcoin"
     },
     {
-      "id": "bitcoin-core", "title": "Bitcoin Core", "version": "28.4",
-      "description": "Reference implementation of the Bitcoin protocol. Run a full node validating and relaying blocks.",
+      "id": "bitcoin-core",
+      "title": "Bitcoin Core",
+      "version": "28.4",
+      "description": "Reference Bitcoin node implementation. Alternative to Bitcoin Knots; uninstall Knots before switching.",
       "icon": "/assets/img/app-icons/bitcoin-core.svg",
-      "author": "Bitcoin Core contributors", "category": "money", "tier": "optional",
-      "dockerImage": "docker.io/bitcoin/bitcoin:28.4",
+      "author": "Bitcoin Core contributors",
+      "category": "money",
+      "tier": "optional",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin:28.4",
       "repoUrl": "https://github.com/bitcoin/bitcoin"
     },
     {
-      "id": "lnd", "title": "LND", "version": "0.18.4",
+      "id": "lnd",
+      "title": "LND",
+      "version": "0.18.4",
       "description": "Lightning Network Daemon. Fast Bitcoin payments through Lightning.",
       "icon": "/assets/img/app-icons/lnd.svg",
-      "author": "Lightning Labs", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/lnd:v0.18.4-beta",
+      "author": "Lightning Labs",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/lnd:v0.18.4-beta",
       "repoUrl": "https://github.com/lightningnetwork/lnd",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "btcpay-server", "title": "BTCPay Server", "version": "1.13.7",
+      "id": "btcpay-server",
+      "title": "BTCPay Server",
+      "version": "1.13.7",
       "description": "Self-hosted Bitcoin payment processor.",
       "icon": "/assets/img/app-icons/btcpay-server.png",
-      "author": "BTCPay Server Foundation", "category": "commerce", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/btcpayserver:1.13.7",
+      "author": "BTCPay Server Foundation",
+      "category": "commerce",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/btcpayserver:1.13.7",
       "repoUrl": "https://github.com/btcpayserver/btcpayserver",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "mempool", "title": "Mempool Explorer", "version": "3.0.0",
+      "id": "mempool",
+      "title": "Mempool Explorer",
+      "version": "3.0.0",
       "description": "Self-hosted Bitcoin blockchain and mempool visualizer.",
       "icon": "/assets/img/app-icons/mempool.webp",
-      "author": "Mempool", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/mempool-frontend:v3.0.0",
+      "author": "Mempool",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/mempool-frontend:v3.0.0",
       "repoUrl": "https://github.com/mempool/mempool",
-      "requires": ["bitcoin-knots", "electrumx"]
+      "requires": [
+        "bitcoin-knots",
+        "electrumx"
+      ]
     },
     {
-      "id": "electrumx", "title": "ElectrumX", "version": "1.18.0",
+      "id": "electrumx",
+      "title": "ElectrumX",
+      "version": "1.18.0",
       "description": "Electrum protocol server. Index the blockchain for fast wallet lookups.",
       "icon": "/assets/img/app-icons/electrumx.webp",
-      "author": "Luke Childs", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/electrumx:v1.18.0",
+      "author": "Luke Childs",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/electrumx:v1.18.0",
       "repoUrl": "https://github.com/spesmilo/electrumx",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "indeedhub", "title": "IndeeHub", "version": "1.0.0",
+      "id": "indeedhub",
+      "title": "IndeeHub",
+      "version": "1.0.0",
       "description": "Bitcoin documentary streaming with Nostr identity.",
       "icon": "/assets/img/app-icons/indeedhub.png",
-      "author": "IndeeHub", "category": "community",
-      "dockerImage": "git.tx1138.com/lfg2025/indeedhub:1.0.0",
+      "author": "IndeeHub",
+      "category": "community",
+      "dockerImage": "146.59.87.168:3000/lfg2025/indeedhub:1.0.0",
       "repoUrl": "https://github.com/indeedhub/indeedhub"
     },
     {
-      "id": "botfights", "title": "BotFights", "version": "1.1.0",
+      "id": "botfights",
+      "title": "BotFights",
+      "version": "1.1.0",
       "description": "Bot arena + 2-player arcade fighter with controller support and Adventure Mode.",
       "icon": "/assets/img/app-icons/botfights.svg",
-      "author": "BotFights", "category": "community",
-      "dockerImage": "git.tx1138.com/lfg2025/botfights:1.1.0",
+      "author": "BotFights",
+      "category": "community",
+      "dockerImage": "146.59.87.168:3000/lfg2025/botfights:1.1.0",
       "repoUrl": "https://botfights.net"
     },
     {
-      "id": "gitea", "title": "Gitea", "version": "1.23",
+      "id": "gitea",
+      "title": "Gitea",
+      "version": "1.23",
       "description": "Self-hosted Git service with container registry, CI/CD, issue tracking.",
       "icon": "/assets/img/app-icons/gitea.svg",
-      "author": "Gitea", "category": "development",
-      "dockerImage": "docker.io/gitea/gitea:1.23",
+      "author": "Gitea",
+      "category": "development",
+      "dockerImage": "146.59.87.168:3000/lfg2025/gitea:1.23",
       "repoUrl": "https://gitea.com"
     },
     {
-      "id": "filebrowser", "title": "File Browser", "version": "2.27.0",
+      "id": "filebrowser",
+      "title": "File Browser",
+      "version": "2.27.0",
       "description": "Web-based file manager.",
       "icon": "/assets/img/app-icons/file-browser.webp",
-      "author": "File Browser", "category": "data", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/filebrowser:v2.27.0",
+      "author": "File Browser",
+      "category": "data",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/filebrowser:v2.27.0",
       "repoUrl": "https://github.com/filebrowser/filebrowser"
     },
     {
-      "id": "vaultwarden", "title": "Vaultwarden", "version": "1.30.0",
+      "id": "vaultwarden",
+      "title": "Vaultwarden",
+      "version": "1.30.0",
       "description": "Self-hosted password vault with zero-knowledge encryption.",
       "icon": "/assets/img/app-icons/vaultwarden.webp",
-      "author": "Vaultwarden", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/vaultwarden:1.30.0-alpine",
+      "author": "Vaultwarden",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/vaultwarden:1.30.0-alpine",
       "repoUrl": "https://github.com/dani-garcia/vaultwarden"
     },
     {
-      "id": "searxng", "title": "SearXNG", "version": "2024.1.0",
+      "id": "searxng",
+      "title": "SearXNG",
+      "version": "2024.1.0",
       "description": "Privacy-respecting metasearch engine.",
       "icon": "/assets/img/app-icons/searxng.png",
-      "author": "SearXNG", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/searxng:latest",
+      "author": "SearXNG",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/searxng:latest",
       "repoUrl": "https://github.com/searxng/searxng"
     },
     {
-      "id": "fedimint", "title": "Fedimint", "version": "0.10.0",
+      "id": "fedimint",
+      "title": "Fedimint",
+      "version": "0.10.0",
       "description": "Federated Bitcoin mint with privacy through federated guardians.",
       "icon": "/assets/img/app-icons/fedimint.png",
-      "author": "Fedimint", "category": "money",
-      "dockerImage": "git.tx1138.com/lfg2025/fedimintd:v0.10.0",
+      "author": "Fedimint",
+      "category": "money",
+      "dockerImage": "146.59.87.168:3000/lfg2025/fedimintd:v0.10.0",
       "repoUrl": "https://github.com/fedimint/fedimint"
     },
     {
-      "id": "ollama", "title": "Ollama", "version": "0.5.4",
-      "description": "Run AI models locally. Private and on your hardware.",
-      "icon": "/assets/img/app-icons/ollama.png",
-      "author": "Ollama", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/ollama:latest",
-      "repoUrl": "https://github.com/ollama/ollama"
-    },
-    {
-      "id": "nextcloud", "title": "Nextcloud", "version": "28",
-      "description": "Your own private cloud. File sync, calendars, contacts.",
-      "icon": "/assets/img/app-icons/nextcloud.webp",
-      "author": "Nextcloud", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/nextcloud:28",
-      "repoUrl": "https://github.com/nextcloud/server"
-    },
-    {
-      "id": "jellyfin", "title": "Jellyfin", "version": "10.8.13",
+      "id": "jellyfin",
+      "title": "Jellyfin",
+      "version": "10.8.13",
       "description": "Free media server. Stream movies, music, and photos.",
       "icon": "/assets/img/app-icons/jellyfin.webp",
-      "author": "Jellyfin", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/jellyfin:10.8.13",
+      "author": "Jellyfin",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/jellyfin:10.8.13",
       "repoUrl": "https://github.com/jellyfin/jellyfin"
     },
     {
-      "id": "immich", "title": "Immich", "version": "1.90.0",
+      "id": "immich",
+      "title": "Immich",
+      "version": "1.90.0",
       "description": "High-performance photo and video backup with ML.",
       "icon": "/assets/img/app-icons/immich.png",
-      "author": "Immich", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/immich-server:release",
+      "author": "Immich",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/immich-server:release",
       "repoUrl": "https://github.com/immich-app/immich"
     },
     {
-      "id": "homeassistant", "title": "Home Assistant", "version": "2024.1",
+      "id": "homeassistant",
+      "title": "Home Assistant",
+      "version": "2024.1",
       "description": "Open-source home automation.",
       "icon": "/assets/img/app-icons/homeassistant.png",
-      "author": "Home Assistant", "category": "home",
-      "dockerImage": "git.tx1138.com/lfg2025/home-assistant:2024.1",
+      "author": "Home Assistant",
+      "category": "home",
+      "dockerImage": "146.59.87.168:3000/lfg2025/home-assistant:2024.1",
       "repoUrl": "https://github.com/home-assistant/core"
     },
     {
-      "id": "grafana", "title": "Grafana", "version": "10.2.0",
+      "id": "grafana",
+      "title": "Grafana",
+      "version": "10.2.0",
       "description": "Analytics and monitoring dashboards.",
       "icon": "/assets/img/app-icons/grafana.png",
-      "author": "Grafana Labs", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/grafana:10.2.0",
+      "author": "Grafana Labs",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/grafana:10.2.0",
       "repoUrl": "https://github.com/grafana/grafana"
     },
     {
-      "id": "tailscale", "title": "Tailscale", "version": "1.78.0",
+      "id": "tailscale",
+      "title": "Tailscale",
+      "version": "1.78.0",
       "description": "Zero-config VPN with WireGuard mesh networking.",
       "icon": "/assets/img/app-icons/tailscale.webp",
-      "author": "Tailscale", "category": "networking", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/tailscale:stable",
+      "author": "Tailscale",
+      "category": "networking",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/tailscale:stable",
       "repoUrl": "https://github.com/tailscale/tailscale"
     },
     {
-      "id": "uptime-kuma", "title": "Uptime Kuma", "version": "1.23.0",
+      "id": "uptime-kuma",
+      "title": "Uptime Kuma",
+      "version": "1.23.0",
       "description": "Self-hosted uptime monitoring.",
       "icon": "/assets/img/app-icons/uptime-kuma.webp",
-      "author": "Uptime Kuma", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/uptime-kuma:1",
+      "author": "Uptime Kuma",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/uptime-kuma:1",
       "repoUrl": "https://github.com/louislam/uptime-kuma"
     },
     {
-      "id": "dwn", "title": "Decentralized Web Node", "version": "0.4.0",
-      "description": "Own your data with DID-based access control.",
-      "icon": "/assets/img/app-icons/dwn.svg",
-      "author": "TBD", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/dwn-server:main",
-      "repoUrl": "https://github.com/TBD54566975/dwn-server"
-    },
-    {
-      "id": "endurain", "title": "Endurain", "version": "0.8.0",
-      "description": "Self-hosted fitness tracking. Strava alternative.",
-      "icon": "/assets/img/app-icons/endurain.png",
-      "author": "Endurain", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/endurain:0.8.0",
-      "repoUrl": "https://github.com/joaovitoriasilva/endurain"
-    },
-    {
-      "id": "photoprism", "title": "PhotoPrism", "version": "240915",
+      "id": "photoprism",
+      "title": "PhotoPrism",
+      "version": "240915",
       "description": "AI-powered photo management with facial recognition.",
       "icon": "/assets/img/app-icons/photoprism.svg",
-      "author": "PhotoPrism", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/photoprism:240915",
+      "author": "PhotoPrism",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/photoprism:240915",
       "repoUrl": "https://github.com/photoprism/photoprism"
+    },
+    {
+      "id": "nextcloud",
+      "title": "Nextcloud",
+      "version": "28",
+      "description": "Your own private cloud. File sync, calendars, contacts.",
+      "icon": "/assets/img/app-icons/nextcloud.webp",
+      "author": "Nextcloud",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/nextcloud:28",
+      "repoUrl": "https://github.com/nextcloud/server"
     }
   ]
 }

apps/archy-nbxplorer/manifest.yml

@@ -47,7 +47,7 @@ app:
     - NBXPLORER_BIND=0.0.0.0:32838
     - NBXPLORER_BTCRPCURL=http://bitcoin-knots:8332
     - NBXPLORER_BTCRPCUSER=archipelago
-    - NBXPLORER_POSTGRES=User ID=btcpay;Password=${BTCPAY_DB_PASS};Host=archy-btcpay-db;Port=5432;Database=nbxplorer;Include Error Detail=true
+    - NBXPLORER_POSTGRES=Username=btcpay;Password=${BTCPAY_DB_PASS};Host=archy-btcpay-db;Port=5432;Database=nbxplorer
 
   health_check:
     type: http

apps/bitcoin-core/manifest.yml

@@ -1,22 +1,27 @@
 app:
   id: bitcoin-core
-  name: Bitcoin Knots
+  name: Bitcoin Core
   version: 28.4.0
-  description: Full Bitcoin Knots node with dynamic prune/full-mode startup based on host disk.
+  description: Reference Bitcoin Core node with dynamic prune/full-mode startup based on host disk.
 
-  container_name: bitcoin-knots
+  container_name: bitcoin-core
 
   container:
-    image: git.tx1138.com/lfg2025/bitcoin-knots:latest
+    image: 146.59.87.168:3000/lfg2025/bitcoin:28.4
     pull_policy: if-not-present
     network: archy-net
     entrypoint: ["sh", "-lc"]
     custom_args:
+      # Sync-speed flags: -par=0 uses every core (was capped at 2 by
+      # --cpus=2, now removed for bitcoin/electrumx). -dbcache sized to
+      # the IBD sweet spot — 4GB on full nodes, 1GB on pruned. Container
+      # --memory=8g (config.rs::get_memory_limit) leaves headroom for
+      # mempool + connections.
       - >-
         if [ "${DISK_GB:-0}" -lt 1000 ]; then
-          exec bitcoind -server=1 -prune=550 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=512 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+          exec bitcoind -server=1 -prune=550 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=1024 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
         else
-          exec bitcoind -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+          exec bitcoind -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
         fi
     derived_env:
       - key: DISK_GB

apps/bitcoin-knots/manifest.yml

@@ -0,0 +1,75 @@
+app:
+  id: bitcoin-knots
+  name: Bitcoin Knots
+  version: 28.1.0
+  description: Full Bitcoin Knots node with dynamic prune/full-mode startup based on host disk.
+
+  container_name: bitcoin-knots
+
+  container:
+    image: 146.59.87.168:3000/lfg2025/bitcoin-knots:latest
+    pull_policy: if-not-present
+    network: archy-net
+    entrypoint: ["sh", "-lc"]
+    custom_args:
+      # Sync-speed flags: -par=0 uses every core (was capped at 2 by
+      # --cpus=2, now removed for bitcoin/electrumx). -dbcache sized to
+      # the IBD sweet spot — 4GB on full nodes, 1GB on pruned. Container
+      # --memory=8g (config.rs::get_memory_limit) leaves headroom for
+      # mempool + connections.
+      - >-
+        if [ "${DISK_GB:-0}" -lt 1000 ]; then
+          exec bitcoind -server=1 -prune=550 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=1024 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+        else
+          exec bitcoind -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 -rpcuser="${BITCOIN_RPC_USER}" -rpcpassword="${BITCOIN_RPC_PASS}";
+        fi
+    derived_env:
+      - key: DISK_GB
+        template: "{{DISK_GB}}"
+    secret_env:
+      - key: BITCOIN_RPC_PASS
+        secret_file: bitcoin-rpc-password
+    data_uid: "100101:100101"
+
+  dependencies:
+    - storage: 500Gi
+
+  resources:
+    cpu_limit: 0
+    memory_limit: 4Gi
+    disk_limit: 500Gi
+
+  security:
+    capabilities: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE]
+    readonly_root: false
+    network_policy: isolated
+
+  ports:
+    - host: 8332
+      container: 8332
+      protocol: tcp
+    - host: 8333
+      container: 8333
+      protocol: tcp
+
+  volumes:
+    - type: bind
+      source: /var/lib/archipelago/bitcoin
+      target: /home/bitcoin/.bitcoin
+      options: [rw]
+
+  environment:
+    - BITCOIN_RPC_USER=archipelago
+
+  health_check:
+    type: tcp
+    endpoint: localhost:8332
+    interval: 30s
+    timeout: 5s
+    retries: 3
+
+  bitcoin_integration:
+    rpc_access: admin
+    sync_required: true
+    testnet_support: false
+    pruning_support: true

apps/btcpay-server/manifest.yml

@@ -51,7 +51,7 @@ app:
     - BTCPAY_BTCEXPLORERURL=http://archy-nbxplorer:32838
     - BTCPAY_BTCRPCURL=http://bitcoin-knots:8332
     - BTCPAY_BTCRPCUSER=archipelago
-    - BTCPAY_POSTGRES=User ID=btcpay;Password=${BTCPAY_DB_PASS};Host=archy-btcpay-db;Port=5432;Database=btcpay;Include Error Detail=true
+    - BTCPAY_POSTGRES=Username=btcpay;Password=${BTCPAY_DB_PASS};Host=archy-btcpay-db;Port=5432;Database=btcpay
 
   health_check:
     type: http

apps/endurain/.dockerignore

@@ -1,6 +0,0 @@
-node_modules
-dist
-*.log
-.git
-.gitignore
-README.md

apps/endurain/Dockerfile

@@ -1,37 +0,0 @@
-FROM node:20-alpine AS builder
-
-WORKDIR /app
-
-# Copy package files
-COPY package*.json ./
-RUN npm ci --only=production
-
-# Copy source code
-COPY . .
-
-# Build the application
-RUN npm run build
-
-# Production stage
-FROM node:20-alpine
-
-WORKDIR /app
-
-# Copy built application
-COPY --from=builder /app/dist ./dist
-COPY --from=builder /app/node_modules ./node_modules
-COPY --from=builder /app/package.json ./
-
-# Create non-root user
-RUN addgroup -g 1000 appuser && \
-    adduser -D -u 1000 -G appuser appuser && \
-    mkdir -p /app/data && \
-    chown -R appuser:appuser /app
-
-USER appuser
-
-EXPOSE 8080
-
-ENV ENDURAIN_DATA_DIR=/app/data
-
-CMD ["node", "dist/index.js"]

apps/endurain/manifest.yml

@@ -1,50 +0,0 @@
-app:
-  id: endurain
-  name: Endurain
-  version: 1.0.0
-  description: Endurain application platform. Custom application runtime.
-  
-  container:
-    image: archipelago/endurain:1.0.0
-    image_signature: cosign://...
-    pull_policy: if-not-present
-    
-  dependencies:
-    - storage: 2Gi
-    
-  resources:
-    cpu_limit: 2
-    memory_limit: 1Gi
-    disk_limit: 2Gi
-    
-  security:
-    capabilities: []
-    readonly_root: true
-    no_new_privileges: true
-    user: 1000
-    seccomp_profile: default
-    network_policy: isolated
-    apparmor_profile: endurain
-    
-  ports:
-    - host: 8085
-      container: 8080
-      protocol: tcp  # Web UI
-    
-  volumes:
-    - type: bind
-      source: /var/lib/archipelago/endurain
-      target: /app/data
-      options: [rw]
-      
-  environment:
-    - ENDURAIN_ENV=production
-    - ENDURAIN_DATA_DIR=/app/data
-    
-  health_check:
-    type: http
-    endpoint: http://localhost:8085
-    path: /health
-    interval: 30s
-    timeout: 5s
-    retries: 3

apps/endurain/package.json

@@ -1,20 +0,0 @@
-{
-  "name": "endurain",
-  "version": "1.0.0",
-  "description": "Endurain application platform",
-  "main": "dist/index.js",
-  "scripts": {
-    "build": "tsc",
-    "start": "node dist/index.js",
-    "dev": "ts-node src/index.ts"
-  },
-  "dependencies": {
-    "express": "^4.18.2"
-  },
-  "devDependencies": {
-    "@types/express": "^4.17.21",
-    "@types/node": "^20.10.0",
-    "typescript": "^5.3.3",
-    "ts-node": "^10.9.2"
-  }
-}

apps/endurain/src/index.ts

@@ -1,27 +0,0 @@
-import express from 'express';
-
-const app = express();
-const port = 8080;
-
-// Middleware
-app.use(express.json());
-
-// Health check endpoint
-app.get('/health', (req, res) => {
-  res.json({ status: 'ok', service: 'endurain', version: '1.0.0' });
-});
-
-// API endpoints
-app.get('/api/info', (req, res) => {
-  res.json({
-    name: 'Endurain',
-    version: '1.0.0',
-    status: 'running'
-  });
-});
-
-// Start server
-app.listen(port, '0.0.0.0', () => {
-  console.log(`Endurain listening on port ${port}`);
-  console.log(`Data directory: ${process.env.ENDURAIN_DATA_DIR || '/app/data'}`);
-});

apps/endurain/tsconfig.json

@@ -1,16 +0,0 @@
-{
-  "compilerOptions": {
-    "target": "ES2020",
-    "module": "commonjs",
-    "lib": ["ES2020"],
-    "outDir": "./dist",
-    "rootDir": "./src",
-    "strict": true,
-    "esModuleInterop": true,
-    "skipLibCheck": true,
-    "forceConsistentCasingInFileNames": true,
-    "resolveJsonModule": true
-  },
-  "include": ["src/**/*"],
-  "exclude": ["node_modules", "dist"]
-}

apps/indeedhub/manifest.yml

@@ -6,7 +6,7 @@ app:
   category: media
 
   container:
-    image: git.tx1138.com/lfg2025/indeedhub:latest
+    image: 146.59.87.168:3000/lfg2025/indeedhub:latest
     pull_policy: always  # Pull from registry; falls back to local build
 
   dependencies:
@@ -27,9 +27,9 @@ app:
     apparmor_profile: default
 
   ports:
-    - host: 7777
-      container: 3000
-      protocol: tcp  # Web UI (Next.js)
+    - host: 7778
+      container: 7777
+      protocol: tcp  # Web UI. Port 7777 on the host is reserved for Nostr relay.
 
   volumes:
     - type: tmpfs
@@ -57,7 +57,7 @@ app:
       name: Web UI
       description: Stream Bitcoin documentaries with Nostr identity
       type: ui
-      port: 7777
+      port: 7778
       protocol: http
       path: /
 

apps/ollama/Dockerfile

@@ -1,5 +0,0 @@
-# Ollama - uses official image
-FROM ollama/ollama:latest
-
-# Default configuration is in the image
-# No additional setup needed

apps/ollama/manifest.yml

@@ -1,50 +0,0 @@
-app:
-  id: ollama
-  name: Ollama
-  version: 0.1.0
-  description: Run large language models locally. Privacy-preserving AI on your node.
-  
-  container:
-    image: ollama/ollama:0.6.2
-    image_signature: cosign://...
-    pull_policy: if-not-present
-    
-  dependencies:
-    - storage: 50Gi  # Models can be large
-    
-  resources:
-    cpu_limit: 4
-    memory_limit: 8Gi  # LLMs need lots of RAM
-    disk_limit: 50Gi
-    
-  security:
-    capabilities: []
-    readonly_root: false  # Ollama needs write access for models
-    no_new_privileges: true
-    user: 1000
-    seccomp_profile: default
-    network_policy: isolated
-    apparmor_profile: ollama
-    
-  ports:
-    - host: 11434
-      container: 11434
-      protocol: tcp  # API
-    
-  volumes:
-    - type: bind
-      source: /var/lib/archipelago/ollama
-      target: /root/.ollama
-      options: [rw]
-      
-  environment:
-    - OLLAMA_HOST=0.0.0.0:11434
-    - OLLAMA_KEEP_ALIVE=24h
-    
-  health_check:
-    type: http
-    endpoint: http://localhost:11434
-    path: /api/tags
-    interval: 30s
-    timeout: 10s
-    retries: 3

core/Cargo.lock

@@ -80,7 +80,7 @@ checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
 
 [[package]]
 name = "archipelago"
-version = "1.7.43-alpha"
+version = "1.7.51-alpha"
 dependencies = [
  "anyhow",
  "archipelago-container",

core/archipelago/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "archipelago"
-version = "1.7.44-alpha"
+version = "1.7.51-alpha"
 edition = "2021"
 description = "Archipelago Bitcoin Node OS - Native backend"
 authors = ["Archipelago Team"]

core/archipelago/src/api/handler/mod.rs

@@ -429,6 +429,7 @@ impl ApiHandler {
 
             // Electrs status — unauthenticated (read-only sync status)
             (Method::GET, "/electrs-status") => Self::handle_electrs_status().await,
+            (Method::GET, "/bitcoin-status") => Self::handle_bitcoin_status().await,
 
             // App-catalog proxy — fetches catalog.json from the configured
             // upstream URLs server-side so the browser doesn't hit CORS

core/archipelago/src/api/handler/proxy.rs

@@ -1,5 +1,6 @@
 use super::build_response;
 use crate::api::rpc::RpcHandler;
+use crate::bitcoin_status;
 use crate::electrs_status;
 use anyhow::Result;
 use hyper::{Response, StatusCode};
@@ -76,11 +77,23 @@ impl ApiHandler {
     pub(super) async fn handle_electrs_status() -> Result<Response<hyper::Body>> {
         let status = electrs_status::get_electrs_sync_status().await;
         let body = serde_json::to_vec(&status).unwrap_or_default();
-        Ok(build_response(
-            StatusCode::OK,
-            "application/json",
-            hyper::Body::from(body),
-        ))
+        Ok(Response::builder()
+            .status(StatusCode::OK)
+            .header("Content-Type", "application/json")
+            .header("Cache-Control", "no-store")
+            .body(hyper::Body::from(body))
+            .unwrap_or_else(|_| Response::new(hyper::Body::from("{}"))))
+    }
+
+    pub(super) async fn handle_bitcoin_status() -> Result<Response<hyper::Body>> {
+        let status = bitcoin_status::get_bitcoin_status().await;
+        let body = serde_json::to_vec(&status).unwrap_or_default();
+        Ok(Response::builder()
+            .status(StatusCode::OK)
+            .header("Content-Type", "application/json")
+            .header("Cache-Control", "no-store")
+            .body(hyper::Body::from(body))
+            .unwrap_or_else(|_| Response::new(hyper::Body::from("{}"))))
     }
 
     pub(super) async fn handle_lnd_connect_info(

core/archipelago/src/api/rpc/package/async_lifecycle.rs

@@ -113,11 +113,26 @@ impl RpcHandler {
                 Err(e) => {
                     error!("package.install {} failed: {:#}", package_id_spawn, e);
                     install_log(&format!("INSTALL FAIL: {} — {:#}", package_id_spawn, e)).await;
-                    // No pre-state to revert to — remove the entry entirely so
-                    // the UI shows the app as not installed. The next package
-                    // scan will re-create it only if podman actually has a
-                    // container for it (partial install recovery).
-                    remove_package_entry(&handler.state_manager, &package_id_spawn).await;
+                    // Don't remove the entry — that's what made the card
+                    // vanish from My Apps mid-install / between retry-loop
+                    // attempts (e.g. tailscale's entrypoint failure). Leave
+                    // the entry visible with state=Stopped + the install
+                    // error in install_progress.message so the user can see
+                    // what went wrong and decide whether to retry or
+                    // uninstall. clear_install_progress would erase the
+                    // message, so we set it explicitly here instead.
+                    let err_msg = format!("Install failed: {:#}", e);
+                    let (mut data, _) = handler.state_manager.get_snapshot().await;
+                    if let Some(entry) = data.package_data.get_mut(&package_id_spawn) {
+                        entry.state = PackageState::Stopped;
+                        entry.install_progress = Some(crate::data_model::InstallProgress {
+                            size: 0,
+                            downloaded: 0,
+                            phase: None,
+                            message: Some(err_msg),
+                        });
+                        handler.state_manager.update_data(data).await;
+                    }
                 }
             }
         });

core/archipelago/src/api/rpc/package/config.rs

@@ -244,13 +244,19 @@ pub(super) fn get_health_check_args(app_id: &str, _rpc_pass: &str) -> Vec<String
 /// Get per-app memory limit.
 pub(super) fn get_memory_limit(app_id: &str) -> &'static str {
     match app_id {
-        // Heavy apps
-        "bitcoin" | "bitcoin-core" | "bitcoin-knots" => "4g",
+        // Heavy apps. Bitcoin: dbcache uses ~4GB; the daemon also needs
+        // headroom for mempool + connection buffers + script-verifier
+        // memory + I/O. 4g caused OOM-cascades during IBD. 8g is the
+        // floor; ideally this would be host-RAM aware (next pass).
+        "bitcoin" | "bitcoin-core" | "bitcoin-knots" => "8g",
+        // ElectrumX: bumped from 1g to 2g so its CACHE_MB has somewhere
+        // to live during initial blockchain indexing. CACHE_MB=2048 in
+        // env vars below requires this much.
+        "electrumx" | "mempool-electrs" | "electrs" => "2g",
         "cryptpad" => "512m",
         "ollama" => "4g",
         // Medium apps
         "lnd" => "512m",
-        "electrumx" | "mempool-electrs" | "electrs" => "1g",
         "nextcloud" => "1g",
         "immich_server" | "immich" => "1g",
         "btcpay-server" | "btcpayserver" => "1g",
@@ -399,6 +405,9 @@ pub(super) async fn get_containers_for_app(package_id: &str) -> Result<Vec<Strin
 pub(super) fn get_data_dirs_for_app(package_id: &str) -> Vec<String> {
     let base = "/var/lib/archipelago";
     match package_id {
+        "bitcoin" | "bitcoin-core" | "bitcoin-knots" => {
+            vec![format!("{}/bitcoin", base), format!("{}/bitcoin-ui", base)]
+        }
         "mempool" | "mempool-web" => vec![
             format!("{}/mempool", base),
             format!("{}/mysql-mempool", base),
@@ -497,6 +506,16 @@ pub(super) async fn get_app_config(
             // only what's in bitcoin.conf + argv. The shared bitcoin.conf
             // carries rpcauth; we inject the networking flags as CLI args so
             // RPC is reachable from the bitcoin-ui companion container.
+            //
+            // Sync-speed flags:
+            //   -dbcache=4096 — UTXO set cache; 4GB is the sweet spot before
+            //     diminishing returns. Container has --memory=8g now so
+            //     there's headroom for mempool + connections.
+            //   -par=0       — use all available cores for script
+            //     verification (defaults to NCPU-1 capped at 16). Was
+            //     effectively pinned at 2 by --cpus=2 (now removed).
+            //   -maxconnections=125 — default but explicit, so ops can
+            //     tune downward on bandwidth-constrained nodes.
             Some(vec![
                 "-server=1".to_string(),
                 "-rpcbind=0.0.0.0".to_string(),
@@ -504,6 +523,9 @@ pub(super) async fn get_app_config(
                 "-rpcport=8332".to_string(),
                 "-printtoconsole=1".to_string(),
                 "-datadir=/home/bitcoin/.bitcoin".to_string(),
+                "-dbcache=4096".to_string(),
+                "-par=0".to_string(),
+                "-maxconnections=125".to_string(),
             ]),
         ),
         "bitcoin" | "bitcoin-knots" => (
@@ -597,6 +619,13 @@ pub(super) async fn get_app_config(
                     "COIN=Bitcoin".to_string(),
                     "DB_DIRECTORY=/data".to_string(),
                     "SERVICES=tcp://:50001,rpc://0.0.0.0:8000".to_string(),
+                    // Sync-speed: bigger LRU/write cache during initial
+                    // history index. Default is 1200MB, container now
+                    // gets 2g (config.rs::get_memory_limit) so 2048 fits.
+                    "CACHE_MB=2048".to_string(),
+                    // Block-fetcher concurrency — defaults are conservative
+                    // for shared hosts; 4 is plenty for one bitcoind backend.
+                    "MAX_SEND=10000000".to_string(),
                 ],
                 None,
                 None,
@@ -768,10 +797,17 @@ pub(super) async fn get_app_config(
             vec!["8240:8240".to_string()],
             vec!["/var/lib/archipelago/tailscale:/var/lib/tailscale".to_string()],
             vec!["TS_STATE_DIR=/var/lib/tailscale".to_string()],
-            Some(
-                "sh -c 'tailscale web --listen 0.0.0.0:8240 & exec tailscaled'".to_string(),
-            ),
+            // Don't use custom_command (Option<String>) — install.rs passes
+            // it as a SINGLE arg to podman, which then treats the whole
+            // "sh -c 'tailscale web …'" string as the executable name and
+            // fails: "executable file `sh -c 'tailscale web …'` not found".
+            // custom_args (Option<Vec<String>>) splits properly.
             None,
+            Some(vec![
+                "sh".to_string(),
+                "-c".to_string(),
+                "tailscale web --listen 0.0.0.0:8240 & exec tailscaled".to_string(),
+            ]),
         ),
         "fedimint" => (
             vec![

core/archipelago/src/api/rpc/package/install.rs

@@ -10,6 +10,7 @@ use super::progress::parse_pull_progress;
 use super::validation::validate_app_id;
 use crate::api::rpc::RpcHandler;
 use crate::data_model::InstallPhase;
+use crate::update::host_sudo;
 use anyhow::{Context, Result};
 use tokio::io::{AsyncBufReadExt, BufReader};
 use tracing::{debug, info, warn};
@@ -31,6 +32,130 @@ pub(in crate::api::rpc) async fn install_log(msg: &str) {
     }
 }
 
+/// Patch the Bitcoin RPC `Authorization: Basic ...` header inside the running
+/// bitcoin-ui container's nginx config and reload nginx. Authoritative
+/// credential injection — runs whether the image was built locally or pulled
+/// from the registry. Without this, registry images ship with whatever auth
+/// header was baked at build time on the publisher's machine, which never
+/// matches the per-node randomly-generated bitcoin-rpc-password.
+///
+/// Implementation note: this used to do `podman exec sed`, but rootless
+/// podman + tightly-confined containers (--cap-drop=ALL, restricted user)
+/// reject the exec because crun can't add a new process to the container's
+/// cgroup ("write cgroup.procs: Permission denied"). Switched to
+/// `podman cp` (storage layer, no cgroup join) + `podman kill --signal=SIGHUP`
+/// (signal to existing PID 1, no new process needed). Verified on .228.
+async fn inject_bitcoin_rpc_auth_into_running_container(container: &str, auth_b64: &str) {
+    use rand::distributions::{Alphanumeric, DistString};
+    let token = Alphanumeric.sample_string(&mut rand::thread_rng(), 8);
+    let host_path = format!("/tmp/archy-{container}-nginx.conf-{token}");
+    let in_container = "/etc/nginx/conf.d/default.conf";
+
+    // 1. Copy the running config out to host
+    let cp_out = tokio::process::Command::new("podman")
+        .args(["cp", &format!("{container}:{in_container}"), &host_path])
+        .output()
+        .await;
+    if let Err(e) = cp_out {
+        warn!("inject auth: podman cp out failed for {}: {}", container, e);
+        return;
+    }
+    if let Ok(ref o) = cp_out {
+        if !o.status.success() {
+            warn!(
+                "inject auth: podman cp out failed for {}: {}",
+                container,
+                String::from_utf8_lossy(&o.stderr)
+            );
+            return;
+        }
+    }
+
+    // 2. Patch the auth line on disk
+    let content = match tokio::fs::read_to_string(&host_path).await {
+        Ok(c) => c,
+        Err(e) => {
+            warn!("inject auth: read {} failed: {}", host_path, e);
+            let _ = tokio::fs::remove_file(&host_path).await;
+            return;
+        }
+    };
+    let mut patched_any = false;
+    let updated: String = content
+        .lines()
+        .map(|line| {
+            if line.contains("proxy_set_header Authorization") && line.contains("Basic") {
+                patched_any = true;
+                format!(
+                    "        proxy_set_header Authorization \"Basic {}\";",
+                    auth_b64
+                )
+            } else {
+                line.to_string()
+            }
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+    if !patched_any {
+        warn!(
+            "inject auth: no Authorization line matched in {}'s nginx.conf",
+            container
+        );
+        let _ = tokio::fs::remove_file(&host_path).await;
+        return;
+    }
+    if let Err(e) = tokio::fs::write(&host_path, format!("{}\n", updated)).await {
+        warn!("inject auth: write back failed: {}", e);
+        let _ = tokio::fs::remove_file(&host_path).await;
+        return;
+    }
+
+    // 3. Copy patched config back into the container
+    let cp_in = tokio::process::Command::new("podman")
+        .args(["cp", &host_path, &format!("{container}:{in_container}")])
+        .output()
+        .await;
+    let _ = tokio::fs::remove_file(&host_path).await;
+    match cp_in {
+        Ok(o) if !o.status.success() => {
+            warn!(
+                "inject auth: podman cp in failed for {}: {}",
+                container,
+                String::from_utf8_lossy(&o.stderr)
+            );
+            return;
+        }
+        Err(e) => {
+            warn!("inject auth: podman cp in errored for {}: {}", container, e);
+            return;
+        }
+        _ => {}
+    }
+
+    // 4. Reload nginx via SIGHUP to PID 1 (no exec/cgroup join needed)
+    let reload = tokio::process::Command::new("podman")
+        .args(["kill", "--signal=SIGHUP", container])
+        .output()
+        .await;
+    match reload {
+        Ok(o) if o.status.success() => {
+            info!(
+                "Injected Bitcoin RPC auth into {} (post-start, cp+SIGHUP)",
+                container
+            );
+        }
+        Ok(o) => warn!(
+            "Patched nginx.conf in {} but SIGHUP failed: {}",
+            container,
+            String::from_utf8_lossy(&o.stderr)
+        ),
+        Err(e) => warn!(
+            "Patched nginx.conf in {} but SIGHUP errored: {}",
+            container, e
+        ),
+    }
+}
+
 impl RpcHandler {
     /// Install a package from a Docker image.
     /// Security: Image verification, resource limits, network isolation.
@@ -83,6 +208,16 @@ impl RpcHandler {
             }
         }
 
+        // Phase: Preparing — emit BEFORE the stack dispatch so multi-container
+        // stacks also flip state to Installing immediately. Without this, the
+        // backend's package state for stack apps stayed empty until the first
+        // podman pull finished, so a hard refresh during the early seconds of
+        // a stack install showed the app as missing entirely (the user
+        // reported "the app disappears from installing if you hard refresh
+        // then sometimes comes back later").
+        self.set_install_phase(package_id, InstallPhase::Preparing)
+            .await;
+
         // Multi-container stacks get their own install path
         if package_id == "immich" {
             return self.install_immich_stack().await;
@@ -97,14 +232,16 @@ impl RpcHandler {
             return self.install_indeedhub_stack().await;
         }
 
-        // Phase: Preparing — validating deps and configs before any slow I/O.
-        self.set_install_phase(package_id, InstallPhase::Preparing)
-            .await;
-
         // Dependency checks
         let deps = detect_running_deps().await?;
         check_install_deps(package_id, &deps)?;
         log_optional_dep_info(package_id, &deps);
+        check_bitcoin_implementation_conflict(package_id).await?;
+        let repaired_bitcoin_conf = if matches!(package_id, "bitcoin" | "bitcoin-core" | "bitcoin-knots") {
+            ensure_bitcoin_rpc_bindings().await?
+        } else {
+            false
+        };
 
         // Check if container already exists
         let check_output = tokio::process::Command::new("podman")
@@ -145,7 +282,35 @@ impl RpcHandler {
                 .trim()
                 .to_string();
 
-            if state != "running" {
+            if state == "running" && repaired_bitcoin_conf {
+                info!(
+                    "Restarting existing container {} after bitcoin.conf RPC repair",
+                    package_id
+                );
+                let restart_output = tokio::process::Command::new("podman")
+                    .args(["restart", package_id])
+                    .output()
+                    .await
+                    .context("Failed to restart existing container after bitcoin.conf repair")?;
+                if !restart_output.status.success() {
+                    let stderr = String::from_utf8_lossy(&restart_output.stderr);
+                    install_log(&format!(
+                        "INSTALL ADOPT FAIL: {} - restart after RPC repair failed: {}",
+                        package_id, stderr
+                    ))
+                    .await;
+                    return Err(anyhow::anyhow!(
+                        "Container {} exists but failed to restart after RPC repair: {}",
+                        package_id,
+                        stderr
+                    ));
+                }
+                let _ = tokio::process::Command::new("podman")
+                    .args(["restart", "archy-bitcoin-ui"])
+                    .output()
+                    .await;
+                wait_for_adopted_container(package_id, package_id).await?;
+            } else if state != "running" {
                 // Start the stopped/exited container
                 info!("Starting existing container {} (was {})", package_id, state);
                 let start_output = tokio::process::Command::new("podman")
@@ -166,6 +331,8 @@ impl RpcHandler {
                         stderr
                     ));
                 }
+
+                wait_for_adopted_container(package_id, package_id).await?;
             }
 
             install_log(&format!(
@@ -443,7 +610,23 @@ impl RpcHandler {
         let memory_limit = get_memory_limit(package_id);
         let mem_arg = format!("--memory={}", memory_limit);
         run_args.push(&mem_arg);
-        run_args.push("--cpus=2");
+        // Bitcoin (and friends) need every core they can get during initial
+        // blockchain download — script verification is parallelizable and
+        // the limiting factor on most home boxes. --cpus=2 was halving sync
+        // speed for 4-8 core machines. ElectrumX likewise scales with cores
+        // during its initial reorg/indexing phase.
+        let cpu_capped = !matches!(
+            package_id,
+            "bitcoin"
+                | "bitcoin-core"
+                | "bitcoin-knots"
+                | "electrumx"
+                | "electrs"
+                | "mempool-electrs"
+        );
+        if cpu_capped {
+            run_args.push("--cpus=2");
+        }
 
         // Uptime Kuma image entrypoint (`extra/entrypoint.sh`) attempts
         // `setpriv --clear-groups` and fails under our rootless + cap-drop
@@ -956,12 +1139,14 @@ impl RpcHandler {
         // user" and skip. Matches the lnd.conf behavior below.
         match tokio::fs::metadata(&conf_path).await {
             Ok(_) => {
-                info!("bitcoin.conf already exists, skipping write");
+                ensure_bitcoin_rpc_bindings().await?;
+                info!("bitcoin.conf already exists, ensured RPC bind settings");
                 return Ok(());
             }
             Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
             Err(_) => {
-                info!("bitcoin.conf path inaccessible (container-owned data dir), skipping write");
+                ensure_bitcoin_rpc_bindings().await?;
+                info!("bitcoin.conf path inaccessible, ensured RPC bind settings via host helper");
                 return Ok(());
             }
         }
@@ -1366,8 +1551,20 @@ autopilot.active=false\n",
             info!("Nextcloud trusted domains configured for {}", host_ip);
         }
 
-        // Pre-build: inject Bitcoin RPC auth into bitcoin-ui nginx.conf
-        if matches!(package_id, "bitcoin" | "bitcoin-core" | "bitcoin-knots") {
+        // Inject Bitcoin RPC auth into bitcoin-ui nginx.conf.
+        // Two paths because the credential is per-node and randomly generated
+        // at first boot, so it can't be baked into the published registry image:
+        //   1. Build-time: rewrite nginx.conf on disk before `podman build`.
+        //      Only fires when /opt/archipelago/docker/bitcoin-ui exists (dev
+        //      box or ISO that shipped the docker tree). Skipped silently in
+        //      production where ui_builds falls through to the registry image.
+        //   2. Post-start: `podman exec` into the running container to patch
+        //      nginx.conf and reload. Authoritative for both paths — runs
+        //      regardless of how the image was built.
+        let bitcoin_rpc_auth_b64: Option<String> = if matches!(
+            package_id,
+            "bitcoin" | "bitcoin-core" | "bitcoin-knots"
+        ) {
             let (rpc_user, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
             use base64::Engine;
             let auth_b64 = base64::engine::general_purpose::STANDARD
@@ -1377,30 +1574,47 @@ autopilot.active=false\n",
                 "/home/archipelago/archy/docker/bitcoin-ui",
             ] {
                 let conf_path = format!("{}/nginx.conf", dir);
-                if let Ok(content) = tokio::fs::read_to_string(&conf_path).await {
-                    // Replace placeholder or previously-injected auth (regex: Basic followed by base64 or placeholder)
-                    let updated = content
-                        .replace("__BITCOIN_RPC_AUTH__", &auth_b64)
-                        .lines()
-                        .map(|line| {
-                            if line.contains("proxy_set_header Authorization")
-                                && line.contains("Basic")
-                            {
-                                format!(
-                                    "        proxy_set_header Authorization \"Basic {}\";",
-                                    auth_b64
-                                )
-                            } else {
-                                line.to_string()
-                            }
-                        })
-                        .collect::<Vec<_>>()
-                        .join("\n");
-                    let _ = tokio::fs::write(&conf_path, format!("{}\n", updated)).await;
-                    info!("Injected Bitcoin RPC auth into {}", conf_path);
+                match tokio::fs::read_to_string(&conf_path).await {
+                    Ok(content) => {
+                        let updated = content
+                            .replace("__BITCOIN_RPC_AUTH__", &auth_b64)
+                            .lines()
+                            .map(|line| {
+                                if line.contains("proxy_set_header Authorization")
+                                    && line.contains("Basic")
+                                {
+                                    format!(
+                                        "        proxy_set_header Authorization \"Basic {}\";",
+                                        auth_b64
+                                    )
+                                } else {
+                                    line.to_string()
+                                }
+                            })
+                            .collect::<Vec<_>>()
+                            .join("\n");
+                        if let Err(e) = tokio::fs::write(&conf_path, format!("{}\n", updated)).await
+                        {
+                            warn!(
+                                "Failed to write {} with injected RPC auth: {}",
+                                conf_path, e
+                            );
+                        } else {
+                            info!("Injected Bitcoin RPC auth into {} (build-time)", conf_path);
+                        }
+                    }
+                    Err(_) => {
+                        debug!(
+                                "No build-time nginx.conf at {} (will patch running container after start)",
+                                conf_path
+                            );
+                    }
                 }
             }
-        }
+            Some(auth_b64)
+        } else {
+            None
+        };
 
         // Build and start companion UI containers for headless services.
         // All UIs proxy to localhost (backend :5678 or bitcoin :8332) so they need --network=host.
@@ -1437,9 +1651,14 @@ autopilot.active=false\n",
             .find(|d| std::path::Path::new(d).join("Dockerfile").exists())
             .unwrap_or_else(|| ui_dir.to_string());
             let image_base = image_base.to_string();
-            let registry = "git.tx1138.com/lfg2025";
+            let registry = "146.59.87.168:3000/lfg2025";
             let registry_image = format!("{}/{}:latest", registry, image_base);
             let local_image = format!("localhost/{}:latest", image_base);
+            let post_start_auth = if name == "archy-bitcoin-ui" {
+                bitcoin_rpc_auth_b64.clone()
+            } else {
+                None
+            };
             tokio::spawn(async move {
                 // Remove existing container
                 let _ = tokio::process::Command::new("podman")
@@ -1487,32 +1706,69 @@ autopilot.active=false\n",
                     }
                 };
 
+                // For bitcoin-ui specifically: render nginx.conf to host BEFORE
+                // starting the container, then bind-mount it. This is the durable
+                // fix for the bitcoin-rpc 401 — the per-node password is in the
+                // file before nginx ever opens it. Survives container recreate,
+                // image update, reboot, --restart=unless-stopped cycles, and
+                // doesn't need any post-start patching that could fail under
+                // tightly-confined cgroup permissions.
+                let mut bitcoin_ui_mount: Option<String> = None;
+                if name == "archy-bitcoin-ui" {
+                    let paths = crate::container::bitcoin_ui::RenderPaths::default();
+                    match crate::container::bitcoin_ui::render(&paths).await {
+                        Ok(outcome) => {
+                            bitcoin_ui_mount = Some(format!(
+                                "{}:/etc/nginx/conf.d/default.conf:ro,Z",
+                                paths.rendered_path.display()
+                            ));
+                            info!(
+                                "bitcoin-ui nginx.conf rendered ({:?}) — will bind-mount at startup",
+                                outcome
+                            );
+                        }
+                        Err(e) => warn!(
+                            "Failed to render bitcoin-ui nginx.conf: {} — \
+                             will fall back to post-start patch (less reliable)",
+                            e
+                        ),
+                    }
+                }
+
                 // Run with --network=host (UIs proxy to localhost backend/bitcoin)
                 // --user 0:0: run as root inside container (still unprivileged on host
                 // in rootless podman) to avoid nginx chown failures
+                let mut args: Vec<String> = vec![
+                    "run".into(),
+                    "-d".into(),
+                    "--name".into(),
+                    name.clone(),
+                    "--restart=unless-stopped".into(),
+                    "--network=host".into(),
+                    "--user=0:0".into(),
+                    "--cap-drop=ALL".into(),
+                    "--cap-add=CHOWN".into(),
+                    "--cap-add=DAC_OVERRIDE".into(),
+                    "--cap-add=NET_BIND_SERVICE".into(),
+                    "--cap-add=SETUID".into(),
+                    "--cap-add=SETGID".into(),
+                    "--memory=128m".into(),
+                ];
+                if let Some(ref mount) = bitcoin_ui_mount {
+                    args.push("-v".into());
+                    args.push(mount.clone());
+                }
+                args.push(image.clone());
                 let run = tokio::process::Command::new("podman")
-                    .args([
-                        "run",
-                        "-d",
-                        "--name",
-                        &name,
-                        "--restart=unless-stopped",
-                        "--network=host",
-                        "--user=0:0",
-                        "--cap-drop=ALL",
-                        "--cap-add=CHOWN",
-                        "--cap-add=DAC_OVERRIDE",
-                        "--cap-add=NET_BIND_SERVICE",
-                        "--cap-add=SETUID",
-                        "--cap-add=SETGID",
-                        "--memory=128m",
-                        &image,
-                    ])
+                    .args(&args)
                     .output()
                     .await;
                 match run {
                     Ok(o) if o.status.success() => {
-                        info!("{} UI container started (host network)", name)
+                        info!("{} UI container started (host network)", name);
+                        if let Some(ref auth) = post_start_auth {
+                            inject_bitcoin_rpc_auth_into_running_container(&name, auth).await;
+                        }
                     }
                     Ok(o) => warn!(
                         "Failed to start {}: {}",
@@ -1755,13 +2011,146 @@ async fn resolve_host_gateway() -> String {
     "--add-host=host.containers.internal:10.0.2.2".to_string()
 }
 
+async fn wait_for_adopted_container(package_id: &str, container_name: &str) -> Result<()> {
+    for _ in 0..12u32 {
+        tokio::time::sleep(std::time::Duration::from_secs(5)).await;
+        let status = tokio::process::Command::new("podman")
+            .args(["inspect", container_name, "--format", "{{.State.Status}}"])
+            .output()
+            .await;
+        let Ok(output) = status else {
+            continue;
+        };
+        let state = String::from_utf8_lossy(&output.stdout).trim().to_string();
+        match state.as_str() {
+            "running" => return Ok(()),
+            "exited" | "dead" => {
+                let logs = tokio::process::Command::new("podman")
+                    .args(["logs", "--tail", "40", container_name])
+                    .output()
+                    .await;
+                let log_output = logs
+                    .map(|o| {
+                        let stdout = String::from_utf8_lossy(&o.stdout);
+                        let stderr = String::from_utf8_lossy(&o.stderr);
+                        format!("{}{}", stdout, stderr)
+                    })
+                    .unwrap_or_default();
+                install_log(&format!(
+                    "INSTALL ADOPT CRASH: {} - existing container {} exited. Logs:\n{}",
+                    package_id,
+                    container_name,
+                    &log_output.chars().take(1000).collect::<String>()
+                ))
+                .await;
+                return Err(anyhow::anyhow!(
+                    "Existing container {} exited after start. Logs: {}",
+                    container_name,
+                    log_output.chars().take(500).collect::<String>()
+                ));
+            }
+            _ => {}
+        }
+    }
+
+    install_log(&format!(
+        "INSTALL ADOPT TIMEOUT: {} - existing container {} did not stay running",
+        package_id, container_name
+    ))
+    .await;
+    Err(anyhow::anyhow!(
+        "Existing container {} did not reach running state within 60s",
+        container_name
+    ))
+}
+
+async fn ensure_bitcoin_rpc_bindings() -> Result<bool> {
+    let script = r#"
+set -eu
+conf=/var/lib/archipelago/bitcoin/bitcoin.conf
+[ -f "$conf" ] || exit 0
+changed=0
+ensure_line() {
+    line="$1"
+    key="${line%%=*}"
+    if ! grep -q "^${key}=" "$conf"; then
+        printf '%s\n' "$line" >> "$conf"
+        changed=1
+    fi
+}
+ensure_line server=1
+ensure_line rpcbind=0.0.0.0
+ensure_line rpcallowip=0.0.0.0/0
+ensure_line rpcport=8332
+ensure_line listen=1
+[ "$changed" -eq 0 ] && exit 0
+exit 2
+"#;
+    let status = host_sudo(&["sh", "-lc", script])
+        .await
+        .context("ensure bitcoin.conf RPC bind settings")?;
+    match status.code() {
+        Some(0) => Ok(false),
+        Some(2) => {
+            install_log("INSTALL REPAIR: bitcoin.conf RPC bind settings added").await;
+            Ok(true)
+        }
+        _ => Err(anyhow::anyhow!(
+            "bitcoin.conf RPC repair helper exited with {}",
+            status
+        )),
+    }
+}
+
 fn should_try_orchestrator_install(package_id: &str, orchestrator_available: bool) -> bool {
     orchestrator_available && uses_orchestrator_install_flow(package_id)
 }
 
+async fn check_bitcoin_implementation_conflict(package_id: &str) -> Result<()> {
+    let other = match package_id {
+        "bitcoin-core" => "bitcoin-knots",
+        "bitcoin-knots" => "bitcoin-core",
+        _ => return Ok(()),
+    };
+
+    let output = tokio::process::Command::new("podman")
+        .args([
+            "ps",
+            "-a",
+            "--format",
+            "{{.Names}}",
+            "--filter",
+            &format!("name=^{}$", other),
+        ])
+        .output()
+        .await
+        .context("Failed to check existing Bitcoin node containers")?;
+
+    if String::from_utf8_lossy(&output.stdout).trim().is_empty() {
+        return Ok(());
+    }
+
+    let current = match other {
+        "bitcoin-core" => "Bitcoin Core",
+        "bitcoin-knots" => "Bitcoin Knots",
+        _ => "another Bitcoin node",
+    };
+    let requested = match package_id {
+        "bitcoin-core" => "Bitcoin Core",
+        "bitcoin-knots" => "Bitcoin Knots",
+        _ => "the requested Bitcoin node",
+    };
+
+    Err(anyhow::anyhow!(
+        "{} is already installed. Stop and uninstall {} before installing {}; both implementations use the same Bitcoin data directory and ports.",
+        current,
+        current,
+        requested
+    ))
+}
+
 fn orchestrator_install_app_id(package_id: &str) -> &str {
     match package_id {
-        "bitcoin-knots" => "bitcoin-core",
         "electrs" | "mempool-electrs" => "electrumx",
         _ => package_id,
     }
@@ -1847,7 +2236,11 @@ mod tests {
 
     #[test]
     fn install_aliases_map_to_manifest_app_ids() {
-        assert_eq!(orchestrator_install_app_id("bitcoin-knots"), "bitcoin-core");
+        assert_eq!(
+            orchestrator_install_app_id("bitcoin-knots"),
+            "bitcoin-knots"
+        );
+        assert_eq!(orchestrator_install_app_id("bitcoin-core"), "bitcoin-core");
         assert_eq!(orchestrator_install_app_id("electrs"), "electrumx");
         assert_eq!(orchestrator_install_app_id("mempool-electrs"), "electrumx");
         assert_eq!(orchestrator_install_app_id("lnd"), "lnd");

core/archipelago/src/api/rpc/package/progress.rs

@@ -25,6 +25,7 @@ impl RpcHandler {
             size,
             downloaded,
             phase: existing_phase,
+            message: None,
         });
         self.state_manager.update_data(data).await;
     }
@@ -55,6 +56,7 @@ impl RpcHandler {
             size,
             downloaded,
             phase: Some(phase),
+            message: None,
         });
         self.state_manager.update_data(data).await;
     }
@@ -97,6 +99,7 @@ impl RpcHandler {
             size: total,
             downloaded,
             phase: existing_phase,
+            message: None,
         });
         state_manager.update_data(data).await;
     }

core/archipelago/src/api/rpc/package/stacks.rs

@@ -4,6 +4,7 @@
 //! containers in dependency order.
 
 use crate::api::rpc::RpcHandler;
+use crate::data_model::InstallPhase;
 use anyhow::{Context, Result};
 use tracing::info;
 
@@ -47,6 +48,12 @@ async fn adopt_stack_if_exists(
                 .await;
         }
     }
+    let existing: Vec<&str> = all_containers
+        .iter()
+        .copied()
+        .filter(|container| names.iter().any(|n| n == container))
+        .collect();
+    wait_for_stack_containers(stack_name, &existing, 60).await?;
 
     install_log(&format!(
         "INSTALL ADOPT OK: {} — started existing containers",
@@ -60,6 +67,107 @@ async fn adopt_stack_if_exists(
     })))
 }
 
+async fn run_required_stack_command(
+    stack_name: &str,
+    label: &str,
+    cmd: &mut tokio::process::Command,
+) -> Result<()> {
+    let output = cmd
+        .output()
+        .await
+        .with_context(|| format!("{}: failed to run {}", stack_name, label))?;
+    if output.status.success() {
+        return Ok(());
+    }
+
+    let stdout = String::from_utf8_lossy(&output.stdout);
+    let stderr = String::from_utf8_lossy(&output.stderr);
+    let msg = format!("{} failed: {}{}", label, stdout, stderr);
+    install_log(&format!("INSTALL FAIL: {} - {}", stack_name, msg.trim())).await;
+    Err(anyhow::anyhow!("{} {}", stack_name, msg.trim()))
+}
+
+async fn wait_for_stack_containers(
+    stack_name: &str,
+    containers: &[&str],
+    timeout_secs: u64,
+) -> Result<()> {
+    let deadline = std::time::Instant::now() + std::time::Duration::from_secs(timeout_secs);
+    loop {
+        let mut pending = Vec::new();
+        for container in containers {
+            let status = tokio::process::Command::new("podman")
+                .args(["inspect", container, "--format", "{{.State.Status}}"])
+                .output()
+                .await;
+            match status {
+                Ok(output) if output.status.success() => {
+                    let state = String::from_utf8_lossy(&output.stdout).trim().to_string();
+                    match state.as_str() {
+                        "running" => {}
+                        "exited" | "dead" => {
+                            let logs = stack_container_logs(container, 40).await;
+                            install_log(&format!(
+                                "INSTALL CRASH: {} - container {} exited. Logs:\n{}",
+                                stack_name,
+                                container,
+                                logs.chars().take(1000).collect::<String>()
+                            ))
+                            .await;
+                            return Err(anyhow::anyhow!(
+                                "{} container {} exited after install. Logs: {}",
+                                stack_name,
+                                container,
+                                logs.chars().take(500).collect::<String>()
+                            ));
+                        }
+                        other => pending.push(format!("{}={}", container, other)),
+                    }
+                }
+                Ok(output) => {
+                    pending.push(format!(
+                        "{}=missing({})",
+                        container,
+                        String::from_utf8_lossy(&output.stderr).trim()
+                    ));
+                }
+                Err(e) => pending.push(format!("{}=inspect-error({})", container, e)),
+            }
+        }
+
+        if pending.is_empty() {
+            return Ok(());
+        }
+        if std::time::Instant::now() >= deadline {
+            install_log(&format!(
+                "INSTALL TIMEOUT: {} - containers not running: {}",
+                stack_name,
+                pending.join(", ")
+            ))
+            .await;
+            return Err(anyhow::anyhow!(
+                "{} containers did not reach running state: {}",
+                stack_name,
+                pending.join(", ")
+            ));
+        }
+        tokio::time::sleep(std::time::Duration::from_secs(2)).await;
+    }
+}
+
+async fn stack_container_logs(container: &str, lines: u32) -> String {
+    tokio::process::Command::new("podman")
+        .args(["logs", "--tail", &lines.to_string(), container])
+        .output()
+        .await
+        .map(|o| {
+            let stdout = String::from_utf8_lossy(&o.stdout);
+            let stderr = String::from_utf8_lossy(&o.stderr);
+            format!("{}{}", stdout, stderr)
+        })
+        .unwrap_or_default()
+}
+
 async fn install_stack_via_orchestrator(
     handler: &RpcHandler,
     stack_name: &str,
@@ -124,7 +232,7 @@ fn mempool_stack_app_ids() -> &'static [&'static str] {
     &["archy-mempool-db", "mempool-api", "archy-mempool-web"]
 }
 
-const REGISTRY: &str = "git.tx1138.com/lfg2025";
+const REGISTRY: &str = "146.59.87.168:3000/lfg2025";
 
 /// Pull an image with retry and exponential backoff (3 attempts).
 async fn pull_image_with_retry(image: &str) -> Result<()> {
@@ -199,13 +307,22 @@ impl RpcHandler {
         }
 
         let images = [
-            "git.tx1138.com/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
-            "docker.io/valkey/valkey:7-alpine",
-            "git.tx1138.com/lfg2025/immich-server:release",
+            "146.59.87.168:3000/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
+            "146.59.87.168:3000/lfg2025/valkey:7-alpine",
+            "146.59.87.168:3000/lfg2025/immich-server:release",
         ];
-        for img in &images {
+        self.set_install_phase("immich", InstallPhase::PullingImage)
+            .await;
+        let n_images = images.len() as u64;
+        for (i, img) in images.iter().enumerate() {
+            self.set_install_progress("immich", i as u64, n_images)
+                .await;
             pull_image_with_retry(img).await?;
         }
+        self.set_install_progress("immich", n_images, n_images)
+            .await;
+        self.set_install_phase("immich", InstallPhase::CreatingContainer)
+            .await;
 
         let _ = tokio::process::Command::new("sudo")
             .args([
@@ -265,7 +382,7 @@ impl RpcHandler {
                 "POSTGRES_USER=postgres",
                 "-e",
                 "POSTGRES_DB=immich",
-                "git.tx1138.com/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
+                "146.59.87.168:3000/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
             ])
             .output()
             .await;
@@ -292,7 +409,7 @@ impl RpcHandler {
                 "--health-cmd=valkey-cli ping || exit 1",
                 "--health-interval=30s",
                 "--health-retries=3",
-                "docker.io/valkey/valkey:7-alpine",
+                "146.59.87.168:3000/lfg2025/valkey:7-alpine",
             ])
             .output()
             .await;
@@ -330,7 +447,7 @@ impl RpcHandler {
                 "REDIS_HOSTNAME=immich_redis",
                 "-e",
                 "UPLOAD_LOCATION=/usr/src/app/upload",
-                "git.tx1138.com/lfg2025/immich-server:release",
+                "146.59.87.168:3000/lfg2025/immich-server:release",
             ])
             .output()
             .await
@@ -341,6 +458,13 @@ impl RpcHandler {
             return Err(anyhow::anyhow!("Failed to start Immich server: {}", stderr));
         }
 
+        self.set_install_phase("immich", InstallPhase::WaitingHealthy)
+            .await;
+        self.set_install_phase("immich", InstallPhase::PostInstall)
+            .await;
+        self.set_install_phase("immich", InstallPhase::Done).await;
+        self.clear_install_progress("immich").await;
+
         info!("Immich stack installed and started");
         Ok(serde_json::json!({
             "success": true,
@@ -384,9 +508,18 @@ impl RpcHandler {
             &format!("{}/nbxplorer:2.6.0", REGISTRY),
             &format!("{}/btcpayserver:1.13.7", REGISTRY),
         ];
-        for img in &images {
+        self.set_install_phase("btcpay-server", InstallPhase::PullingImage)
+            .await;
+        let n_images = images.len() as u64;
+        for (i, img) in images.iter().enumerate() {
+            self.set_install_progress("btcpay-server", i as u64, n_images)
+                .await;
             pull_image_with_retry(img).await?;
         }
+        self.set_install_progress("btcpay-server", n_images, n_images)
+            .await;
+        self.set_install_phase("btcpay-server", InstallPhase::CreatingContainer)
+            .await;
 
         // Create data dirs (chown to current user so rootless podman can write)
         let _ = tokio::process::Command::new("sudo")
@@ -541,6 +674,14 @@ impl RpcHandler {
             return Err(anyhow::anyhow!("Failed to start BTCPay Server: {}", stderr));
         }
 
+        self.set_install_phase("btcpay-server", InstallPhase::WaitingHealthy)
+            .await;
+        self.set_install_phase("btcpay-server", InstallPhase::PostInstall)
+            .await;
+        self.set_install_phase("btcpay-server", InstallPhase::Done)
+            .await;
+        self.clear_install_progress("btcpay-server").await;
+
         install_log("INSTALL OK: btcpay-server stack").await;
         info!("BTCPay stack installed and started");
         Ok(serde_json::json!({
@@ -590,9 +731,18 @@ impl RpcHandler {
             &format!("{}/mempool-backend:v3.0.0", REGISTRY),
             &format!("{}/mempool-frontend:v3.0.0", REGISTRY),
         ];
-        for img in &images {
+        self.set_install_phase("mempool", InstallPhase::PullingImage)
+            .await;
+        let n_images = images.len() as u64;
+        for (i, img) in images.iter().enumerate() {
+            self.set_install_progress("mempool", i as u64, n_images)
+                .await;
             pull_image_with_retry(img).await?;
         }
+        self.set_install_progress("mempool", n_images, n_images)
+            .await;
+        self.set_install_phase("mempool", InstallPhase::CreatingContainer)
+            .await;
 
         // Create data dirs (chown to current user so rootless podman can write)
         let _ = tokio::process::Command::new("sudo")
@@ -750,6 +900,13 @@ impl RpcHandler {
             return Err(anyhow::anyhow!("Failed to start Mempool: {}", stderr));
         }
 
+        self.set_install_phase("mempool", InstallPhase::WaitingHealthy)
+            .await;
+        self.set_install_phase("mempool", InstallPhase::PostInstall)
+            .await;
+        self.set_install_phase("mempool", InstallPhase::Done).await;
+        self.clear_install_progress("mempool").await;
+
         install_log("INSTALL OK: mempool stack").await;
         info!("Mempool stack installed and started");
         Ok(serde_json::json!({
@@ -769,7 +926,7 @@ impl RpcHandler {
             .into_iter()
             .find(|r| r.enabled)
             .map(|r| r.url)
-            .unwrap_or_else(|| "git.tx1138.com/lfg2025".to_string());
+            .unwrap_or_else(|| "146.59.87.168:3000/lfg2025".to_string());
 
         let user_tmp = format!(
             "{}/.local/share/containers/tmp",
@@ -794,12 +951,22 @@ impl RpcHandler {
         // Pull all images with retry; fail the install if any image can't be pulled.
         // Previously this just logged a warning and continued, leaving the stack
         // broken and the user seeing "failed" with no recovery path.
-        for img in &images {
+        self.set_install_phase("indeedhub", InstallPhase::PullingImage)
+            .await;
+        let n_images = images.len() as u64;
+        for (i, img) in images.iter().enumerate() {
+            // set_install_progress fills the byte-counter fallback the UI uses
+            // when it can't read podman's pull output — gives the bar a clear
+            // X-of-N step as each image lands.
+            self.set_install_progress("indeedhub", i as u64, n_images)
+                .await;
             info!("Pulling {}", img);
             pull_image_with_retry(img)
                 .await
                 .with_context(|| format!("Failed to pull IndeedHub image: {}", img))?;
         }
+        self.set_install_progress("indeedhub", n_images, n_images)
+            .await;
 
         // Remove any leftover containers from a previous partial install (or
         // from the first-boot frontend stub that used to race the installer).
@@ -826,6 +993,12 @@ impl RpcHandler {
             .status()
             .await;
 
+        // Phase: CreatingContainer — pulls done, network rebuilt, now spinning
+        // up the 7 stack containers. Bar advances from PullingImage band into
+        // CreatingContainer band so the user sees movement.
+        self.set_install_phase("indeedhub", InstallPhase::CreatingContainer)
+            .await;
+
         // Create indeedhub-net
         let _ = tokio::process::Command::new("podman")
             .args(["network", "create", "indeedhub-net"])
@@ -839,7 +1012,8 @@ impl RpcHandler {
         let minio_pass = super::config::read_or_generate_secret("indeedhub-minio-password").await;
 
         // 1. Postgres
-        let _ = tokio::process::Command::new("podman")
+        let mut postgres_cmd = tokio::process::Command::new("podman");
+        postgres_cmd
             .args([
                 "run",
                 "-d",
@@ -861,12 +1035,12 @@ impl RpcHandler {
                 "indeedhub-postgres-data:/var/lib/postgresql/data",
                 &format!("{}/postgres:16.13-alpine", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create postgres", &mut postgres_cmd).await?;
 
         // 2. Redis
-        let _ = tokio::process::Command::new("podman")
+        let mut redis_cmd = tokio::process::Command::new("podman");
+        redis_cmd
             .args([
                 "run",
                 "-d",
@@ -882,12 +1056,12 @@ impl RpcHandler {
                 "indeedhub-redis-data:/data",
                 &format!("{}/redis:7.4.8-alpine", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create redis", &mut redis_cmd).await?;
 
         // 3. MinIO
-        let _ = tokio::process::Command::new("podman")
+        let mut minio_cmd = tokio::process::Command::new("podman");
+        minio_cmd
             .args([
                 "run",
                 "-d",
@@ -909,12 +1083,12 @@ impl RpcHandler {
                 "server",
                 "/data",
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create minio", &mut minio_cmd).await?;
 
         // 4. Nostr relay
-        let _ = tokio::process::Command::new("podman")
+        let mut relay_cmd = tokio::process::Command::new("podman");
+        relay_cmd
             .args([
                 "run",
                 "-d",
@@ -930,12 +1104,12 @@ impl RpcHandler {
                 "indeedhub-relay-data:/usr/src/app/db",
                 &format!("{}/nostr-rs-relay:0.9.0", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create relay", &mut relay_cmd).await?;
 
         // 5. API
-        let _ = tokio::process::Command::new("podman")
+        let mut api_cmd = tokio::process::Command::new("podman");
+        api_cmd
             .args([
                 "run",
                 "-d",
@@ -952,35 +1126,47 @@ impl RpcHandler {
                 "-e",
                 "DATABASE_HOST=postgres",
                 "-e",
+                "DATABASE_PORT=5432",
+                "-e",
                 "DATABASE_USER=indeedhub",
                 "-e",
                 &format!("DATABASE_PASSWORD={}", db_pass),
                 "-e",
                 "DATABASE_NAME=indeedhub",
                 "-e",
-                "REDIS_HOST=redis",
+                "QUEUE_HOST=redis",
+                "-e",
+                "QUEUE_PORT=6379",
                 "-e",
                 "S3_ENDPOINT=http://minio:9000",
                 "-e",
+                "AWS_REGION=us-east-1",
+                "-e",
                 &format!("AWS_ACCESS_KEY={}", minio_user),
                 "-e",
                 &format!("AWS_SECRET_KEY={}", minio_pass),
                 "-e",
                 "S3_PUBLIC_BUCKET_NAME=indeedhub-public",
                 "-e",
+                "S3_PRIVATE_BUCKET_NAME=indeedhub-private",
+                "-e",
                 "S3_PUBLIC_BUCKET_URL=/storage",
                 "-e",
                 &format!("NOSTR_JWT_SECRET={}", jwt_secret),
                 "-e",
+                "NOSTR_JWT_EXPIRES_IN=7d",
+                "-e",
+                "AES_MASTER_SECRET=0123456789abcdef0123456789abcdef",
+                "-e",
                 "ENVIRONMENT=production",
                 &format!("{}/indeedhub-api:1.0.0", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create api", &mut api_cmd).await?;
 
         // 6. FFmpeg worker
-        let _ = tokio::process::Command::new("podman")
+        let mut ffmpeg_cmd = tokio::process::Command::new("podman");
+        ffmpeg_cmd
             .args([
                 "run",
                 "-d",
@@ -993,6 +1179,8 @@ impl RpcHandler {
                 "-e",
                 "DATABASE_HOST=postgres",
                 "-e",
+                "DATABASE_PORT=5432",
+                "-e",
                 "DATABASE_USER=indeedhub",
                 "-e",
                 &format!("DATABASE_PASSWORD={}", db_pass),
@@ -1001,6 +1189,8 @@ impl RpcHandler {
                 "-e",
                 "QUEUE_HOST=redis",
                 "-e",
+                "QUEUE_PORT=6379",
+                "-e",
                 "S3_ENDPOINT=http://minio:9000",
                 "-e",
                 &format!("AWS_ACCESS_KEY={}", minio_user),
@@ -1011,20 +1201,22 @@ impl RpcHandler {
                 "-e",
                 "S3_PUBLIC_BUCKET_NAME=indeedhub-public",
                 "-e",
+                "S3_PRIVATE_BUCKET_NAME=indeedhub-private",
+                "-e",
                 "ENVIRONMENT=production",
                 "-e",
                 "AES_MASTER_SECRET=0123456789abcdef0123456789abcdef",
                 &format!("{}/indeedhub-ffmpeg:1.0.0", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .status()
-            .await;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create ffmpeg worker", &mut ffmpeg_cmd).await?;
 
         // Wait for backend services to start
         tokio::time::sleep(std::time::Duration::from_secs(5)).await;
 
         // 7. Frontend (nginx)
-        let run = tokio::process::Command::new("podman")
+        let mut frontend_cmd = tokio::process::Command::new("podman");
+        frontend_cmd
             .args([
                 "run",
                 "-d",
@@ -1038,15 +1230,35 @@ impl RpcHandler {
                 "7778:7777",
                 &format!("{}/indeedhub:1.0.0", registry),
             ])
-            .env("TMPDIR", &user_tmp)
-            .output()
-            .await
-            .context("Failed to create indeedhub container")?;
+            .env("TMPDIR", &user_tmp);
+        run_required_stack_command("indeedhub", "create frontend", &mut frontend_cmd).await?;
 
-        if !run.status.success() {
-            let err = String::from_utf8_lossy(&run.stderr);
-            return Err(anyhow::anyhow!("IndeedHub frontend failed: {}", err));
-        }
+        wait_for_stack_containers(
+            "indeedhub",
+            &[
+                "indeedhub-postgres",
+                "indeedhub-redis",
+                "indeedhub-minio",
+                "indeedhub-relay",
+                "indeedhub-api",
+                "indeedhub-ffmpeg",
+                "indeedhub",
+            ],
+            60,
+        )
+        .await?;
+
+        // Phase: WaitingHealthy → PostInstall → clear. The actual readiness
+        // gate is the package scanner's next sweep; this just gives the UI a
+        // truthful end-of-install signal so the bar settles at 95→100→done
+        // instead of sitting at "Queued… 2%" forever.
+        self.set_install_phase("indeedhub", InstallPhase::WaitingHealthy)
+            .await;
+        self.set_install_phase("indeedhub", InstallPhase::PostInstall)
+            .await;
+        self.set_install_phase("indeedhub", InstallPhase::Done)
+            .await;
+        self.clear_install_progress("indeedhub").await;
 
         install_log("INSTALL OK: indeedhub stack").await;
         info!("IndeedHub stack installed");

core/archipelago/src/api/rpc/package/update.rs

@@ -453,7 +453,7 @@ fn candidate_app_ids_for_container(container_name: &str) -> Vec<String> {
     };
 
     match container_name {
-        "bitcoin-knots" => {
+        "bitcoin-knots" | "bitcoin-core" => {
             push("bitcoin-core");
             push("bitcoin-knots");
         }

core/archipelago/src/api/rpc/tor/mod.rs

@@ -355,7 +355,7 @@ pub(in crate::api::rpc) fn known_service_port(name: &str) -> u16 {
         "penpot" => 9001,
         "nginx-proxy-manager" => 81,
         "vaultwarden" => 8343,
-        "indeedhub" => 7777,
+        "indeedhub" => 7778,
         _ => 0,
     }
 }

core/archipelago/src/bitcoin_status.rs

@@ -0,0 +1,186 @@
+//! Cached Bitcoin node status for browser UIs.
+//!
+//! The bitcoin-ui should not poll Bitcoin RPC directly for display state.
+//! During container restarts, reindexing, and IBD, direct browser RPC polling
+//! turns short RPC gaps into visible UI failures. This module owns the RPC
+//! polling loop, caches the last successful snapshot, and serves stale-but-known
+//! state while the node is reconnecting.
+
+use anyhow::{Context, Result};
+use serde::Serialize;
+use std::sync::OnceLock;
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tokio::sync::RwLock;
+use tracing::{debug, warn};
+
+const CACHE_REFRESH_SECS: u64 = 5;
+
+#[derive(Debug, Clone, Serialize)]
+pub struct BitcoinNodeStatus {
+    pub ok: bool,
+    pub stale: bool,
+    pub updated_at_ms: u64,
+    pub error: Option<String>,
+    pub blockchain_info: Option<serde_json::Value>,
+    pub network_info: Option<serde_json::Value>,
+    pub index_info: Option<serde_json::Value>,
+    pub zmq_notifications: Option<serde_json::Value>,
+}
+
+impl Default for BitcoinNodeStatus {
+    fn default() -> Self {
+        Self {
+            ok: false,
+            stale: false,
+            updated_at_ms: 0,
+            error: Some("Connecting to Bitcoin node...".to_string()),
+            blockchain_info: None,
+            network_info: None,
+            index_info: None,
+            zmq_notifications: None,
+        }
+    }
+}
+
+static STATUS_CACHE: OnceLock<RwLock<BitcoinNodeStatus>> = OnceLock::new();
+
+fn cache() -> &'static RwLock<BitcoinNodeStatus> {
+    STATUS_CACHE.get_or_init(|| RwLock::new(BitcoinNodeStatus::default()))
+}
+
+fn now_ms() -> u64 {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .unwrap_or_default()
+        .as_millis() as u64
+}
+
+fn transient_error(err_msg: &str) -> bool {
+    let lower = err_msg.to_lowercase();
+    lower.contains("connect")
+        || lower.contains("reset")
+        || lower.contains("refused")
+        || lower.contains("timed out")
+        || lower.contains("timeout")
+        || lower.contains("broken pipe")
+        || lower.contains("eof")
+        || lower.contains("500 internal server error")
+}
+
+pub fn spawn_status_cache() {
+    tokio::spawn(async {
+        loop {
+            let fresh = fetch_bitcoin_status().await;
+            let mut cached = cache().write().await;
+            match fresh {
+                Ok(mut status) => {
+                    status.ok = true;
+                    status.stale = false;
+                    status.error = None;
+                    *cached = status;
+                }
+                Err(e) => {
+                    let err_msg = e.to_string();
+                    if transient_error(&err_msg) {
+                        debug!("Bitcoin status: transient RPC failure: {}", err_msg);
+                    } else {
+                        warn!("Bitcoin status: RPC failure: {}", err_msg);
+                    }
+
+                    if cached.blockchain_info.is_some() {
+                        cached.ok = false;
+                        cached.stale = true;
+                        cached.error = Some(format!(
+                            "Bitcoin node is reconnecting; showing last known state: {}",
+                            err_msg
+                        ));
+                    } else {
+                        *cached = BitcoinNodeStatus {
+                            ok: false,
+                            stale: false,
+                            updated_at_ms: now_ms(),
+                            error: Some(format!("Connecting to Bitcoin node: {}", err_msg)),
+                            ..BitcoinNodeStatus::default()
+                        };
+                    }
+                }
+            }
+            drop(cached);
+            tokio::time::sleep(Duration::from_secs(CACHE_REFRESH_SECS)).await;
+        }
+    });
+}
+
+pub async fn get_bitcoin_status() -> BitcoinNodeStatus {
+    cache().read().await.clone()
+}
+
+async fn fetch_bitcoin_status() -> Result<BitcoinNodeStatus> {
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(8))
+        .build()
+        .context("build Bitcoin status HTTP client")?;
+
+    let blockchain_info = bitcoin_rpc_call(&client, "getblockchaininfo", serde_json::json!([]))
+        .await
+        .context("getblockchaininfo")?;
+    let network_info = bitcoin_rpc_call(&client, "getnetworkinfo", serde_json::json!([]))
+        .await
+        .context("getnetworkinfo")
+        .ok();
+    let index_info = bitcoin_rpc_call(&client, "getindexinfo", serde_json::json!([]))
+        .await
+        .context("getindexinfo")
+        .ok();
+    let zmq_notifications =
+        bitcoin_rpc_call(&client, "getzmqnotifications", serde_json::json!([]))
+            .await
+            .context("getzmqnotifications")
+            .ok();
+
+    Ok(BitcoinNodeStatus {
+        ok: true,
+        stale: false,
+        updated_at_ms: now_ms(),
+        error: None,
+        blockchain_info: Some(blockchain_info),
+        network_info,
+        index_info,
+        zmq_notifications,
+    })
+}
+
+async fn bitcoin_rpc_call(
+    client: &reqwest::Client,
+    method: &str,
+    params: serde_json::Value,
+) -> Result<serde_json::Value> {
+    let (rpc_user, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
+    let body = serde_json::json!({
+        "jsonrpc": "1.0",
+        "id": "bitcoin-status",
+        "method": method,
+        "params": params,
+    });
+
+    let resp = client
+        .post(crate::constants::BITCOIN_RPC_URL)
+        .basic_auth(rpc_user, Some(rpc_pass))
+        .header("Content-Type", "application/json")
+        .json(&body)
+        .send()
+        .await
+        .context("Bitcoin RPC request failed")?;
+
+    let status = resp.status();
+    let json: serde_json::Value = resp.json().await.context("decode Bitcoin RPC JSON")?;
+    if !status.is_success() {
+        anyhow::bail!("Bitcoin RPC returned {}: {}", status, json);
+    }
+    if let Some(error) = json.get("error").filter(|e| !e.is_null()) {
+        anyhow::bail!("Bitcoin RPC {} error: {}", method, error);
+    }
+    json.get("result")
+        .cloned()
+        .context("missing Bitcoin RPC result")
+}

core/archipelago/src/bootstrap.rs

@@ -15,7 +15,7 @@
 //! best-effort — failures are logged but never abort the backend.
 
 use anyhow::{Context, Result};
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use tokio::fs;
 use tracing::{debug, info, warn};
 
@@ -31,6 +31,7 @@ const DOCTOR_SERVICE_PATH: &str = "/etc/systemd/system/archipelago-doctor.servic
 const DOCTOR_TIMER_PATH: &str = "/etc/systemd/system/archipelago-doctor.timer";
 
 const NGINX_CONF_PATH: &str = "/etc/nginx/sites-available/archipelago";
+const RUNTIME_ASSETS_DIR: &str = "/opt/archipelago/web-ui/archipelago-runtime";
 
 /// Inserted into every server block of the nginx config that lacks the
 /// `/api/app-catalog` proxy. Kept in sync with the canonical block in
@@ -40,6 +41,11 @@ const NGINX_APP_CATALOG_BLOCK: &str = "\n    # App Store catalog proxy — backe
 /// Entry point called from main startup. Never returns an error to the caller —
 /// failing to bootstrap host artifacts must not prevent the backend from serving.
 pub async fn ensure_doctor_installed() {
+    match run_runtime_assets().await {
+        Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
+        Ok(_) => debug!("No OTA runtime payload to synchronize"),
+        Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
+    }
     match run().await {
         Ok(changed) if changed => info!("Doctor artifacts synchronized with binary"),
         Ok(_) => debug!("Doctor artifacts already in sync"),
@@ -50,6 +56,170 @@ pub async fn ensure_doctor_installed() {
         Ok(false) => debug!("Nginx already has /api/app-catalog block"),
         Err(e) => warn!("Nginx bootstrap failed (non-fatal): {:#}", e),
     }
+    match run_bitcoin_rpc_repair().await {
+        Ok(true) => info!("Repaired Bitcoin RPC bind settings and restarted Bitcoin containers"),
+        Ok(false) => debug!("Bitcoin RPC bind settings already usable"),
+        Err(e) => warn!("Bitcoin RPC repair failed (non-fatal): {:#}", e),
+    }
+}
+
+async fn run_runtime_assets() -> Result<bool> {
+    // The v1.7.50 OTA bridge puts scripts/apps/docker assets inside the
+    // frontend tarball because older binaries only know how to apply the
+    // backend binary and frontend archive. Once the new backend starts, it
+    // promotes that payload into /opt so app installs use the matching specs.
+    let runtime_dir = Path::new(RUNTIME_ASSETS_DIR);
+    if !runtime_dir.exists() {
+        return Ok(false);
+    }
+
+    let mut changed = false;
+    for (relative, dest) in [
+        ("apps", "/opt/archipelago/apps"),
+        ("scripts", "/opt/archipelago/scripts"),
+        ("docker", "/opt/archipelago/docker"),
+    ] {
+        let src = runtime_dir.join(relative);
+        if src.exists() {
+            replace_dir_from_runtime(&src, dest).await?;
+            if relative == "scripts" {
+                let _ = host_sudo(&[
+                    "find", dest, "-type", "f", "-name", "*.sh", "-exec", "chmod", "755", "{}", "+",
+                ])
+                .await;
+                let image_versions = format!("{}/image-versions.sh", dest);
+                if Path::new(&image_versions).exists() {
+                    let _ =
+                        host_sudo(&["cp", &image_versions, "/opt/archipelago/image-versions.sh"])
+                            .await;
+                }
+            }
+            changed = true;
+        }
+    }
+
+    let configs = runtime_dir.join("image-recipe/configs");
+    for unit in ["archipelago-doctor.service", "archipelago-doctor.timer"] {
+        let src = configs.join(unit);
+        if src.exists() {
+            let src_s = src.to_string_lossy().to_string();
+            let dest = format!("/etc/systemd/system/{}", unit);
+            let status = host_sudo(&["install", "-m", "644", &src_s, &dest])
+                .await
+                .with_context(|| format!("install {}", unit))?;
+            if !status.success() {
+                anyhow::bail!("install {} exited with {}", unit, status);
+            }
+            changed = true;
+        }
+    }
+
+    if changed {
+        let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
+        let _ = host_sudo(&["systemctl", "enable", "--now", "archipelago-doctor.timer"]).await;
+    }
+    Ok(changed)
+}
+
+async fn replace_dir_from_runtime(src: &Path, dest: &str) -> Result<()> {
+    let tmp = format!("{}.new.{}", dest, chrono::Utc::now().timestamp_millis());
+    let src_dot = path_dot(src);
+    let mkdir = host_sudo(&["mkdir", "-p", &tmp])
+        .await
+        .with_context(|| format!("mkdir {}", tmp))?;
+    if !mkdir.success() {
+        anyhow::bail!("mkdir {} exited with {}", tmp, mkdir);
+    }
+    let copy = host_sudo(&["cp", "-a", &src_dot, &tmp])
+        .await
+        .with_context(|| format!("copy runtime {} -> {}", src.display(), tmp))?;
+    if !copy.success() {
+        let _ = host_sudo(&["rm", "-rf", &tmp]).await;
+        anyhow::bail!("copy runtime {} exited with {}", src.display(), copy);
+    }
+    let _ = host_sudo(&["mkdir", "-p", dest]).await;
+    let cleanup = host_sudo(&[
+        "find",
+        dest,
+        "-mindepth",
+        "1",
+        "-maxdepth",
+        "1",
+        "-exec",
+        "rm",
+        "-rf",
+        "{}",
+        "+",
+    ])
+    .await
+    .with_context(|| format!("clean {}", dest))?;
+    if !cleanup.success() {
+        let _ = host_sudo(&["rm", "-rf", &tmp]).await;
+        anyhow::bail!("clean {} exited with {}", dest, cleanup);
+    }
+    let tmp_dot = format!("{}/.", tmp);
+    let promote = host_sudo(&["cp", "-a", &tmp_dot, dest])
+        .await
+        .with_context(|| format!("promote {} -> {}", tmp, dest))?;
+    let _ = host_sudo(&["rm", "-rf", &tmp]).await;
+    if !promote.success() {
+        anyhow::bail!("promote {} exited with {}", dest, promote);
+    }
+    Ok(())
+}
+
+fn path_dot(path: &Path) -> String {
+    let mut p = PathBuf::from(path);
+    p.push(".");
+    p.to_string_lossy().to_string()
+}
+
+async fn run_bitcoin_rpc_repair() -> Result<bool> {
+    // Older installs can have a container-owned bitcoin.conf with only rpcauth
+    // and printtoconsole. In that state bitcoind is healthy internally, but the
+    // host-network bitcoin-ui proxy to 127.0.0.1:8332 gets connection resets.
+    // Repair it at startup so OTA fixes existing nodes without a manual
+    // uninstall/reinstall.
+    let script = r#"
+set -eu
+conf=/var/lib/archipelago/bitcoin/bitcoin.conf
+[ -f "$conf" ] || exit 0
+changed=0
+ensure_line() {
+    line="$1"
+    key="${line%%=*}"
+    if ! grep -q "^${key}=" "$conf"; then
+        printf '%s\n' "$line" >> "$conf"
+        changed=1
+    fi
+}
+ensure_line server=1
+ensure_line rpcbind=0.0.0.0
+ensure_line rpcallowip=0.0.0.0/0
+ensure_line rpcport=8332
+ensure_line listen=1
+[ "$changed" -eq 0 ] && exit 0
+exit 2
+"#;
+    let status = host_sudo(&["sh", "-lc", script])
+        .await
+        .context("repair bitcoin.conf RPC bind settings")?;
+    match status.code() {
+        Some(0) => Ok(false),
+        Some(2) => {
+            for name in ["bitcoin-knots", "bitcoin-core", "archy-bitcoin-ui"] {
+                let _ = tokio::process::Command::new("podman")
+                    .args(["restart", name])
+                    .status()
+                    .await;
+            }
+            Ok(true)
+        }
+        _ => {
+            warn!("Bitcoin RPC repair helper exited with {}", status);
+            Ok(false)
+        }
+    }
 }
 
 async fn run() -> Result<bool> {

core/archipelago/src/container/bitcoin_ui_nginx.conf.template

@@ -15,5 +15,13 @@ server {
         add_header Access-Control-Allow-Headers "Content-Type, Authorization";
         if ($request_method = OPTIONS) { return 204; }
     }
+    location /bitcoin-status {
+        proxy_pass http://127.0.0.1:5678/bitcoin-status;
+        proxy_http_version 1.1;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        add_header Cache-Control "no-store";
+    }
     location / { try_files $uri $uri/ /index.html; }
 }

core/archipelago/src/container/registry.rs

@@ -11,6 +11,8 @@ use tokio::fs;
 use tracing::{debug, info};
 
 const REGISTRY_FILE: &str = "config/registries.json";
+const OVH_REGISTRY_URL: &str = "146.59.87.168:3000/lfg2025";
+const TX1138_REGISTRY_URL: &str = "git.tx1138.com/lfg2025";
 
 /// A single container registry.
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -44,14 +46,14 @@ impl Default for RegistryConfig {
         Self {
             registries: vec![
                 Registry {
-                    url: "146.59.87.168:3000/lfg2025".to_string(),
+                    url: OVH_REGISTRY_URL.to_string(),
                     name: "Server 1 (OVH)".to_string(),
                     tls_verify: false,
                     enabled: true,
                     priority: 0,
                 },
                 Registry {
-                    url: "git.tx1138.com/lfg2025".to_string(),
+                    url: TX1138_REGISTRY_URL.to_string(),
                     name: "Server 2 (tx1138)".to_string(),
                     tls_verify: true,
                     enabled: true,
@@ -139,6 +141,19 @@ pub async fn load_registries(data_dir: &Path) -> Result<RegistryConfig> {
             changed = true;
         }
     }
+    let before_order: Vec<(String, bool, u32)> = config
+        .registries
+        .iter()
+        .map(|r| (r.url.clone(), r.enabled, r.priority))
+        .collect();
+    force_ovh_registry_primary(&mut config);
+    changed = changed
+        || before_order
+            != config
+                .registries
+                .iter()
+                .map(|r| (r.url.clone(), r.enabled, r.priority))
+                .collect::<Vec<_>>();
     if changed {
         // Persist so the next load doesn't have to re-merge.
         let _ = save_registries(data_dir, &config).await;
@@ -146,6 +161,37 @@ pub async fn load_registries(data_dir: &Path) -> Result<RegistryConfig> {
     Ok(config)
 }
 
+fn force_ovh_registry_primary(config: &mut RegistryConfig) {
+    let defaults = RegistryConfig::default();
+    for def in defaults.registries {
+        if !config.registries.iter().any(|r| r.url == def.url) {
+            config.registries.push(def);
+        }
+    }
+
+    for registry in config.registries.iter_mut() {
+        match registry.url.as_str() {
+            OVH_REGISTRY_URL => {
+                registry.name = "Server 1 (OVH)".to_string();
+                registry.tls_verify = false;
+                registry.enabled = true;
+                registry.priority = 0;
+            }
+            TX1138_REGISTRY_URL => {
+                registry.name = "Server 2 (tx1138)".to_string();
+                registry.tls_verify = true;
+                registry.enabled = true;
+                registry.priority = 10;
+            }
+            _ => {
+                if registry.priority <= 10 {
+                    registry.priority = registry.priority.saturating_add(20);
+                }
+            }
+        }
+    }
+}
+
 /// Save registry config to disk.
 pub async fn save_registries(data_dir: &Path, config: &RegistryConfig) -> Result<()> {
     let dir = data_dir.join("config");

core/archipelago/src/data_model.rs

@@ -255,6 +255,12 @@ pub struct InstallProgress {
     /// a fixed UI percentage and a descriptive label.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub phase: Option<InstallPhase>,
+    /// Optional explicit message — used to surface install failures so
+    /// the UI can keep the app card visible with an error description
+    /// instead of silently removing the entry on fail. UI's PHASE_INFO
+    /// label takes precedence when phase is set.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub message: Option<String>,
 }
 
 /// Phases of the install / update pipeline, surfaced to the UI so users

core/archipelago/src/electrs_status.rs

@@ -30,9 +30,11 @@ async fn bitcoin_rpc_auth() -> String {
 #[derive(Debug, Clone, Serialize)]
 pub struct ElectrsSyncStatus {
     pub indexed_height: u64,
+    pub bitcoin_height: u64,
     pub network_height: u64,
     pub progress_pct: f64,
     pub status: String,
+    pub stale: bool,
     pub error: Option<String>,
     /// Index data size in human-readable format (e.g. "11.2 GB")
     pub index_size: Option<String>,
@@ -44,9 +46,11 @@ impl Default for ElectrsSyncStatus {
     fn default() -> Self {
         Self {
             indexed_height: 0,
+            bitcoin_height: 0,
             network_height: 0,
             progress_pct: 0.0,
             status: "starting".to_string(),
+            stale: false,
             error: None,
             index_size: None,
             tor_onion: None,
@@ -64,15 +68,33 @@ fn cache() -> &'static RwLock<ElectrsSyncStatus> {
 /// Spawn background task that refreshes ElectrumX status every CACHE_REFRESH_SECS.
 pub fn spawn_status_cache() {
     tokio::spawn(async {
-        // Initial delay — let services start up before first query
-        tokio::time::sleep(Duration::from_secs(5)).await;
-
-        let mut interval = tokio::time::interval(Duration::from_secs(CACHE_REFRESH_SECS));
         loop {
-            interval.tick().await;
-            let fresh = fetch_electrs_sync_status().await;
+            let mut fresh = fetch_electrs_sync_status().await;
             let mut cached = cache().write().await;
+            if fresh.indexed_height == 0
+                && cached.indexed_height > 0
+                && matches!(fresh.status.as_str(), "indexing" | "waiting")
+            {
+                fresh.indexed_height = cached.indexed_height;
+                if fresh.network_height == 0 {
+                    fresh.network_height = cached.network_height;
+                }
+                if fresh.bitcoin_height == 0 {
+                    fresh.bitcoin_height = cached.bitcoin_height;
+                }
+                if fresh.progress_pct <= 0.0 {
+                    fresh.progress_pct = cached.progress_pct;
+                }
+                fresh.stale = true;
+                fresh.error = Some(
+                    fresh
+                        .error
+                        .unwrap_or_else(|| "ElectrumX is reconnecting; showing last known indexed height.".to_string()),
+                );
+            }
             *cached = fresh;
+            drop(cached);
+            tokio::time::sleep(Duration::from_secs(CACHE_REFRESH_SECS)).await;
         }
     });
 }
@@ -187,13 +209,69 @@ async fn electrumx_indexed_height() -> Result<u64> {
     Ok(height)
 }
 
-/// Fetch Bitcoin network height via JSON-RPC.
-async fn bitcoin_network_height() -> Result<u64> {
+fn parse_electrumx_height_from_logs(logs: &str) -> Option<u64> {
+    let mut height = None;
+
+    for line in logs.lines() {
+        if let Some(idx) = line.find("BlockProcessor:our height:") {
+            let rest = &line[idx + "BlockProcessor:our height:".len()..];
+            if let Some(parsed) = parse_first_u64_token(rest) {
+                height = Some(parsed);
+            }
+            continue;
+        }
+
+        if let Some(idx) = line.find("DB:height:") {
+            let rest = &line[idx + "DB:height:".len()..];
+            if let Some(parsed) = parse_first_u64_token(rest) {
+                height = Some(parsed);
+            }
+        }
+    }
+
+    height
+}
+
+fn parse_first_u64_token(input: &str) -> Option<u64> {
+    let token: String = input
+        .trim_start()
+        .chars()
+        .take_while(|c| c.is_ascii_digit() || *c == ',')
+        .filter(|c| *c != ',')
+        .collect();
+
+    if token.is_empty() {
+        None
+    } else {
+        token.parse().ok()
+    }
+}
+
+async fn electrumx_log_indexed_height() -> Result<u64> {
+    let output = tokio::process::Command::new("podman")
+        .args(["logs", "--tail", "500", "electrumx"])
+        .output()
+        .await
+        .context("Failed to read ElectrumX logs")?;
+
+    if !output.status.success() {
+        anyhow::bail!(
+            "podman logs electrumx failed: {}",
+            String::from_utf8_lossy(&output.stderr).trim()
+        );
+    }
+
+    let logs = String::from_utf8_lossy(&output.stdout);
+    parse_electrumx_height_from_logs(&logs).context("No ElectrumX indexed height in logs")
+}
+
+/// Fetch Bitcoin local block height and best-known network header height via JSON-RPC.
+async fn bitcoin_chain_heights() -> Result<(u64, u64)> {
     let client = reqwest::Client::new();
     let body = serde_json::json!({
         "jsonrpc": "1.0",
         "id": "electrs-status",
-        "method": "getblockcount",
+        "method": "getblockchaininfo",
         "params": []
     });
     let resp = client
@@ -211,11 +289,18 @@ async fn bitcoin_network_height() -> Result<u64> {
     }
 
     let json: serde_json::Value = resp.json().await?;
-    let height = json
+    let result = json
         .get("result")
-        .and_then(|r| r.as_u64())
         .context("Missing result in Bitcoin RPC")?;
-    Ok(height)
+    let blocks = result
+        .get("blocks")
+        .and_then(|h| h.as_u64())
+        .context("Missing blocks in Bitcoin RPC")?;
+    let headers = result
+        .get("headers")
+        .and_then(|h| h.as_u64())
+        .unwrap_or(blocks);
+    Ok((blocks, headers.max(blocks)))
 }
 
 /// Fetch fresh ElectrumX sync status (called by background cache task).
@@ -260,8 +345,8 @@ async fn fetch_electrs_sync_status() -> ElectrsSyncStatus {
         onion
     };
 
-    let network_height = match bitcoin_network_height().await {
-        Ok(h) => h,
+    let (bitcoin_blocks, network_height) = match bitcoin_chain_heights().await {
+        Ok(heights) => heights,
         Err(e) => {
             let err_msg = e.to_string();
             if is_transient_error(&err_msg) {
@@ -271,9 +356,11 @@ async fn fetch_electrs_sync_status() -> ElectrsSyncStatus {
             }
             return ElectrsSyncStatus {
                 indexed_height: 0,
+                bitcoin_height: 0,
                 network_height: 0,
                 progress_pct: 0.0,
                 status: "waiting".to_string(),
+                stale: false,
                 error: Some("Waiting for Bitcoin node...".to_string()),
                 index_size,
                 tor_onion,
@@ -283,7 +370,9 @@ async fn fetch_electrs_sync_status() -> ElectrsSyncStatus {
 
     let indexed_height = match electrumx_indexed_height().await {
         Ok(h) => h,
-        Err(e) => {
+        Err(e) => match electrumx_log_indexed_height().await {
+            Ok(h) if h > 0 => h,
+            _ => {
             let err_msg = e.to_string();
             if is_transient_error(&err_msg) {
                 // ElectrumX is starting up or busy — estimate from data size
@@ -295,9 +384,11 @@ async fn fetch_electrs_sync_status() -> ElectrsSyncStatus {
                 let size_str = index_size.clone().unwrap_or_else(|| "0 MB".to_string());
                 return ElectrsSyncStatus {
                     indexed_height: 0,
+                    bitcoin_height: bitcoin_blocks,
                     network_height,
                     progress_pct,
                     status: "indexing".to_string(),
+                    stale: false,
                     error: Some(format!(
                         "Building index ({} / ~130 GB estimated). Electrum RPC will be available when complete.",
                         size_str
@@ -310,35 +401,85 @@ async fn fetch_electrs_sync_status() -> ElectrsSyncStatus {
             warn!("ElectrumX status: unexpected error: {}", err_msg);
             return ElectrsSyncStatus {
                 indexed_height: 0,
+                bitcoin_height: bitcoin_blocks,
                 network_height,
                 progress_pct: 0.0,
                 status: "error".to_string(),
+                stale: false,
                 error: Some(format!("ElectrumX: {}", err_msg)),
                 index_size,
                 tor_onion,
             };
         }
+        },
     };
 
-    let progress_pct = if network_height > 0 {
-        (indexed_height as f64 / network_height as f64) * 100.0
+    let observed_header_height = network_height.max(indexed_height);
+    let bitcoin_catching_up = bitcoin_blocks > 0 && bitcoin_blocks < observed_header_height;
+    let electrum_waiting_on_bitcoin =
+        bitcoin_catching_up && indexed_height >= bitcoin_blocks.saturating_sub(1);
+    let sync_target_height = if bitcoin_blocks > 0 {
+        bitcoin_blocks
+    } else {
+        observed_header_height
+    };
+
+    let progress_pct = if electrum_waiting_on_bitcoin && observed_header_height > 0 {
+        ((bitcoin_blocks as f64 / observed_header_height as f64) * 100.0).min(99.9)
+    } else if sync_target_height > 0 {
+        ((indexed_height as f64 / sync_target_height as f64) * 100.0).min(100.0)
     } else {
         0.0
     };
 
-    let status = if indexed_height >= network_height.saturating_sub(1) {
+    let status = if sync_target_height == 0 {
+        "waiting"
+    } else if electrum_waiting_on_bitcoin {
+        "waiting"
+    } else if indexed_height >= sync_target_height.saturating_sub(1) {
         "synced"
     } else {
         "syncing"
     };
 
+    let error = if electrum_waiting_on_bitcoin {
+        Some(format!(
+            "ElectrumX is indexed to {:}; waiting for the local Bitcoin node to catch up from {:} to known header {:}.",
+            indexed_height, bitcoin_blocks, observed_header_height
+        ))
+    } else if status == "syncing" && bitcoin_blocks < observed_header_height {
+        Some(format!(
+            "Indexing local Bitcoin node height {:} of {:}. Bitcoin node is still catching up to known header {:}.",
+            indexed_height, bitcoin_blocks, observed_header_height
+        ))
+    } else {
+        None
+    };
+
     ElectrsSyncStatus {
         indexed_height,
-        network_height,
+        bitcoin_height: bitcoin_blocks,
+        network_height: observed_header_height,
         progress_pct,
         status: status.to_string(),
-        error: None,
+        stale: false,
+        error,
         index_size,
         tor_onion,
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::parse_electrumx_height_from_logs;
+
+    #[test]
+    fn parses_latest_electrumx_progress_height_from_logs() {
+        let logs = r#"
+INFO:DB:height: 228,238
+INFO:BlockProcessor:our height: 228,248 daemon: 731,568 UTXOs 1MB hist 1MB
+INFO:BlockProcessor:our height: 232,117 daemon: 732,108 UTXOs 281MB hist 83MB
+"#;
+        assert_eq!(parse_electrumx_height_from_logs(logs), Some(232_117));
+    }
+}

core/archipelago/src/health_monitor.rs

@@ -539,6 +539,20 @@ pub fn spawn_health_monitor(state: Arc<StateManager>, data_dir: PathBuf) {
                         debug!("Skipping uninstalled container: {}", container.name);
                         continue;
                     }
+                } else {
+                    // Orphan: container exists in podman but archipelago has
+                    // no package_data entry for it. Common after a variant
+                    // switch (bitcoin-core ↔ bitcoin-knots) where the
+                    // uninstall removed the package entry but the prior
+                    // variant's container survived in stopped state. Without
+                    // this guard the health monitor pages every minute with
+                    // "Auto-restart failed (attempt N/10)" for an app the
+                    // user can no longer see in the dashboard.
+                    debug!(
+                        "Skipping orphan container (not in package_data): {}",
+                        container.name
+                    );
+                    continue;
                 }
 
                 if container.healthy {

core/archipelago/src/main.rs

@@ -29,6 +29,7 @@ mod auth;
 mod avatar;
 mod backup;
 mod bitcoin_rpc;
+mod bitcoin_status;
 mod blobs;
 mod bootstrap;
 mod config;
@@ -289,6 +290,7 @@ async fn main() -> Result<()> {
 
     // Spawn ElectrumX status cache (refreshes every 15s, serves cached data to avoid race conditions)
     electrs_status::spawn_status_cache();
+    bitcoin_status::spawn_status_cache();
 
     let startup_ms = startup_start.elapsed().as_millis();
     info!(

core/archipelago/src/server.rs

@@ -985,6 +985,17 @@ async fn scan_and_update_packages(
     let current_ids: Vec<String> = merged.keys().cloned().collect();
     for id in current_ids {
         if !packages.contains_key(&id) {
+            // Don't evict packages mid-transition: Installing/Updating/Removing
+            // legitimately have no live container yet (image still pulling) or
+            // briefly (during recreate). The absence-eviction here was racing
+            // installs and removing apps from the UI 14s in. The transitional
+            // owner (spawn_task) is responsible for clearing state, not us.
+            if let Some(entry) = merged.get(&id) {
+                if is_transitional(&entry.state) {
+                    absence_tracker.remove(&id);
+                    continue;
+                }
+            }
             let count = absence_tracker.entry(id.clone()).or_insert(0);
             *count += 1;
             if *count >= CONTAINER_ABSENCE_THRESHOLD {

core/archipelago/src/update.rs

@@ -65,13 +65,11 @@ fn is_newer(candidate: &str, current: &str) -> bool {
 }
 
 const DEFAULT_UPDATE_MANIFEST_URL: &str =
-    "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/manifest.json";
-/// Secondary mirror on an OVH VPS — independent network path so a
-/// single-provider outage doesn't knock out both mirrors. Promoted to
-/// primary default on 2026-04-23 after the Hetzner .160 VPS was
-/// decommissioned.
-const DEFAULT_SECONDARY_MIRROR_URL: &str =
     "http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json";
+/// Secondary mirror on tx1138 gitea — independent network path so a
+/// single-provider outage doesn't knock out both mirrors.
+const DEFAULT_SECONDARY_MIRROR_URL: &str =
+    "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/manifest.json";
 const UPDATE_STATE_FILE: &str = "update_state.json";
 const UPDATE_MIRRORS_FILE: &str = "update-mirrors.json";
 /// Marker written by apply_update() just before the service restart and
@@ -111,11 +109,11 @@ fn mirrors_path(data_dir: &Path) -> std::path::PathBuf {
 fn default_mirrors() -> Vec<UpdateMirror> {
     vec![
         UpdateMirror {
-            url: DEFAULT_SECONDARY_MIRROR_URL.to_string(),
+            url: DEFAULT_UPDATE_MANIFEST_URL.to_string(),
             label: "Server 1 (OVH)".to_string(),
         },
         UpdateMirror {
-            url: DEFAULT_UPDATE_MANIFEST_URL.to_string(),
+            url: DEFAULT_SECONDARY_MIRROR_URL.to_string(),
             label: "Server 2 (tx1138)".to_string(),
         },
     ]
@@ -165,12 +163,40 @@ pub async fn load_mirrors(data_dir: &Path) -> Result<Vec<UpdateMirror>> {
             changed = true;
         }
     }
+    let before_order: Vec<String> = list.iter().map(|m| m.url.clone()).collect();
+    force_ovh_update_primary(&mut list);
+    changed = changed || before_order != list.iter().map(|m| m.url.clone()).collect::<Vec<_>>();
     if changed {
         let _ = save_mirrors(data_dir, &list).await;
     }
     Ok(list)
 }
 
+fn force_ovh_update_primary(list: &mut Vec<UpdateMirror>) {
+    let defaults = default_mirrors();
+    for def in &defaults {
+        if !list.iter().any(|m| m.url == def.url) {
+            list.push(def.clone());
+        }
+    }
+    for mirror in list.iter_mut() {
+        if mirror.url == DEFAULT_UPDATE_MANIFEST_URL {
+            mirror.label = "Server 1 (OVH)".to_string();
+        } else if mirror.url == DEFAULT_SECONDARY_MIRROR_URL {
+            mirror.label = "Server 2 (tx1138)".to_string();
+        }
+    }
+    list.sort_by_key(|m| {
+        if m.url == DEFAULT_UPDATE_MANIFEST_URL {
+            0
+        } else if m.url == DEFAULT_SECONDARY_MIRROR_URL {
+            1
+        } else {
+            2
+        }
+    });
+}
+
 pub async fn save_mirrors(data_dir: &Path, mirrors: &[UpdateMirror]) -> Result<()> {
     fs::create_dir_all(data_dir)
         .await
@@ -573,15 +599,10 @@ pub async fn check_for_updates(data_dir: &Path) -> Result<UpdateState> {
                                 mirror = %manifest_url,
                                 "No newer version in manifest"
                             );
-                            if state.available_update.is_some() {
-                                // A later mirror might still have a
-                                // newer version — don't clobber what an
-                                // earlier mirror told us. But also don't
-                                // break: another mirror could be ahead.
-                                continue 'mirrors;
-                            }
                             state.manifest_mirror = None;
                             state.available_update = None;
+                            handled = true;
+                            continue 'mirrors;
                         }
                         handled = true;
                         break 'mirrors;
@@ -1175,6 +1196,125 @@ pub async fn apply_update(data_dir: &Path) -> Result<()> {
                 }
                 info!(name = %name, "Frontend archive extracted to /opt/archipelago/web-ui");
             }
+            _ if name.contains("runtime") && name.ends_with(".tar.gz") => {
+                let ts = chrono::Utc::now().timestamp_millis();
+                let staging_new = format!("/opt/archipelago/runtime.new.{}", ts);
+                let archive = src.to_string_lossy().to_string();
+
+                let mk = host_sudo(&["mkdir", "-p", &staging_new])
+                    .await
+                    .context("Failed to create runtime staging dir")?;
+                if !mk.success() {
+                    anyhow::bail!("mkdir {} failed", staging_new);
+                }
+
+                let extract = host_sudo(&["tar", "-xzf", &archive, "-C", &staging_new])
+                    .await
+                    .with_context(|| format!("Failed to extract {}", name))?;
+                if !extract.success() {
+                    let _ = host_sudo(&["rm", "-rf", &staging_new]).await;
+                    anyhow::bail!("tar extraction failed for {}", name);
+                }
+
+                let runtime_paths = [
+                    ("apps", "apps"),
+                    ("scripts", "scripts"),
+                    ("docker", "docker"),
+                    (
+                        "image-recipe/configs/archipelago-doctor.service",
+                        "archipelago-doctor.service",
+                    ),
+                    (
+                        "image-recipe/configs/archipelago-doctor.timer",
+                        "archipelago-doctor.timer",
+                    ),
+                ];
+
+                for (relative, label) in runtime_paths {
+                    let staged_path = format!("{}/{}", staging_new, relative);
+                    if !Path::new(&staged_path).exists() {
+                        tracing::debug!(path = %relative, "Runtime artifact path absent, skipping");
+                        continue;
+                    }
+
+                    match label {
+                        "apps" | "scripts" | "docker" => {
+                            let dest = format!("/opt/archipelago/{}", label);
+                            let tmp_dest =
+                                format!("{}.new.{}", dest, chrono::Utc::now().timestamp_millis());
+                            let _ = host_sudo(&["mkdir", "-p", &tmp_dest]).await;
+                            let staged_dot = format!("{}/.", staged_path);
+                            let copy = host_sudo(&["cp", "-a", &staged_dot, &tmp_dest])
+                                .await
+                                .with_context(|| format!("Failed to copy runtime {}", label))?;
+                            if !copy.success() {
+                                let _ = host_sudo(&["rm", "-rf", &tmp_dest]).await;
+                                anyhow::bail!("runtime copy failed for {}", label);
+                            }
+                            let _ = host_sudo(&["mkdir", "-p", &dest]).await;
+                            let clean = host_sudo(&[
+                                "find",
+                                &dest,
+                                "-mindepth",
+                                "1",
+                                "-maxdepth",
+                                "1",
+                                "-exec",
+                                "rm",
+                                "-rf",
+                                "{}",
+                                "+",
+                            ])
+                            .await
+                            .with_context(|| format!("Failed to clean runtime {}", label))?;
+                            if !clean.success() {
+                                let _ = host_sudo(&["rm", "-rf", &tmp_dest]).await;
+                                anyhow::bail!("runtime clean failed for {}", label);
+                            }
+                            let tmp_dot = format!("{}/.", tmp_dest);
+                            let promote = host_sudo(&["cp", "-a", &tmp_dot, &dest])
+                                .await
+                                .with_context(|| format!("Failed to promote runtime {}", label))?;
+                            let _ = host_sudo(&["rm", "-rf", &tmp_dest]).await;
+                            if !promote.success() {
+                                anyhow::bail!("runtime promote failed for {}", label);
+                            }
+                            if label == "scripts" {
+                                let _ = host_sudo(&[
+                                    "find", &dest, "-type", "f", "-name", "*.sh", "-exec", "chmod",
+                                    "755", "{}", "+",
+                                ])
+                                .await;
+                            }
+                        }
+                        "archipelago-doctor.service" | "archipelago-doctor.timer" => {
+                            let dest = format!("/etc/systemd/system/{}", label);
+                            let install = host_sudo(&["install", "-m", "644", &staged_path, &dest])
+                                .await
+                                .with_context(|| format!("Failed to install {}", label))?;
+                            if !install.success() {
+                                anyhow::bail!("runtime unit install failed for {}", label);
+                            }
+                        }
+                        _ => {}
+                    }
+                }
+
+                if Path::new(&format!("{}/scripts/image-versions.sh", staging_new)).exists() {
+                    let _ = host_sudo(&[
+                        "cp",
+                        &format!("{}/scripts/image-versions.sh", staging_new),
+                        "/opt/archipelago/image-versions.sh",
+                    ])
+                    .await;
+                }
+
+                let _ = host_sudo(&["systemctl", "daemon-reload"]).await;
+                let _ =
+                    host_sudo(&["systemctl", "enable", "--now", "archipelago-doctor.timer"]).await;
+                let _ = host_sudo(&["rm", "-rf", &staging_new]).await;
+                info!(name = %name, "Runtime assets applied to /opt/archipelago");
+            }
             _ => {
                 debug!(name = %name, "Unknown component, skipping");
             }

docker/bitcoin-ui/index.html

@@ -606,7 +606,8 @@
         console.log('[Bitcoin UI] Script loaded, initializing...');
         
         // RPC Configuration - Use local Nginx proxy within container
-        const RPC_ENDPOINT = '/bitcoin-rpc/';
+        const RPC_ENDPOINT = 'bitcoin-rpc/';
+        const STATUS_ENDPOINT = 'bitcoin-status';
         console.log('[Bitcoin UI] RPC Endpoint:', RPC_ENDPOINT);
 
         // Make RPC call to Bitcoin node via local proxy
@@ -645,6 +646,14 @@
             }
         }
 
+        async function fetchBitcoinStatus() {
+            const response = await fetch(STATUS_ENDPOINT, { cache: 'no-store' });
+            if (!response.ok) {
+                throw new Error(`status HTTP ${response.status}`);
+            }
+            return response.json();
+        }
+
         // Implementation branding — detected from getnetworkinfo.subversion.
         // Bitcoin Knots identifies as "/Satoshi:<ver>/Knots:<date>/", Bitcoin Core as "/Satoshi:<ver>/".
         let brandingApplied = false;
@@ -672,22 +681,62 @@
 
         // Track last block count for animations
         let lastBlockCount = 0;
+        let consecutiveRpcFailures = 0;
+        let lastSuccessfulUpdateAt = 0;
+
+        function formatPercent(value) {
+            if (!Number.isFinite(value) || value <= 0) return '0.00';
+            if (value < 0.01) return '<0.01';
+            return value.toFixed(2);
+        }
+
+        function formatBytes(bytes) {
+            if (!Number.isFinite(bytes) || bytes <= 0) return null;
+            const units = ['B', 'KB', 'MB', 'GB', 'TB'];
+            let value = bytes;
+            let unit = 0;
+            while (value >= 1000 && unit < units.length - 1) {
+                value /= 1000;
+                unit += 1;
+            }
+            return `${value.toFixed(unit >= 3 ? 1 : 0)} ${units[unit]}`;
+        }
 
         // Update blockchain info
         async function updateBlockchainInfo() {
             console.log('[Bitcoin UI] updateBlockchainInfo() called');
             try {
-                const blockchainInfo = await callRPC('getblockchaininfo');
+                const status = await fetchBitcoinStatus();
+                const blockchainInfo = status.blockchain_info;
                 console.log('[Bitcoin UI] blockchainInfo:', blockchainInfo);
                 
                 if (!blockchainInfo) {
                     console.error('[Bitcoin UI] No blockchain info received');
-                    document.getElementById('syncStatusText').textContent = 'Unable to connect to Bitcoin node';
-                    document.getElementById('syncStatusText').className = 'text-red-400 text-sm';
+                    consecutiveRpcFailures += 1;
+                    const syncStatusText = document.getElementById('syncStatusText');
+                    const syncIcon = document.getElementById('syncIcon');
+                    if (syncStatusText) {
+                        if (status.stale) {
+                            syncStatusText.textContent = status.error || 'Bitcoin node is reconnecting... showing last known values';
+                            syncStatusText.className = 'text-yellow-300 text-sm font-medium';
+                        } else if (consecutiveRpcFailures < 6) {
+                            syncStatusText.textContent = status.error || 'Connecting to Bitcoin node...';
+                            syncStatusText.className = 'text-yellow-300 text-sm font-medium';
+                        } else {
+                            syncStatusText.textContent = status.error || 'Bitcoin node is not responding yet';
+                            syncStatusText.className = 'text-red-400 text-sm font-medium';
+                        }
+                    }
+                    if (syncIcon) {
+                        syncIcon.classList.add('animate-spin-slow');
+                        syncIcon.classList.remove('text-green-500');
+                    }
                     return;
                 }
+                consecutiveRpcFailures = 0;
+                lastSuccessfulUpdateAt = Date.now();
 
-                const networkInfo = await callRPC('getnetworkinfo');
+                const networkInfo = status.network_info;
 
                 applyImplBranding(networkInfo && networkInfo.subversion);
 
@@ -743,44 +792,51 @@
                 }
 
                 // Populate Settings — Transaction Index, ZMQ, RPC (fire-and-forget)
-                (async () => {
-                    const txIndexEl = document.getElementById('settingsTxIndex');
-                    if (txIndexEl) {
-                        const idx = await callRPC('getindexinfo');
-                        if (idx && typeof idx === 'object') {
-                            const names = Object.keys(idx);
-                            txIndexEl.textContent = names.length
-                                ? `Enabled: ${names.join(', ')}`
-                                : 'Disabled';
-                        } else {
-                            txIndexEl.textContent = 'Disabled';
-                        }
+                const txIndexEl = document.getElementById('settingsTxIndex');
+                if (txIndexEl) {
+                    const idx = status.index_info;
+                    if (idx && typeof idx === 'object') {
+                        const names = Object.keys(idx);
+                        txIndexEl.textContent = names.length
+                            ? `Enabled: ${names.join(', ')}`
+                            : 'Disabled';
+                    } else {
+                        txIndexEl.textContent = 'Unavailable while node starts';
                     }
-                    const zmqEl = document.getElementById('settingsZmq');
-                    if (zmqEl) {
-                        const zmq = await callRPC('getzmqnotifications');
-                        if (Array.isArray(zmq) && zmq.length) {
-                            zmqEl.textContent = zmq.map(z => `${z.type}@${z.address}`).join('; ');
-                        } else {
-                            zmqEl.textContent = 'Not enabled';
-                        }
+                }
+                const zmqEl = document.getElementById('settingsZmq');
+                if (zmqEl) {
+                    const zmq = status.zmq_notifications;
+                    if (Array.isArray(zmq) && zmq.length) {
+                        zmqEl.textContent = zmq.map(z => `${z.type}@${z.address}`).join('; ');
+                    } else if (Array.isArray(zmq)) {
+                        zmqEl.textContent = 'Not enabled';
+                    } else {
+                        zmqEl.textContent = 'Unavailable while node starts';
                     }
-                    const rpcEl = document.getElementById('settingsRpc');
-                    if (rpcEl && networkInfo) {
-                        const port = chain === 'main' ? 8332 : (chain === 'test' ? 18332 : (chain === 'signet' ? 38332 : 18443));
-                        rpcEl.textContent = `Reachable on port ${port}`;
-                    }
-                })();
+                }
+                const rpcEl = document.getElementById('settingsRpc');
+                if (rpcEl) {
+                    const port = chain === 'main' ? 8332 : (chain === 'test' ? 18332 : (chain === 'signet' ? 38332 : 18443));
+                    rpcEl.textContent = status.stale
+                        ? `Reconnecting on port ${port}`
+                        : `Reachable on port ${port}`;
+                }
 
                 // Update sync status
                 const blocks = blockchainInfo.blocks || 0;
                 const headers = blockchainInfo.headers || 0;
                 const verificationProgress = blockchainInfo.verificationprogress || 0;
-                const isSynced = blocks >= headers - 1;
+                const initialBlockDownload = blockchainInfo.initialblockdownload === true;
+                const isSynced = headers > 0 && blocks >= headers - 1 && !initialBlockDownload;
+                const diskSize = formatBytes(blockchainInfo.size_on_disk || 0);
+                const appearsToBeReindexing = initialBlockDownload && blocks === 0 && headers > 0 && (blockchainInfo.size_on_disk || 0) > 1024 * 1024 * 1024;
                 
                 // Calculate actual sync percentage based on blocks/headers
-                const actualSyncPercentage = headers > 0 ? ((blocks / headers) * 100).toFixed(2) : '0.00';
-                const verificationPercentage = (verificationProgress * 100).toFixed(2);
+                const actualSyncValue = headers > 0 ? (blocks / headers) * 100 : 0;
+                const actualSyncPercentage = formatPercent(actualSyncValue);
+                const progressWidth = Math.max(0, Math.min(100, actualSyncValue));
+                const verificationPercentage = formatPercent(verificationProgress * 100);
 
                 // Animate block count if it changed
                 const currentHeightElem = document.getElementById('currentHeight');
@@ -795,16 +851,27 @@
                 document.getElementById('headers').textContent = headers.toLocaleString();
                 document.getElementById('verificationProgress').textContent = `${verificationPercentage}%`;
                 document.getElementById('syncPercentage').textContent = `${actualSyncPercentage}%`;
-                document.getElementById('currentBlock').textContent = `Block ${blocks.toLocaleString()}`;
-                document.getElementById('syncProgressBar').style.width = `${actualSyncPercentage}%`;
+                document.getElementById('currentBlock').textContent = appearsToBeReindexing
+                    ? 'Reindexing from disk'
+                    : `Block ${blocks.toLocaleString()}`;
+                document.getElementById('syncProgressBar').style.width = `${progressWidth}%`;
 
                 // Update sync status text and icon
                 const syncStatusText = document.getElementById('syncStatusText');
                 const syncIcon = document.getElementById('syncIcon');
                 
-                if (isSynced) {
-                    syncStatusText.textContent = '✓ Fully synchronized with the network';
-                    syncStatusText.className = 'text-green-400 text-sm font-medium';
+                if (appearsToBeReindexing) {
+                    syncStatusText.textContent = `Reindexing local block files${diskSize ? ` (${diskSize} on disk)` : ''}`;
+                    syncStatusText.className = 'text-orange-400 text-sm font-medium';
+                    if (syncIcon) {
+                        syncIcon.classList.add('animate-spin-slow');
+                        syncIcon.classList.remove('text-green-500');
+                    }
+                } else if (isSynced) {
+                    syncStatusText.textContent = status.stale
+                        ? 'Bitcoin node is reconnecting... showing last known synchronized state'
+                        : '✓ Fully synchronized with the network';
+                    syncStatusText.className = status.stale ? 'text-yellow-300 text-sm font-medium' : 'text-green-400 text-sm font-medium';
                     // Stop spinning when synced
                     if (syncIcon) {
                         syncIcon.classList.remove('animate-spin-slow');
@@ -812,8 +879,12 @@
                     }
                 } else {
                     const remaining = headers - blocks;
-                    syncStatusText.textContent = `Syncing... ${remaining.toLocaleString()} blocks remaining`;
-                    syncStatusText.className = 'text-orange-400 text-sm font-medium';
+                    syncStatusText.textContent = status.stale
+                        ? 'Bitcoin node is reconnecting... showing last known sync state'
+                        : initialBlockDownload
+                        ? `Initial block download... ${remaining.toLocaleString()} blocks remaining`
+                        : `Syncing... ${remaining.toLocaleString()} blocks remaining`;
+                    syncStatusText.className = status.stale ? 'text-yellow-300 text-sm font-medium' : 'text-orange-400 text-sm font-medium';
                     // Keep spinning while syncing
                     if (syncIcon) {
                         syncIcon.classList.add('animate-spin-slow');
@@ -834,8 +905,15 @@
 
             } catch (error) {
                 console.error('Failed to update blockchain info:', error);
-                document.getElementById('syncStatusText').textContent = 'Unable to fetch blockchain data';
-                document.getElementById('syncStatusText').className = 'text-red-400 text-sm';
+                consecutiveRpcFailures += 1;
+                const syncStatusText = document.getElementById('syncStatusText');
+                if (syncStatusText) {
+                    const hasRecentData = lastSuccessfulUpdateAt > 0 && Date.now() - lastSuccessfulUpdateAt < 120000;
+                    syncStatusText.textContent = hasRecentData
+                        ? 'Bitcoin status bridge is reconnecting... keeping last known values'
+                        : 'Connecting to Bitcoin status bridge...';
+                    syncStatusText.className = 'text-yellow-300 text-sm font-medium';
+                }
             }
         }
 

docker/electrs-ui/index.html

@@ -68,6 +68,7 @@
         @media (min-width: 768px) {
             .md-flex-row { flex-direction: row; }
             .md-grid-cols-4 { grid-template-columns: repeat(4, 1fr); }
+            .md-grid-cols-5 { grid-template-columns: repeat(5, 1fr); }
         }
 
         /* Connection details */
@@ -147,13 +148,17 @@
                 </div>
             </div>
 
-            <div class="grid grid-cols-2 md-grid-cols-4 gap-3">
+            <div class="grid grid-cols-2 md-grid-cols-5 gap-3">
                 <div class="info-card">
-                    <p class="text-xs text-white-60 mb-1">Indexed Height</p>
+                    <p class="text-xs text-white-60 mb-1">Electrum Indexed</p>
                     <p class="text-lg font-semibold text-white" id="indexedHeight">-</p>
                 </div>
                 <div class="info-card">
-                    <p class="text-xs text-white-60 mb-1">Network Height</p>
+                    <p class="text-xs text-white-60 mb-1">Bitcoin Node</p>
+                    <p class="text-lg font-semibold text-white" id="bitcoinHeight">-</p>
+                </div>
+                <div class="info-card">
+                    <p class="text-xs text-white-60 mb-1">Known Headers</p>
                     <p class="text-lg font-semibold text-white" id="networkHeight">-</p>
                 </div>
                 <div class="info-card">
@@ -370,15 +375,36 @@
             }
 
             var indexedH = data.indexed_height || 0;
-            var networkH = data.network_height || 0;
+            var bitcoinH = data.bitcoin_height || 0;
+            var reportedNetworkH = data.network_height || 0;
+            var knownHeaderH = Math.max(reportedNetworkH, indexedH, bitcoinH);
+            var targetH = bitcoinH > 0 ? bitcoinH : knownHeaderH;
             var pct = data.progress_pct || 0;
+            var hasIndexedHeight = indexedH > 0 || data.stale;
+            var indexedLabel = hasIndexedHeight
+                ? indexedH.toLocaleString()
+                : (data.status === 'indexing' ? 'Pending' : '-');
+            var currentBlockLabel;
+            if (hasIndexedHeight && bitcoinH > 0 && indexedH > bitcoinH) {
+                currentBlockLabel = 'Bitcoin node ' + bitcoinH.toLocaleString()
+                    + (knownHeaderH > 0 ? ' of known headers ' + knownHeaderH.toLocaleString() : '')
+                    + '; Electrum index ' + indexedH.toLocaleString();
+            } else if (hasIndexedHeight) {
+                currentBlockLabel = 'Indexed ' + indexedH.toLocaleString() + ' of '
+                    + (targetH > 0 ? targetH.toLocaleString() : 'Bitcoin node height');
+            } else {
+                currentBlockLabel = data.index_size
+                    ? 'Index building from disk (' + data.index_size + ')'
+                    : 'Waiting for Electrum index height';
+            }
 
-            document.getElementById('indexedHeight').textContent = indexedH > 0 ? indexedH.toLocaleString() : (data.status === 'indexing' ? 'Building...' : '-');
-            document.getElementById('networkHeight').textContent = networkH > 0 ? networkH.toLocaleString() : '-';
+            document.getElementById('indexedHeight').textContent = indexedLabel;
+            document.getElementById('bitcoinHeight').textContent = bitcoinH > 0 ? bitcoinH.toLocaleString() : 'Checking...';
+            document.getElementById('networkHeight').textContent = knownHeaderH > 0 ? knownHeaderH.toLocaleString() : 'Checking...';
             document.getElementById('indexSize').textContent = data.index_size || '-';
-            document.getElementById('progressPct').textContent = pct > 0 ? pct.toFixed(1) + '%' : '-';
-            document.getElementById('currentBlock').textContent = indexedH > 0 ? 'Block ' + indexedH.toLocaleString() : (data.index_size ? 'Index: ' + data.index_size : 'Block 0');
-            document.getElementById('syncPercentage').textContent = pct > 0 ? pct.toFixed(1) + '%' : '0%';
+            document.getElementById('progressPct').textContent = (knownHeaderH > 0 || pct > 0) ? pct.toFixed(1) + '%' : '-';
+            document.getElementById('currentBlock').textContent = currentBlockLabel;
+            document.getElementById('syncPercentage').textContent = (knownHeaderH > 0 || pct > 0) ? pct.toFixed(1) + '%' : '0%';
             document.getElementById('syncProgressBar').style.width = Math.max(pct, 0.5) + '%';
 
             var statusTextEl = document.getElementById('syncStatusText');
@@ -389,14 +415,16 @@
                 statusTextEl.textContent = data.error || 'Starting up...';
                 statusTextEl.style.color = '#fbbf24';
                 statusDot.className = 'status-dot bg-yellow animate-pulse';
-                document.getElementById('statusText').textContent = 'Starting';
+                document.getElementById('statusText').textContent = data.status === 'waiting' ? 'Waiting' : 'Starting';
                 syncIcon.classList.add('animate-spin-slow');
                 document.getElementById('connSubtitle').textContent = 'Connections will be available once ElectrumX has completed syncing.';
             } else if (data.status === 'indexing') {
-                statusTextEl.textContent = data.error || 'Building index...';
+                statusTextEl.textContent = data.stale
+                    ? (data.error || 'ElectrumX is reconnecting; showing last known indexed height.')
+                    : (data.error || 'Building index. Indexed height will appear when Electrum RPC is ready.');
                 statusTextEl.style.color = '#fbbf24';
                 statusDot.className = 'status-dot bg-amber animate-pulse';
-                document.getElementById('statusText').textContent = 'Indexing';
+                document.getElementById('statusText').textContent = data.stale ? 'Reconnecting' : 'Indexing';
                 syncIcon.classList.add('animate-spin-slow');
                 document.getElementById('connSubtitle').textContent = 'Connections will be available once ElectrumX has completed syncing.';
             } else if (data.status === 'error') {
@@ -414,8 +442,10 @@
                 syncIcon.style.color = '#4ade80';
                 document.getElementById('connSubtitle').textContent = 'Use the following details to connect your wallet or application to ElectrumX.';
             } else {
-                var remaining = networkH - indexedH;
-                statusTextEl.textContent = 'Syncing... ' + remaining.toLocaleString() + ' blocks remaining';
+                var remaining = Math.max(targetH - indexedH, 0);
+                statusTextEl.textContent = data.error || (targetH > 0
+                    ? 'Syncing... ' + remaining.toLocaleString() + ' blocks remaining'
+                    : 'Waiting for Bitcoin network height...');
                 statusTextEl.style.color = '#fb923c';
                 statusDot.className = 'status-dot bg-yellow';
                 document.getElementById('statusText').textContent = 'Syncing';

image-recipe/_archived/build-auto-installer-iso.sh

@@ -2143,6 +2143,24 @@ chown -R 1000:1000 /mnt/target/var/lib/archipelago
 
 echo "   ✅ Data partition encrypted with LUKS2 ($LUKS_CIPHER)"
 
+# Allocate swap space on the encrypted data partition. Without swap, large
+# container image builds (immich, indeedhub) and brief memory spikes can
+# OOM-kill containers or trigger cgroup cascades. Sized to RAM, capped at
+# 8GB (above which swap is rarely useful), floored at 2GB so even
+# constrained nodes have headroom. Lives on the LUKS partition so it's
+# encrypted at rest.
+step "Allocating swap"
+RAM_MB=$(($(awk '/^MemTotal:/ {print $2}' /proc/meminfo) / 1024))
+SWAP_MB=$RAM_MB
+[ "$SWAP_MB" -lt 2048 ] && SWAP_MB=2048
+[ "$SWAP_MB" -gt 8192 ] && SWAP_MB=8192
+SWAPFILE=/mnt/target/var/lib/archipelago/swapfile
+echo "   Allocating ${SWAP_MB}MB swap at /var/lib/archipelago/swapfile"
+run dd if=/dev/zero of="$SWAPFILE" bs=1M count=$SWAP_MB status=none
+run chmod 600 "$SWAPFILE"
+run mkswap "$SWAPFILE"
+echo "   ✅ ${SWAP_MB}MB swap allocated"
+
 # Configure auto-unlock via crypttab (key file on root partition)
 step "Configuring system"
 DATA_UUID=$(blkid -s UUID -o value "$DATA_PART")
@@ -2208,6 +2226,8 @@ cat > /mnt/target/etc/fstab <<EOF
 UUID=$(blkid -s UUID -o value "$ROOT_PART")  /                    ext4  errors=remount-ro  0  1
 UUID=$(blkid -s UUID -o value "$EFI_PART")   /boot/efi            vfat  umask=0077         0  1
 /dev/mapper/archipelago-data                 /var/lib/archipelago  ext4  defaults,nofail,x-systemd.device-timeout=60  0  2
+# Swap on encrypted data partition — activated after LUKS unlock
+/var/lib/archipelago/swapfile                none                 swap  sw,nofail  0  0
 EOF
 
 # Configure hostname
@@ -2241,11 +2261,11 @@ cat > /mnt/target/home/archipelago/.config/containers/registries.conf <<'REGCONF
 unqualified-search-registries = ["docker.io"]
 
 [[registry]]
-location = "git.tx1138.com"
+location = "146.59.87.168:3000"
 insecure = true
 
 [[registry]]
-location = "146.59.87.168:3000"
+location = "git.tx1138.com"
 insecure = true
 REGCONF
 chown -R 1000:1000 /mnt/target/home/archipelago/.config
@@ -2255,8 +2275,8 @@ mkdir -p /mnt/target/var/lib/archipelago/config
 cat > /mnt/target/var/lib/archipelago/config/registries.json <<'DYNREG'
 {
   "registries": [
-    {"url": "git.tx1138.com/lfg2025", "name": "Archipelago Primary", "tls_verify": true, "enabled": true, "priority": 0},
-    {"url": "146.59.87.168:3000/lfg2025", "name": "Archipelago Fallback", "tls_verify": false, "enabled": true, "priority": 10}
+    {"url": "146.59.87.168:3000/lfg2025", "name": "Archipelago Primary", "tls_verify": false, "enabled": true, "priority": 0},
+    {"url": "git.tx1138.com/lfg2025", "name": "Archipelago Fallback", "tls_verify": true, "enabled": true, "priority": 10}
   ]
 }
 DYNREG

image-recipe/configs/archipelago-doctor.service

@@ -7,6 +7,6 @@ Type=oneshot
 # Runs as root: needs to kill orphaned conmon processes, fix permissions
 User=root
 ExecStart=/home/archipelago/archy/scripts/container-doctor.sh --local
-TimeoutStartSec=120
+TimeoutStartSec=300
 StandardOutput=journal
 StandardError=journal

image-recipe/configs/archipelago-doctor.timer

@@ -2,9 +2,11 @@
 Description=Archipelago container doctor (periodic)
 
 [Timer]
-# First run 5 minutes after boot, then every 30 minutes
-OnBootSec=5min
-OnUnitActiveSec=30min
+# First run 2 minutes after boot, then every 5 minutes. The doctor is
+# idempotent and exits quickly when no drift exists; this keeps vanished
+# rootless port listeners and stopped containers from remaining broken.
+OnBootSec=2min
+OnUnitActiveSec=5min
 # Jitter to avoid load spikes
 RandomizedDelaySec=60
 

image-recipe/configs/archipelago.service

@@ -10,7 +10,7 @@ Environment="ARCHIPELAGO_BIND=127.0.0.1:5678"
 # DEV_MODE disabled in production — enabled via override.conf on dev servers
 Environment="XDG_RUNTIME_DIR=/run/user/1000"
 # + prefix runs these as root (needed for chown/mkdir outside ReadWritePaths)
-ExecStartPre=+/bin/bash -c 'mkdir -p /run/user/1000 && chown archipelago:archipelago /run/user/1000 && chmod 700 /run/user/1000'
+ExecStartPre=+/bin/bash -c 'mkdir -p /run/user/1000 /run/containers /var/lib/containers && chown archipelago:archipelago /run/user/1000 && chmod 700 /run/user/1000'
 ExecStartPre=+/bin/bash -c 'mkdir -p /var/lib/archipelago && chown archipelago:archipelago /var/lib/archipelago && echo "ARCHIPELAGO_HOST_IP=$(hostname -I 2>/dev/null | awk "{print $$1}")" > /var/lib/archipelago/host-ip.env && chown archipelago:archipelago /var/lib/archipelago/host-ip.env'
 ExecStart=/usr/local/bin/archipelago
 Restart=on-failure

neode-ui/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "neode-ui",
-  "version": "1.7.44-alpha",
+  "version": "1.7.51-alpha",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "neode-ui",
-      "version": "1.7.44-alpha",
+      "version": "1.7.51-alpha",
       "dependencies": {
         "@types/dompurify": "^3.0.5",
         "@vue-leaflet/vue-leaflet": "^0.10.1",

neode-ui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "neode-ui",
   "private": true,
-  "version": "1.7.44-alpha",
+  "version": "1.7.51-alpha",
   "type": "module",
   "scripts": {
     "start": "./start-dev.sh",

neode-ui/public/catalog.json

@@ -1,7 +1,7 @@
 {
   "version": 2,
   "updated": "2026-04-22T00:00:00Z",
-  "registry": "git.tx1138.com/lfg2025",
+  "registry": "146.59.87.168:3000/lfg2025",
   "featured": {
     "id": "indeedhub",
     "banner": "/assets/img/featured/indeedhub-banner.jpg",
@@ -11,200 +11,260 @@
   },
   "apps": [
     {
-      "id": "bitcoin-knots", "title": "Bitcoin Knots", "version": "28.1.0",
+      "id": "bitcoin-knots",
+      "title": "Bitcoin Knots",
+      "version": "28.1.0",
       "description": "Run a full Bitcoin node. Validate and relay blocks and transactions.",
       "icon": "/assets/img/app-icons/bitcoin-knots.webp",
-      "author": "Bitcoin Knots", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/bitcoin-knots:latest",
+      "author": "Bitcoin Knots",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin-knots:latest",
       "repoUrl": "https://github.com/bitcoinknots/bitcoin"
     },
     {
-      "id": "bitcoin-core", "title": "Bitcoin Core", "version": "28.4",
-      "description": "Reference implementation of the Bitcoin protocol. Run a full node validating and relaying blocks.",
+      "id": "bitcoin-core",
+      "title": "Bitcoin Core",
+      "version": "28.4",
+      "description": "Reference Bitcoin node implementation. Alternative to Bitcoin Knots; uninstall Knots before switching.",
       "icon": "/assets/img/app-icons/bitcoin-core.svg",
-      "author": "Bitcoin Core contributors", "category": "money", "tier": "optional",
-      "dockerImage": "docker.io/bitcoin/bitcoin:28.4",
+      "author": "Bitcoin Core contributors",
+      "category": "money",
+      "tier": "optional",
+      "dockerImage": "146.59.87.168:3000/lfg2025/bitcoin:28.4",
       "repoUrl": "https://github.com/bitcoin/bitcoin"
     },
     {
-      "id": "lnd", "title": "LND", "version": "0.18.4",
+      "id": "lnd",
+      "title": "LND",
+      "version": "0.18.4",
       "description": "Lightning Network Daemon. Fast Bitcoin payments through Lightning.",
       "icon": "/assets/img/app-icons/lnd.svg",
-      "author": "Lightning Labs", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/lnd:v0.18.4-beta",
+      "author": "Lightning Labs",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/lnd:v0.18.4-beta",
       "repoUrl": "https://github.com/lightningnetwork/lnd",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "btcpay-server", "title": "BTCPay Server", "version": "1.13.7",
+      "id": "btcpay-server",
+      "title": "BTCPay Server",
+      "version": "1.13.7",
       "description": "Self-hosted Bitcoin payment processor.",
       "icon": "/assets/img/app-icons/btcpay-server.png",
-      "author": "BTCPay Server Foundation", "category": "commerce", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/btcpayserver:1.13.7",
+      "author": "BTCPay Server Foundation",
+      "category": "commerce",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/btcpayserver:1.13.7",
       "repoUrl": "https://github.com/btcpayserver/btcpayserver",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "mempool", "title": "Mempool Explorer", "version": "3.0.0",
+      "id": "mempool",
+      "title": "Mempool Explorer",
+      "version": "3.0.0",
       "description": "Self-hosted Bitcoin blockchain and mempool visualizer.",
       "icon": "/assets/img/app-icons/mempool.webp",
-      "author": "Mempool", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/mempool-frontend:v3.0.0",
+      "author": "Mempool",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/mempool-frontend:v3.0.0",
       "repoUrl": "https://github.com/mempool/mempool",
-      "requires": ["bitcoin-knots", "electrumx"]
+      "requires": [
+        "bitcoin-knots",
+        "electrumx"
+      ]
     },
     {
-      "id": "electrumx", "title": "ElectrumX", "version": "1.18.0",
+      "id": "electrumx",
+      "title": "ElectrumX",
+      "version": "1.18.0",
       "description": "Electrum protocol server. Index the blockchain for fast wallet lookups.",
       "icon": "/assets/img/app-icons/electrumx.webp",
-      "author": "Luke Childs", "category": "money", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/electrumx:v1.18.0",
+      "author": "Luke Childs",
+      "category": "money",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/electrumx:v1.18.0",
       "repoUrl": "https://github.com/spesmilo/electrumx",
-      "requires": ["bitcoin-knots"]
+      "requires": [
+        "bitcoin-knots"
+      ]
     },
     {
-      "id": "indeedhub", "title": "IndeeHub", "version": "1.0.0",
+      "id": "indeedhub",
+      "title": "IndeeHub",
+      "version": "1.0.0",
       "description": "Bitcoin documentary streaming with Nostr identity.",
       "icon": "/assets/img/app-icons/indeedhub.png",
-      "author": "IndeeHub", "category": "community",
-      "dockerImage": "git.tx1138.com/lfg2025/indeedhub:1.0.0",
+      "author": "IndeeHub",
+      "category": "community",
+      "dockerImage": "146.59.87.168:3000/lfg2025/indeedhub:1.0.0",
       "repoUrl": "https://github.com/indeedhub/indeedhub"
     },
     {
-      "id": "botfights", "title": "BotFights", "version": "1.1.0",
+      "id": "botfights",
+      "title": "BotFights",
+      "version": "1.1.0",
       "description": "Bot arena + 2-player arcade fighter with controller support and Adventure Mode.",
       "icon": "/assets/img/app-icons/botfights.svg",
-      "author": "BotFights", "category": "community",
-      "dockerImage": "git.tx1138.com/lfg2025/botfights:1.1.0",
+      "author": "BotFights",
+      "category": "community",
+      "dockerImage": "146.59.87.168:3000/lfg2025/botfights:1.1.0",
       "repoUrl": "https://botfights.net"
     },
     {
-      "id": "gitea", "title": "Gitea", "version": "1.23",
+      "id": "gitea",
+      "title": "Gitea",
+      "version": "1.23",
       "description": "Self-hosted Git service with container registry, CI/CD, issue tracking.",
       "icon": "/assets/img/app-icons/gitea.svg",
-      "author": "Gitea", "category": "development",
-      "dockerImage": "docker.io/gitea/gitea:1.23",
+      "author": "Gitea",
+      "category": "development",
+      "dockerImage": "146.59.87.168:3000/lfg2025/gitea:1.23",
       "repoUrl": "https://gitea.com"
     },
     {
-      "id": "filebrowser", "title": "File Browser", "version": "2.27.0",
+      "id": "filebrowser",
+      "title": "File Browser",
+      "version": "2.27.0",
       "description": "Web-based file manager.",
       "icon": "/assets/img/app-icons/file-browser.webp",
-      "author": "File Browser", "category": "data", "tier": "core",
-      "dockerImage": "git.tx1138.com/lfg2025/filebrowser:v2.27.0",
+      "author": "File Browser",
+      "category": "data",
+      "tier": "core",
+      "dockerImage": "146.59.87.168:3000/lfg2025/filebrowser:v2.27.0",
       "repoUrl": "https://github.com/filebrowser/filebrowser"
     },
     {
-      "id": "vaultwarden", "title": "Vaultwarden", "version": "1.30.0",
+      "id": "vaultwarden",
+      "title": "Vaultwarden",
+      "version": "1.30.0",
       "description": "Self-hosted password vault with zero-knowledge encryption.",
       "icon": "/assets/img/app-icons/vaultwarden.webp",
-      "author": "Vaultwarden", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/vaultwarden:1.30.0-alpine",
+      "author": "Vaultwarden",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/vaultwarden:1.30.0-alpine",
       "repoUrl": "https://github.com/dani-garcia/vaultwarden"
     },
     {
-      "id": "searxng", "title": "SearXNG", "version": "2024.1.0",
+      "id": "searxng",
+      "title": "SearXNG",
+      "version": "2024.1.0",
       "description": "Privacy-respecting metasearch engine.",
       "icon": "/assets/img/app-icons/searxng.png",
-      "author": "SearXNG", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/searxng:latest",
+      "author": "SearXNG",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/searxng:latest",
       "repoUrl": "https://github.com/searxng/searxng"
     },
     {
-      "id": "fedimint", "title": "Fedimint", "version": "0.10.0",
+      "id": "fedimint",
+      "title": "Fedimint",
+      "version": "0.10.0",
       "description": "Federated Bitcoin mint with privacy through federated guardians.",
       "icon": "/assets/img/app-icons/fedimint.png",
-      "author": "Fedimint", "category": "money",
-      "dockerImage": "git.tx1138.com/lfg2025/fedimintd:v0.10.0",
+      "author": "Fedimint",
+      "category": "money",
+      "dockerImage": "146.59.87.168:3000/lfg2025/fedimintd:v0.10.0",
       "repoUrl": "https://github.com/fedimint/fedimint"
     },
     {
-      "id": "ollama", "title": "Ollama", "version": "0.5.4",
-      "description": "Run AI models locally. Private and on your hardware.",
-      "icon": "/assets/img/app-icons/ollama.png",
-      "author": "Ollama", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/ollama:latest",
-      "repoUrl": "https://github.com/ollama/ollama"
-    },
-    {
-      "id": "nextcloud", "title": "Nextcloud", "version": "28",
-      "description": "Your own private cloud. File sync, calendars, contacts.",
-      "icon": "/assets/img/app-icons/nextcloud.webp",
-      "author": "Nextcloud", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/nextcloud:28",
-      "repoUrl": "https://github.com/nextcloud/server"
-    },
-    {
-      "id": "jellyfin", "title": "Jellyfin", "version": "10.8.13",
+      "id": "jellyfin",
+      "title": "Jellyfin",
+      "version": "10.8.13",
       "description": "Free media server. Stream movies, music, and photos.",
       "icon": "/assets/img/app-icons/jellyfin.webp",
-      "author": "Jellyfin", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/jellyfin:10.8.13",
+      "author": "Jellyfin",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/jellyfin:10.8.13",
       "repoUrl": "https://github.com/jellyfin/jellyfin"
     },
     {
-      "id": "immich", "title": "Immich", "version": "1.90.0",
+      "id": "immich",
+      "title": "Immich",
+      "version": "1.90.0",
       "description": "High-performance photo and video backup with ML.",
       "icon": "/assets/img/app-icons/immich.png",
-      "author": "Immich", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/immich-server:release",
+      "author": "Immich",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/immich-server:release",
       "repoUrl": "https://github.com/immich-app/immich"
     },
     {
-      "id": "homeassistant", "title": "Home Assistant", "version": "2024.1",
+      "id": "homeassistant",
+      "title": "Home Assistant",
+      "version": "2024.1",
       "description": "Open-source home automation.",
       "icon": "/assets/img/app-icons/homeassistant.png",
-      "author": "Home Assistant", "category": "home",
-      "dockerImage": "git.tx1138.com/lfg2025/home-assistant:2024.1",
+      "author": "Home Assistant",
+      "category": "home",
+      "dockerImage": "146.59.87.168:3000/lfg2025/home-assistant:2024.1",
       "repoUrl": "https://github.com/home-assistant/core"
     },
     {
-      "id": "grafana", "title": "Grafana", "version": "10.2.0",
+      "id": "grafana",
+      "title": "Grafana",
+      "version": "10.2.0",
       "description": "Analytics and monitoring dashboards.",
       "icon": "/assets/img/app-icons/grafana.png",
-      "author": "Grafana Labs", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/grafana:10.2.0",
+      "author": "Grafana Labs",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/grafana:10.2.0",
       "repoUrl": "https://github.com/grafana/grafana"
     },
     {
-      "id": "tailscale", "title": "Tailscale", "version": "1.78.0",
+      "id": "tailscale",
+      "title": "Tailscale",
+      "version": "1.78.0",
       "description": "Zero-config VPN with WireGuard mesh networking.",
       "icon": "/assets/img/app-icons/tailscale.webp",
-      "author": "Tailscale", "category": "networking", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/tailscale:stable",
+      "author": "Tailscale",
+      "category": "networking",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/tailscale:stable",
       "repoUrl": "https://github.com/tailscale/tailscale"
     },
     {
-      "id": "uptime-kuma", "title": "Uptime Kuma", "version": "1.23.0",
+      "id": "uptime-kuma",
+      "title": "Uptime Kuma",
+      "version": "1.23.0",
       "description": "Self-hosted uptime monitoring.",
       "icon": "/assets/img/app-icons/uptime-kuma.webp",
-      "author": "Uptime Kuma", "category": "data", "tier": "recommended",
-      "dockerImage": "git.tx1138.com/lfg2025/uptime-kuma:1",
+      "author": "Uptime Kuma",
+      "category": "data",
+      "tier": "recommended",
+      "dockerImage": "146.59.87.168:3000/lfg2025/uptime-kuma:1",
       "repoUrl": "https://github.com/louislam/uptime-kuma"
     },
     {
-      "id": "dwn", "title": "Decentralized Web Node", "version": "0.4.0",
-      "description": "Own your data with DID-based access control.",
-      "icon": "/assets/img/app-icons/dwn.svg",
-      "author": "TBD", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/dwn-server:main",
-      "repoUrl": "https://github.com/TBD54566975/dwn-server"
-    },
-    {
-      "id": "endurain", "title": "Endurain", "version": "0.8.0",
-      "description": "Self-hosted fitness tracking. Strava alternative.",
-      "icon": "/assets/img/app-icons/endurain.png",
-      "author": "Endurain", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/endurain:0.8.0",
-      "repoUrl": "https://github.com/joaovitoriasilva/endurain"
-    },
-    {
-      "id": "photoprism", "title": "PhotoPrism", "version": "240915",
+      "id": "photoprism",
+      "title": "PhotoPrism",
+      "version": "240915",
       "description": "AI-powered photo management with facial recognition.",
       "icon": "/assets/img/app-icons/photoprism.svg",
-      "author": "PhotoPrism", "category": "data",
-      "dockerImage": "git.tx1138.com/lfg2025/photoprism:240915",
+      "author": "PhotoPrism",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/photoprism:240915",
       "repoUrl": "https://github.com/photoprism/photoprism"
+    },
+    {
+      "id": "nextcloud",
+      "title": "Nextcloud",
+      "version": "28",
+      "description": "Your own private cloud. File sync, calendars, contacts.",
+      "icon": "/assets/img/app-icons/nextcloud.webp",
+      "author": "Nextcloud",
+      "category": "data",
+      "dockerImage": "146.59.87.168:3000/lfg2025/nextcloud:28",
+      "repoUrl": "https://github.com/nextcloud/server"
     }
   ]
 }

neode-ui/src/stores/appLauncher.ts

@@ -13,7 +13,7 @@ const NEW_TAB_PORTS = new Set([
   '8085',  // Nextcloud — X-Frame-Options: SAMEORIGIN
   '3002',  // Uptime Kuma — X-Frame-Options: SAMEORIGIN
   '9001',  // Penpot — not reachable
-  // IndeedHub (7777) uses proxy path for NIP-07 nostr-provider.js — NOT new tab
+  // Port 7777 is the Nostr relay; IndeeHub's web UI is exposed on 7778.
 ])
 
 const NEW_TAB_APP_IDS = new Set([
@@ -34,6 +34,7 @@ function mustOpenInNewTab(url: string): boolean {
 function inferAppIdFromTitle(title?: string): string | null {
   const t = (title || '').toLowerCase()
   if (!t) return null
+  if (t.includes('indeehub') || t.includes('indeedhub')) return 'indeedhub'
   if ((t.includes('uptime') && t.includes('kuma')) || t.includes('uptime-kuma')) return 'uptime-kuma'
   if ((t.includes('nginx') && t.includes('proxy') && t.includes('manager')) || t.includes('nginx-proxy-manager')) return 'nginx-proxy-manager'
   if (t.includes('gitea')) return 'gitea'
@@ -47,6 +48,10 @@ function normalizeLaunchUrl(urlStr: string, appIdHint?: string | null): string {
     const normalizedPath = u.pathname === '/' ? '' : u.pathname
     const rebuilt = (port: string) => `${u.protocol}//${u.hostname}:${port}${normalizedPath}${u.search}${u.hash}`
 
+    if (sameHost && appIdHint === 'indeedhub' && u.port === '7777') {
+      return rebuilt('7778')
+    }
+
     if (sameHost && appIdHint === 'uptime-kuma' && u.port === '3001') {
       return rebuilt('3002')
     }
@@ -87,7 +92,7 @@ const PORT_TO_APP_ID: Record<string, string> = {
   '8175': 'fedimint',
   '8176': 'fedimint-gateway',
   '3100': 'dwn',
-  '7777': 'indeedhub',
+  '7778': 'indeedhub',
   '50002': 'electrumx',
   '3010': 'thunderhub',
 }

neode-ui/src/stores/container.ts

@@ -31,7 +31,7 @@ export const BUNDLED_APPS: BundledApp[] = [
   {
     id: 'bitcoin-knots',
     name: 'Bitcoin Knots',
-    image: 'git.tx1138.com/lfg2025/bitcoin-knots:latest',
+    image: '146.59.87.168:3000/lfg2025/bitcoin-knots:latest',
     description: 'Full Bitcoin node with additional features',
     icon: '₿',
     ports: [{ host: 8334, container: 80 }],

neode-ui/src/stores/server.ts

@@ -63,18 +63,44 @@ export const useServerStore = defineStore('server', () => {
           if (progress.phase) {
             const info = PHASE_INFO[progress.phase]
             if (info) {
+              // Within the PullingImage band (20→70%), interpolate the
+              // bar based on how many images / bytes have landed so far.
+              // Without this, multi-container stacks (indeedhub: 7,
+              // mempool: 3, btcpay: 4) just sit at 20% for the entire
+              // pull duration — exactly what the user reported as
+              // "Downloading sticks at 20% mostly". X-of-N progress
+              // comes from set_install_progress(i, n) in stacks.rs.
+              let bandProgress = info.progress
+              if (progress.phase === 'pulling-image' && progress.size > 0) {
+                const fraction = Math.min(progress.downloaded / progress.size, 1)
+                // PullingImage band: 20% → 70%, so 50pp to interpolate over.
+                bandProgress = 20 + Math.round(fraction * 50)
+              }
               // Only advance forward — never let the bar step backward
               // between patches (can happen briefly during scan merges).
-              const nextProgress = Math.max(current.progress, info.progress)
+              const nextProgress = Math.max(current.progress, bandProgress)
+              // Show explicit message when set (e.g. install-fail descriptions
+              // surfaced via install_progress.message) — otherwise PHASE_INFO label.
+              const label = progress.message || info.message
               installingApps.value.set(appId, {
                 ...current,
                 status: info.status,
                 progress: nextProgress,
-                message: info.message,
+                message: label,
               })
               continue
             }
           }
+          // No phase but message is set (install-fail path) — show the message
+          // even if PHASE_INFO doesn't apply. Status stays whatever the watcher
+          // currently has.
+          if (progress.message) {
+            installingApps.value.set(appId, {
+              ...current,
+              message: progress.message,
+            })
+            continue
+          }
           // Fallback: byte counters (rare — podman usually doesn't
           // emit parseable progress on a piped stderr).
           const pct = progress.size > 0 ? Math.round((progress.downloaded / progress.size) * 100) : 0

neode-ui/src/types/api.ts

@@ -166,6 +166,9 @@ export interface InstallProgress {
    *  counters — podman pull doesn't emit parseable progress when
    *  stderr is piped, so byte counters are usually (0,0). */
   phase?: InstallPhase
+  /** Optional explicit message — surfaced on install failures so the
+   *  UI can show what went wrong instead of silently removing the card. */
+  message?: string
 }
 
 // RPC Request/Response types

neode-ui/src/utils/dummyApps.ts

@@ -504,7 +504,7 @@ export const dummyApps: Record<string, PackageDataEntry> = {
       'interface-addresses': {
         main: {
           'tor-address': '',
-          'lan-address': 'http://localhost:8190'
+          'lan-address': 'http://localhost:7778'
         }
       },
       status: ServiceStatus.Running
@@ -749,4 +749,3 @@ export const dummyApps: Record<string, PackageDataEntry> = {
     }
   }
 }
-

neode-ui/src/views/AppDetails.vue

@@ -144,7 +144,6 @@ import {
   WEB_ONLY_APP_URLS,
   PACKAGE_ALIASES,
   BITCOIN_DEPENDENT_APPS,
-  APP_URLS,
   resolvePackageKey,
   isRealOnionAddress,
 } from './appDetails/appDetailsData'
@@ -285,7 +284,6 @@ function goBack() {
 
 function launchApp() {
   if (!pkg.value) return
-  const isDev = import.meta.env.DEV
   const id = appId.value
 
   const webOnlyUrl = WEB_ONLY_APP_URLS[id]
@@ -294,22 +292,12 @@ function launchApp() {
     return
   }
 
-  if (APP_URLS[id]) {
-    let url = isDev ? APP_URLS[id].dev : APP_URLS[id].prod
-    if (url.includes('localhost')) {
-      url = url.replace('localhost', window.location.hostname)
-    }
-    useAppLauncherStore().open({ url, title: pkg.value.manifest.title })
-    return
-  }
-
-  const torAddress = pkg.value.manifest.interfaces?.main?.['tor-config']
-  const lanConfig = pkg.value.manifest.interfaces?.main?.['lan-config']
-  if (torAddress || lanConfig) {
-    showActionError(t('appDetails.noLaunchUrl'))
-  }
+  // Container apps should launch through session routing so protocol/path
+  // handling stays centralized in appSessionConfig.
+  useAppLauncherStore().openSession(id)
 }
 
+
 async function startApp() {
   try {
     await store.startPackage(appId.value)

neode-ui/src/views/Discover.vue

@@ -221,7 +221,6 @@ const categories = computed(() => [
 // local watcher that duplicated logic using byte counters only — it has
 // been removed in favour of the store's phase-aware mapping.
 const installingApps = serverStore.installingApps
-const maxAttempts = ref(60)
 
 function selectCategory(id: string) {
   selectedCategory.value = id
@@ -415,7 +414,6 @@ function viewAppDetails(app: MarketplaceApp) {
 
 // Timer management
 const activeTimers: ReturnType<typeof setTimeout>[] = []
-const activeIntervals: ReturnType<typeof setInterval>[] = []
 
 function trackTimeout(fn: () => void, ms: number) {
   const id = setTimeout(() => {
@@ -427,86 +425,68 @@ function trackTimeout(fn: () => void, ms: number) {
   return id
 }
 
-function trackInterval(fn: () => void, ms: number) {
-  const id = setInterval(fn, ms)
-  activeIntervals.push(id)
-  return id
-}
-
-function clearTrackedInterval(id: ReturnType<typeof setInterval>) {
-  clearInterval(id)
-  const idx = activeIntervals.indexOf(id)
-  if (idx !== -1) activeIntervals.splice(idx, 1)
-}
-
 onBeforeUnmount(() => {
   for (const t of activeTimers) clearTimeout(t)
   activeTimers.length = 0
-  for (const i of activeIntervals) clearInterval(i)
-  activeIntervals.length = 0
 })
 
-function startInstallPolling(appId: string, statusMessage: string) {
-  const interval = trackInterval(() => {
-    const current = installingApps.get(appId)
-    if (!current) { clearTrackedInterval(interval); return }
-    const newAttempt = current.attempt + 1
-    installingApps.set(appId, { ...current, attempt: newAttempt, progress: Math.min(60 + (newAttempt * 0.5), 95), message: statusMessage })
-    if (isInstalled(appId)) {
-      clearTrackedInterval(interval)
-      installingApps.set(appId, { ...current, status: 'complete', progress: 100, message: 'Installation complete!' })
-      trackTimeout(() => { installingApps.delete(appId) }, 2000)
-    } else if (newAttempt >= maxAttempts.value) {
-      clearTrackedInterval(interval)
-      installingApps.set(appId, { ...current, status: 'error', progress: 0, message: 'Installation timeout' })
-      trackTimeout(() => { installingApps.delete(appId) }, 5000)
-    }
-  }, 1000)
-}
-
 const toast = useToast()
 
+function queueInstall(app: MarketplaceApp) {
+  serverStore.setInstallProgress(app.id, {
+    id: app.id,
+    title: app.title ?? app.id,
+    status: 'downloading',
+    progress: 2,
+    message: 'Queued…',
+    attempt: 0,
+  })
+}
+
+function failInstall(app: MarketplaceApp, err: unknown) {
+  const message = "Failed: " + (err instanceof Error ? err.message : String(err))
+  serverStore.setInstallProgress(app.id, {
+    id: app.id,
+    title: app.title ?? app.id,
+    status: 'error',
+    progress: 0,
+    message,
+    attempt: 0,
+  })
+  trackTimeout(() => { serverStore.clearInstallProgress(app.id) }, 5000)
+}
+
 async function installApp(app: MarketplaceApp) {
   if (installingApps.has(app.id) || isInstalled(app.id)) return
-  installingApps.set(app.id, { id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Preparing installation...', attempt: 0 })
-  toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
+  queueInstall(app)
+  toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
   router.push('/dashboard/apps').catch(() => {})
   try {
     const installUrl = app.url || app.manifestUrl || app.s9pkUrl
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 30, message: 'Downloading package...' })
     await rpcClient.call({ method: 'package.install', params: { id: app.id, url: installUrl, version: app.version }, timeout: 15000 })
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Installing package...' })
-    startInstallPolling(app.id, 'Starting application...')
   } catch (err) {
     if (import.meta.env.DEV) console.error('Installation failed:', err)
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
-    trackTimeout(() => { installingApps.delete(app.id) }, 5000)
+    failInstall(app, err)
   }
 }
 
 async function installCommunityApp(app: MarketplaceApp) {
   if (installingApps.has(app.id) || isInstalled(app.id) || !app.dockerImage) return
-  installingApps.set(app.id, { id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Pulling Docker image...', attempt: 0 })
-  toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
+  queueInstall(app)
+  toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
   router.push('/dashboard/apps').catch(() => {})
   try {
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 20, message: 'Downloading container image...' })
-    // Pass containerConfig from catalog if available (allows dynamic apps without hardcoded backend config)
     const installParams: Record<string, unknown> = { id: app.id, dockerImage: app.dockerImage, version: app.version }
     if ((app as Record<string, unknown>).containerConfig) {
       installParams.containerConfig = (app as Record<string, unknown>).containerConfig
     }
     await rpcClient.call({ method: 'package.install', params: installParams, timeout: 15000 })
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Starting container...' })
-    startInstallPolling(app.id, 'Initializing application...')
   } catch (err) {
     if (import.meta.env.DEV) console.error('[Discover] Installation failed:', err)
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
-    trackTimeout(() => { installingApps.delete(app.id) }, 5000)
+    failInstall(app, err)
   }
 }
 
-
 onMounted(() => {
   discoverAnimationDone = true
   if (communityApps.value.length === 0 && !loadingCommunity.value) {

neode-ui/src/views/Kiosk.vue

@@ -83,9 +83,11 @@ const launchableApps = computed<KioskApp[]>(() => {
   const pkgs = store.data?.['package-data'] || {}
   const apps: KioskApp[] = []
 
-  // App URL mappings — use nginx proxy paths for local apps
+  // App URL mappings. Bitcoin UI uses its direct host-network port; loading it
+  // through /app/bitcoin-ui/ can render a blank shell because its assets are
+  // rooted at /.
   const urlMap: Record<string, string> = {
-    'bitcoin-knots': '/app/bitcoin-ui/',
+    'bitcoin-knots': 'http://' + window.location.hostname + ':8334',
     'lnd': '/app/lnd/',
     'mempool': '/app/mempool/',
     'btcpay-server': '/app/btcpay/',
@@ -107,7 +109,7 @@ const launchableApps = computed<KioskApp[]>(() => {
     'fedimint': '/app/fedimint/',
     'fedimint-gateway': '/app/fedimint-gateway/',
     'dwn': '/app/dwn/',
-    'indeedhub': 'http://localhost:8190',
+    'indeedhub': 'http://localhost:7778',
     'botfights': 'http://localhost:9100',
     'nwnn': 'https://nwnn.l484.com',
     '484-kitchen': 'https://484.kitchen',

neode-ui/src/views/Marketplace.vue

@@ -157,7 +157,6 @@ const categories = computed(() => [
 
 // Installation state — uses global store so it persists across navigation
 const installingApps = server.installingApps
-const maxAttempts = ref(60)
 
 // Install progress tracking is now in serverStore (global watcher on WebSocket data)
 // so it works regardless of which page is active
@@ -338,7 +337,6 @@ function viewAppDetails(app: MarketplaceApp) {
 }
 
 const activeTimers: ReturnType<typeof setTimeout>[] = []
-const activeIntervals: ReturnType<typeof setInterval>[] = []
 
 function trackTimeout(fn: () => void, ms: number) {
   const id = setTimeout(() => {
@@ -350,115 +348,74 @@ function trackTimeout(fn: () => void, ms: number) {
   return id
 }
 
-function trackInterval(fn: () => void, ms: number) {
-  const id = setInterval(fn, ms)
-  activeIntervals.push(id)
-  return id
-}
-
-function clearTrackedInterval(id: ReturnType<typeof setInterval>) {
-  clearInterval(id)
-  const idx = activeIntervals.indexOf(id)
-  if (idx !== -1) activeIntervals.splice(idx, 1)
-}
-
 onBeforeUnmount(() => {
   for (const t of activeTimers) clearTimeout(t)
   activeTimers.length = 0
-  for (const i of activeIntervals) clearInterval(i)
-  activeIntervals.length = 0
 })
 
-function startInstallPolling(appId: string, statusMessage: string) {
-  const interval = trackInterval(() => {
-    const current = installingApps.get(appId)
-    if (!current) { clearTrackedInterval(interval); return }
+function queueInstall(app: MarketplaceApp) {
+  server.setInstallProgress(app.id, {
+    id: app.id,
+    title: app.title ?? app.id,
+    status: 'downloading',
+    progress: 2,
+    message: 'Queued…',
+    attempt: 0,
+  })
+}
 
-    const newAttempt = current.attempt + 1
-    const state = getInstalledState(appId)
-
-    // Update message based on actual backend state
-    let message = statusMessage
-    if (state === 'starting') message = 'Starting application...'
-    else if (state === 'running') message = 'Installation complete!'
-
-    installingApps.set(appId, {
-      ...current,
-      attempt: newAttempt,
-      progress: Math.min(60 + (newAttempt * 0.5), 95),
-      message
-    })
-
-    // Only clear when fully running — server store watcher handles the actual delete
-    if (state === 'running') {
-      clearTrackedInterval(interval)
-      installingApps.set(appId, { ...current, status: 'complete', progress: 100, message: 'Installation complete!' })
-      // Server store watcher will clear installingApps when it sees 'running'
-    } else if (newAttempt >= maxAttempts.value) {
-      clearTrackedInterval(interval)
-      installingApps.set(appId, { ...current, status: 'error', progress: 0, message: 'Installation timeout — check My Apps' })
-      trackTimeout(() => { installingApps.delete(appId) }, 5000)
-    }
-  }, 1000)
+function failInstall(app: MarketplaceApp, err: unknown) {
+  const message = "Failed: " + (err instanceof Error ? err.message : String(err))
+  server.setInstallProgress(app.id, {
+    id: app.id,
+    title: app.title ?? app.id,
+    status: 'error',
+    progress: 0,
+    message,
+    attempt: 0,
+  })
+  trackTimeout(() => { server.clearInstallProgress(app.id) }, 5000)
 }
 
 async function installApp(app: MarketplaceApp) {
   if (installingApps.has(app.id) || isInstalled(app.id)) return
 
-  installingApps.set(app.id, {
-    id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Preparing installation...', attempt: 0
-  })
-
-  // Navigate to My Apps immediately and show toast
-  toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
+  queueInstall(app)
+  toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
   router.push('/dashboard/apps').catch(() => {})
 
   try {
     const installUrl = app.url || app.manifestUrl || app.s9pkUrl
-
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 30, message: 'Downloading package...' })
-
-    await rpcClient.call({ method: 'package.install', params: { id: app.id, url: installUrl, version: app.version }, timeout: 15000 })
-
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Installing package...' })
-
-    startInstallPolling(app.id, 'Starting application...')
+    await rpcClient.call({
+      method: 'package.install',
+      params: { id: app.id, url: installUrl, version: app.version },
+      timeout: 15000,
+    })
   } catch (err) {
     if (import.meta.env.DEV) console.error('Installation failed:', err)
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
-    trackTimeout(() => { installingApps.delete(app.id) }, 5000)
+    failInstall(app, err)
   }
 }
 
 async function installCommunityApp(app: MarketplaceApp) {
   if (installingApps.has(app.id) || isInstalled(app.id) || !app.dockerImage) return
 
-  installingApps.set(app.id, {
-    id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Pulling Docker image...', attempt: 0
-  })
-
-  // Navigate to My Apps immediately and show toast
-  toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
+  queueInstall(app)
+  toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
   router.push('/dashboard/apps').catch(() => {})
 
   try {
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 20, message: 'Downloading container image...' })
-
     await rpcClient.call({
       method: 'package.install',
       params: { id: app.id, dockerImage: app.dockerImage, version: app.version },
-      timeout: 15000
+      timeout: 15000,
     })
-
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Starting container...' })
-
-    startInstallPolling(app.id, 'Initializing application...')
   } catch (err) {
     if (import.meta.env.DEV) console.error('[Marketplace] Installation failed:', err)
-    installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
-    trackTimeout(() => { installingApps.delete(app.id) }, 5000)
+    failInstall(app, err)
   }
 }
+
 </script>
 
 <style scoped>

neode-ui/src/views/MarketplaceAppDetails.vue

@@ -444,7 +444,7 @@ const features = computed(() => {
 })
 
 /** App dependency definitions */
-const R = 'git.tx1138.com/lfg2025'
+const R = '146.59.87.168:3000/lfg2025'
 const APP_DEPENDENCIES: Record<string, { id: string; title: string; dockerImage: string }[]> = {
   'electrumx': [{ id: 'bitcoin-knots', title: 'Bitcoin Knots', dockerImage: `${R}/bitcoin-knots:latest` }],
   'lnd': [{ id: 'bitcoin-knots', title: 'Bitcoin Knots', dockerImage: `${R}/bitcoin-knots:latest` }],
@@ -607,4 +607,3 @@ async function installApp() {
   }
 }
 </script>
-

neode-ui/src/views/appDetails/appDetailsData.ts

@@ -6,7 +6,6 @@ import { PackageState } from '@/types/api'
 
 /** Web-only app detection (no container -- external websites) */
 export const WEB_ONLY_APP_URLS: Record<string, string> = {
-  'indeedhub': `${window.location.protocol}//${window.location.hostname}:7777`,
   'nwnn': 'https://nwnn.l484.com',
   '484-kitchen': 'https://484.kitchen',
   'call-the-operator': 'https://cta.tx1138.com',
@@ -65,7 +64,6 @@ export const APP_URLS: Record<string, { dev: string; prod: string }> = {
   'lorabell': { dev: 'http://192.168.1.166', prod: 'http://192.168.1.166' },
   'atob': { dev: 'http://localhost:8102', prod: 'https://app.atobitcoin.io' },
   'k484': { dev: 'http://localhost:8103', prod: 'http://localhost:8103' },
-  'indeedhub': { dev: 'https://archipelago.indeehub.studio', prod: 'https://archipelago.indeehub.studio' },
   'bitcoin': { dev: 'http://localhost:8332', prod: 'http://localhost:8332' },
   'btcpay-server': { dev: 'http://localhost:23000', prod: 'http://localhost:23000' },
   'homeassistant': { dev: 'http://localhost:8123', prod: 'http://localhost:8123' },

neode-ui/src/views/appSession/appSessionConfig.ts

@@ -45,8 +45,7 @@ export const APP_PORTS: Record<string, number> = {
 }
 
 /** Apps that need nginx proxy for iframe embedding.
- *  IndeedHub loads via /app/indeedhub/ proxy for nostr-provider.js injection
- *  from the container's internal nginx so iframe works on all servers. */
+ *  IndeeHub web UI is on 7778. Port 7777 is the Nostr relay. */
 export const PROXY_APPS: Record<string, string> = {
   'gitea': '/app/gitea/',
   'nginx-proxy-manager': '/app/nginx-proxy-manager/',
@@ -56,13 +55,11 @@ export const PROXY_APPS: Record<string, string> = {
 /** Nginx proxy paths -- used on HTTPS to avoid mixed content (HTTPS parent + HTTP port iframe).
  *  On HTTP, direct port access is used instead (faster, no proxy). */
 export const HTTPS_PROXY_PATHS: Record<string, string> = {
-  'bitcoin-knots': '/app/bitcoin-ui/',
-  'bitcoin-core': '/app/bitcoin-ui/',
-  'bitcoin-ui': '/app/bitcoin-ui/',
   'lnd': '/app/lnd/',
-  'electrumx': '/app/electrs/',
-  'electrs': '/app/electrs/',
-  'mempool-electrs': '/app/electrs/',
+  'electrumx': '/app/electrumx/',
+  'electrs': '/app/electrumx/',
+  'archy-electrs-ui': '/app/electrumx/',
+  'mempool-electrs': '/app/electrumx/',
   'mempool': '/app/mempool/',
   'mempool-web': '/app/mempool/',
   'archy-mempool-web': '/app/mempool/',
@@ -87,7 +84,6 @@ export const HTTPS_PROXY_PATHS: Record<string, string> = {
   'btcpay-server': '/app/btcpay/',
   'nextcloud': '/app/nextcloud/',
   'grafana': '/app/grafana/',
-  'indeedhub': '/app/indeedhub/',
   'botfights': '/app/botfights/',
   'gitea': '/app/gitea/',
 }
@@ -138,11 +134,26 @@ export function resolveAppUrl(id: string, routeQueryPath?: string): string {
   const ext = EXTERNAL_URLS[id]
   if (ext) return ext
 
-  // Local apps: always launch by host port
+  // Bitcoin UI is a host-network companion on :8334. Do not launch it via
+  // /app/bitcoin-ui/: the static UI is built for root and renders a blank
+  // shell when proxied under a path prefix on some nodes.
+  if (id === 'bitcoin-knots' || id === 'bitcoin-core' || id === 'bitcoin-ui') {
+    return 'http://' + window.location.hostname + ':8334'
+  }
+
+  // HTTPS pages cannot embed plain HTTP port origins (mixed-content).
+  if (window.location.protocol === 'https:') {
+    const proxyPath = HTTPS_PROXY_PATHS[id]
+    if (proxyPath) {
+      return window.location.protocol + '//' + window.location.hostname + proxyPath
+    }
+  }
+
+  // Local apps on HTTP pages launch by host port.
   const port = APP_PORTS[id]
   if (!port) return ''
 
-  let base = `${window.location.protocol}//${window.location.hostname}:${port}`
+  let base = window.location.protocol + '//' + window.location.hostname + ':' + String(port)
   if (routeQueryPath) base += routeQueryPath
   return base
 }

neode-ui/src/views/discover/curatedApps.ts

@@ -1,6 +1,6 @@
 import type { MarketplaceApp } from './types'
 
-const R = 'git.tx1138.com/lfg2025'
+const R = '146.59.87.168:3000/lfg2025'
 
 // ---------- Dynamic catalog from registry ----------
 export interface CatalogFeatured {

neode-ui/src/views/marketplace/marketplaceData.ts

@@ -376,7 +376,7 @@ export function getCuratedAppList(): MarketplaceApp[] {
       description: 'Bitcoin documentary streaming platform with Nostr identity sign-in. Stream God Bless Bitcoin and other educational content about sovereignty and decentralized technology.',
       icon: '/assets/img/app-icons/indeedhub.png',
       author: 'Indeehub Team',
-      dockerImage: 'git.tx1138.com/lfg2025/indeedhub:latest',
+      dockerImage: '146.59.87.168:3000/lfg2025/indeedhub:latest',
       manifestUrl: undefined,
       repoUrl: 'https://github.com/indeedhub/indeedhub'
     },

release-manifest.json

@@ -1,31 +1,29 @@
 {
-  "version": "1.7.44-alpha",
-  "release_date": "2026-04-28",
+  "version": "1.7.51-alpha",
+  "release_date": "2026-05-01",
   "changelog": [
-    "43de3b73 feat(orchestrator): complete container migration and release hardening",
-    "ce39430b feat(self-update): sync and rebuild UI containers on OTA",
-    "72dec5aa fix(lnd-ui): align container port across all specs",
-    "83aacdf2 chore(release): archive ISO build recipes, tarball-only releases",
-    "All notable changes to Archipelago will be documented in this file.",
-    "The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),",
-    "and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)."
+    "Install success now requires adopted containers and IndeedHub stack containers to stay running; failed starts surface logs instead of disappearing from My Apps.",
+    "Bitcoin uninstall removes the shared Bitcoin data/UI directories when data is not preserved, preventing stale partial installs from being adopted as success.",
+    "Bitcoin RPC bind settings are repaired on startup and before adopting existing Bitcoin containers, fixing older nodes where bitcoin-ui showed endless getblockchaininfo/502.",
+    "Bitcoin Core/Knots launch the Bitcoin UI on direct port 8334 instead of the /app/bitcoin-ui path proxy.",
+    "Nodes force OVH as the primary update mirror and app registry on next startup, with tx1138 retained as fallback."
   ],
   "components": [
     {
       "name": "archipelago",
-      "current_version": "1.7.44-alpha",
-      "new_version": "1.7.44-alpha",
-      "download_url": "https://github.com/archipelago-os/releases/releases/download/v1.7.44-alpha/archipelago",
-      "sha256": "ea9167d376210bdc416dd0dcf3c1737e2fa2b1bdb9f192e473f2edaaf0d046a7",
-      "size_bytes": 41373032
+      "current_version": "1.7.50-alpha",
+      "new_version": "1.7.51-alpha",
+      "download_url": "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/v1.7.51-alpha/archipelago",
+      "sha256": "f761e659d661f0a83cd3a67a086bb2279398bc05e50ee3c52e769e52d11e476c",
+      "size_bytes": 41637536
     },
     {
-      "name": "archipelago-frontend-1.7.44-alpha.tar.gz",
-      "current_version": "1.7.44-alpha",
-      "new_version": "1.7.44-alpha",
-      "download_url": "https://github.com/archipelago-os/releases/releases/download/v1.7.44-alpha/archipelago-frontend-1.7.44-alpha.tar.gz",
-      "sha256": "28bdbccd8151bfa7834761adb5363862ef54b76c6010d1446fb8cca4ed4fe18c",
-      "size_bytes": 162089899
+      "name": "archipelago-frontend-1.7.51-alpha.tar.gz",
+      "current_version": "1.7.50-alpha",
+      "new_version": "1.7.51-alpha",
+      "download_url": "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/v1.7.51-alpha/archipelago-frontend-1.7.51-alpha.tar.gz",
+      "sha256": "3403f4e38202bf56c53407dd62e66899693ee73252bf203475715532ac6ae326",
+      "size_bytes": 165155462
     }
   ]
 }

scripts/app-surface-smoke-test.sh

@@ -0,0 +1,161 @@
+#!/bin/bash
+#
+# App surface smoke test.
+#
+# Verifies that installed containers have their published host ports listening
+# and that known nginx app proxy paths return a non-5xx response. This catches
+# the common "container is running but UI disappeared" failure mode.
+#
+# Usage:
+#   scripts/app-surface-smoke-test.sh --target archipelago@192.168.1.228 --ssh-key /path/key
+
+set -euo pipefail
+
+TARGET=""
+SSH_KEY="${ARCHIPELAGO_SSH_KEY:-}"
+SSH_EXTRA=()
+
+while [ "$#" -gt 0 ]; do
+  case "$1" in
+    --target) TARGET="${2:-}"; shift 2 ;;
+    --ssh-key) SSH_KEY="${2:-}"; shift 2 ;;
+    --ssh-option) SSH_EXTRA+=("-o" "${2:-}"); shift 2 ;;
+    -h|--help) sed -n '1,12p' "$0"; exit 0 ;;
+    *) echo "unknown argument: $1" >&2; exit 2 ;;
+  esac
+done
+
+[ -n "$TARGET" ] || { echo "--target is required" >&2; exit 2; }
+
+SSH_OPTS=(-F /dev/null -o BatchMode=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no)
+[ -n "$SSH_KEY" ] && SSH_OPTS+=(-i "$SSH_KEY")
+SSH_OPTS+=("${SSH_EXTRA[@]}")
+
+ssh_run() {
+  ssh "${SSH_OPTS[@]}" "$TARGET" "$@"
+}
+
+ssh_run 'bash -s' <<'REMOTE'
+set -u
+
+pass=0
+fail=0
+
+ok() { echo "  PASS $*"; pass=$((pass + 1)); }
+bad() { echo "  FAIL $*"; fail=$((fail + 1)); }
+
+container_exists() {
+  podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$1"
+}
+
+port_listening() {
+  ss -ltn 2>/dev/null | awk '{print $4}' | grep -Eq "(^|:)$1$"
+}
+
+http_code() {
+  local url="$1" code
+  for _ in 1 2 3; do
+    code=$(curl -ksS -o /dev/null -w '%{http_code}' --max-time 12 "$url" 2>/dev/null || true)
+    [ -n "$code" ] || code=000
+    [ "$code" != "000" ] && { echo "$code"; return; }
+    sleep 2
+  done
+  echo "$code"
+}
+
+http_post_code() {
+  local url="$1" code
+  for _ in 1 2 3; do
+    code=$(curl -ksS -o /dev/null -w '%{http_code}' --max-time 25 \
+      -H 'Content-Type: application/json' \
+      -d '{"jsonrpc":"2.0","id":1,"method":"getblockchaininfo","params":[]}' \
+      "$url" 2>/dev/null || true)
+    [ -n "$code" ] || code=000
+    [ "$code" != "000" ] && { echo "$code"; return; }
+    sleep 2
+  done
+  echo "$code"
+}
+
+assert_http() {
+  local label="$1" url="$2" code
+  code=$(http_code "$url")
+  case "$code" in
+    200|204|301|302|307|308|401|403) ok "$label HTTP $code" ;;
+    *) bad "$label HTTP $code ($url)" ;;
+  esac
+}
+
+assert_http_post() {
+  local label="$1" url="$2" code
+  code=$(http_post_code "$url")
+  case "$code" in
+    200|204|401|403) ok "$label HTTP POST $code" ;;
+    *) bad "$label HTTP POST $code ($url)" ;;
+  esac
+}
+
+assert_container_ports() {
+  local name="$1" ports port missing=0
+  container_exists "$name" || return 0
+  ports=$(podman inspect "$name" --format '{{range $p,$bindings := .NetworkSettings.Ports}}{{if $bindings}}{{range $bindings}}{{.HostPort}}{{"\n"}}{{end}}{{end}}{{end}}' 2>/dev/null | sort -u)
+  [ -n "$ports" ] || return 0
+  while IFS= read -r port; do
+    [ -n "$port" ] || continue
+    if port_listening "$port"; then
+      ok "$name port $port listening"
+    else
+      bad "$name port $port missing listener"
+      missing=1
+    fi
+  done <<< "$ports"
+  return "$missing"
+}
+
+assert_env_contains() {
+  local name="$1" key="$2" needle="$3" val
+  container_exists "$name" || return 0
+  val=$(podman inspect "$name" --format '{{range .Config.Env}}{{println .}}{{end}}' 2>/dev/null | sed -n "s/^${key}=//p" | head -n 1)
+  if [ -n "$val" ] && printf '%s' "$val" | grep -qF "$needle"; then
+    ok "$name env $key"
+  else
+    bad "$name env $key missing $needle"
+  fi
+}
+
+echo "[surface] host=$(hostname) ip=$(hostname -I 2>/dev/null | awk '{print $1}')"
+
+for c in $(podman ps -a --format '{{.Names}}' 2>/dev/null | sort); do
+  assert_container_ports "$c" || true
+done
+
+container_exists archy-bitcoin-ui && {
+  assert_http "bitcoin-ui" "http://127.0.0.1/app/bitcoin-ui/"
+  assert_http "bitcoin status" "http://127.0.0.1/app/bitcoin-ui/bitcoin-status"
+  assert_http_post "bitcoin rpc proxy" "http://127.0.0.1/app/bitcoin-ui/bitcoin-rpc/"
+}
+
+container_exists archy-electrs-ui && {
+  assert_http "electrumx ui" "http://127.0.0.1/app/electrumx/"
+  assert_http "electrumx status" "http://127.0.0.1/app/electrumx/electrs-status"
+  assert_http "electrs legacy status" "http://127.0.0.1/app/electrs/electrs-status"
+}
+
+container_exists mempool && assert_http "mempool ui" "http://127.0.0.1/app/mempool/"
+container_exists indeedhub && assert_http "indeedhub ui" "http://127.0.0.1:7778/"
+container_exists uptime-kuma && assert_http "uptime-kuma" "http://127.0.0.1/app/uptime-kuma/"
+container_exists filebrowser && assert_http "filebrowser" "http://127.0.0.1/app/filebrowser/"
+container_exists searxng && assert_http "searxng" "http://127.0.0.1/app/searxng/"
+container_exists grafana && assert_http "grafana" "http://127.0.0.1/app/grafana/"
+container_exists portainer && assert_http "portainer" "http://127.0.0.1/app/portainer/"
+container_exists vaultwarden && assert_http "vaultwarden" "http://127.0.0.1/app/vaultwarden/"
+container_exists nextcloud && assert_http "nextcloud" "http://127.0.0.1/app/nextcloud/"
+container_exists archy-nbxplorer && assert_env_contains "archy-nbxplorer" "NBXPLORER_POSTGRES" "Database=nbxplorer"
+container_exists btcpay-server && {
+  assert_env_contains "btcpay-server" "BTCPAY_POSTGRES" "Database=btcpay"
+  assert_http "btcpay" "http://127.0.0.1/app/btcpay/"
+}
+
+echo "[surface] summary: pass=$pass fail=$fail"
+[ "$fail" -eq 0 ]
+REMOTE

scripts/bitcoin-stack-lifecycle-test.sh

@@ -0,0 +1,249 @@
+#!/bin/bash
+#
+# Bitcoin stack lifecycle test.
+#
+# Exercises the production Bitcoin stack under repeated stop/start and
+# remove/recreate cycles while asserting the actual user-facing surfaces:
+# Bitcoin RPC, bitcoin-ui /bitcoin-rpc, ElectrumX status, and electrs-ui.
+#
+# This intentionally removes containers but not data volumes. It is safe for
+# installed nodes, but it will briefly interrupt Bitcoin/ElectrumX service.
+#
+# Usage:
+#   scripts/bitcoin-stack-lifecycle-test.sh --target archipelago@192.168.1.228
+#   scripts/bitcoin-stack-lifecycle-test.sh --target archipelago@192.168.1.116 --cycles 5
+
+set -euo pipefail
+
+TARGET=""
+SSH_KEY="${ARCHIPELAGO_SSH_KEY:-}"
+CYCLES=3
+SSH_EXTRA=()
+
+while [ "$#" -gt 0 ]; do
+  case "$1" in
+    --target)
+      TARGET="${2:-}"
+      shift 2
+      ;;
+    --ssh-key)
+      SSH_KEY="${2:-}"
+      shift 2
+      ;;
+    --cycles)
+      CYCLES="${2:-}"
+      shift 2
+      ;;
+    --ssh-option)
+      SSH_EXTRA+=("-o" "${2:-}")
+      shift 2
+      ;;
+    -h|--help)
+      sed -n '1,22p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "unknown argument: $1" >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [ -z "$TARGET" ]; then
+  echo "--target is required, for example archipelago@192.168.1.228" >&2
+  exit 2
+fi
+
+SSH=(ssh -F /dev/null -o BatchMode=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null)
+if [ -n "$SSH_KEY" ]; then
+  SSH+=("-i" "$SSH_KEY")
+fi
+SSH+=("${SSH_EXTRA[@]}")
+
+"${SSH[@]}" "$TARGET" "CYCLES='$CYCLES' bash -s" <<'REMOTE'
+set -euo pipefail
+
+PODMAN="${PODMAN:-podman}"
+SCRIPTS_DIR="/opt/archipelago/scripts"
+if [ ! -x "$SCRIPTS_DIR/reconcile-containers.sh" ]; then
+  SCRIPTS_DIR="$HOME/archy/scripts"
+fi
+RECONCILE="$SCRIPTS_DIR/reconcile-containers.sh"
+
+pass_count=0
+fail_count=0
+
+log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
+pass() { pass_count=$((pass_count + 1)); printf '  PASS %s\n' "$*"; }
+fail() { fail_count=$((fail_count + 1)); printf '  FAIL %s\n' "$*" >&2; }
+
+retry() {
+  local timeout="$1" label="$2"
+  shift 2
+  local end=$((SECONDS + timeout))
+  local out rc
+  while [ "$SECONDS" -lt "$end" ]; do
+    set +e
+    out=$("$@" 2>&1)
+    rc=$?
+    set -e
+    if [ "$rc" -eq 0 ]; then
+      pass "$label"
+      return 0
+    fi
+    sleep 2
+  done
+  fail "$label: $out"
+  return 1
+}
+
+rpc_pass() {
+  cat /var/lib/archipelago/secrets/bitcoin-rpc-password
+}
+
+json_rpc_reachable_or_warming() {
+  local url="$1" auth_arg=() body rc
+  if [ "${2:-}" = "auth" ]; then
+    auth_arg=(--user "archipelago:$(rpc_pass)")
+  fi
+  set +e
+  body=$(curl --connect-timeout 3 --max-time 20 -sS "${auth_arg[@]}" \
+    -H "Content-Type: application/json" \
+    --data-binary '{"jsonrpc":"1.0","id":"lifecycle-test","method":"getblockchaininfo","params":[]}' \
+    "$url" 2>&1)
+  rc=$?
+  set -e
+  [ "$rc" -eq 0 ] || {
+    echo "$body"
+    return 1
+  }
+  echo "$body" | grep -q '"result"' && return 0
+  echo "$body" | grep -q '"code":-28' && return 0
+  echo "$body"
+  return 1
+}
+
+bitcoin_status_usable() {
+  local url="$1"
+  local body
+  body=$(curl --connect-timeout 3 --max-time 20 -fsS "$url")
+  echo "$body" | grep -q '"ok":\(true\|false\)' || {
+    echo "$body"
+    return 1
+  }
+  echo "$body" | grep -q '"blockchain_info"' || echo "$body" | grep -q '"error"'
+}
+
+http_ok() {
+  local url="$1"
+  curl --connect-timeout 3 --max-time 20 -fsS -o /dev/null "$url"
+}
+
+electrs_status_ok() {
+  local url="${1:-http://127.0.0.1:50002/electrs-status}"
+  local body
+  body=$(curl --connect-timeout 3 --max-time 20 -fsS "$url")
+  echo "$body" | grep -q '"network_height":[1-9]' || {
+    echo "$body"
+    return 1
+  }
+  echo "$body" | grep -q '"status":"\(indexing\|syncing\|synced\|waiting\)"'
+}
+
+container_running() {
+  local name="$1"
+  [ "$($PODMAN inspect "$name" --format '{{.State.Status}}' 2>/dev/null || true)" = "running" ]
+}
+
+container_healthy_or_starting() {
+  local name="$1"
+  local health
+  health=$($PODMAN inspect "$name" --format '{{if .State.Health}}{{.State.Health.Status}}{{end}}' 2>/dev/null || true)
+  [ "$health" = "healthy" ] || [ "$health" = "starting" ] || [ -z "$health" ]
+}
+
+assert_bitcoin_stack() {
+  retry 90 "bitcoin-knots running" container_running bitcoin-knots
+  retry 90 "bitcoin-knots healthy/starting" container_healthy_or_starting bitcoin-knots
+  retry 90 "host Bitcoin RPC reachable/ready" json_rpc_reachable_or_warming http://127.0.0.1:8332/ auth
+  retry 90 "backend Bitcoin status bridge usable" bitcoin_status_usable http://127.0.0.1:5678/bitcoin-status
+  retry 90 "bitcoin-ui page" http_ok http://127.0.0.1:8334/
+  retry 90 "bitcoin-ui status bridge usable" bitcoin_status_usable http://127.0.0.1:8334/bitcoin-status
+  retry 90 "bitcoin-ui app-session status bridge usable" bitcoin_status_usable http://127.0.0.1/app/bitcoin-ui/bitcoin-status
+  retry 90 "bitcoin-ui RPC proxy reachable/ready" json_rpc_reachable_or_warming http://127.0.0.1:8334/bitcoin-rpc/
+  retry 90 "bitcoin-ui app-session RPC proxy reachable/ready" json_rpc_reachable_or_warming http://127.0.0.1/app/bitcoin-ui/bitcoin-rpc/
+}
+
+assert_electrum_stack() {
+  retry 120 "electrumx running" container_running electrumx
+  retry 120 "electrumx healthy/starting" container_healthy_or_starting electrumx
+  retry 90 "electrs-ui page" http_ok http://127.0.0.1:50002/
+  retry 120 "electrs status has network height" electrs_status_ok
+  retry 120 "electrs app-session status has network height" electrs_status_ok http://127.0.0.1/app/electrumx/electrs-status
+  retry 120 "electrs legacy app-session status has network height" electrs_status_ok http://127.0.0.1/app/electrs/electrs-status
+}
+
+reconcile_one() {
+  local name="$1"
+  "$RECONCILE" --container="$name" --force --force-recreate --create-missing
+}
+
+restart_container() {
+  local name="$1"
+  log "restart $name"
+  $PODMAN restart "$name" >/dev/null || {
+    log "podman restart failed for $name; using stop/start"
+    $PODMAN stop "$name" >/dev/null 2>&1 || true
+    sleep 3
+    $PODMAN start "$name" >/dev/null
+  }
+}
+
+remove_and_reconcile() {
+  local name="$1"
+  log "remove/recreate $name"
+  $PODMAN rm -f "$name" >/dev/null 2>&1 || true
+  reconcile_one "$name"
+}
+
+log "target $(hostname) cycles=$CYCLES"
+log "using reconciler: $RECONCILE"
+
+assert_bitcoin_stack
+assert_electrum_stack
+
+for i in $(seq 1 "$CYCLES"); do
+  log "cycle $i/$CYCLES: bitcoin restart"
+  restart_container bitcoin-knots
+  assert_bitcoin_stack
+  assert_electrum_stack
+
+  log "cycle $i/$CYCLES: bitcoin remove/reconcile"
+  remove_and_reconcile bitcoin-knots
+  assert_bitcoin_stack
+  assert_electrum_stack
+
+  log "cycle $i/$CYCLES: bitcoin UI remove/reconcile"
+  remove_and_reconcile archy-bitcoin-ui
+  assert_bitcoin_stack
+
+  log "cycle $i/$CYCLES: electrumx restart"
+  restart_container electrumx
+  assert_electrum_stack
+
+  log "cycle $i/$CYCLES: electrumx remove/reconcile"
+  remove_and_reconcile electrumx
+  assert_electrum_stack
+
+  log "cycle $i/$CYCLES: electrs UI remove/reconcile"
+  remove_and_reconcile archy-electrs-ui
+  assert_electrum_stack
+done
+
+log "final container state"
+$PODMAN ps -a --format 'table {{.Names}}\t{{.State}}\t{{.Status}}' \
+  | grep -E 'bitcoin-knots|electrumx|archy-bitcoin-ui|archy-electrs-ui' || true
+
+log "summary: pass=$pass_count fail=$fail_count"
+[ "$fail_count" -eq 0 ]
+REMOTE

scripts/container-doctor.sh

@@ -15,6 +15,7 @@
 #   6. Bitcoin Knots prune+txindex conflict
 #   7. Containers stuck with exit code 127 (binary not found)
 #   8. Stopped core containers (rootless restart policy workaround)
+#   9. Missing rootless port listeners while Podman still shows published ports
 #
 # Safe to run multiple times (idempotent). Never blocks deploy (exit 0 always).
 #
@@ -31,6 +32,21 @@ FIX_NAMES=()
 
 log() { echo "[$(date +%H:%M:%S)] DOCTOR: $*"; }
 
+podman_rootless() {
+    if [ "$(id -u)" = "0" ] && id archipelago >/dev/null 2>&1; then
+        local archi_uid
+        archi_uid=$(id -u archipelago)
+        sudo -u archipelago env XDG_RUNTIME_DIR="/run/user/$archi_uid" podman "$@"
+    else
+        podman "$@"
+    fi
+}
+
+port_is_listening() {
+    local port="$1"
+    ss -ltn 2>/dev/null | awk '{print $4}' | grep -Eq "(^|:)$port$"
+}
+
 run_fix() {
     local name="$1"
     shift
@@ -374,6 +390,11 @@ print(' '.join(['\"' + a + '\"' if ' ' in a else a for a in args[2:]]))
 # at 0 peers; package pulls fail. The only reliable repair is a stop-all/
 # start-all cycle so pasta + aardvark-dns rebuild the netns from scratch.
 fix_rootless_netns_egress() {
+    # Needs root for nsenter. When doctor runs as the rootless container owner,
+    # a failed nsenter probe is a permissions artifact, not evidence of broken
+    # egress; do not cycle the fleet from that context.
+    [ "$(id -u)" = "0" ] || return 1
+
     local archi_uid
     archi_uid=$(id -u archipelago 2>/dev/null) || return 1
 
@@ -453,6 +474,44 @@ fix_stopped_core_containers() {
     [ ${#restarted[@]} -gt 0 ] && return 0 || return 1
 }
 
+# ── Fix 10: Missing rootless port listeners ─────────────────
+# Rootless Podman can leave a container running with PortBindings still present
+# while the host-side rootlessport process has disappeared. Nginx then returns
+# 502 and direct app ports refuse connections even though `podman ps` looks OK.
+fix_missing_rootless_ports() {
+    local containers
+    containers=$(podman_rootless ps --format '{{.Names}}' 2>/dev/null || true)
+    [ -n "$containers" ] || return 1
+
+    local fixed=false
+    local name
+    for name in $containers; do
+        local ports
+        ports=$(podman_rootless inspect "$name" --format '{{range $p,$bindings := .NetworkSettings.Ports}}{{if $bindings}}{{range $bindings}}{{.HostPort}}{{"\n"}}{{end}}{{end}}{{end}}' 2>/dev/null | sort -u)
+        [ -n "$ports" ] || continue
+
+        local missing=()
+        local port
+        for port in $ports; do
+            [ -n "$port" ] || continue
+            if ! port_is_listening "$port"; then
+                missing+=("$port")
+            fi
+        done
+
+        if [ ${#missing[@]} -gt 0 ]; then
+            log "Restarting $name: missing rootlessport listener(s): ${missing[*]}"
+            if podman_rootless restart "$name" >/dev/null 2>&1; then
+                fixed=true
+            else
+                log "WARN: failed to restart $name for missing rootlessport listener(s)"
+            fi
+        fi
+    done
+
+    $fixed && return 0 || return 1
+}
+
 # ── Main ─────────────────────────────────────────────────────
 
 # If remote host provided, run via SSH
@@ -481,6 +540,7 @@ run_fix "bitcoin-txindex" fix_bitcoin_txindex
 run_fix "exit-127"        fix_exit_127
 run_fix "netns-egress"    fix_rootless_netns_egress
 run_fix "stopped-core"    fix_stopped_core_containers
+run_fix "rootless-ports" fix_missing_rootless_ports
 
 echo ""
 if [ $FIXES_APPLIED -gt 0 ]; then

scripts/container-specs.sh

@@ -252,7 +252,7 @@ load_spec_archy-nbxplorer() {
   SPEC_VOLUMES="/var/lib/archipelago/nbxplorer:/data"
   SPEC_MEMORY="$(mem_limit archy-nbxplorer)"
   SPEC_HEALTH_CMD="curl -sf http://localhost:32838/ || exit 1"
-  SPEC_ENV="NBXPLORER_DATADIR=/data NBXPLORER_NETWORK=mainnet NBXPLORER_CHAINS=btc NBXPLORER_BIND=0.0.0.0:32838 NBXPLORER_BTCRPCURL=http://bitcoin-knots:8332 NBXPLORER_BTCRPCUSER=$BITCOIN_RPC_USER NBXPLORER_BTCRPCPASSWORD=$BITCOIN_RPC_PASS NBXPLORER_POSTGRES=User ID=btcpay;Password=$BTCPAY_DB_PASS;Host=archy-btcpay-db;Port=5432;Database=nbxplorer;Include Error Detail=true"
+  SPEC_ENV="NBXPLORER_DATADIR=/data NBXPLORER_NETWORK=mainnet NBXPLORER_CHAINS=btc NBXPLORER_BIND=0.0.0.0:32838 NBXPLORER_BTCRPCURL=http://bitcoin-knots:8332 NBXPLORER_BTCRPCUSER=$BITCOIN_RPC_USER NBXPLORER_BTCRPCPASSWORD=$BITCOIN_RPC_PASS NBXPLORER_POSTGRES=Username=btcpay;Password=$BTCPAY_DB_PASS;Host=archy-btcpay-db;Port=5432;Database=nbxplorer"
   SPEC_TIER="2"
   SPEC_DATA_DIR="/var/lib/archipelago/nbxplorer"
   SPEC_DEPENDS="bitcoin-knots archy-btcpay-db"
@@ -268,7 +268,7 @@ load_spec_btcpay-server() {
   SPEC_VOLUMES="/var/lib/archipelago/btcpay:/datadir"
   SPEC_MEMORY="$(mem_limit btcpay-server)"
   SPEC_HEALTH_CMD="curl -sf http://localhost:49392/ || exit 1"
-  SPEC_ENV="ASPNETCORE_URLS=http://0.0.0.0:49392 BTCPAY_PROTOCOL=http BTCPAY_HOST=$HOST_IP:23000 BTCPAY_CHAINS=btc BTCPAY_BTCEXPLORERURL=http://archy-nbxplorer:32838 BTCPAY_BTCRPCURL=http://bitcoin-knots:8332 BTCPAY_BTCRPCUSER=$BITCOIN_RPC_USER BTCPAY_BTCRPCPASSWORD=$BITCOIN_RPC_PASS BTCPAY_POSTGRES=User ID=btcpay;Password=$BTCPAY_DB_PASS;Host=archy-btcpay-db;Port=5432;Database=btcpay;Include Error Detail=true"
+  SPEC_ENV="ASPNETCORE_URLS=http://0.0.0.0:49392 BTCPAY_PROTOCOL=http BTCPAY_HOST=$HOST_IP:23000 BTCPAY_CHAINS=btc BTCPAY_BTCEXPLORERURL=http://archy-nbxplorer:32838 BTCPAY_BTCRPCURL=http://bitcoin-knots:8332 BTCPAY_BTCRPCUSER=$BITCOIN_RPC_USER BTCPAY_BTCRPCPASSWORD=$BITCOIN_RPC_PASS BTCPAY_POSTGRES=Username=btcpay;Password=$BTCPAY_DB_PASS;Host=archy-btcpay-db;Port=5432;Database=btcpay"
   SPEC_TIER="2"
   SPEC_DATA_DIR="/var/lib/archipelago/btcpay"
   SPEC_DEPENDS="archy-nbxplorer archy-btcpay-db"
@@ -344,7 +344,7 @@ load_spec_homeassistant() {
   SPEC_ENV="TZ=UTC"
   SPEC_TIER="3"
   SPEC_DATA_DIR="/var/lib/archipelago/home-assistant"
-  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE"
+  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE NET_BIND_SERVICE"
   SPEC_OPTIONAL="true"
 }
 
@@ -362,7 +362,7 @@ load_spec_grafana() {
   SPEC_TIER="3"
   SPEC_DATA_DIR="/var/lib/archipelago/grafana"
   SPEC_DATA_UID="100472:100472"
-  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE"
+  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE NET_BIND_SERVICE"
   SPEC_OPTIONAL="true"
 }
 
@@ -370,7 +370,7 @@ load_spec_uptime-kuma() {
   reset_spec
   SPEC_NAME="uptime-kuma"
   SPEC_IMAGE="${UPTIME_KUMA_IMAGE}"
-  SPEC_PORTS="3001:3001"
+  SPEC_PORTS="3002:3001"
   SPEC_VOLUMES="/var/lib/archipelago/uptime-kuma:/app/data"
   SPEC_MEMORY="$(mem_limit uptime-kuma)"
   SPEC_HEALTH_CMD="curl -sf http://localhost:3001/ || exit 1"
@@ -434,7 +434,7 @@ load_spec_nextcloud() {
   SPEC_HEALTH_CMD="curl -sf http://localhost:80/ || exit 1"
   SPEC_TIER="3"
   SPEC_DATA_DIR="/var/lib/archipelago/nextcloud"
-  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE"
+  SPEC_CAPS="CHOWN SETUID SETGID DAC_OVERRIDE NET_BIND_SERVICE"
   SPEC_OPTIONAL="true"
 }
 
@@ -539,6 +539,7 @@ load_spec_archy-bitcoin-ui() {
   SPEC_NAME="archy-bitcoin-ui"
   SPEC_IMAGE="localhost/bitcoin-ui:local"
   SPEC_NETWORK="host"
+  SPEC_VOLUMES="/var/lib/archipelago/bitcoin-ui/nginx.conf:/etc/nginx/conf.d/default.conf:ro"
   SPEC_MEMORY="$(mem_limit archy-bitcoin-ui)"
   SPEC_TIER="4"
   SPEC_LOCAL_IMAGE="true"

scripts/create-release-manifest.sh

@@ -87,6 +87,30 @@ if [ -z "$FRONTEND_ARCHIVE" ]; then
             echo "  Including AIUI from demo/aiui/"
             cp -r "$PROJECT_ROOT/demo/aiui" "$STAGING_DIR/aiui"
         fi
+        # OTA bridge for nodes running older updaters: they only know how to
+        # apply the backend binary and frontend archive. Carry host runtime
+        # assets inside the frontend tarball; the new backend promotes them
+        # from /opt/archipelago/web-ui/archipelago-runtime on first startup.
+        RUNTIME_DIR="$STAGING_DIR/archipelago-runtime"
+        mkdir -p "$RUNTIME_DIR"
+        for runtime_path in apps scripts docker; do
+            if [ -d "$PROJECT_ROOT/$runtime_path" ]; then
+                echo "  Including runtime $runtime_path/"
+                cp -r "$PROJECT_ROOT/$runtime_path" "$RUNTIME_DIR/$runtime_path"
+            fi
+        done
+        if [ -f "$PROJECT_ROOT/image-recipe/configs/archipelago-doctor.service" ] || \
+           [ -f "$PROJECT_ROOT/image-recipe/configs/archipelago-doctor.timer" ]; then
+            mkdir -p "$RUNTIME_DIR/image-recipe/configs"
+            for unit in archipelago-doctor.service archipelago-doctor.timer; do
+                if [ -f "$PROJECT_ROOT/image-recipe/configs/$unit" ]; then
+                    echo "  Including runtime unit $unit"
+                    cp "$PROJECT_ROOT/image-recipe/configs/$unit" "$RUNTIME_DIR/image-recipe/configs/$unit"
+                fi
+            done
+        fi
+        rm -rf "$RUNTIME_DIR/scripts/resilience/reports"
+        find "$RUNTIME_DIR" -type f \( -name '*.bak' -o -name '._*' -o -name '*.log' \) -delete
         # Force world-readable perms on every entry BEFORE tar, so the
         # archive's internal mode bits are 755/644 regardless of what
         # the staging dir's umask gave us. Without this, mktemp -d

scripts/image-versions.sh

@@ -5,13 +5,13 @@
 # Usage: source /opt/archipelago/image-versions.sh 2>/dev/null || true
 #        source "$(dirname "$0")/image-versions.sh" 2>/dev/null || true
 #
-# Tags MUST match what's actually in the registry at git.tx1138.com/lfg2025/
-# Run: podman images --format '{{.Repository}}:{{.Tag}}' | grep 'git.tx1138' | sort
+# Tags MUST match what's actually in the registry at 146.59.87.168:3000/lfg2025/
+# Run: podman images --format '{{.Repository}}:{{.Tag}}' | grep '146.59.87.168:3000' | sort
 # to verify against the registry.
 
 # Archipelago app registries (primary + fallback)
-ARCHY_REGISTRY="git.tx1138.com/lfg2025"
-ARCHY_REGISTRY_FALLBACK="146.59.87.168:3000/lfg2025"
+ARCHY_REGISTRY="146.59.87.168:3000/lfg2025"
+ARCHY_REGISTRY_FALLBACK="git.tx1138.com/lfg2025"
 
 # Bitcoin stack
 BITCOIN_KNOTS_IMAGE="$ARCHY_REGISTRY/bitcoin-knots:latest"

scripts/nginx-https-app-proxies.conf

@@ -183,6 +183,26 @@ location /app/electrs/ {
     proxy_hide_header X-Frame-Options;
     proxy_hide_header Content-Security-Policy;
 }
+location /app/electrumx/ {
+    proxy_pass http://127.0.0.1:50002/;
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_hide_header X-Frame-Options;
+    proxy_hide_header Content-Security-Policy;
+}
+location /app/electrs-ui/ {
+    proxy_pass http://127.0.0.1:50002/;
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_hide_header X-Frame-Options;
+    proxy_hide_header Content-Security-Policy;
+}
 location /app/nginx-proxy-manager/ {
     proxy_pass http://127.0.0.1:81/;
     proxy_http_version 1.1;

scripts/reconcile-containers.sh

@@ -8,6 +8,7 @@
 #   sudo ./reconcile-containers.sh                 # Fix everything
 #   sudo ./reconcile-containers.sh --check-only    # Audit only, no changes
 #   sudo ./reconcile-containers.sh --force          # Override user-stopped
+#   sudo ./reconcile-containers.sh --force-recreate # Recreate matched containers
 #   sudo ./reconcile-containers.sh --tier=2         # Only reconcile tier 2
 #   sudo ./reconcile-containers.sh --container=lnd  # Only reconcile lnd
 #
@@ -18,6 +19,7 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 # ── Parse arguments ──────────────────────────────────────────────────
 CHECK_ONLY=false
 FORCE=false
+FORCE_RECREATE=false
 CREATE_MISSING=false
 FILTER_TIER=""
 FILTER_CONTAINER=""
@@ -25,14 +27,18 @@ for arg in "$@"; do
   case "$arg" in
     --check-only)         CHECK_ONLY=true ;;
     --force)              FORCE=true ;;
+    --force-recreate)     FORCE_RECREATE=true ;;
     --create-missing)     CREATE_MISSING=true ;;
     --tier=*)             FILTER_TIER="${arg#*=}" ;;
     --container=*)        FILTER_CONTAINER="${arg#*=}" ;;
     -h|--help)
-      echo "Usage: $0 [--check-only] [--force] [--create-missing] [--tier=N] [--container=NAME]"
+      echo "Usage: $0 [--check-only] [--force] [--force-recreate] [--create-missing] [--tier=N] [--container=NAME]"
       echo ""
       echo "  --check-only      Audit only, no changes."
       echo "  --force           Override user-stopped state."
+      echo "  --force-recreate  Recreate matched existing containers even if they"
+      echo "                    otherwise match the spec. Use with --container or"
+      echo "                    --tier for scoped image/config refreshes."
       echo "  --create-missing  Override SPEC_OPTIONAL for containers that have on-disk"
       echo "                    data but no live container (recovery from failed updates)."
       echo "  --tier=N          Only reconcile containers in tier N."
@@ -110,6 +116,14 @@ container_image() {
   $PODMAN inspect "$1" --format '{{.ImageName}}' 2>/dev/null
 }
 
+container_image_id() {
+  $PODMAN inspect "$1" --format '{{.Image}}' 2>/dev/null
+}
+
+spec_image_id() {
+  $PODMAN image inspect "$SPEC_IMAGE" --format '{{.Id}}' 2>/dev/null
+}
+
 container_network() {
   # Use actual Networks map — NetworkMode is unreliable (always shows 'bridge' in rootless)
   local nets
@@ -122,6 +136,34 @@ container_memory() {
   $PODMAN inspect "$1" --format '{{.HostConfig.Memory}}' 2>/dev/null
 }
 
+container_health_cmd() {
+  $PODMAN inspect "$1" --format '{{with .Config.Healthcheck}}{{range .Test}}{{println .}}{{end}}{{end}}' 2>/dev/null \
+    | awk 'NR > 1 { print }' \
+    | paste -sd ' ' -
+}
+
+normalize_health_cmd() {
+  printf '%s' "$1" | sed 's/\\"/"/g; s/[[:space:]][[:space:]]*/ /g; s/^ //; s/ $//'
+}
+
+host_port_listening() {
+  local port="$1"
+  ss -ltn 2>/dev/null | awk -v p=":$port" '
+    $4 == p || $4 ~ p "$" { found=1 }
+    END { exit found ? 0 : 1 }
+  '
+}
+
+container_has_mount() {
+  local name="$1" source="$2" target="$3"
+  $PODMAN inspect "$name" --format '{{range .Mounts}}{{println .Source "|" .Destination}}{{end}}' 2>/dev/null \
+    | awk -F'|' -v src="$source" -v dst="$target" '
+        { gsub(/[[:space:]]+$/, "", $1); gsub(/^[[:space:]]+/, "", $2); }
+        $1 == src && $2 == dst { found=1 }
+        END { exit found ? 0 : 1 }
+      '
+}
+
 # Read one environment variable's current value from a running/stopped container.
 # Returns empty string if the var is not set.
 container_env_val() {
@@ -153,6 +195,36 @@ image_exists() {
   echo "$images" | grep -qF "$1"
 }
 
+resolve_spec_image() {
+  image_exists "$SPEC_IMAGE" && return
+
+  local image_path image_name image_tag candidate repo
+  image_path="${SPEC_IMAGE#*/}"
+  image_name="${SPEC_IMAGE##*/}"
+  image_tag="${image_name#*:}"
+  image_name="${image_name%%:*}"
+
+  for candidate in \
+    "${ARCHY_REGISTRY_FALLBACK:-}/${image_path}" \
+    "80.71.235.15:3000/archipelago/${image_name}:${image_tag}" \
+    "80.71.235.15:3000/lfg2025/${image_name}:${image_tag}"; do
+    [ "$candidate" = "/" ] && continue
+    if image_exists "$candidate"; then
+      info "$SPEC_NAME — using local image alias $candidate"
+      SPEC_IMAGE="$candidate"
+      return
+    fi
+  done
+
+  repo=$($PODMAN images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null \
+    | grep -E "/${image_name}:${image_tag}$" \
+    | head -1 || true)
+  if [ -n "$repo" ]; then
+    info "$SPEC_NAME — using local image alias $repo"
+    SPEC_IMAGE="$repo"
+  fi
+}
+
 # Convert memory string to bytes for comparison
 mem_to_bytes() {
   local m="$1"
@@ -262,6 +334,10 @@ reconcile() {
     return
   fi
 
+  # Resolve registry aliases before create/recreate. ISOs and older installers
+  # may seed the same image under a fallback registry tag.
+  resolve_spec_image
+
   # Local images: skip if image doesn't exist and container doesn't exist
   if [ "$SPEC_LOCAL_IMAGE" = "true" ]; then
     if ! image_exists "$SPEC_IMAGE" && ! container_exists "$name"; then
@@ -284,14 +360,28 @@ reconcile() {
   local reasons=""
 
   if container_exists "$name"; then
-    local cur_image cur_network cur_memory
+    local cur_image cur_image_id want_image_id cur_network cur_memory
     cur_image=$(container_image "$name")
+    cur_image_id=$(container_image_id "$name")
+    want_image_id=$(spec_image_id)
     cur_network=$(container_network "$name")
     cur_memory=$(container_memory "$name")
     local spec_memory_bytes expected_network
 
     spec_memory_bytes=$(mem_to_bytes "$SPEC_MEMORY")
 
+    if [ "$FORCE_RECREATE" = "true" ]; then
+      action="RECREATE"
+      reasons+="force-recreate "
+    fi
+
+    # Same-tag local rebuilds leave running containers on the old image ID.
+    # Recreate when the currently tagged spec image points at a different ID.
+    if [ "$action" = "OK" ] && [ -n "$want_image_id" ] && [ -n "$cur_image_id" ] && [ "$cur_image_id" != "$want_image_id" ]; then
+      action="RECREATE"
+      reasons+="image-id "
+    fi
+
     # Check network mismatch
     # For archy-net and host: exact match required
     # For bridge/default: accept any non-archy-net, non-host network
@@ -319,6 +409,19 @@ reconcile() {
       reasons+="memory(none→$SPEC_MEMORY) "
     fi
 
+    # Healthcheck drift matters: a stale check can leave an otherwise working
+    # service permanently unhealthy (for example ElectrumX images do not ship
+    # curl, so the healthcheck must use python's socket module).
+    if [ "$action" = "OK" ] && [ -n "$SPEC_HEALTH_CMD" ]; then
+      local cur_health spec_health
+      cur_health=$(normalize_health_cmd "$(container_health_cmd "$name")")
+      spec_health=$(normalize_health_cmd "$SPEC_HEALTH_CMD")
+      if [ "$cur_health" != "$spec_health" ]; then
+        action="RECREATE"
+        reasons+="healthcheck "
+      fi
+    fi
+
     # Check URL/HOST env drift — catches stale network topology baked into
     # container env (fedimint April-11 bug: FM_P2P_URL pointed at old IP).
     # Only checks URL-shaped keys; other env drift (passwords rotated, etc.)
@@ -342,6 +445,40 @@ reconcile() {
       done
     fi
 
+    # Check bind mounts. This catches companion UIs recreated from older specs,
+    # especially bitcoin-ui: its image intentionally does not bake nginx.conf,
+    # so the rendered RPC proxy config must be mounted from the host.
+    if [ "$action" = "OK" ] && [ -n "$SPEC_VOLUMES" ]; then
+      for v in $SPEC_VOLUMES; do
+        local mount_source mount_rest mount_target
+        mount_source="${v%%:*}"
+        mount_rest="${v#*:}"
+        mount_target="${mount_rest%%:*}"
+        [ -n "$mount_source" ] && [ -n "$mount_target" ] || continue
+        if ! container_has_mount "$name" "$mount_source" "$mount_target"; then
+          action="RECREATE"
+          reasons+="mount($mount_target) "
+          break
+        fi
+      done
+    fi
+
+    # Rootless Podman can occasionally leave a container running while its
+    # rootlessport listener is gone. The container still looks healthy in
+    # `podman ps`, but host-network UIs and backend status probes fail against
+    # 127.0.0.1. Treat missing host listeners as spec drift.
+    if [ "$action" = "OK" ] && [ -n "$SPEC_PORTS" ]; then
+      for p in $SPEC_PORTS; do
+        local host_port="${p%%:*}"
+        [ -n "$host_port" ] || continue
+        if ! host_port_listening "$host_port"; then
+          action="RECREATE"
+          reasons+="port($host_port-not-listening) "
+          break
+        fi
+      done
+    fi
+
     # Check if running
     if ! container_running "$name" && [ "$action" = "OK" ]; then
       action="START"
@@ -476,7 +613,7 @@ ensure_secrets() {
 ensure_bitcoin_conf() {
   local BITCOIN_CONF="/var/lib/archipelago/bitcoin/bitcoin.conf"
   sudo mkdir -p /var/lib/archipelago/bitcoin 2>/dev/null
-  if [ ! -f "$BITCOIN_CONF" ] || ! grep -q "^rpcauth=" "$BITCOIN_CONF" 2>/dev/null; then
+  if [ ! -f "$BITCOIN_CONF" ] || ! sudo grep -q "^rpcauth=" "$BITCOIN_CONF" 2>/dev/null; then
     if ! $CHECK_ONLY && [ -n "$BITCOIN_RPC_PASS" ]; then
       local salt hash rpcauth
       salt=$(openssl rand -hex 16)
@@ -491,10 +628,14 @@ BTCEOF
       info "Generated bitcoin.conf"
     fi
   fi
-  # Strip duplicate server/rpc/listen lines from existing conf to avoid conflicts with custom args
-  if [ -f "$BITCOIN_CONF" ]; then
-    sudo sed -i '/^server=/d; /^rpcbind=/d; /^rpcallowip=/d; /^rpcport=/d; /^listen=/d' "$BITCOIN_CONF" 2>/dev/null
-  fi
+  # Strip duplicate server/rpc/listen lines from existing conf files to avoid
+  # conflicts with custom args. Knots can persist runtime args in
+  # bitcoin_rw.conf, so clean both files.
+  for conf in "$BITCOIN_CONF" "/var/lib/archipelago/bitcoin/bitcoin_rw.conf"; do
+    if [ -f "$conf" ]; then
+      sudo sed -i '/^server=/d; /^txindex=/d; /^rpcbind=/d; /^rpcallowip=/d; /^rpcport=/d; /^listen=/d; /^bind=/d; /^dbcache=/d' "$conf" 2>/dev/null
+    fi
+  done
   sudo chown -R 100101:100101 /var/lib/archipelago/bitcoin 2>/dev/null
 }
 
@@ -531,6 +672,63 @@ LNDEOF
   fi
 }
 
+# ── Ensure bitcoin-ui nginx.conf ────────────────────────────────────
+ensure_bitcoin_ui_nginx_conf() {
+  local CONF_DIR="/var/lib/archipelago/bitcoin-ui"
+  local CONF_PATH="$CONF_DIR/nginx.conf"
+  [ -n "$BITCOIN_RPC_PASS" ] || return
+  if $CHECK_ONLY; then
+    [ -f "$CONF_PATH" ] || info "Would generate bitcoin-ui nginx.conf"
+    return
+  fi
+
+  local auth_b64 tmp
+  auth_b64=$(printf '%s' "${BITCOIN_RPC_USER}:${BITCOIN_RPC_PASS}" | base64 | tr -d '\n')
+  sudo mkdir -p "$CONF_DIR" 2>/dev/null
+  tmp="${CONF_PATH}.tmp.$$"
+  sudo tee "$tmp" >/dev/null << EOF
+server {
+    listen 8334;
+    server_name _;
+    root /usr/share/nginx/html;
+    index index.html;
+
+    location /bitcoin-rpc/ {
+        proxy_pass http://127.0.0.1:8332/;
+        proxy_http_version 1.1;
+        proxy_set_header Host \$host;
+        proxy_set_header X-Real-IP \$remote_addr;
+        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+        proxy_set_header Authorization "Basic ${auth_b64}";
+        add_header Access-Control-Allow-Origin *;
+        add_header Access-Control-Allow-Methods "POST, GET, OPTIONS";
+        add_header Access-Control-Allow-Headers "Content-Type, Authorization";
+        if (\$request_method = OPTIONS) { return 204; }
+    }
+
+    location /bitcoin-status {
+        proxy_pass http://127.0.0.1:5678/bitcoin-status;
+        proxy_http_version 1.1;
+        proxy_set_header Host \$host;
+        proxy_set_header X-Real-IP \$remote_addr;
+        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+        add_header Cache-Control "no-store";
+    }
+
+    location / {
+        try_files \$uri \$uri/ /index.html;
+    }
+}
+EOF
+  if ! sudo cmp -s "$tmp" "$CONF_PATH" 2>/dev/null; then
+    sudo mv "$tmp" "$CONF_PATH"
+    sudo chmod 644 "$CONF_PATH"
+    info "Generated bitcoin-ui nginx.conf"
+  else
+    sudo rm -f "$tmp"
+  fi
+}
+
 # ── Ensure BTCPay databases ─────────────────────────────────────────
 ensure_btcpay_db() {
   if container_running "archy-btcpay-db"; then
@@ -548,8 +746,10 @@ START_TIME=$(date +%s)
 
 header "Phase 0: Prerequisites"
 ensure_secrets
+detect_environment
 ensure_bitcoin_conf
 ensure_lnd_conf
+ensure_bitcoin_ui_nginx_conf
 
 TIER_NAMES=("Databases" "Core Infrastructure" "Services" "Applications" "Frontend UIs")
 

scripts/resilience/README.md

@@ -0,0 +1,109 @@
+# Resilience Harness
+
+Black-box state-machine tester for archipelago app containers.
+
+Drives the live RPC against a real archipelago + podman runtime on a target
+host. For each app in `app-catalog/catalog.json`, runs every state transition
+a user could trigger and asserts the system stays in the expected state.
+
+## Why this exists
+
+We shipped v1.7.43-alpha on .228 with three independent bugs that no unit test
+caught:
+
+1. `indeedhub-api` crashlooped 8500+ times because `stacks.rs` was missing 5
+   env vars (`QUEUE_HOST`/`QUEUE_PORT`/`DATABASE_PORT`/`S3_PRIVATE_BUCKET_NAME`/
+   `AES_MASTER_SECRET`) — the install "succeeded" (containers running) but the
+   API never became healthy.
+2. `bitcoin-ui` shipped with a stale baked-in `Authorization: Basic …` header
+   from the registry image, so every `/bitcoin-rpc/` call returned 401.
+3. The container-absence scanner evicted apps from the UI 14 seconds into
+   install (before image pull finished).
+
+All three were exactly the kind of bug a "did the user-visible flow actually
+work end to end?" test would catch — and the kind a single-file unit test
+will never catch. This harness is the gate.
+
+## Running
+
+Against the .228 test node:
+
+    scripts/resilience/resilience.sh archipelago@192.168.1.228
+
+Or non-interactive (CI):
+
+    RESILIENCE_SSH_PASS=… RESILIENCE_UI_PASS=… \
+      scripts/resilience/resilience.sh archipelago@192.168.1.228
+
+Filters:
+
+    # Smoke test (3 apps, no reboot, ~15min)
+    scripts/resilience/resilience.sh archipelago@192.168.1.228 smoke
+
+    # Single app
+    scripts/resilience/resilience.sh archipelago@192.168.1.228 bitcoin-knots
+
+    # Subset
+    scripts/resilience/resilience.sh archipelago@192.168.1.228 bitcoin-knots,lnd
+
+Without a filter, the harness sweeps **every** app in the catalog
+(~24 apps × 7 per-app transitions + 2 batch transitions) and runs the
+batch transitions (archipelago.service restart, host reboot) at the end.
+Full sweep is ~3-4 hours and **reboots the target host** as part of the
+run — only point it at a dedicated test node.
+
+## What it tests
+
+Per-app transitions:
+
+| # | Transition           | Pass criteria                                  |
+|---|----------------------|------------------------------------------------|
+| 1 | install              | All containers reach `running` within 10 min  |
+| 2 | ui_probe             | HTTP 2xx/3xx via `https://<host>/app/<id>/`   |
+| 3 | auth_probe           | (bitcoin-rpc only) returns 200 not 401        |
+| 4 | stop                 | All containers reach `exited` state            |
+| 5 | start                | All containers reach `running` state           |
+| 6 | restart              | All containers `running` after restart         |
+| 7 | uninstall            | All containers absent, no residue              |
+
+Batch transitions (full sweep only):
+
+| # | Transition                    | Pass criteria                       |
+|---|-------------------------------|-------------------------------------|
+| 8 | archipelago.service restart   | Container set unchanged across     |
+| 9 | host reboot                   | Container set unchanged across     |
+
+Coverage by design — discovery rather than encoded metadata. The harness
+snapshots `podman ps -a` before install, again after install stabilizes,
+and the difference IS this app's container set. Works equally well for
+single-container apps and 7-container stacks (indeedhub) without per-app
+configuration.
+
+## Output
+
+JSON-lines results at `scripts/resilience/reports/<run_ts>/results.jsonl`:
+
+    {"ts":"…","app":"bitcoin-knots","transition":"install","status":"PASS","detail":"bitcoin-knots,archy-bitcoin-ui"}
+    {"ts":"…","app":"bitcoin-knots","transition":"auth_probe","status":"PASS","detail":"bitcoin-rpc HTTP 200"}
+
+Exit code: `0` if every cell green, `1` if any red, `2` if setup failed
+before tests began. Use as a release gate — refuse to tag if any cell red.
+
+## Auth flow
+
+The harness uses the same `auth.login` RPC that the UI uses, then carries
+`session=…` and `csrf_token=…` cookies plus the `X-CSRF-Token` header on
+every subsequent call. Re-logs in after archipelago.service restart and
+host reboot.
+
+## Caveats / known gaps
+
+- App proxy probe (`/app/<id>/`) only validates the proxy responds — for
+  apps with deeper protocol behavior (lnd, fedimint, mempool) this only
+  catches "container alive, proxy reachable", not "the protocol is healthy".
+- Multi-container stack assertions: the harness checks **every** new
+  container is `running`, so it would catch the indeedhub-api restart loop
+  while postgres/redis/minio looked fine.
+- Host reboot test is destructive and slow — runs once at end of full sweep.
+- `package.start`/`stop`/`restart` RPC methods may not exist for all apps;
+  failures are recorded and the harness continues.

scripts/resilience/lib.sh

@@ -0,0 +1,297 @@
+#!/bin/bash
+# Resilience harness shared helpers.
+# Sourced by resilience.sh — do not invoke directly.
+
+# Required env (set by resilience.sh before sourcing):
+#   TARGET     — ssh target, e.g. archipelago@192.168.1.228
+#   RPC_URL    — http://<host>:5678/rpc/v1
+#   COOKIE_JAR — path for curl cookie store
+#   SSH_PASS   — sshpass password
+#   UI_PASS    — archipelago UI password
+#   OUT_DIR    — report output dir
+
+# ── ssh ─────────────────────────────────────────────────────────
+ssh_run() {
+    # -n: redirect stdin from /dev/null so ssh doesn't gobble up our parent's
+    #     stdin. Without this, ssh inside a `while read … done <<< "$LIST"`
+    #     consumes the heredoc on the first call, ending the loop after one
+    #     iteration. Cost us a smoke run that only tested filebrowser instead
+    #     of all three smoke apps.
+    sshpass -p "$SSH_PASS" ssh -n -o StrictHostKeyChecking=accept-new \
+        -o ConnectTimeout=10 -o LogLevel=ERROR "$TARGET" "$@"
+}
+
+# Run a command and tolerate ssh failure (host rebooting, etc.).
+ssh_try() {
+    sshpass -p "$SSH_PASS" ssh -n -o StrictHostKeyChecking=accept-new \
+        -o ConnectTimeout=5 -o LogLevel=ERROR "$TARGET" "$@" 2>/dev/null || echo "__SSH_FAIL__"
+}
+
+ssh_wait_ready() {
+    local deadline=$(($(date +%s) + ${1:-180}))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        if [ "$(ssh_try 'echo OK')" = "OK" ]; then return 0; fi
+        sleep 3
+    done
+    return 1
+}
+
+# ── rpc ─────────────────────────────────────────────────────────
+rpc_login() {
+    local resp
+    resp=$(curl -ksS -c "$COOKIE_JAR" -H "Content-Type: application/json" \
+        -d "{\"jsonrpc\":\"2.0\",\"method\":\"auth.login\",\"params\":{\"password\":\"$UI_PASS\"},\"id\":1}" \
+        "$RPC_URL")
+    if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
+        echo "ERROR: login failed: $(echo "$resp" | jq -c .)" >&2
+        return 1
+    fi
+    CSRF_TOKEN=$(awk '/csrf_token/ {print $7}' "$COOKIE_JAR" | head -1)
+    [ -n "$CSRF_TOKEN" ] || { echo "ERROR: no CSRF token after login" >&2; return 1; }
+    export CSRF_TOKEN
+}
+
+# Make an RPC call. Args: method, json_params, timeout_secs (optional, default 90).
+# Prints raw JSON response. Caller asserts success via jq.
+#
+# CSRF rotates per-response: the server may issue a new csrf_token on every
+# state-changing call, so we re-read it from the cookie jar before each call
+# rather than caching the value from login. Also retries once on nginx-served
+# BACKEND_UNAVAILABLE (5xx fallback) for transient stalls.
+rpc_call() {
+    local method="$1"
+    # NOTE: don't use ${2:-{}} — bash matches the first unescaped `}` as the
+    # end of the expansion, so the trailing `}` becomes a literal char and
+    # corrupts every params value into invalid JSON. Use an if-check instead.
+    local params="${2-}"
+    [ -z "$params" ] && params='{}'
+    local timeout="${3:-90}"
+    local attempt
+    for attempt in 1 2 3 4; do
+        local csrf
+        csrf=$(awk '/^[^#]/ && /csrf_token/ {print $7; exit}' "$COOKIE_JAR")
+        local resp
+        resp=$(curl -ksS -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
+            -H "Content-Type: application/json" \
+            -H "X-CSRF-Token: $csrf" \
+            -d "{\"jsonrpc\":\"2.0\",\"method\":\"$method\",\"params\":$params,\"id\":1}" \
+            --max-time "$timeout" \
+            "$RPC_URL")
+        # Retry on transient errors:
+        #   BACKEND_UNAVAILABLE — nginx 5xx fallback (archipelago briefly stalled)
+        #   429 — nginx rate limiter exceeded (burst=40 in /etc/nginx/sites-enabled/*)
+        if echo "$resp" | jq -e '.error.code == "BACKEND_UNAVAILABLE" or .error.code == 429' >/dev/null 2>&1; then
+            [ "$attempt" -eq 4 ] && { echo "$resp"; return; }
+            # Exponential-ish backoff: 5s, 15s, 30s. Plenty of time for the
+            # nginx rate window (1s) and any archipelago restart to clear.
+            sleep $((attempt * 10))
+            continue
+        fi
+        echo "$resp"
+        return
+    done
+}
+
+# After a service restart the session may need re-establishing.
+rpc_relogin_if_needed() {
+    local probe
+    probe=$(rpc_call "package.list" '{}' 2>/dev/null)
+    if echo "$probe" | jq -e '.error.code == -32001' >/dev/null 2>&1; then
+        rpc_login || return 1
+    fi
+}
+
+# ── per-app metadata ────────────────────────────────────────────
+# Mappings the harness needs that aren't expressible from catalog.json alone:
+# multi-container stack rosters, alias/variant container names (bitcoin-knots
+# vs bitcoin-core install the same slots), and the actual nginx UI proxy path
+# (which often differs from /app/<id>/, e.g. `bitcoin-knots` → `/app/bitcoin-ui/`).
+#
+# Keep these tables in sync with the install code in package/stacks.rs and
+# the `*_IMAGE` companion handling in install.rs (the `archy-<x>-ui` set).
+
+# Containers an app installs. Used for app_already_installed detection AND
+# for state assertions when the snapshot-diff falls back (variant apps don't
+# create new containers when their alternate is already present).
+expected_containers_for() {
+    case "$1" in
+        bitcoin-knots) echo "bitcoin-knots archy-bitcoin-ui" ;;
+        bitcoin-core)  echo "bitcoin-core archy-bitcoin-ui" ;;
+        lnd)           echo "lnd archy-lnd-ui" ;;
+        electrumx|electrs|mempool-electrs)
+                       echo "electrs archy-electrs-ui" ;;
+        btcpay-server) echo "archy-btcpay-server archy-btcpay-db archy-nbxplorer archy-btcpay-ui" ;;
+        mempool)       echo "mempool archy-mempool-web archy-mempool-db" ;;
+        immich)        echo "immich_server immich_machine_learning immich_postgres immich_redis" ;;
+        penpot|penpot-frontend)
+                       echo "penpot-frontend penpot-backend penpot-exporter penpot-postgres penpot-redis" ;;
+        indeedhub)     echo "indeedhub indeedhub-api indeedhub-ffmpeg indeedhub-postgres indeedhub-redis indeedhub-minio indeedhub-relay" ;;
+        *)             echo "$1" ;;
+    esac
+}
+
+# UI proxy URL path on the HTTPS frontend. Most apps live at /app/<id>/ but
+# Bitcoin/LND/Electrs proxy through their UI companion containers, and BTCPay
+# uses its own short path.
+ui_proxy_path_for() {
+    case "$1" in
+        bitcoin-knots|bitcoin-core)  echo "/app/bitcoin-ui/" ;;
+        electrumx|electrs)           echo "/app/electrumx/" ;;
+        lnd)                         echo "/app/lnd-ui/" ;;
+        btcpay-server)               echo "/app/btcpay/" ;;
+        *)                           echo "/app/$1/" ;;
+    esac
+}
+
+# Authenticated probe for credentialed UIs. Echoes the HTTP status code if
+# defined, otherwise returns 1 (caller records SKIP). PASS = code in
+# {200,401,403} for endpoints that prove the proxy reaches the backend
+# (401/403 from app's own auth ≠ 502 from broken proxy).
+auth_probe_for() {
+    local app="$1"
+    local host; host="$(echo "$TARGET" | cut -d@ -f2)"
+    case "$app" in
+        bitcoin-knots|bitcoin-core)
+            # Direct bitcoin-rpc proxy on :8334 inside .228 — credential
+            # plumbing is the .228 bug we just shipped, must return 200.
+            ssh_run 'curl -s -o /dev/null -w "%{http_code}" --max-time 5 -X POST http://127.0.0.1:8334/bitcoin-rpc/ -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"getblockchaininfo\",\"params\":[]}"'
+            ;;
+        btcpay-server)
+            # BTCPay's own auth returns 401 for unauthenticated API calls;
+            # 502 means proxy broken / backend down.
+            curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
+                "https://$host/app/btcpay/api/v1/server/info"
+            ;;
+        lnd)
+            # LND has a /lnd-connect-info passthrough on archipelago itself —
+            # returns lndconnect URI when LND is up. 200 = backend reachable.
+            curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
+                "https://$host/lnd-connect-info"
+            ;;
+        electrumx|electrs)
+            # ElectrumX is plain TCP (electrum protocol) — no HTTPS auth path.
+            # archipelago exposes /electrs-status which queries the daemon.
+            curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
+                "https://$host/electrs-status"
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+# Whether an auth_probe HTTP code counts as a pass.
+auth_probe_pass_codes() {
+    case "$1" in
+        bitcoin-knots|bitcoin-core) echo "200" ;;
+        btcpay-server)              echo "200 401 403" ;;
+        lnd|electrumx|electrs)      echo "200" ;;
+        *)                          echo "200" ;;
+    esac
+}
+
+# ── probes (state assertions) ───────────────────────────────────
+# Returns container Status string ("running","exited","absent",…).
+probe_container_state() {
+    local name="$1"
+    ssh_run "podman inspect '$name' --format '{{.State.Status}}' 2>/dev/null || echo absent"
+}
+
+# Returns RestartCount as integer.
+probe_container_restart_count() {
+    local name="$1"
+    ssh_run "podman inspect '$name' --format '{{.RestartCount}}' 2>/dev/null || echo -1"
+}
+
+# Probe the app's UI proxy on the HTTPS frontend. Returns HTTP code.
+# Uses ui_proxy_path_for so apps with non-default proxy paths (bitcoin-ui,
+# lnd-ui, electrs-ui, btcpay) get probed at the right URL.
+probe_app_proxy() {
+    local app_id="$1"
+    local host
+    host="$(echo "$TARGET" | cut -d@ -f2)"
+    local path
+    path=$(ui_proxy_path_for "$app_id")
+    curl -ks -o /dev/null -w "%{http_code}" --max-time 5 "https://$host$path" || echo "000"
+}
+
+# Check that ZERO containers are leftover for this app — catches uninstall residue.
+probe_no_residue() {
+    local prefix="$1"
+    ssh_run "podman ps -a --format '{{.Names}}' | grep -E '^${prefix}(-|$)' | wc -l"
+}
+
+# ── waiters ─────────────────────────────────────────────────────
+# Wait for the package's state in the RPC list to match expected, with timeout.
+wait_for_package_state() {
+    local pkg="$1"; local want="$2"; local timeout="${3:-300}"
+    local deadline=$(($(date +%s) + timeout))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        local got
+        got=$(rpc_call "package.list" '{}' \
+              | jq -r ".result.package_data[\"$pkg\"].state // \"absent\"")
+        case "$want" in
+            Running) [ "$got" = "Running" ] && return 0 ;;
+            Stopped) [ "$got" = "Stopped" ] && return 0 ;;
+            absent)  [ "$got" = "absent" ] && return 0 ;;
+        esac
+        sleep 4
+    done
+    echo "TIMEOUT waiting for $pkg → $want (last seen: $got)" >&2
+    return 1
+}
+
+# Wait for podman state of a specific container.
+wait_for_container_state() {
+    local name="$1"; local want="$2"; local timeout="${3:-180}"
+    local deadline=$(($(date +%s) + timeout))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        local got
+        got=$(probe_container_state "$name")
+        [ "$got" = "$want" ] && return 0
+        sleep 3
+    done
+    echo "TIMEOUT waiting for container $name → $want (last seen: $got)" >&2
+    return 1
+}
+
+# Wait until restart count is stable for `stable_secs` seconds — proxy for "no crashloop".
+wait_restart_count_stable() {
+    local name="$1"; local stable_secs="${2:-30}"; local timeout="${3:-180}"
+    local deadline=$(($(date +%s) + timeout))
+    local last; local last_change_ts
+    last=$(probe_container_restart_count "$name")
+    last_change_ts=$(date +%s)
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        sleep 5
+        local now
+        now=$(probe_container_restart_count "$name")
+        if [ "$now" != "$last" ]; then
+            last="$now"
+            last_change_ts=$(date +%s)
+        elif [ $(( $(date +%s) - last_change_ts )) -ge "$stable_secs" ]; then
+            return 0
+        fi
+    done
+    echo "TIMEOUT waiting for $name restart-count stable (last=$last)" >&2
+    return 1
+}
+
+# ── result recording ────────────────────────────────────────────
+# Append a result row to the JSON-lines report.
+# Args: app_id, transition, status (PASS/FAIL/SKIP), detail
+record() {
+    local app="$1"; local transition="$2"; local status="$3"; local detail="${4:-}"
+    local ts
+    ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+    jq -nc --arg ts "$ts" --arg app "$app" --arg t "$transition" --arg s "$status" --arg d "$detail" \
+        '{ts:$ts, app:$app, transition:$t, status:$s, detail:$d}' >> "$OUT_DIR/results.jsonl"
+    local marker
+    case "$status" in
+        PASS) marker="✅" ;;
+        FAIL) marker="❌" ;;
+        SKIP) marker="⏭" ;;
+        *)    marker="•"  ;;
+    esac
+    printf '%s [%-15s] %-30s %s%s\n' "$marker" "$app" "$transition" "$status" "${detail:+ — $detail}"
+}

scripts/resilience/resilience.sh

@@ -0,0 +1,508 @@
+#!/bin/bash
+# Archipelago resilience harness — black-box state-machine tester for app containers.
+#
+# Drives the live archipelago RPC against a real podman runtime on a target
+# host. For each app in the catalog, runs every state transition a user could
+# trigger (install / probe / stop / start / restart / archipelago-restart /
+# host-reboot / uninstall / reinstall / vanish-watch) and asserts the system
+# remains in the expected state at every step.
+#
+# Usage:
+#     scripts/resilience/resilience.sh archipelago@192.168.1.228 [filter]
+#
+# `filter` is a comma-separated list of app IDs (or "smoke" for the curated
+# fast subset). Default: every app in app-catalog/catalog.json.
+#
+# Exit codes:
+#     0  every cell green
+#     1  any cell red — release should not ship
+#     2  setup/auth error before tests began
+
+set -uo pipefail
+
+# ── args ─────────────────────────────────────────────────────────
+TARGET="${1:?usage: $0 <user@host> [filter]}"
+FILTER="${2:-}"
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+HERE="$ROOT/scripts/resilience"
+RUN_TS="$(date -u +%Y%m%dT%H%M%SZ)"
+OUT_DIR="$HERE/reports/$RUN_TS"
+mkdir -p "$OUT_DIR"
+COOKIE_JAR="$OUT_DIR/cookies.txt"
+
+HOST="$(echo "$TARGET" | cut -d@ -f2)"
+# RPC reaches archipelago through nginx on 443 (which proxies to localhost:5678).
+# Direct :5678 is bound to 127.0.0.1 on the target so we can't curl it from here.
+RPC_URL="https://$HOST/rpc/v1"
+
+export TARGET RPC_URL COOKIE_JAR OUT_DIR
+
+# shellcheck source=lib.sh
+. "$HERE/lib.sh"
+
+# ── credentials ──────────────────────────────────────────────────
+# Pull from env first (so this script can be called from CI). Fall back to
+# interactive prompts.
+SSH_PASS="${RESILIENCE_SSH_PASS:-}"
+UI_PASS="${RESILIENCE_UI_PASS:-}"
+if [ -z "$SSH_PASS" ]; then
+    read -rsp "SSH password for $TARGET: " SSH_PASS; echo
+fi
+if [ -z "$UI_PASS" ]; then
+    read -rsp "Archipelago UI password: " UI_PASS; echo
+fi
+export SSH_PASS UI_PASS
+
+command -v sshpass >/dev/null || { echo "sshpass required"; exit 2; }
+command -v jq >/dev/null      || { echo "jq required"; exit 2; }
+
+ssh_run 'echo ok' >/dev/null || { echo "ssh to $TARGET failed"; exit 2; }
+rpc_login                    || exit 2
+
+echo "Resilience harness — target $TARGET, run $RUN_TS"
+echo "Output: $OUT_DIR/results.jsonl"
+echo "─────────────────────────────────────────────────────────────"
+
+# ── catalog & filter ─────────────────────────────────────────────
+CATALOG="$ROOT/app-catalog/catalog.json"
+ALL_APPS=$(jq -r '.apps[].id' "$CATALOG")
+
+# Topo-sort the catalog by `requires`. Outputs app IDs in install order
+# (deps first, then dependents). Kahn's algorithm via python — keeps the
+# bash side simple and the deps logic obvious for next-time-readers.
+topo_order() {
+    python3 -c "
+import json
+with open('$CATALOG') as f: c = json.load(f)
+deps = {a['id']: list(a.get('requires', [])) for a in c['apps']}
+order = []
+remaining = set(deps)
+while remaining:
+    ready = sorted(a for a in remaining if all(d not in remaining for d in deps[a]))
+    if not ready:  # cycle (shouldn't happen) — emit whatever's left
+        order.extend(sorted(remaining)); break
+    order.extend(ready); remaining.difference_update(ready)
+print('\n'.join(order))
+"
+}
+
+apps_to_test() {
+    local order; order=$(topo_order)
+    if [ -z "$FILTER" ]; then
+        # Full sweep — but skip bitcoin-core since it shares container slots
+        # with bitcoin-knots; testing both back-to-back would just churn the
+        # same containers. bitcoin-knots is the canonical entry.
+        echo "$order" | grep -v '^bitcoin-core$'
+    elif [ "$FILTER" = "smoke" ]; then
+        # Fast subset exercising the bug classes we just fixed:
+        # single-container, multi-container stack, credentialed UI.
+        echo -e "filebrowser\nbitcoin-knots\nindeedhub"
+    else
+        echo "$order" | grep -E "^($(echo "$FILTER" | tr ',' '|'))$"
+    fi
+}
+
+# Resolve `requires` chain for $1 in install-order (deps first).
+deps_for_app() {
+    local app="$1"
+    python3 -c "
+import json
+with open('$CATALOG') as f: c = json.load(f)
+deps_map = {a['id']: list(a.get('requires', [])) for a in c['apps']}
+visited, order = set(), []
+def visit(x):
+    if x in visited or x not in deps_map: return
+    visited.add(x)
+    for d in deps_map.get(x, []): visit(d)
+    order.append(x)
+for d in deps_map.get('$app', []): visit(d)
+print('\n'.join(order))
+"
+}
+
+# ── per-app transitions ──────────────────────────────────────────
+# Diff helper: capture container names matching a sane prefix for $app_id.
+# Approach: snapshot before install, snapshot after, take the difference =
+# this app's containers.
+snapshot_containers() {
+    ssh_run "podman ps -a --format '{{.Names}}' | sort"
+}
+
+# Whether $app currently has ALL of its expected containers running. Uses
+# the per-app metadata table in lib.sh (expected_containers_for) so variant
+# apps (bitcoin-knots/bitcoin-core sharing slots) and stacks are detected
+# correctly. Falls back to name-prefix match for apps the table doesn't know.
+#
+# Returns true only when every expected container is present. Earlier
+# versions returned true on ANY match — that caused dep installs (e.g.
+# bitcoin-knots required by btcpay) to be declared "installed" as soon as
+# the backend container appeared, before the UI companion (archy-bitcoin-ui)
+# was up. The before-snapshot then missed the companion, the after-snapshot
+# caught it, and it leaked into the dependent app's "new containers" set,
+# false-positive-FAILing stop/uninstall when the companion (correctly) did
+# not respond to the dependent app's package.stop.
+app_already_installed() {
+    local app="$1"
+    local snap; snap=$(snapshot_containers)
+    local expected
+    expected=$(expected_containers_for "$app")
+    if [ -n "$expected" ] && [ "$expected" != "$app" ]; then
+        local c missing=0
+        for c in $expected; do
+            echo "$snap" | grep -qxF "$c" || missing=1
+        done
+        [ "$missing" -eq 0 ] && return 0
+        # Fall through to prefix match if the expected_containers list has
+        # gaps; a partial install still counts as "installed enough" for
+        # preclean purposes.
+    fi
+    # Generic prefix fallback for apps not in the expected_containers_for table.
+    echo "$snap" | grep -qE "^(${app}|${app}-|archy-${app}|archy-${app}-)"
+}
+
+# Install missing deps for $app via the regular install path. Idempotent —
+# already-installed deps are skipped. Records dep_install per dep so we can
+# tell from the report whether the bitcoin pre-req was actually green by the
+# time lnd's matrix started.
+ensure_deps_installed() {
+    local app="$1"
+    local dep
+    for dep in $(deps_for_app "$app"); do
+        if app_already_installed "$dep"; then
+            continue
+        fi
+        echo "  · dep install: $dep (required by $app)"
+        local img ver resp
+        img=$(jq -r --arg id "$dep" '.apps[] | select(.id==$id) | .dockerImage // ""' "$CATALOG")
+        ver=$(jq -r --arg id "$dep" '.apps[] | select(.id==$id) | .version // ""' "$CATALOG")
+        if [ -z "$img" ]; then
+            record "$app" "dep_$dep" FAIL "no dockerImage in catalog for dep $dep"
+            return 1
+        fi
+        resp=$(rpc_call "package.install" "$(jq -nc \
+            --arg id "$dep" --arg img "$img" --arg ver "$ver" \
+            '{id:$id, dockerImage:$img, version:$ver}')")
+        if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
+            record "$app" "dep_$dep" FAIL "rpc error: $(echo "$resp" | jq -c '.error')"
+            return 1
+        fi
+        # Wait for at least one expected container to appear running.
+        local deadline=$(($(date +%s) + 600))
+        while [ "$(date +%s)" -lt "$deadline" ]; do
+            if app_already_installed "$dep"; then
+                record "$app" "dep_$dep" PASS "installed"
+                break
+            fi
+            sleep 5
+        done
+        if ! app_already_installed "$dep"; then
+            record "$app" "dep_$dep" FAIL "containers did not appear within 10min"
+            return 1
+        fi
+    done
+    return 0
+}
+
+# Pre-clean: if the app is currently installed, uninstall it and wait for
+# all containers to disappear. We can't measure install correctness without
+# starting from a clean slate. Fail-soft — if the uninstall RPC errors we
+# log but proceed; the install step will catch any residual state.
+preclean_app() {
+    local app="$1"
+    if ! app_already_installed "$app"; then
+        return 0
+    fi
+    echo "  · pre-clean: $app already installed, uninstalling first"
+    local resp; resp=$(rpc_call "package.uninstall" "{\"id\":\"$app\"}")
+    if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
+        echo "    pre-clean uninstall RPC error: $(echo "$resp" | jq -c '.error')"
+    fi
+    # Multi-container stacks (indeedhub: 7, immich: 5, mempool: 3, btcpay: 6)
+    # take noticeably longer to tear down than single-container apps. 240s was
+    # too tight for indeedhub's 7-container teardown — bump to 10 min for
+    # safety; per-container timeout is still bounded inside archipelago itself.
+    local deadline=$(($(date +%s) + 600))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        if ! app_already_installed "$app"; then return 0; fi
+        sleep 5
+    done
+    echo "    pre-clean: timeout waiting for $app to uninstall"
+    return 1
+}
+
+# Run the full per-app matrix. Records a row per transition.
+run_app_matrix() {
+    local app="$1"
+    echo
+    echo "═══ $app ═══"
+
+    if ! ensure_deps_installed "$app"; then
+        record "$app" install FAIL "dep install failed; skipping rest of matrix"
+        return
+    fi
+    preclean_app "$app" || record "$app" preclean FAIL "uninstall before test did not complete"
+
+    # ── 01 install ───────────────────────────────────────────────
+    local before after new_containers
+    before=$(snapshot_containers)
+    # The install handler requires `id` + `dockerImage` from the catalog
+    # entry. Match what the UI passes (Discover.vue / MarketplaceAppDetails.vue).
+    local docker_image version
+    docker_image=$(jq -r --arg id "$app" '.apps[] | select(.id==$id) | .dockerImage // ""' "$CATALOG")
+    version=$(jq -r --arg id "$app" '.apps[] | select(.id==$id) | .version // ""' "$CATALOG")
+    if [ -z "$docker_image" ]; then
+        record "$app" install FAIL "no dockerImage in catalog for $app"
+        return
+    fi
+    local install_resp
+    install_resp=$(rpc_call "package.install" "$(jq -nc \
+        --arg id "$app" --arg img "$docker_image" --arg ver "$version" \
+        '{id:$id, dockerImage:$img, version:$ver}')")
+    if echo "$install_resp" | jq -e '.error' >/dev/null 2>&1; then
+        record "$app" install FAIL "rpc error: $(echo "$install_resp" | jq -c '.error')"
+        return  # cannot continue this app
+    fi
+
+    # Wait for the EXPECTED containers (per expected_containers_for) to all
+    # appear. The old "snapshot stable for 10s + count > before" heuristic
+    # terminated early on apps with deps: e.g. mempool's wait would break
+    # when archy-electrs-ui (electrumx dep companion) appeared, long before
+    # mempool's own containers were created (those take ~10min to pull and
+    # start). Waiting on the expected-set is exact, not heuristic.
+    #
+    # Cap at 15 minutes — mempool stack with cold image cache needs ~12 min.
+    local expected; expected=$(expected_containers_for "$app")
+    local deadline=$(($(date +%s) + 900))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        after=$(snapshot_containers)
+        local missing=0
+        for c in $expected; do
+            echo "$after" | grep -qxF "$c" || missing=1
+        done
+        [ "$missing" -eq 0 ] && break
+        sleep 5
+    done
+    new_containers=$(comm -13 <(echo "$before") <(echo "$after"))
+    if [ -z "$new_containers" ]; then
+        record "$app" install FAIL "no containers created within 10min"
+        return
+    fi
+    # Assert each new container is in 'running' state.
+    local install_ok=1; local detail=""
+    while read -r c; do
+        [ -z "$c" ] && continue
+        local s
+        s=$(probe_container_state "$c")
+        if [ "$s" != "running" ]; then
+            install_ok=0
+            detail="$detail $c=$s"
+        fi
+    done <<< "$new_containers"
+    if [ "$install_ok" -eq 1 ]; then
+        record "$app" install PASS "$(echo "$new_containers" | tr '\n' ',' | sed 's/,$//')"
+    else
+        record "$app" install FAIL "containers not running:$detail"
+    fi
+
+    # ── 02 ui_probe ──────────────────────────────────────────────
+    # Retry with backoff — install just finished, but the app's backend
+    # (fedimint, immich, mempool stack) may take 30+s to be ready to serve
+    # HTTP. Probing immediately false-positive-FAILed those apps; pass on
+    # first 2xx/3xx within 60s.
+    local code
+    local ui_deadline=$(($(date +%s) + 60))
+    while :; do
+        code=$(probe_app_proxy "$app")
+        [[ "$code" =~ ^(2[0-9][0-9]|3[0-9][0-9])$ ]] && break
+        [ "$(date +%s)" -ge "$ui_deadline" ] && break
+        sleep 5
+    done
+    # Accept all 2xx/3xx — proxy reaches backend, app may redirect to login,
+    # serve OAuth flow (307), or use 308 permanent. 401/403 still fail because
+    # those mean "backend reached, app rejected request" which is the
+    # credential-plumbing failure mode we DO want to catch.
+    if [[ "$code" =~ ^(2[0-9][0-9]|3[0-9][0-9])$ ]]; then
+        record "$app" ui_probe PASS "HTTP $code"
+    else
+        record "$app" ui_probe FAIL "HTTP $code (expected 2xx/3xx, retried 60s)"
+    fi
+
+    # ── 03 auth_probe (only for apps with a credentialed/data endpoint) ──
+    # Same backoff treatment: bitcoin-ui's nginx config bind-mount is
+    # picked up at start, but the bitcoin-core backend may not have
+    # accepted RPC connections yet on a fresh install.
+    local probe_code; local pass_codes
+    pass_codes=$(auth_probe_pass_codes "$app")
+    if probe_code=$(auth_probe_for "$app" 2>/dev/null) && [ -n "$probe_code" ]; then
+        local auth_deadline=$(($(date +%s) + 60))
+        while :; do
+            echo " $pass_codes " | grep -qF " $probe_code " && break
+            [ "$(date +%s)" -ge "$auth_deadline" ] && break
+            sleep 5
+            probe_code=$(auth_probe_for "$app" 2>/dev/null) || break
+        done
+        if echo " $pass_codes " | grep -qF " $probe_code "; then
+            record "$app" auth_probe PASS "HTTP $probe_code"
+        else
+            record "$app" auth_probe FAIL "HTTP $probe_code (expected one of: $pass_codes; retried 60s — credential plumbing broken)"
+        fi
+    else
+        record "$app" auth_probe SKIP "no authenticated probe defined"
+    fi
+
+    # ── 04 stop ──────────────────────────────────────────────────
+    local stop_resp
+    stop_resp=$(rpc_call "package.stop" "{\"id\":\"$app\"}")
+    if echo "$stop_resp" | jq -e '.error' >/dev/null 2>&1; then
+        record "$app" stop FAIL "rpc error: $(echo "$stop_resp" | jq -c '.error')"
+    else
+        local all_stopped=1
+        while read -r c; do
+            [ -z "$c" ] && continue
+            wait_for_container_state "$c" "exited" 60 || all_stopped=0
+        done <<< "$new_containers"
+        if [ "$all_stopped" -eq 1 ]; then
+            record "$app" stop PASS
+        else
+            record "$app" stop FAIL "not all containers reached exited state"
+        fi
+    fi
+
+    # ── 05 start ─────────────────────────────────────────────────
+    local start_resp
+    start_resp=$(rpc_call "package.start" "{\"id\":\"$app\"}")
+    if echo "$start_resp" | jq -e '.error' >/dev/null 2>&1; then
+        record "$app" start FAIL "rpc error: $(echo "$start_resp" | jq -c '.error')"
+    else
+        local all_started=1
+        while read -r c; do
+            [ -z "$c" ] && continue
+            wait_for_container_state "$c" "running" 90 || all_started=0
+        done <<< "$new_containers"
+        if [ "$all_started" -eq 1 ]; then
+            record "$app" start PASS
+        else
+            record "$app" start FAIL "not all containers reached running state"
+        fi
+    fi
+
+    # ── 06 restart_container ─────────────────────────────────────
+    # `package.restart` returns immediately and spawns the actual restart.
+    # `podman restart -t <stop_timeout>` blocks for up to stop_timeout
+    # seconds (e.g. 600s for bitcoin-core). Polling once after sleep 5
+    # races on slow-stopping apps and false-positive-FAILs them. Poll
+    # each container up to 90s for "running" instead.
+    local restart_resp
+    restart_resp=$(rpc_call "package.restart" "{\"id\":\"$app\"}")
+    if echo "$restart_resp" | jq -e '.error' >/dev/null 2>&1; then
+        record "$app" restart FAIL "rpc error: $(echo "$restart_resp" | jq -c '.error')"
+    else
+        local all_running=1
+        while read -r c; do
+            [ -z "$c" ] && continue
+            wait_for_container_state "$c" "running" 90 || all_running=0
+        done <<< "$new_containers"
+        if [ "$all_running" -eq 1 ]; then
+            record "$app" restart PASS
+        else
+            record "$app" restart FAIL "container not running 90s after restart"
+        fi
+    fi
+
+    # ── 09 uninstall (skip 07 archipelago-restart and 08 host-reboot
+    #    here — those are batch tests run once across all installed apps) ─
+    local uninst_resp
+    uninst_resp=$(rpc_call "package.uninstall" "{\"id\":\"$app\"}")
+    if echo "$uninst_resp" | jq -e '.error' >/dev/null 2>&1; then
+        record "$app" uninstall FAIL "rpc error: $(echo "$uninst_resp" | jq -c '.error')"
+    else
+        # Wait for all this-app containers to be absent.
+        local all_gone=1
+        while read -r c; do
+            [ -z "$c" ] && continue
+            wait_for_container_state "$c" "absent" 120 || all_gone=0
+        done <<< "$new_containers"
+        if [ "$all_gone" -eq 1 ]; then
+            record "$app" uninstall PASS
+        else
+            record "$app" uninstall FAIL "not all containers removed"
+        fi
+    fi
+}
+
+# ── batch transitions (run after per-app loop) ───────────────────
+batch_archipelago_service_restart() {
+    echo
+    echo "═══ batch: archipelago.service restart ═══"
+    local before; before=$(snapshot_containers)
+    if ! ssh_run 'sudo systemctl restart archipelago'; then
+        record "_batch" archipelago_restart FAIL "systemctl restart errored"
+        return
+    fi
+    ssh_wait_ready 60 || { record "_batch" archipelago_restart FAIL "ssh did not return"; return; }
+    sleep 30  # let containers re-stabilize
+    rpc_login || { record "_batch" archipelago_restart FAIL "rpc relogin failed"; return; }
+    local after; after=$(snapshot_containers)
+    if [ "$before" = "$after" ]; then
+        record "_batch" archipelago_restart PASS "container set unchanged"
+    else
+        record "_batch" archipelago_restart FAIL "container set drifted across restart"
+    fi
+}
+
+batch_host_reboot() {
+    echo
+    echo "═══ batch: host reboot ═══"
+    local before; before=$(snapshot_containers)
+    ssh_run 'sudo systemctl reboot' || true  # ssh disconnects immediately
+    sleep 30
+    # 5 min was too short — .228 took ~9min for full BIOS+kernel+systemd+
+    # rootless-podman boot. 12 min gives margin for slower hardware.
+    ssh_wait_ready 720 || { record "_batch" host_reboot FAIL "host did not come back in 12min"; return; }
+    sleep 60  # let containers auto-restart
+    rpc_login || { record "_batch" host_reboot FAIL "rpc unreachable after reboot"; return; }
+    local after; after=$(snapshot_containers)
+    if [ "$before" = "$after" ]; then
+        record "_batch" host_reboot PASS "all containers came back"
+    else
+        local missing
+        missing=$(comm -23 <(echo "$before") <(echo "$after") | tr '\n' ',' | sed 's/,$//')
+        record "_batch" host_reboot FAIL "missing: $missing"
+    fi
+}
+
+# ── main ─────────────────────────────────────────────────────────
+APPS_LIST=$(apps_to_test)
+if [ -z "$APPS_LIST" ]; then
+    echo "no apps match filter '$FILTER'" >&2; exit 2
+fi
+
+while read -r app; do
+    [ -z "$app" ] && continue
+    run_app_matrix "$app"
+done <<< "$APPS_LIST"
+
+# Batch transitions only run on full sweep (skip in filtered/smoke mode).
+if [ -z "$FILTER" ]; then
+    batch_archipelago_service_restart
+    batch_host_reboot
+fi
+
+# ── summary ──────────────────────────────────────────────────────
+echo
+echo "═══ summary ═══"
+count_status() {
+    local pat="$1"
+    [ -s "$OUT_DIR/results.jsonl" ] || { echo 0; return; }
+    awk -v pat="$pat" '$0 ~ pat { n++ } END { print n+0 }' "$OUT_DIR/results.jsonl"
+}
+PASS=$(count_status '"status":"PASS"')
+FAIL=$(count_status '"status":"FAIL"')
+SKIP=$(count_status '"status":"SKIP"')
+TOTAL=$((PASS + FAIL + SKIP))
+echo "PASS: $PASS / FAIL: $FAIL / SKIP: $SKIP / TOTAL: $TOTAL"
+echo "Report: $OUT_DIR/results.jsonl"
+
+[ "$FAIL" -eq 0 ] || exit 1
+exit 0

scripts/self-update.sh

@@ -186,7 +186,7 @@ fi
 # for backward compatibility with older binaries that still look there.
 SCRIPTS_DEST="/opt/archipelago/scripts"
 sudo mkdir -p "$SCRIPTS_DEST"
-for script in image-versions.sh reconcile-containers.sh container-specs.sh; do
+for script in image-versions.sh reconcile-containers.sh container-specs.sh container-doctor.sh app-surface-smoke-test.sh bitcoin-stack-lifecycle-test.sh; do
     src="$REPO_DIR/scripts/$script"
     if [ -f "$src" ]; then
         sudo install -m 755 "$src" "$SCRIPTS_DEST/$script"
@@ -200,6 +200,19 @@ if [ -f "$REPO_DIR/scripts/image-versions.sh" ]; then
     sudo cp "$REPO_DIR/scripts/image-versions.sh" /opt/archipelago/image-versions.sh
 fi
 
+# Sync app manifests and app-local build contexts into the canonical
+# production manifest root. The backend orchestrator loads install specs from
+# /opt/archipelago/apps; updating only the binary/frontend can leave a node
+# with new installer logic but stale or missing app manifests.
+APPS_DEST="/opt/archipelago/apps"
+if [ -d "$REPO_DIR/apps" ]; then
+    sudo mkdir -p "$APPS_DEST"
+    sudo rsync -a --delete "$REPO_DIR/apps/" "$APPS_DEST/"
+    ok "App manifests synced"
+else
+    warn "Apps directory not found at $REPO_DIR/apps — install manifests may be stale"
+fi
+
 # Update first-boot-containers.sh too (the canonical first-boot orchestrator).
 # Nodes run it once on install, but keeping a fresh copy on disk means any
 # future boot or reconciler invocation uses current port specs and caps.
@@ -299,6 +312,25 @@ if [ -f "$REPO_DIR/image-recipe/configs/archipelago.service" ]; then
     fi
 fi
 
+# Keep the doctor timer/service current too. Container uptime fixes rely on
+# these units as much as on the helper scripts themselves.
+DOCTOR_UNITS_CHANGED=false
+for unit in archipelago-doctor.service archipelago-doctor.timer; do
+    src="$REPO_DIR/image-recipe/configs/$unit"
+    dst="/etc/systemd/system/$unit"
+    [ -f "$src" ] || continue
+    if [ ! -f "$dst" ] || ! diff -q "$src" "$dst" &>/dev/null; then
+        sudo install -m 644 "$src" "$dst"
+        DOCTOR_UNITS_CHANGED=true
+        ok "Updated $unit"
+    fi
+done
+if [ "$DOCTOR_UNITS_CHANGED" = "true" ]; then
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now archipelago-doctor.timer 2>>"$LOG_FILE" || \
+        warn "Failed to enable archipelago-doctor.timer"
+fi
+
 # Install/refresh tmpfiles.d rules. The logs rule creates
 # /var/log/archipelago/ + container-installs.log with archipelago:archipelago
 # ownership so the non-root backend can append install audit lines.