fix: overhaul container lifecycle — recovery, health, uninstall, UI state

Container recovery: - Health monitor: MAX_RESTART_ATTEMPTS 3→10, interval 60s→120s - Dependency-aware restarts: won't restart services before their deps - Reset dependent counters when a dependency recovers - Handle "created" state containers (were invisible to health monitor) - Added IndeedHub, mempool-api, mysql to tier system - Crash recovery: podman start timeout 30s→120s with retry - Podman client: socket timeout 5s→30s, added restart policy UI state representation: - Exit code 0 shows "stopped" (gray), not "crashed" (red) - Exit code 137 shows "killed (OOM)" - Non-zero exit shows "crashed" (red) - Added exit_code field to PackageDataEntry Install/uninstall fixes: - Install returns error when container doesn't start (was silent success) - Post-install hooks awaited instead of fire-and-forget tokio::spawn - Uninstall: graceful rm before force, volume prune, network cleanup - Uninstall returns error on partial failure (was 200 OK) Config consistency: - DB passwords read from /var/lib/archipelago/secrets/ (was hardcoded) - Bitcoin: added ZMQ ports 28332/28333 for LND block notifications - IndeedHub port 7777→8190 (was conflicting with strfry) - Marketplace versions: LND 0.17.4→0.18.4, Mempool 2.5.0→3.0.0 Performance: - Metrics collector interval 60s→300s (was duplicating health monitor) - Podman client: proper error propagation instead of unwrap_or_default Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 07:03:57 +01:00
parent cdff10a8bc
commit 64b57dca7d
65 changed files with 3950 additions and 298 deletions
@@ -0,0 +1,80 @@
+# ISO Boot Branding — Archipelago
+
+Design and build the visual boot experience from USB power-on to web UI.
+
+## Brand Identity
+
+**Archipelago** = self-sovereign Bitcoin node OS. Floating islands in the sky.
+
+| Element | Value |
+|---------|-------|
+| Primary accent | `#fb923c` (Bitcoin orange) |
+| Secondary accent | `#f7931a` (deeper orange) |
+| Success | `#4ade80` (green) |
+| Background | `#0a0a0a` -> `#050505` (near-black) |
+| Text | `#ffffff` (white), `#aaaaaa` (dim), `#555555` (subtle) |
+| Glass | `rgba(255,255,255,0.06)` frost overlay |
+| Style | Pixel art cyberpunk, dark glass morphism, CRT scanlines |
+| Logo | Pixel-art lowercase "a" (from SVG favicon) |
+
+## Boot Stages & What's Customizable
+
+### 1. GRUB Menu (UEFI boot)
+- **Background**: `branding/grub-theme/background.png` — any PNG, GRUB scales it
+- **Theme**: `branding/grub-theme/theme.txt` — colors, layout, labels
+- **Fonts**: Generated with `grub-mkfont` during build, .pf2 format
+- **Config**: Written by build script in Step 5 (`grub.cfg` heredoc)
+
+GRUB theme.txt properties that work:
+```
+desktop-color: "#rrggbb"
+desktop-image: "background.png"
+title-text: ""
+
+ boot_menu { left/top/width/height = N%; item_color/selected_item_color = "#rrggbb" }
+ label { left/top/width = N%; text = "string"; color = "#rrggbb"; align = "center" }
+```
+
+**IMPORTANT**: Do NOT reference font names in theme.txt unless you know the exact internal name from grub-mkfont output.
+
+### 2. ISOLINUX Menu (BIOS boot)
+- Text-only ANSI-style `MENU COLOR` directives
+- Use `vesamenu.c32` for graphical, `menu.c32` for compatibility
+
+### 3. Plymouth Splash (kernel boot -> login)
+- Theme: `branding/plymouth-theme/archipelago.script`
+- Logo: `branding/plymouth-theme/logo.png` (PNG with transparency)
+- Config: `branding/plymouth-theme/archipelago.plymouth`
+- Kernel param `splash` must be present
+
+### 4. Console Banner (TTY login)
+- ASCII art in `/etc/profile.d/archipelago.sh`
+- Uses ANSI escape codes for color
+
+### 5. Installer Prompt
+- In systemd service wrapper: `/usr/local/bin/archipelago-start-installer`
+
+## Image Specs
+
+| Asset | Format | Size | Notes |
+|-------|--------|------|-------|
+| GRUB background | PNG | 1024x768 recommended | Large images slow boot |
+| Plymouth logo | PNG (RGBA) | 256x256 recommended | Transparent background |
+| GRUB fonts | .pf2 | Generated | `grub-mkfont -s SIZE -o out.pf2 input.ttf` |
+
+## Build Integration
+
+GRUB theme: Step 2 (copied from `branding/grub-theme/`, fonts generated with `grub-mkfont`)
+Plymouth theme: Step 3 (component copy) + Step 4 (auto-install.sh copies to target)
+GRUB on target: auto-install.sh copies to `/mnt/target/boot/grub/themes/archipelago/`
+
+## What to Edit
+
+| File | Affects |
+|------|---------|
+| `branding/grub-theme/background.png` | GRUB boot screen image |
+| `branding/grub-theme/theme.txt` | GRUB menu colors, layout |
+| `branding/plymouth-theme/logo.png` | Plymouth boot logo |
+| `branding/plymouth-theme/archipelago.script` | Plymouth animation/progress |
+| `branding/generate-grub-background.py` | Procedural background generator |
+| `branding/generate-plymouth-logo.py` | Procedural logo generator |
@@ -0,0 +1,89 @@
+---
+name: podman
+description: Rootless Podman container management — diagnose, fix, and harden uptime. Use for container issues, port problems, UID mapping, health checks, or uptime hardening.
+disable-model-invocation: true
+allowed-tools: Bash, Read, Edit, Write, Glob, Grep
+argument-hint: "[diagnose|fix|uptime] [container-name]"
+---
+
+# Podman — Container Management
+
+Archipelago runs rootless Podman as `archipelago` user (UID 1000). All `podman` commands run without sudo. UID mapping: container UID N → host UID (100000 + N).
+
+**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`
+
+## Diagnose
+
+```bash
+# Container status
+podman ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}\t{{.Networks}}"
+
+# Restart policies (must be "unless-stopped")
+for c in $(podman ps -a --format "{{.Names}}"); do
+  echo -n "$c: "; podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}"
+done
+
+# Health checks
+for c in $(podman ps --format "{{.Names}}"); do
+  health=$(podman inspect "$c" --format "{{.State.Health.Status}}" 2>/dev/null)
+  [ -n "$health" ] && [ "$health" != "<no value>" ] && echo "$c: $health"
+done
+
+# Resource usage + recent deaths
+podman stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
+podman events --filter event=died --since 24h 2>/dev/null | tail -10
+
+# Rootless prerequisites
+echo "XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR"  # must be /run/user/1000
+grep archipelago /etc/subuid                # must show archipelago:100000:65536
+ls /var/lib/systemd/linger/ | grep archipelago  # must exist
+grep DEFAULT_FORWARD_POLICY /etc/default/ufw    # must be ACCEPT
+```
+
+Cross-check 4 layers for port consistency: Backend config (package.rs) → Podman ports → Nginx proxy → Frontend appLauncher.ts. See `references/port-map.md`.
+
+## Fix
+
+**Restart policy missing**: `podman update --restart unless-stopped CONTAINER_NAME`
+
+**UID mapping (permission denied)**: `sudo chown -R HOST_UID:HOST_UID /var/lib/archipelago/APP`. Formula: host_uid = 100000 + container_uid. See `references/uid-mapping.md`.
+
+**Port conflict**: `ss -tlnp | grep :PORT` to find offender. Can't add ports to running container — must recreate.
+
+**Network missing**: `podman network connect archy-net CONTAINER_NAME`
+
+**UFW blocking LAN**: `sudo sed -i 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw && sudo ufw reload`
+
+**Stale processes**: `pgrep -c -f "podman ps"` — if >10, kill stuck processes.
+
+See `references/common-failures.md` for the full error→cause→fix lookup table.
+
+## Uptime Hardening
+
+### Layer 1: Restart policies
+```bash
+for c in $(podman ps -a --format "{{.Names}}"); do
+  policy=$(podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
+  [ "$policy" = "no" ] || [ -z "$policy" ] && podman update --restart unless-stopped "$c"
+done
+```
+
+### Layer 2: Watchdog timer
+Create `/usr/local/bin/archipelago-container-watchdog.sh` that restarts stopped/unhealthy containers every 2 minutes via systemd timer. Script runs as archipelago user with `XDG_RUNTIME_DIR=/run/user/1000`.
+
+### Layer 3: Ordered startup
+Bitcoin stack has dependency chain: bitcoin-knots → electrumx + lnd → mempool + btcpay + fedimint → UI containers. Create `/usr/local/bin/archipelago-ordered-start.sh` with wait-for-container logic between tiers.
+
+### Verification
+```bash
+sudo reboot  # then SSH back after 3 min
+podman ps --format "{{.Names}}" | sort  # should match pre-reboot list
+```
+
+## Systemd Requirements
+
+The archipelago.service needs these for rootless Podman:
+- `ProtectHome=no` (podman stores in ~/.local/share/containers/)
+- `PrivateTmp=no` (runtime in /tmp/podman-run-1000/)
+- Do not set `RestrictNamespaces=` or `SystemCallFilter=`
+- `Environment=XDG_RUNTIME_DIR=/run/user/1000`
@@ -0,0 +1,102 @@
+# Common Podman Failure Patterns
+
+## Rootless Podman Specific Failures
+
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `ERRO[0000] cannot find UID/GID for user` | subuid/subgid not configured | Add `archipelago:100000:65536` to `/etc/subuid` and `/etc/subgid` |
+| `Error: unshare: operation not permitted` | Systemd `RestrictNamespaces` blocks user namespaces | Remove `RestrictNamespaces=` from `archipelago.service` |
+| `Error: could not get runtime: creating runtime` | XDG_RUNTIME_DIR not set or /run/user/1000 missing | Set `Environment=XDG_RUNTIME_DIR=/run/user/1000` in service, ensure `loginctl enable-linger archipelago` |
+| `permission denied` on volume mount | Wrong UID ownership — must use mapped UIDs | `sudo chown -R 100000:100000 /var/lib/archipelago/APP` (see UID mapping table) |
+| `ERRO[0000] rootless containers not supported` | Podman not configured for rootless | Run `podman system migrate`, check `/etc/subuid` |
+| `Error: creating container storage: layer not known` | Corrupted rootless storage | `podman system reset` (destroys all containers — last resort) |
+| `Error: stat /tmp/podman-run-1000/...: no such file` | PrivateTmp=yes in systemd isolates /tmp | Set `PrivateTmp=no` in `archipelago.service` |
+| Container ports unreachable from LAN | UFW DEFAULT_FORWARD_POLICY="DROP" | Change to "ACCEPT" in `/etc/default/ufw`, then `sudo ufw reload` |
+| `Error: error creating network namespace` | Systemd `SystemCallFilter` blocks clone/unshare | Remove `SystemCallFilter=` from `archipelago.service` |
+| Containers lose network after service restart | podman runtime dir in /tmp cleaned | Ensure `PrivateTmp=no` so /tmp/podman-run-1000/ persists |
+
+## Container Won't Start
+
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `exec format error` | Binary built on wrong arch | Rebuild on the Linux server |
+| `address already in use` | Port conflict | `ss -tlnp \| grep :PORT` to find offender |
+| `permission denied` | Missing capability, wrong UID ownership, or read-only root | Check capabilities, check volume ownership with mapped UID, add tmpfs |
+| `OCI runtime error` | Corrupt container state | `podman rm -f NAME && recreate` |
+| `image not known` | Image not pulled | `podman pull IMAGE:TAG` |
+| `no such network` | Network missing | `podman network create archy-net` |
+| `Error: netavark: ...subnet overlap` | Network CIDR conflict | `podman network rm archy-net && podman network create archy-net` |
+
+## Container Starts But App Unreachable
+
+| Symptom | Check Layer | Fix |
+|---------|------------|-----|
+| Direct port works, /app/ doesn't | Nginx config | Add `/app/{id}/` location block |
+| Neither works | Podman ports | `podman port NAME` — verify mapping exists |
+| Port mapped but refused | Container logs | App crashing internally — check logs |
+| Works sometimes | Resources | Check OOM kills, CPU, disk space |
+| 502 Bad Gateway | Nginx→Container | Wrong port in proxy_pass or container restarted |
+| Works locally but not from LAN | UFW forward policy | Set `DEFAULT_FORWARD_POLICY="ACCEPT"` in `/etc/default/ufw` |
+
+## Container Keeps Dying
+
+| Pattern | Cause | Fix |
+|---------|-------|-----|
+| Exits immediately (code 1) | Config error | Check `podman logs NAME` |
+| Dies after minutes | OOM killed | Increase `--memory` limit |
+| Dies when dep restarts | No restart policy | Add `--restart unless-stopped` |
+| Crash loop | Repeated crash | Fix root cause, don't just restart |
+| Exit code 127 | Missing binary in container | Wrong image tag or corrupted image — re-pull |
+| Exit code 137 | Killed by OOM or signal | Check `dmesg` for OOM kill, check `podman inspect` for OOMKilled |
+
+## Network Issues
+
+| Problem | Cause | Fix |
+|---------|-------|-----|
+| Can't resolve container names | Not on archy-net | Recreate with `--network=archy-net` |
+| Can't reach internet | DNS missing | Add `--dns 1.1.1.1` |
+| Container-to-container timeout | Different networks | Put both on same network |
+| Bitcoin RPC refused from container | rpcallowip wrong subnet | Use `rpcallowip=0.0.0.0/0` (safe: port mapped, not exposed) |
+| Old containers can't find new network | Subnet changed (rootful→rootless) | Recreate containers on new archy-net (rootless uses 10.89.x.x) |
+
+## Volume Permission Patterns (Rootless UID Mapping)
+
+Formula: **host_uid = 100000 + container_uid**
+
+| Container UID | Host UID | Apps | Data Directory |
+|---|---|---|---|
+| 0 (root) | 100000 | lnd, fedimint, homeassistant, jellyfin, vaultwarden, photoprism, ollama, filebrowser, electrumx, btcpay, immich | `/var/lib/archipelago/{app}` |
+| 70 | 100070 | postgres (btcpay-db, immich-db, penpot-postgres) | `/var/lib/archipelago/postgres-*` |
+| 101 | 100101 | bitcoin-knots | `/var/lib/archipelago/bitcoin` |
+| 472 | 100472 | grafana | `/var/lib/archipelago/grafana` |
+| 999 | 100999 | MariaDB (mysql-mempool) | `/var/lib/archipelago/mysql-mempool` |
+
+## Capability Reference
+
+| Capability | Apps That Need It | Failure Mode |
+|-----------|------------------|-------------|
+| CHOWN | nextcloud, homeassistant, btcpay, jellyfin, portainer | Can't chown during setup |
+| SETUID/SETGID | nextcloud, homeassistant, btcpay, jellyfin | Can't switch to service user |
+| DAC_OVERRIDE | nextcloud, homeassistant, btcpay | Can't access cross-UID files |
+| FOWNER | bitcoin-knots, lnd, fedimint | Can't modify data dir perms |
+| NET_BIND_SERVICE | nginx-proxy-manager, vaultwarden | Can't bind ports <1024 |
+| NET_ADMIN + NET_RAW | tailscale | Can't create TUN device or manage routes |
+
+## Read-Only Safe Apps
+
+Only these apps can run with `--read-only` + tmpfs: searxng, grafana, filebrowser, electrumx, mempool-electrs, electrs, nostr-rs-relay, ollama, indeedhub
+
+All others need writable root or will fail silently.
+
+## Systemd Sandbox Requirements for Rootless Podman
+
+These systemd service settings MUST be configured for rootless Podman to work:
+
+| Setting | Required Value | Why |
+|---------|---------------|-----|
+| `ProtectHome=` | `no` | Podman stores images in `~/.local/share/containers/` |
+| `PrivateTmp=` | `no` | Podman runtime lives in `/tmp/podman-run-1000/` |
+| `RestrictNamespaces=` | NOT SET | Rootless podman creates user namespaces |
+| `SystemCallFilter=` | NOT SET | Rootless podman needs clone/unshare syscalls |
+| `ReadWritePaths=` | Include `/var/lib/archipelago /run/user /tmp /etc/containers /var/lib/containers /run/containers` | Volume data + podman runtime paths |
+| `Environment=` | `XDG_RUNTIME_DIR=/run/user/1000` | Podman socket location |
@@ -0,0 +1,71 @@
+# Archipelago Canonical Port Map
+
+All port assignments across the 4 configuration layers. When adding or debugging an app, every row must be consistent across all columns.
+
+## Bitcoin Stack
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| bitcoin-knots | 8332, 8333 | 8332, 8333 | archy-net | /app/bitcoin-knots/ | 8332→bitcoin-knots |
+| bitcoin-ui | 8334 | 80 | bridge | /app/bitcoin-ui/ | 8334→bitcoin-knots |
+| electrs | 50001 | 50001 | archy-net | /app/electrs/ | 50001→electrs |
+| lnd | 9735, 10009, 8080 | 9735, 10009, 8080 | archy-net | /app/lnd/ | 10009→lnd |
+| lnd-ui (RTL) | 8081 | 80 | bridge | /app/lnd-ui/ | 8081→lnd |
+
+## Lightning & Payment
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| btcpay-server | 23000 | 49392 | archy-net | /app/btcpay/ | 23000→btcpay-server |
+| nbxplorer | 24444 | 32838 | archy-net | N/A (internal) | N/A |
+| fedimint | 8173, 8174, 8175 | 8173, 8174, 8175 | archy-net | /app/fedimint/ | 8174→fedimint |
+| fedimint-gateway | 8175 | 8175 | archy-net | /app/fedimint-gateway/ | 8175→fedimint-gateway |
+
+## Explorer & Monitoring
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| mempool | 4080 | 8080 | archy-net | /app/mempool/ | 4080→mempool |
+| grafana | 3000 | 3000 | bridge | /app/grafana/ | 3000→grafana (new tab) |
+
+## Self-Hosted Apps
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| nextcloud | 8085 | 80 | bridge | /app/nextcloud/ | 8085→nextcloud |
+| vaultwarden | 8082 | 80 | bridge | /app/vaultwarden/ | 8082→vaultwarden (new tab) |
+| filebrowser | 8083 | 80 | bridge | /app/filebrowser/ | 8083→filebrowser |
+| searxng | 8888 | 8080 | bridge | /app/searxng/ | 8888→searxng |
+| photoprism | 2342 | 2342 | bridge | /app/photoprism/ | 2342→photoprism (new tab) |
+| jellyfin | 8096 | 8096 | bridge | /app/jellyfin/ | 8096→jellyfin |
+| homeassistant | 8123 | 8123 | bridge | /app/homeassistant/ | 8123→homeassistant (new tab) |
+| ollama | 11434 | 11434 | archy-net | /app/ollama/ | 11434→ollama |
+| open-webui | 3080 | 8080 | archy-net | /app/open-webui/ | 3080→open-webui |
+
+## Nostr & Social
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| nostr-rs-relay | 7000 | 8080 | archy-net | /app/nostr-rs-relay/ | 7000→nostr-rs-relay |
+| indeedhub | 3001 | 3000 | archy-net | /app/indeedhub/ | 3001→indeedhub |
+
+## System
+
+| App | Host Port(s) | Container Port(s) | Network | Nginx Path | Frontend Map |
+|-----|-------------|-------------------|---------|------------|-------------|
+| tailscale | 8240 | 8240 | host | /app/tailscale/ | N/A |
+| nginx-proxy-manager | 81, 8443 | 81, 443 | bridge | N/A | 81→nginx-proxy-manager |
+
+## Multi-Container Stacks
+
+**Immich**: immich-server (2283), immich-postgres (internal 5432), immich-redis (internal 6379) — all on immich-net
+**Penpot**: penpot-frontend (9001→80), penpot-backend, penpot-exporter, penpot-postgres, penpot-mailcatch — all on penpot-net
+**Mempool**: mempool (4080→8080), mempool-db (internal 3306) — on archy-net
+**BTCPay**: btcpay-server (23000→49392), nbxplorer (24444→32838), btcpay-postgres (internal 5432) — on archy-net
+
+## Key Notes
+
+- **archy-net apps** resolve each other by container name (e.g., `bitcoin-knots:8332`)
+- **bridge apps** are standalone — access services via host IP/port
+- **host network** (tailscale only) — shares host namespace, no port mapping
+- **New tab apps**: btcpay (23000), grafana (3000), vaultwarden (8082), photoprism (2342), homeassistant (8123) — X-Frame-Options blocks iframe
@@ -0,0 +1,93 @@
+# Rootless Podman UID Mapping Reference
+
+## How Rootless UID Mapping Works
+
+When Podman runs as the `archipelago` user (UID 1000), container processes don't run as their "apparent" UID on the host. Instead, Linux user namespaces remap UIDs.
+
+**Mapping formula**: `host_uid = 100000 + container_uid`
+
+This is configured in `/etc/subuid` and `/etc/subgid`:
+```
+archipelago:100000:65536
+```
+
+This means:
+- Container UID 0 (root inside container) → Host UID 100000 (unprivileged on host)
+- Container UID 70 (postgres) → Host UID 100070
+- Container UID 101 (bitcoin) → Host UID 100101
+- etc.
+
+## Why This Matters
+
+Volume directories (bind mounts) on the host must be owned by the **mapped** UID, not the container UID. If Bitcoin runs as UID 101 inside its container, the host directory must be owned by UID 100101.
+
+If ownership is wrong, the container gets `permission denied` when trying to read/write its data.
+
+## Complete UID Mapping Table
+
+| Container UID | Host UID | Containers | Fix Command |
+|---|---|---|---|
+| 0 (root) | 100000 | lnd, fedimint, fedimint-gateway, homeassistant, jellyfin, vaultwarden, photoprism, ollama, filebrowser, electrumx, btcpay-server, nbxplorer, immich, nostr-rs-relay, strfry, nextcloud, searxng, onlyoffice, tailscale, uptime-kuma | `sudo chown -R 100000:100000 /var/lib/archipelago/{app}` |
+| 70 | 100070 | postgres (btcpay-db, immich-db, penpot-postgres) | `sudo chown -R 100070:100070 /var/lib/archipelago/postgres-*` |
+| 101 | 100101 | bitcoin-knots, bitcoin-core | `sudo chown -R 100101:100101 /var/lib/archipelago/bitcoin` |
+| 472 | 100472 | grafana | `sudo chown -R 100472:100472 /var/lib/archipelago/grafana` |
+| 999 | 100999 | MariaDB (mysql-mempool) | `sudo chown -R 100999:100999 /var/lib/archipelago/mysql-mempool` |
+
+## How to Find a Container's UID
+
+If you encounter a new container with permission issues:
+
+```bash
+# Check what user the container runs as
+podman inspect CONTAINER_NAME --format "{{.Config.User}}"
+
+# If empty, it runs as root (UID 0) → host UID 100000
+
+# If it shows a username, find the UID inside the image
+podman run --rm IMAGE_NAME id
+
+# Then calculate: host_uid = 100000 + container_uid
+```
+
+## Fix Script
+
+Run this after any fresh install, migration, or when containers have permission errors:
+
+```bash
+#!/bin/bash
+# Fix all rootless podman volume ownership
+
+# UID 0 → 100000 (most containers)
+for dir in lnd fedimint fedimint-gateway homeassistant jellyfin vaultwarden photoprism \
+           ollama filebrowser electrumx btcpay nbxplorer immich nostr-rs-relay nextcloud \
+           searxng onlyoffice uptime-kuma; do
+  [ -d "/var/lib/archipelago/$dir" ] && sudo chown -R 100000:100000 "/var/lib/archipelago/$dir"
+done
+
+# UID 101 → 100101 (Bitcoin)
+[ -d "/var/lib/archipelago/bitcoin" ] && sudo chown -R 100101:100101 /var/lib/archipelago/bitcoin
+
+# UID 70 → 100070 (PostgreSQL)
+for dir in /var/lib/archipelago/postgres-* /var/lib/archipelago/btcpay-db /var/lib/archipelago/immich-db; do
+  [ -d "$dir" ] && sudo chown -R 100070:100070 "$dir"
+done
+
+# UID 999 → 100999 (MariaDB)
+[ -d "/var/lib/archipelago/mysql-mempool" ] && sudo chown -R 100999:100999 /var/lib/archipelago/mysql-mempool
+
+# UID 472 → 100472 (Grafana)
+[ -d "/var/lib/archipelago/grafana" ] && sudo chown -R 100472:100472 /var/lib/archipelago/grafana
+```
+
+## Rootful vs Rootless Comparison
+
+| Aspect | Rootful (old) | Rootless (current) |
+|--------|---------------|-------------------|
+| Podman command | `sudo podman` | `podman` (as archipelago user) |
+| Container storage | `/var/lib/containers/storage` | `~/.local/share/containers/storage` |
+| Container subnet | `10.88.0.0/16` | `10.89.0.0/16` |
+| Volume ownership | Container UID directly | Mapped UID (100000 + container_uid) |
+| Requires root? | Yes | No (except fixing volume ownership) |
+| XDG_RUNTIME_DIR | Not needed | Required: `/run/user/1000` |
+| User lingering | Not needed | Required: `loginctl enable-linger` |
+| Systemd restrictions | All can be enabled | Must disable: RestrictNamespaces, SystemCallFilter |
@@ -0,0 +1,27 @@
+# Polish: Backend Quality
+
+All changes built on dev server, not macOS: `./scripts/deploy-to-target.sh --live`
+
+## Priority 1: Eliminate panics
+```bash
+ssh archipelago@192.168.1.228 "grep -rn 'unwrap()\|\.expect(' ~/archy/core/archipelago/src/ --include='*.rs' | grep -v test | grep -v '_test.rs'"
+```
+Replace with `?` + `.context()` or `.map_err()`.
+
+## Priority 2: Add timeouts
+- Container ops: `tokio::time::timeout(Duration::from_secs(30), op).await`
+- HTTP/RPC calls: `reqwest::Client::builder().timeout(Duration::from_secs(10))`
+
+## Priority 3: Connection pooling
+Store reusable `reqwest::Client` in RpcHandler instead of creating per-request.
+
+## Priority 4: Clippy
+```bash
+ssh archipelago@192.168.1.228 "cd ~/archy && cargo clippy --all-targets --all-features 2>&1"
+```
+
+## Priority 5: Replace println with tracing
+`println!` → `tracing::info!`, `eprintln!` → `tracing::warn!`
+
+## Verify
+Zero clippy warnings, zero unwrap/expect in prod code, zero println.
@@ -0,0 +1,26 @@
+# Polish: Deployment Pipeline
+
+## Pre-Deploy Checks
+Add to deploy-to-target.sh: SSH key exists, target reachable, 2GB free disk space.
+
+## Backup Before Deploy
+```bash
+sudo cp /usr/local/bin/archipelago /usr/local/bin/archipelago.backup
+sudo cp -a /opt/archipelago/web-ui /opt/archipelago/web-ui.backup
+sudo cp /etc/nginx/sites-available/archipelago /etc/nginx/sites-available/archipelago.backup
+```
+
+## Health Check After Deploy
+Loop up to 15 attempts, 2s apart, checking `curl http://localhost:5678/health` returns 200.
+
+## Rollback on Failure
+If health check fails: restore binary, frontend, nginx from .backup files, restart services.
+
+## Deployment Lock
+Use `flock` on `/tmp/archipelago-deploy.lock` to prevent concurrent deploys.
+
+## Nginx Validation
+Always `sudo nginx -t` before reload. If invalid, restore backup config.
+
+## Integration Flow
+1. acquire_lock → 2. pre_deploy_checks → 3. backup_current → 4. build + deploy → 5. validate_nginx → 6. restart services → 7. health_check || rollback
@@ -0,0 +1,23 @@
+# Polish: Error Handling
+
+## Find
+- Silent catches: `grep -rn "catch.*=>.*{}" --include="*.vue" --include="*.ts" src/`
+- Empty try/catch: `grep -rn "catch.*{$" -A1` looking for immediate `}`
+- Missing error states in views: check each view has `errorMessage` ref
+
+## Fix Pattern
+```typescript
+.catch((err) => {
+  console.error('[ComponentName] operation failed:', err)
+  errorMessage.value = err instanceof Error ? err.message : 'Operation failed'
+})
+```
+
+Template: `<p v-if="errorMessage" class="text-red-400 text-sm mt-2">{{ errorMessage }}</p>`
+
+## Backend
+- Replace `unwrap_or_default()` on serialization with proper error propagation
+- Consistent RPC error structure: `{ error: { code: string, message: string } }`
+
+## Verify
+Both should return zero: silent catches and empty catch blocks.
@@ -0,0 +1,30 @@
+# Polish: Form Validation
+
+## Pattern
+```typescript
+const isSubmitting = ref(false)
+const passwordErrors = computed(() => {
+  const errors: string[] = []
+  if (password.value.length > 0 && password.value.length < 8)
+    errors.push('Must be at least 8 characters')
+  return errors
+})
+
+async function submit() {
+  if (isSubmitting.value) return
+  isSubmitting.value = true
+  try { await rpcClient.call(...) }
+  catch (err) { errorMessage.value = formatError(err) }
+  finally { isSubmitting.value = false }
+}
+```
+
+## Checklist per form
+- Real-time validation as user types (debounced 300ms)
+- Submit button disabled during operation and when validation fails
+- All text inputs trimmed before submission
+- Error messages are user-friendly (no raw error strings)
+- TOTP: `inputmode="numeric"`, auto-submit at 6 digits
+
+## Forms to polish
+Login.vue (password setup, TOTP), Settings.vue (password change), any other form inputs.
@@ -0,0 +1,26 @@
+# Polish: Loading States
+
+Every async view needs 3 states: loading skeleton, empty state, timeout warning.
+
+## Skeleton Pattern
+```vue
+<div v-if="isLoading"><!-- skeleton matching layout --></div>
+<div v-else-if="items.length === 0" class="glass-card text-center py-12">
+  <p class="text-white/60">No items yet</p>
+</div>
+<div v-else><!-- real content --></div>
+```
+
+## Timeout Warning
+After 15s show "Taking longer than expected...", after 30s show troubleshooting.
+```typescript
+const loadingTooLong = ref(false)
+const timeout = setTimeout(() => { loadingTooLong.value = true }, 15000)
+watch(isLoading, (val) => { if (!val) clearTimeout(timeout) })
+```
+
+## Priority Views
+Apps.vue, AppDetails.vue, Marketplace.vue, Dashboard.vue, Cloud.vue, Settings.vue, Server.vue
+
+## Verify
+Each view has: `isLoading` ref, skeleton section, empty state, timeout warning. Use global classes only.
@@ -0,0 +1,22 @@
+# Polish: Security Hardening
+
+## 1. Systemd Service
+Add to `image-recipe/configs/archipelago.service`:
+`NoNewPrivileges=true`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/archipelago`
+Verify: `ssh ... "sudo systemd-analyze security archipelago"` — score < 5.0
+
+## 2. Nginx Headers
+- HSTS (HTTPS only): `add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;`
+- Rate limiting zones: `limit_req_zone $binary_remote_addr zone=auth:10m rate=5r/m;`
+- Custom log format stripping tokens
+
+## 3. Secrets Management
+Replace hardcoded `archipelago123` with generated secrets:
+- Generate on first boot: `openssl rand -base64 24 > /var/lib/archipelago/secrets/bitcoin-rpc-pass`
+- Backend reads from env var: `std::env::var("ARCHIPELAGO_BITCOIN_RPC_PASS")`
+
+## 4. SSH Hardening
+Replace `StrictHostKeyChecking=no` with `StrictHostKeyChecking=accept-new` in deploy script.
+
+## Verify
+`grep -rn 'archipelago123' scripts/ core/` should return zero. Nginx headers pass curl check. Rate limiting returns 429 on rapid auth requests.
@@ -0,0 +1,25 @@
+# Polish: WebSocket & Real-Time
+
+## 1. Connection Status Indicator
+Add to App.vue header: green dot (connected), amber pulse (reconnecting), red (disconnected).
+Connect to actual WebSocket state from websocket.ts.
+
+## 2. Reconnection UX
+After max reconnect attempts, show persistent banner "Connection lost. Click to retry."
+Add `forceReconnect()` method that resets attempt counter.
+
+## 3. Heartbeat
+Active ping every 30s with 5s pong timeout (replace passive 60s stale detection).
+Backend must respond to `ping` with `pong` — check handler.rs.
+
+## 4. Session Timeout
+In rpc-client.ts base `call()`: on 401/403 response, redirect to `/login`.
+
+## 5. Race Condition Fix
+Use listener deduplication (Set) or remove-all-then-resubscribe on reconnect.
+
+## 6. Message Queuing
+Queue subscription requests while disconnected, replay on reconnect.
+
+## Verify
+Kill backend → shows "Disconnected" → restart → auto-reconnects. Toggle wifi → status updates. Session timeout → redirects to login.