fix: overhaul container lifecycle — recovery, health, uninstall, UI state

Container recovery:
- Health monitor: MAX_RESTART_ATTEMPTS 3→10, interval 60s→120s
- Dependency-aware restarts: won't restart services before their deps
- Reset dependent counters when a dependency recovers
- Handle "created" state containers (were invisible to health monitor)
- Added IndeedHub, mempool-api, mysql to tier system
- Crash recovery: podman start timeout 30s→120s with retry
- Podman client: socket timeout 5s→30s, added restart policy

UI state representation:
- Exit code 0 shows "stopped" (gray), not "crashed" (red)
- Exit code 137 shows "killed (OOM)"
- Non-zero exit shows "crashed" (red)
- Added exit_code field to PackageDataEntry

Install/uninstall fixes:
- Install returns error when container doesn't start (was silent success)
- Post-install hooks awaited instead of fire-and-forget tokio::spawn
- Uninstall: graceful rm before force, volume prune, network cleanup
- Uninstall returns error on partial failure (was 200 OK)

Config consistency:
- DB passwords read from /var/lib/archipelago/secrets/ (was hardcoded)
- Bitcoin: added ZMQ ports 28332/28333 for LND block notifications
- IndeedHub port 7777→8190 (was conflicting with strfry)
- Marketplace versions: LND 0.17.4→0.18.4, Mempool 2.5.0→3.0.0

Performance:
- Metrics collector interval 60s→300s (was duplicating health monitor)
- Podman client: proper error propagation instead of unwrap_or_default

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dorian
2026-03-31 07:03:57 +01:00
co-authored by Claude Opus 4.6
parent cdff10a8bc
commit 64b57dca7d
65 changed files with 3950 additions and 298 deletions
@@ -0,0 +1,27 @@
# Polish: Backend Quality
All changes built on dev server, not macOS: `./scripts/deploy-to-target.sh --live`
## Priority 1: Eliminate panics
```bash
ssh archipelago@192.168.1.228 "grep -rn 'unwrap()\|\.expect(' ~/archy/core/archipelago/src/ --include='*.rs' | grep -v test | grep -v '_test.rs'"
```
Replace with `?` + `.context()` or `.map_err()`.
## Priority 2: Add timeouts
- Container ops: `tokio::time::timeout(Duration::from_secs(30), op).await`
- HTTP/RPC calls: `reqwest::Client::builder().timeout(Duration::from_secs(10))`
## Priority 3: Connection pooling
Store reusable `reqwest::Client` in RpcHandler instead of creating per-request.
## Priority 4: Clippy
```bash
ssh archipelago@192.168.1.228 "cd ~/archy && cargo clippy --all-targets --all-features 2>&1"
```
## Priority 5: Replace println with tracing
`println!``tracing::info!`, `eprintln!``tracing::warn!`
## Verify
Zero clippy warnings, zero unwrap/expect in prod code, zero println.
@@ -0,0 +1,26 @@
# Polish: Deployment Pipeline
## Pre-Deploy Checks
Add to deploy-to-target.sh: SSH key exists, target reachable, 2GB free disk space.
## Backup Before Deploy
```bash
sudo cp /usr/local/bin/archipelago /usr/local/bin/archipelago.backup
sudo cp -a /opt/archipelago/web-ui /opt/archipelago/web-ui.backup
sudo cp /etc/nginx/sites-available/archipelago /etc/nginx/sites-available/archipelago.backup
```
## Health Check After Deploy
Loop up to 15 attempts, 2s apart, checking `curl http://localhost:5678/health` returns 200.
## Rollback on Failure
If health check fails: restore binary, frontend, nginx from .backup files, restart services.
## Deployment Lock
Use `flock` on `/tmp/archipelago-deploy.lock` to prevent concurrent deploys.
## Nginx Validation
Always `sudo nginx -t` before reload. If invalid, restore backup config.
## Integration Flow
1. acquire_lock → 2. pre_deploy_checks → 3. backup_current → 4. build + deploy → 5. validate_nginx → 6. restart services → 7. health_check || rollback
@@ -0,0 +1,23 @@
# Polish: Error Handling
## Find
- Silent catches: `grep -rn "catch.*=>.*{}" --include="*.vue" --include="*.ts" src/`
- Empty try/catch: `grep -rn "catch.*{$" -A1` looking for immediate `}`
- Missing error states in views: check each view has `errorMessage` ref
## Fix Pattern
```typescript
.catch((err) => {
console.error('[ComponentName] operation failed:', err)
errorMessage.value = err instanceof Error ? err.message : 'Operation failed'
})
```
Template: `<p v-if="errorMessage" class="text-red-400 text-sm mt-2">{{ errorMessage }}</p>`
## Backend
- Replace `unwrap_or_default()` on serialization with proper error propagation
- Consistent RPC error structure: `{ error: { code: string, message: string } }`
## Verify
Both should return zero: silent catches and empty catch blocks.
+30
View File
@@ -0,0 +1,30 @@
# Polish: Form Validation
## Pattern
```typescript
const isSubmitting = ref(false)
const passwordErrors = computed(() => {
const errors: string[] = []
if (password.value.length > 0 && password.value.length < 8)
errors.push('Must be at least 8 characters')
return errors
})
async function submit() {
if (isSubmitting.value) return
isSubmitting.value = true
try { await rpcClient.call(...) }
catch (err) { errorMessage.value = formatError(err) }
finally { isSubmitting.value = false }
}
```
## Checklist per form
- Real-time validation as user types (debounced 300ms)
- Submit button disabled during operation and when validation fails
- All text inputs trimmed before submission
- Error messages are user-friendly (no raw error strings)
- TOTP: `inputmode="numeric"`, auto-submit at 6 digits
## Forms to polish
Login.vue (password setup, TOTP), Settings.vue (password change), any other form inputs.
@@ -0,0 +1,26 @@
# Polish: Loading States
Every async view needs 3 states: loading skeleton, empty state, timeout warning.
## Skeleton Pattern
```vue
<div v-if="isLoading"><!-- skeleton matching layout --></div>
<div v-else-if="items.length === 0" class="glass-card text-center py-12">
<p class="text-white/60">No items yet</p>
</div>
<div v-else><!-- real content --></div>
```
## Timeout Warning
After 15s show "Taking longer than expected...", after 30s show troubleshooting.
```typescript
const loadingTooLong = ref(false)
const timeout = setTimeout(() => { loadingTooLong.value = true }, 15000)
watch(isLoading, (val) => { if (!val) clearTimeout(timeout) })
```
## Priority Views
Apps.vue, AppDetails.vue, Marketplace.vue, Dashboard.vue, Cloud.vue, Settings.vue, Server.vue
## Verify
Each view has: `isLoading` ref, skeleton section, empty state, timeout warning. Use global classes only.
@@ -0,0 +1,22 @@
# Polish: Security Hardening
## 1. Systemd Service
Add to `image-recipe/configs/archipelago.service`:
`NoNewPrivileges=true`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/archipelago`
Verify: `ssh ... "sudo systemd-analyze security archipelago"` — score < 5.0
## 2. Nginx Headers
- HSTS (HTTPS only): `add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;`
- Rate limiting zones: `limit_req_zone $binary_remote_addr zone=auth:10m rate=5r/m;`
- Custom log format stripping tokens
## 3. Secrets Management
Replace hardcoded `archipelago123` with generated secrets:
- Generate on first boot: `openssl rand -base64 24 > /var/lib/archipelago/secrets/bitcoin-rpc-pass`
- Backend reads from env var: `std::env::var("ARCHIPELAGO_BITCOIN_RPC_PASS")`
## 4. SSH Hardening
Replace `StrictHostKeyChecking=no` with `StrictHostKeyChecking=accept-new` in deploy script.
## Verify
`grep -rn 'archipelago123' scripts/ core/` should return zero. Nginx headers pass curl check. Rate limiting returns 429 on rapid auth requests.
@@ -0,0 +1,25 @@
# Polish: WebSocket & Real-Time
## 1. Connection Status Indicator
Add to App.vue header: green dot (connected), amber pulse (reconnecting), red (disconnected).
Connect to actual WebSocket state from websocket.ts.
## 2. Reconnection UX
After max reconnect attempts, show persistent banner "Connection lost. Click to retry."
Add `forceReconnect()` method that resets attempt counter.
## 3. Heartbeat
Active ping every 30s with 5s pong timeout (replace passive 60s stale detection).
Backend must respond to `ping` with `pong` — check handler.rs.
## 4. Session Timeout
In rpc-client.ts base `call()`: on 401/403 response, redirect to `/login`.
## 5. Race Condition Fix
Use listener deduplication (Set) or remove-all-then-resubscribe on reconnect.
## 6. Message Queuing
Queue subscription requests while disconnected, replay on reconnect.
## Verify
Kill backend → shows "Disconnected" → restart → auto-reconnects. Toggle wifi → status updates. Session timeout → redirects to login.