diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 00000000..904c84b6 --- /dev/null +++ b/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,333 @@ + +# Architecture + +**Analysis Date:** 2026-07-29 + +## System Overview + +```text +┌────────────────────────────────────────────────────────────────┐ +│ Frontend Layer (Vue 3) │ +│ `neode-ui/src` (TypeScript + SPA) │ +│ Routes → Views → Components → Composables → RPC Client │ +└────────────────┬─────────────────────────────────────────────┘ + │ WebSocket + HTTP(S) + │ JSON-RPC 2.0 protocol + ▼ +┌────────────────────────────────────────────────────────────────┐ +│ HTTP Server Layer (Hyper) │ +│ `core/archipelago/src/server.rs` │ +│ TCP Listener → Hyper → Router → ApiHandler/RpcHandler │ +└────────────────┬─────────────────────────────────────────────┘ + │ + ┌──────────┴──────────┬──────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌──────────┐ ┌────────────┐ + │ WebSocket │ RPC │ │ Content │ + │ Handler │ Handler │ │ Proxy │ + │ (state sync) │ (methods)│ │ (app URIs) │ + └─────────┘ └──────────┘ └────────────┘ + │ │ + └──────────┬───────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Service Layer (Async Tasks) │ + │ `core/archipelago/src/api/rpc/*` │ + │ │ + │ • auth, identity, secrets │ + │ • container orchestration │ + │ • bitcoin, lightning, wallet │ + │ • mesh, federation, FIPS │ + │ • content, backup, settings │ + └─────────────┬───────────────────────┘ + │ + ┌───────────┼───────────┬──────────────┐ + │ │ │ │ + ▼ ▼ ▼ ▼ + ┌─────────┐ ┌──────────┐ ┌────────┐ ┌──────────┐ + │Container│ │ State │ │BlobStore + │Orch. │ │Manager │ │ │ │Identity │ + │(Podman) │ │(Broadcast + │ │ │ channels)│ │ ContentClient Manager │ + └─────────┘ └──────────┘ └────────┘ └──────────┘ + │ │ │ │ + └───────────┼───────────┼─────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Persistent Storage Layer │ + │ │ + │ • Data directory files (YAML/JSON) │ + │ • SQLite (session store) │ + │ • Blob store (content-addressed) │ + │ • Podman container state │ + │ • Secret vaults (encrypted) │ + └─────────────────────────────────────┘ +``` + +## Component Responsibilities + +| Component | Responsibility | File | +|-----------|----------------|------| +| **Server** | HTTP listener, connection multiplexing, TLS/encryption | `core/archipelago/src/server.rs` | +| **ApiHandler** | HTTP request routing, authentication, response formatting | `core/archipelago/src/api/handler/mod.rs` | +| **RpcHandler** | JSON-RPC 2.0 dispatch, method registration, rate limiting | `core/archipelago/src/api/rpc/mod.rs` | +| **ContainerOrchestrator** | Podman lifecycle, manifest reconciliation, adoption | `core/archipelago/src/container/prod_orchestrator.rs` | +| **StateManager** | Central state broadcast channel, revision tracking | `core/archipelago/src/state.rs` | +| **AuthManager** | User credentials, session validation, password hashing | `core/archipelago/src/auth.rs` | +| **Identity Manager** | Node Ed25519 keys, seed derivation, Tor address | `core/archipelago/src/identity_manager.rs` | +| **BootReconciler** | Periodic manifest sync loop, adoption, remediation | `core/archipelago/src/container/boot_reconciler.rs` | +| **Frontend Router** | Vue Router, page navigation, deep linking | `neode-ui/src/router/index.ts` | +| **Frontend Stores** | Pinia state (apps, settings, user, mesh) | `neode-ui/src/stores/` | +| **Frontend Components** | UI elements, modals, cards, layout primitives | `neode-ui/src/components/` | + +## Pattern Overview + +**Overall:** Multi-tier async architecture with centralized request dispatch and broadcast state synchronization. + +**Key Characteristics:** +- **Async-first (Tokio)** - All I/O operations are non-blocking; task spawning for background work +- **RPC-driven API** - Frontend communicates via JSON-RPC 2.0 (not REST); single `/api/v0` WebSocket + HTTP endpoint +- **State as broadcast** - Global state changes flow through Tokio broadcast channels to all connected WebSocket clients +- **Manifest-driven containers** - App lifecycle controlled by declarative YAML manifests (Archipelago-specific extensions) +- **Plugin architecture** - Apps are isolated Podman containers with declarative interfaces (web UI, ports, secrets) + +## Layers + +**HTTP/Transport Layer:** +- Purpose: Accept inbound connections, handle TLS termination, demultiplex HTTP/WebSocket +- Location: `core/archipelago/src/server.rs` +- Contains: Hyper listener, TCP accept loop, connection state tracking +- Depends on: Tokio, Hyper, TLS/mTLS libraries (rustls/openssl) +- Used by: All external clients (web UI, companion app, API consumers) + +**Request Routing & Auth Layer:** +- Purpose: Dispatch HTTP requests to handlers, validate sessions, enforce CSRF, rate-limit login +- Location: `core/archipelago/src/api/` (handler + rpc submodules) +- Contains: Route matching, middleware chain, cookie extraction, error formatting +- Depends on: Server, StateManager, SessionStore +- Used by: All request paths; gates API access + +**RPC Dispatch Layer:** +- Purpose: Deserialize JSON-RPC 2.0 requests, call appropriate service method, serialize responses +- Location: `core/archipelago/src/api/rpc/mod.rs` + subdirectories (auth.rs, container.rs, bitcoin.rs, etc.) +- Contains: Method table, parameter validation, response formatting, rate limit checks +- Depends on: All service modules +- Used by: Frontend (WebSocket + HTTP POST to /api/v0), internal tools + +**Service Layer:** +- Purpose: Implement business logic — container lifecycle, identity, auth, content sync, mesh discovery +- Location: `core/archipelago/src/api/rpc/*` (one RPC module per domain), plus `core/archipelago/src/` (background tasks) +- Contains: ~40 RPC method modules + 50+ core service modules (bootstrap.rs, health_monitor.rs, crash_recovery.rs, etc.) +- Depends on: StateManager, ContainerOrchestrator, config/secrets, external services (Bitcoin, Lightning, FIPS) +- Used by: RPC layer; other services for cross-cutting concerns (mesh, federation, webhooks) + +**State Management Layer:** +- Purpose: Hold canonical application state, broadcast changes to all connected clients, persist snapshots +- Location: `core/archipelago/src/state.rs` (StateManager + data_model.rs) +- Contains: RwLock, broadcast channel, revision counter +- Depends on: DataModel (serde-serializable struct tree) +- Used by: All services that mutate state (container ops, auth, settings) + +**Container Orchestration Layer:** +- Purpose: Podman lifecycle management, image verification, secret injection, crash recovery, adoption +- Location: `core/archipelago/src/container/prod_orchestrator.rs` (1M+ lines; split across boot_reconciler.rs, quadlet.rs, docker_packages.rs, etc.) +- Contains: Manifest parsing, image pull/verify, container create/start/stop, volume mounts, networking +- Depends on: Podman CLI + socket, config parser, image registries, local filesystem +- Used by: RPC container.* methods, BootReconciler loop, crash recovery + +**Frontend Layer (Vue 3):** +- Purpose: Render UI, dispatch RPC calls, maintain local UI state, handle user input +- Location: `neode-ui/src/` +- Contains: Views (pages), Components (reusable UI), Composables (logic hooks), Stores (Pinia), Router +- Depends on: Vue 3, Vue Router, Pinia, RPC client library (custom), D3/Leaflet (charts/maps) +- Used by: Browser clients (desktop, mobile, companion app via WebView) + +## Data Flow + +### Primary Request Path (User Action → Backend → State Sync) + +1. **Frontend user interaction** (click button, type input) → Vue component event handler + - Location: `neode-ui/src/views/*.vue` or `neode-ui/src/components/*.vue` + +2. **Composable dispatches RPC** (e.g., `useContainerInstall()` calls `rpc.container.install()`) + - Location: `neode-ui/src/composables/` (custom or imported from `api/rpc-client.ts`) + +3. **RPC client serializes → HTTP/WebSocket POST to /api/v0** + - Location: `neode-ui/src/api/rpc-client.ts` + - Payload: `{ jsonrpc: "2.0", method: "container.install", params: {...}, id: ... }` + +4. **HTTP Server receives, routes to ApiHandler** + - Location: `core/archipelago/src/server.rs` (listener) → `core/archipelago/src/api/handler/mod.rs` (dispatch) + +5. **ApiHandler checks auth**, extracts body, calls RpcHandler + - Location: `core/archipelago/src/api/handler/mod.rs:handle_request()` + +6. **RpcHandler dispatches by method name** to specific RPC module + - Location: `core/archipelago/src/api/rpc/mod.rs:call()` → routing to `core/archipelago/src/api/rpc/container.rs:install()` + +7. **Service method executes** (e.g., `container.rs:install()` calls orchestrator, updates state) + - Location: `core/archipelago/src/api/rpc/container.rs` (calls methods on ContainerOrchestrator) + +8. **StateManager.update_data()** broadcasts the new state to all WebSocket subscribers + - Location: `core/archipelago/src/state.rs:update_data()` → broadcast channel + - All connected WebSocket clients receive `{ rev: N, data: {...} }` update + +9. **Frontend receives state update**, updates Pinia stores, re-renders UI + - Location: `neode-ui/src/stores/` (Pinia stores mutate) → Vue reactivity chain → DOM update + +**State Management:** +- All reads from `StateManager` go through `get_snapshot()` which acquires read-lock +- All writes go through `update_data()` which acquires write-lock + increments revision +- Broadcast channel has ~100-message buffer; slow subscribers may lose old updates (by design — UI only needs latest) +- WebSocket clients re-sync on reconnect via `get_snapshot()` call (full state transfer) + +### Secondary Flow: Scheduled Reconciliation (Convergence Loop) + +1. **BootReconciler spawned at startup** in `main.rs` + - Location: `core/archipelago/src/main.rs` (line ~338-348) + +2. **Reconciler runs every `RECONCILER_DEFAULT_INTERVAL`** (~30s typical) + - Location: `core/archipelago/src/container/boot_reconciler.rs:run_forever()` + +3. **Compares desired manifests (disk + registry catalog) vs actual Podman state** + - Looks for: containers missing, containers orphaned, image updates, secret changes + +4. **Applies remediation** (create, delete, restart containers) + - Calls: orchestrator.reconcile_*() methods + +5. **Logs changes, broadcasts state update if anything changed** + - Frontend receives update, shows user the reconciled app state + +This ensures apps survive crashes, OTA updates, or manual Podman edits — the desired state always converges. + +## Key Abstractions + +**ContainerOrchestrator trait:** +- Purpose: Abstract container lifecycle behind a trait so Prod (Podman-based) and Dev (in-memory) modes can coexist +- Examples: `core/archipelago/src/container/prod_orchestrator.rs`, `core/archipelago/src/container/dev_orchestrator.rs` +- Pattern: Trait-based strategy; RpcHandler holds `Arc`, switches at runtime +- Methods: create, start, stop, delete, adopt, list, reconcile, install, upgrade + +**Manifest (YAML-based declarative app):** +- Purpose: Fully describe an app's container, dependencies, secrets, ports, UI in one file +- Examples: `/opt/archipelago/apps/*/manifest.yml` (on-disk) or registry-delivered catalogs +- Pattern: Custom extensions over OCI/Docker Compose (e.g., `interfaces.main.ui`, `generated_secrets`) +- Parsed into: `container::manifest::Manifest` struct, consumed by orchestrator + +**RPC Method Modules:** +- Purpose: Group related JSON-RPC methods by domain (auth, container, bitcoin, mesh, etc.) +- Examples: `core/archipelago/src/api/rpc/auth.rs`, `core/archipelago/src/api/rpc/bitcoin.rs` +- Pattern: Each module exports `pub async fn method_name(handler, params) -> Result` +- Registration: Hardcoded dispatch in `RpcHandler::call()` (no reflection; methods are explicit) + +**BlobStore (Content-Addressed):** +- Purpose: Store attachments/files by SHA-256 hash; issue time-limited capability tokens for access +- Examples: Used by mesh.send-content, federation attachments, backup archives +- Pattern: Capability-based access control (CBAC); tokens scoped to issuer pubkey + hash +- Located: `core/archipelago/src/blobs.rs` + `core/archipelago/src/content_server.rs` + +**StateManager + DataModel:** +- Purpose: Single source of truth for UI state; broadcast updates to all clients +- Pattern: Read-write lock over a serde-serializable struct tree; broadcast channel for efficiency +- Persistence: Most state is ephemeral (app listings, UI settings); durable state persists to disk separately +- Clients: Frontend (WebSocket subscriber), internal services (read via get_snapshot), monitoring/debug + +**Session Store:** +- Purpose: Track authenticated HTTP sessions (cookie → user identity mapping) +- Examples: SQLite-backed or in-memory store +- Pattern: Session token issued at login, validated on each request, expires after TTL +- Used by: ApiHandler auth check, rate limiter (per IP + per user) + +## Entry Points + +**Backend Daemon (Binary):** +- Location: `core/archipelago/src/main.rs` +- Triggers: `systemd start archipelago.service` or manual `./archipelago` on development node +- Responsibilities: Parse config, init tracing, load/reconcile containers, start HTTP server, spawn background tasks +- Key setup: Load identity → setup auth → spawn orchestrator → load manifests → start reconciler → start server + +**Frontend SPA:** +- Location: `neode-ui/src/main.ts` +- Triggers: Browser loads `/index.html` (served by HTTP server from `/opt/archipelago/web-ui/`) +- Responsibilities: Boot Vue app, setup Router, setup Pinia stores, establish WebSocket to backend +- Key setup: Mount app → router ready → fetch initial state → subscribe to updates + +**RPC Endpoints (HTTP + WebSocket):** +- Location: `core/archipelago/src/api/` (handler routes requests here) +- Endpoint: `/api/v0` (JSON-RPC 2.0 POST or WebSocket upgrade) +- Methods: ~200+ RPCs across domains (auth, container, bitcoin, mesh, federation, etc.) +- Example: `POST /api/v0` with body `{"jsonrpc": "2.0", "method": "auth.login", "params": {...}, "id": 1}` + +**Background Tasks (Spawned at startup):** +- BootReconciler: Periodic manifest reconciliation loop +- Health Monitor: Periodic app health checks + restart +- Update Scheduler: Periodic app update checks +- Mesh Service: P2P mesh listener + sender (federation, LoRa) +- Webhook Relay: Listens for inbound webhooks, broadcasts to subscribers +- WebSocket Listener: Upgraded HTTP connections → broadcast state subscriber +- See: `core/archipelago/src/main.rs` (lines ~400-450 show the spawned tasks) + +## Architectural Constraints + +- **Single event loop** — All I/O-bound work runs on a single Tokio multi-threaded runtime; no worker threads by default (some container ops are blocking, run in tokio::task::spawn_blocking) +- **Global state via broadcast** — StateManager broadcasts to all WebSocket clients; no request-response for state changes (async by design) +- **Container state mutability** — Podman state can drift from manifest (manual edits, crashes); reconciler runs periodically to converge +- **No in-process data consistency** — Multiple services can mutate StateManager concurrently; last write wins (fine for UI; critical ops use locks) +- **Shared blob store** — All services that need to share content use the same BlobStore instance (single cap_key, single root directory) +- **Rate limiting per IP + method** — Prevents brute-force login, but shared IPs see shared limits (edge case: family users, proxies) +- **Session cookie same-site** — WebSocket + HTTP POST must be same-origin; CORS headers controlled by ApiHandler + +## Anti-Patterns + +### Circular RPC Dispatches + +**What happens:** An RPC method calls back into another RPC method, forming a cycle (e.g., auth.login → container.list → auth.check_permission → auth.login) +**Why it's wrong:** Deadlocks on RwLocks, infinite loops on state broadcasts, unclear error messages, hard to debug +**Do this instead:** Pass check result as a side-effect from the outer method; compute permissions once at the start. Use composable patterns in frontend instead (e.g., `useCanInstall()` checks perms once per component mount). + +### Synchronous blocking in RPC handlers + +**What happens:** RPC method calls `.unwrap()` on Podman command result, blocking the entire event loop +**Why it's wrong:** One slow container op (e.g., large image pull) blocks all concurrent users +**Do this instead:** Use `tokio::task::spawn_blocking()` for I/O that may take >100ms. See `core/archipelago/src/container/docker_packages.rs` for examples. + +### Hardcoding paths in app RPC modules + +**What happens:** `bitcoin.rs` hardcodes `/opt/archipelago/data/bitcoin.conf` instead of using `config.data_dir` +**Why it's wrong:** Dev mode, tests, and alternate installs all fail with "not found" +**Do this instead:** Read from `Config` struct, which is passed to every RPC method. See `core/archipelago/src/api/rpc/bitcoin.rs:status()` for correct pattern. + +### Frontend state outside Pinia stores + +**What happens:** Components use component-local ref<> for app list, duplicate the StateManager's data +**Why it's wrong:** Stale data after OTA updates, inconsistent with other users on the same node, race conditions on install/uninstall +**Do this instead:** Always derive from Pinia stores (e.g., `useAppStore().apps`). Stores subscribe to WebSocket updates. See `neode-ui/src/stores/appStore.ts`. + +### Not handling WebSocket reconnection + +**What happens:** Frontend goes offline for 10s (network glitch), WebSocket closes, frontend doesn't re-sync state +**Why it's wrong:** UI shows stale data (app still "installing" when actually done), user clicks again, double-action happens +**Do this instead:** WebSocket reconnect handler should re-fetch full state (`node.status`, etc.), re-subscribe. See `neode-ui/src/api/rpc-client.ts` for the reconnect loop. + +## Error Handling + +**Strategy:** Defensive layering — errors are caught at each tier, logged, and converted to user-facing messages. + +**Patterns:** +- HTTP layer: 4xx/5xx with JSON error (no 500s for logic errors; only for crashes) +- RPC layer: Serialize error as `{ error: { code: N, message: "...", data: {...} } }` per JSON-RPC spec +- Service layer: Use `anyhow::Result` + `?` operator for early exit; convert to `RpcError` at handler boundary +- Frontend: Catch RPC errors, show toast/modal, log to console (never crash the app) + +**Critical paths:** +- Auth failure: 401 Unauthorized + "Invalid password" (no "user not found" to leak usernames) +- Container ops: If reconciler sees drift, logs it but continues (never crashes the daemon) +- Image pull failure: Fallback to last-cached version if network timeout (user is never blocked on external registries) +- Podman socket unavailable: Return 503 Service Unavailable (user sees "Archipelago is starting") + +--- + +*Architecture analysis: 2026-07-29* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md new file mode 100644 index 00000000..33b335c3 --- /dev/null +++ b/.planning/codebase/CONCERNS.md @@ -0,0 +1,195 @@ +# Codebase Concerns + +**Analysis Date:** 2026-07-29 + +## Tech Debt + +**Federation node removal tombstone gap:** +- Issue: `federation::remove_node()` (`core/archipelago/src/federation/storage.rs:180-197`) calls `tombstone_did()` at line 193 but explicitly drops the error with `let _ = …`. If tombstone write fails (disk I/O, permission, transient), the peer is removed from `nodes.json` but never actually recorded as removed, so the next background sync/notify-join silently re-adds it. +- Files: `core/archipelago/src/federation/storage.rs:180-197`, `core/archipelago/src/api/rpc/federation/handlers.rs:272-300` +- Impact: Federation peers marked for removal can reappear after the next sync cycle, confusing the operator and potentially re-establishing unwanted connections. +- Fix approach: Surface the tombstone-write failure instead of swallowing it; consider retry logic with backoff; add integration test via `tests/multinode/smoke.sh` to verify removal sticks across sync cycles. + +**Container reconciler observability gap:** +- Issue: No metrics distinguish "settling after restart" from "flapping" — container thrashing is invisible until anecdotal reports. No per-app restart counter or log line when an app restarts >N times in M minutes. +- Files: `core/archipelago/src/container/prod_orchestrator.rs` (reconciler loop), `core/archipelago/src/health_monitor.rs` +- Impact: Silent restart storms go unnoticed; users see frequent service interruptions without diagnostics; operator can't distinguish normal convergence from a crash loop. +- Fix approach: Add per-app restart counter + log line when threshold exceeded; emit metric on each restart; wire restart count into health/status RPC output. + +**Failed systemd unit self-healing gap:** +- Issue: When a Quadlet-backed app's `.service` unit enters `failed` state (e.g., exit 255), the reconciler does not automatically `reset-failed` + `start` it. The unit sits failed until the operator manually intervenes or the service restarts. +- Files: `core/archipelago/src/container/prod_orchestrator.rs` (reconcile loop) +- Impact: Apps with transient failures go down and stay down; no automatic recovery; operator must manually reset or restart the orchestrator. +- Fix approach: Add reconcile step: quadlet-backed app whose `.service` is `failed` and not user-stopped → call `systemctl --user reset-failed ` + `start`; add backoff to avoid busy-loop on persistent failures. + +**Bitcoin RPC credentials not retrieved from config/secrets:** +- Issue: `core/container/src/bitcoin_simulator.rs:158` has a TODO marking hardcoded (or missing) RPC credentials in the Bitcoin simulator real-mode path. Credentials should be fetched from the secret store. +- Files: `core/container/src/bitcoin_simulator.rs:155-165` +- Impact: Bitcoin simulator in real mode (Testnet/Mainnet) cannot authenticate to the node; RPC calls fail. +- Fix approach: Inject `SecretsProvider` into `BitcoinSimulator::new()` or pass credentials as constructor args; fetch via `config/secrets` at runtime; handle credential rotation. + +**Container security policies not wired in:** +- Issue: `core/security/src/container_policies.rs` generates AppArmor/SELinux profiles but the `apply_profile()` function has a TODO at line 71: "Configure Podman to use the profile" — the profiles are generated but never applied to running containers. +- Files: `core/security/src/container_policies.rs:63-75` +- Impact: Security profiles exist but provide zero protection; containers run without the intended isolation constraints. +- Fix approach: Pass `--security-opt apparmor=` (or SELinux equivalent) to Podman at container creation; verify profile loads via `apparmor_status`; add CI check that profiles compile cleanly. + +**Dynamic resource adjustment not implemented:** +- Issue: `core/performance/src/resource_manager.rs:86` has a TODO for dynamic resource adjustment based on usage. The allocator is static; no adaptive rebalancing when load patterns shift. +- Files: `core/performance/src/resource_manager.rs:86-88` +- Impact: Resource allocation is rigid; a node with skewed usage (e.g., one app consuming all memory) has no mechanism to rebalance dynamically. +- Fix approach: Monitor per-app resource usage via cgroup stats; implement feedback loop to adjust limits; gate on production deployment (likely Phase 3+). + +## Known Bugs + +**Multinode RPC robustness gap:** +- Symptoms: The `node_rpc()` function in `tests/multinode/lib/multinode.bash` lacks `--max-time` on curl calls — a slow server-side RPC can hang the test suite indefinitely with zero feedback. +- Files: `tests/multinode/lib/multinode.bash` (exact line TBD; see grep for `node_rpc`) +- Trigger: Run multinode federation/mesh test against a slow or overloaded node; curl will block forever. +- Workaround: Manually kill the test process and diagnose the hanging RPC manually; no automatic timeout recovery. +- Fix approach: Add `--max-time 30` to all curl calls in `node_rpc()`; re-run `tests/multinode/smoke.sh` to verify. + +## Security Considerations + +**Secrets environment variable exposure risk:** +- Risk: Bitcoin and other service credentials are materialized as env vars in `ARCHIPELAGO_*` (e.g., `BITCOIN_RPC_PASSWORD`). Env vars are visible via `/proc//environ` and potentially logged. +- Files: `core/archipelago/src/container/prod_orchestrator.rs`, `core/container/src/manifest.rs`, `core/archipelago/src/api/rpc/package/config.rs` +- Current mitigation: Secrets are declared as `generated_secrets` in manifests and materialized 0600/rootless; the orchestrator avoids logging values. +- Recommendations: Audit all env-var passing to containers; consider switching high-sensitivity secrets (bitcoin RPC, LND macaroons) to file-based secrets mounted read-only; add audit logging for secret access. + +**Federation DID validation incomplete:** +- Risk: Federation peer DIDs are added via the RPC without cryptographic verification of ownership. A compromised peer could advertise arbitrary DIDs. +- Files: `core/archipelago/src/api/rpc/federation/handlers.rs` (add-node path), `core/archipelago/src/federation/storage.rs` +- Current mitigation: DIDs are stored locally; transitive federation discovery uses the tombstone list to block removed peers. +- Recommendations: Add DID-ownership proof (e.g., signed proof-of-identity) before accepting a peer's advertised DID; document the trust model; consider user warnings when adding peers. + +**AppArmor profiles overly permissive:** +- Risk: Generated AppArmor profiles use blanket `network,` instead of per-port/protocol rules. Readonly flag is checkbox only, not enforced per actual app needs. +- Files: `core/security/src/container_policies.rs:46-54` +- Current mitigation: None (profiles not applied). +- Recommendations: Refine per-app capabilities based on manifest's declared needs; add integration test verifying readonly mounts are enforced; apply profiles in development before prod. + +## Performance Bottlenecks + +**Container thrashing during reconcile:** +- Problem: Restarting `archipelago.service` SIGKILLs every container, forcing a full rebuild over several minutes. Uninstall + reinstall loops can cascade-trigger restarts. +- Files: `core/archipelago/src/container/prod_orchestrator.rs` (the reconciler's desired-state machine) +- Cause: Pre-Phase-3 architecture: containers run in systemd cgroup, not as independent Quadlet units. +- Improvement path: Phase-3 Quadlet default-flip (`config.rs:256`) — each app becomes an independent `.container` unit; restart only the affected app, not the entire cgroup. + +**Reconciler churn on boot:** +- Problem: Boot reconciler makes multiple passes reconciling drift; during each pass, containers may be recreated. Post-OTA health checks deliberately skip per-app container assertions because of restart-storm unpredictability. +- Files: `core/archipelago/src/container/prod_orchestrator.rs`, `core/archipelago/src/bootstrap.rs` +- Cause: Multi-pass reconciliation + no incremental diff detection. +- Improvement path: Consolidate reconciler into single pass for boot; cache manifest/config diffs to avoid redundant comparisons; add boot-only fast-path. + +**Bitcoin IBD on .198 stalled (disk I/O):** +- Problem: .198 bitcoin is mid-IBD with only 21% progress; disk is 448GB (below 1TB archival threshold); load is high (~3–5). +- Files: `tests/multinode-testing-plan.md` (documented issue) +- Cause: Undersized/slow disk; concurrent workload. +- Improvement path: User decision required: swap in a different node (already done for gate run, using .5 instead) or add storage + wait for sync. Not a code issue. + +## Fragile Areas + +**Uninstall + reinstall lifecycle:** +- Files: `core/archipelago/src/api/rpc/package/install.rs`, `core/archipelago/src/container/quadlet.rs:disable_remove()`, `neode-ui/src/components/AppCard.vue` +- Why fragile: Pre-2026-07-26, `quadlet::disable_remove()` called systemd + podman with no timeouts, causing hangs. Fixed by commit `71cc9ac4` (added `QUADLET_STOP_TIMEOUT`, SIGKILL escalation, reset-failed). AppCard was hardcoding uninstall bar to "stuck full-red" (fixed `9f17ba68`). Tests for reinstall/cascade are still opt-in. +- Safe modification: Any changes to the uninstall path must be tested via `cascade-uninstall.bats` (7/7 on .228); extend coverage to multi-container stacks (immich, btcpay). Verify on .228 before fleet roll. +- Test coverage: `tests/lifecycle/bats/cascade-uninstall.bats` exists but not in canonical gate; must opt-in with `ARCHY_GATE_CASCADE=1`. + +**Production orchestrator state machine:** +- Files: `core/archipelago/src/container/prod_orchestrator.rs` (6291 lines) +- Why fragile: Largest file in the codebase; owns install/start/stop/restart/remove/upgrade for every app; per-app mutex + RwLock concurrency model; complex dependency resolution, adoption scan, Quadlet rendering, and host-port-wait logic interleaved. +- Safe modification: Understand the per-app mutex protocol before touching state mutation; test all changes via the lifecycle gate on .228; use the adoption scan + manifest merge logic for any new manifest evolution. +- Test coverage: 667 unit tests green (2026-07-01); lifecycle gate covers ~8 core apps; ~30 apps untested in gate. + +**Mesh radio configuration + boot race:** +- Files: `core/archipelago/src/mesh/meshtastic.rs`, `core/archipelago/src/mesh/mod.rs`, tests at `tests/lifecycle/bats/meshtastic.bats` +- Why fragile: Radio boot-race fixed (2026-07-28, `a8c4694c`/`3f76b496`); on-air config apply must finish before device is used. Earlier versions had probe-boot-race + live config propagation issues. Must verify on real hardware. +- Safe modification: Any mesh changes require E2E test on real LoRa radios (dev-box ↔ x250-dev, or fleet broadcast); unit tests alone won't catch RF timing issues. +- Test coverage: 8-stage on-air smoke test in `tests/multinode/meshtastic.sh` (run manually; not in canonical gate). + +**Lightning payment state machine:** +- Files: `core/archipelago/src/api/rpc/lnd/wallet.rs:payinvoice()` +- Why fragile: Slow multi-hop payments (>15s) previously surfaced as "failed" while settling in background; client-side 15s timeout was aborting the wait. Fixed by commit `614a0f5a` (120s wait, pending status, lnd.paymentstatus poll). Must verify on Framework PT with real multi-hop. +- Safe modification: Any lnd state changes must test full payment lifecycle: invoice creation, encoding, send, multi-hop wait, settlement confirmation. Verify on Framework PT before release. +- Test coverage: Local LND payinvoice smoke test; no multinode lightning routing test in gate. + +## Scaling Limits + +**Uninstall progress bar truthfulness:** +- Current capacity: Uninstall now has timeouts (fixed 2026-07-26) but progress-bar still reports fake stages (full-red full-opacity). +- Limit: Long uninstalls (>30s) show no real progress; bar claims "uninstalling" for the full duration. +- Scaling path: Backend must emit real progress events (% complete, stage name); UI must poll + display truthfully; integrate into all 5 gate iterations (not just 1 throw-away app). + +**Federation node list deduplication on disk bloat:** +- Current capacity: `federation/storage.rs:dedup_nodes_by_onion()` reads entire nodes.json into memory each time a node is added/synced. At N federated peers, O(N) memory + O(N²) comparisons per operation. +- Limit: No hard limit measured; scales fine up to hundreds of peers. Beyond 1000+ peers, memory/time may become visible. +- Scaling path: Switch to a disk-backed database (e.g., rocksdb) for federation state if peer count grows; or implement incremental dedup on disk writes (preserve dedup state, only recompute on load). + +**Lifecycle gate iteration count:** +- Current capacity: `ARCHY_ITERATIONS=5` runs 5 full cycles (stop/start/restart/survive per app). Entire run takes ~8–12 hours on .228. +- Limit: Cannot easily scale to 10+ iterations without timeout risks; per-app timeout tuning is manual. +- Scaling path: Add per-app timeout tuning (manifest field); parallelize per-app tests where safe (currently serial to avoid contention). + +## Dependencies at Risk + +**Reticulum transport daemon process group:** +- Risk: Pre-fix (before `be50c886`), process group wasn't cleaned up on drop. Fork-bombs or dangling processes possible under error conditions. +- Impact: Stale reticulum processes accumulating over time; resource leaks on node. +- Migration plan: Code fix already deployed (commit `7a7fec21`); no active risk. Monitor fleet for stale python processes post-deployment. + +**Podman socket mount security model:** +- Risk: Apps mounting `/run/podman/podman.sock` get full container-management access. Not restricted by the security policy (AppArmor profiles not applied). +- Files: `core/archipelago/src/container/prod_orchestrator.rs:135-137` (detection), manifests for apps with podman mounts (e.g., portainer) +- Impact: A compromised app with podman socket access can start/stop/delete any container on the node. +- Recommendation: Restrict podman socket mounts to admin-only apps (portainer, docker-api tools); document risk; consider socket filtering layer (selinux context, etc.) once AppArmor is wired. + +**Bitcoin version multi-version branch not fleet-wide:** +- Risk: Branch `bitcoin-version-bulletproof` (base `095a76cd`) carries multi-version support but hasn't been deployed fleet-wide yet. .228 carries it; others still run single version. +- Impact: Users on single-version nodes can't switch versions; version mismatch across fleet breaks federation. +- Migration plan: Coordinated OTA + catalog publish + `:latest` repoint sequencing per `docs/bitcoin-version-bulletproof-rollout.md`. Awaiting user decision on timing. + +## Missing Critical Features + +**Developer tooling CLI suite:** +- Problem: Third-party developers need `archy app validate/render/local-install/lifecycle-test` tooling before external registry launches. +- Blocks: External marketplace (workstream C); external developer onboarding. +- Status: Not yet built; documented in APP-PACKAGING-MIGRATION-PLAN.md step 5. + +**Manifest-distributed registry flip:** +- Problem: Manifests still travel via OTA disk rsync. The signed catalog currently distributes only image overrides, not full manifests. Workstream B phases 1+2 done; not yet fleet-deployed. +- Blocks: Cannot confidently add/bump apps without re-signing the catalog. +- Status: Code ready; flip awaits authorization + timing call from user. + +**Phase-3 Quadlet default-flip:** +- Problem: Orchestrator still uses legacy cgroup-based container management; Phase-3 `use_quadlet_backends` switch exists but is opt-in only. +- Blocks: Resolves container thrashing; unlocks independent app restarts; unblocks lifecycle perfection (workstream F). +- Status: Code validated on .228/.198 (commit pending); ready to flip when multinode gate passes. + +## Test Coverage Gaps + +**~30 apps with zero app-specific assertions:** +- What's not tested: Apps like grafana, jellyfin, vaultwarden, penpot, nextcloud, photoprism, uptime-kuma, homeassistant, etc. have no app-specific health checks beyond "container running." +- Files: `tests/lifecycle/bats/all-apps-matrix.bats`, `tests/lifecycle/bats/all-apps-lifecycle.bats` (generic baseline coverage) +- Risk: App-specific bugs (API down, data corruption, dependency failure) go unnoticed until user encounters them. +- Priority: Medium — baseline coverage is a real safety net; app-specific assertions are a "nice to harden" backlog item, not a gate blocker. +- Approach: Add per-app health RPC endpoints or HTTP probes; wire into the gate as opt-in per-app test suites. + +**Progress UI assertions incomplete:** +- What's not tested: Install + uninstall must report monotonic, truthful progress. No stage/percentage assertions in the gate. +- Files: `neode-ui/src/components/AppCard.vue`, `core/archipelago/src/api/rpc/package/install.rs` (backend progress events) +- Risk: Silent hangs or fake progress bars are invisible to the gate. +- Priority: High — immich/grafana uninstall was stuck full-red (fixed); progress truthfulness is part of definition of done for workstream F. +- Approach: Backend must emit real progress events; UI must display & test them; integrate into canonical gate (currently opt-in). + +**All-apps matrix in cascade gate:** +- What's not tested: `ARCHY_GATE_CASCADE=1` runs ONE throwaway app's uninstall/reinstall. Must extend to multi-container stacks (immich, btcpay, mempool) and all ~40 installed apps. +- Files: `tests/lifecycle/bats/cascade-uninstall.bats` (single-app variant) +- Risk: Multi-container app uninstall bugs (e.g., orphan postgres container) go undetected. +- Priority: High — part of workstream F definition of done. +- Approach: Parametrize cascade test over all manifest IDs; run 5 cascades total (not 5 per app to save time); gate-pass requires zero ghost containers post-uninstall. + +--- + +*Analysis based on codebase state 2026-07-29. Issues tracked in `docs/UNIFIED-TASK-TRACKER.md` (day-to-day) and `docs/PRODUCTION-MASTER-PLAN.md` (historical narrative).* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md new file mode 100644 index 00000000..2cd4f060 --- /dev/null +++ b/.planning/codebase/CONVENTIONS.md @@ -0,0 +1,159 @@ +# Coding Conventions + +**Analysis Date:** 2026-07-29 + +## Naming Patterns + +**Files:** +- TypeScript/Vue: PascalCase for components (e.g., `ToggleSwitch.vue`, `SendBitcoinModal.vue`), camelCase for composables and stores (e.g., `useFileType.ts`, `controller.ts`) +- Rust: snake_case for modules and files (e.g., `bitcoin_rpc.rs`, `storage_crypto.rs`) +- Test files: co-located with source in `__tests__/` subdirectories with `.test.ts` or `.spec.ts` suffix for Vitest, `.bats` for shell tests +- Constants in TypeScript use UPPER_SNAKE_CASE within modules (e.g., `IMAGE_EXTS`, `CATEGORY_COLORS` in `useFileType.ts`) + +**Functions:** +- TypeScript/Vue: camelCase for all functions (e.g., `getFileCategory`, `formatSize`, `useFileType`) +- Composables: `use` prefix for Vue composables (e.g., `useFileType`, `useToast`, `useMessageToast`) — exported as named exports or default exports +- Store functions (Pinia): defined with snake_case action names, exported from `defineStore` factory +- Rust: snake_case for all functions and methods (e.g., `doesnt_reallocate`, following Rust conventions) + +**Variables:** +- TypeScript: camelCase for local variables and reactive refs (e.g., `modelValue`, `isActive`, `gamepadCount`) +- Refs (Vue 3): prefix not required, but convention is lowercase start (e.g., `const ext = ref('jpg')`) +- Computed properties: camelCase, explicit `.value` suffix in templates when needed +- Parameters: camelCase, typed explicitly in TypeScript (e.g., `password: string`, `isDir: Ref`) + +**Types:** +- TypeScript: PascalCase for type aliases and interfaces (e.g., `RPCOptions`, `FileCategory`, `CatalogVersionInfo`) +- Union types: PascalCase (e.g., `PendingState = 'pending' | 'sent' | 'approved'`) +- Component props: typed with `defineProps<{ ... }>()` syntax in `