docs(13): add validation strategy
This commit is contained in:
archipelago
+111
@@ -0,0 +1,111 @@
|
||||
---
|
||||
phase: 13
|
||||
slug: aiui-functional-conversational-node-control-and-content-surf
|
||||
# status lifecycle: draft (seeded by plan-phase) → validated (set by validate-phase §6)
|
||||
# audit-milestone §5.5 distinguishes NOT-VALIDATED (draft) from PARTIAL (validated + nyquist_compliant: false) (#2117)
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
wave_0_complete: false
|
||||
created: 2026-08-03
|
||||
---
|
||||
|
||||
# Phase 13 — Validation Strategy
|
||||
|
||||
> Per-phase validation contract for feedback sampling during execution.
|
||||
> Seeded from `13-RESEARCH.md` § Validation Architecture. Task IDs are filled in by the planner.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
This phase spans **three** test surfaces in **two** repositories.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework (Rust)** | `cargo test` — in-tree unit/integration tests (precedent: `swarm/payment.rs` `#[tokio::test]`, `pine_ha.rs` `#[test]`) |
|
||||
| **Framework (neode-ui)** | Vitest 3.1 — `neode-ui/package.json` `"test": "vitest run"` |
|
||||
| **Framework (AIUI repo)** | ⚠️ UNCONFIRMED — `packages/app/src/__tests__/` and `composables/__tests__/` exist (`contentExtraction.test.ts`, `useAI.test.ts`) but the test command was not verified. **Wave 0 must confirm before any wave depends on it.** |
|
||||
| **Config file** | `core/Cargo.toml` (Rust) · `neode-ui/vitest.config.ts` (frontend) |
|
||||
| **Quick run command** | `cd core && cargo test --package archipelago assistant::` · `cd neode-ui && npx vitest run src/services/__tests__/contextBroker.test.ts` |
|
||||
| **Full suite command** | `cd core && cargo test` · `cd neode-ui && npm run test` |
|
||||
| **Estimated runtime** | Rust full suite ~minutes; Vitest targeted ~seconds |
|
||||
|
||||
**Build gotcha (CLAUDE.md):** if `cargo test` hits `rust-lld: undefined hidden symbol`, that is incremental-cache corruption — rebuild with `CARGO_INCREMENTAL=0`. Not a real failure.
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **After every task commit:** the quick-run command for the touched module (`cargo test assistant::`, or the specific Vitest file)
|
||||
- **After every plan wave:** full `cargo test` + `npm run test` (neode-ui) + AIUI's own test command (once confirmed in Wave 0)
|
||||
- **Before `/gsd-verify-work`:** full suite green **and** the AIUI-06 on-device pass on archi-dev-box (desktop + mobile)
|
||||
- **Max feedback latency:** targeted Vitest < 30s; Rust module tests < 120s
|
||||
|
||||
---
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
Requirement-level map seeded from research. **The planner fills Task ID / Plan / Wave / Threat Ref columns** as it decomposes; every row below must end up owned by at least one task.
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||
| TBD | TBD | TBD | AIUI-01 | — | Typed chat request executes a real read-only tool ("how much space is left" → `system.disk-status`) and returns the real result | integration (Rust) | `cargo test assistant::tests::disk_status_tool_executes` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-01 | D-07/D-11 | A write request ("restart bitcoin") produces a **pending confirmation**, never an executed action, until the human confirms | integration (Rust) + component (Vue) | `cargo test assistant::tests::destructive_tool_requires_confirm`; `npx vitest run src/services/__tests__/toolConfirm.test.ts` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-01 / AIUI-04 | Phase-10 D-01..D-04 | An unauthenticated caller cannot reach any new `assistant.*` RPC method | integration (Rust) | `cargo test rpc::middleware::tests::assistant_methods_require_session` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | **AIUI-04** | **live exposure** | `/aiui/api/claude/` and `/aiui/api/openrouter/` are **no longer reachable without a session** (see Manual-Only + note below) | integration/shell | `curl -s -o /dev/null -w '%{http_code}' http://<node>/aiui/api/claude/` returns 401/403 with no cookie | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-02 | D-16 | A conversational settings change is scoped to a granted permission category and **refused** when not granted | unit (Rust) | `cargo test assistant::tools::tests::settings_tool_respects_category_grant` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-04 | D-10 | Peer-supplied text inside untrusted-content delimiters cannot escalate tool authority; an injected "restart bitcoin" still requires a human confirm naming the real action | unit (Rust) | `cargo test assistant::tests::injected_instruction_does_not_grant_authority` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-03 | — | `content.*` RPC data renders in `FilmGrid`/`SongGrid` through the new adapter (pins the shape mismatch found in research) | unit (Vue/TS) | `npx vitest run src/composables/__tests__/archyContentAdapter.test.ts` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-03 | — | Audio routes to the global bottom-bar player, never the lightbox (regression-pins the rule enforced in 5 call sites) | unit (Vue/TS) | `npx vitest run src/composables/__tests__/useAudioPlayer.test.ts` | ⚠️ partial | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-05 | D-15 | Build enforces `VITE_BASE_PATH=/aiui/`; script exits non-zero if unset | shell/CI | `scripts/build-aiui.sh` (new) | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-05 | D-15 | Post-deploy check **fetches a live asset over HTTP** rather than trusting a directory listing | shell | `curl` a hashed asset resolved via `sw.js`, assert 200 + content | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | AIUI-06 | — | Embedded iframe on archi-dev-box, desktop + mobile | manual | N/A | — | ⬜ pending |
|
||||
|
||||
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
---
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] `core/archipelago/src/assistant/` + its `#[cfg(test)]` module — the tool-calling loop is net-new; **zero** existing coverage
|
||||
- [ ] `neode-ui/src/services/__tests__/toolConfirm.test.ts` — new confirm-flow coverage, extending the `contextBroker.test.ts` pattern
|
||||
- [ ] `neode-ui/src/composables/__tests__/archyContentAdapter.test.ts` — pins the `ContentItem` → `Film`/`Song`/`Podcast` mapping
|
||||
- [ ] `scripts/build-aiui.sh` (or equivalent) — does not exist; D-15's `VITE_BASE_PATH` enforcement + commit-pinning have no automated check today
|
||||
- [ ] **Confirm AIUI's own test command** before any wave assumes Vitest parity — unverified in research
|
||||
- [ ] Keep green: `contextBroker.test.ts`, `chatAiuiEmbed.test.ts`
|
||||
|
||||
---
|
||||
|
||||
## Manual-Only Verifications
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| Embedded AIUI works in the real iframe | AIUI-06 | Real-device rendering in the actual embed context; `dev:mock` does not reproduce it | Load neode-ui Chat view on archi-dev-box, desktop **and** mobile viewport; exercise a read tool, a confirmed write, and a content grid |
|
||||
| Frontend bundle actually shipped | AIUI-05 | Node `assets/` is a never-pruned graveyard — a disk grep reports "deployed" before the deploy | Resolve live chunks via `sw.js`, fetch over HTTP, grep the **fetched** bytes for the new string |
|
||||
| Confirm dialog is un-spoofable by the iframe | AIUI-04 / D-11 | Anti-spoofing is a visual/trust property of the host chrome | Verify the dialog renders outside the iframe, Teleports to body, full-screen backdrop, text drawn from the node's description — not model-authored |
|
||||
| Routstr pays a live request | D-04 / D-05 | Research confidence on the Routstr protocol is MEDIUM — cited from docs, never run against a live provider | Spike against a real provider before the integration is trusted; budget ceiling must hard-stop |
|
||||
|
||||
---
|
||||
|
||||
## Open Questions Blocking Full Validation
|
||||
|
||||
Carried from `13-RESEARCH.md` § Open Questions — each needs a planner decision, and two change what "validated" even means:
|
||||
|
||||
1. **The port-3142 proxy** — `/aiui/api/claude/` and `/aiui/api/openrouter/` are proxied with **no session gate** (`image-recipe/configs/nginx-archipelago.conf`, verified). Anyone reaching the node's web port can spend the owner's API budget. Removed, gated, or superseded by D-01's node-side loop?
|
||||
2. **Iframe sandbox mechanism** — AIUI is same-origin today, no `sandbox` attribute, permissive CSP. AIUI-04's "sandboxed by construction" is currently a code-discipline convention, not browser-enforced. Attribute, CSP, or accepted-and-documented risk?
|
||||
3. **Routstr protocol accuracy** — needs a spike against a live provider before it is load-bearing.
|
||||
4. **RBAC integration** — should new `assistant.*` RPCs go through the existing `role.can_access()` check?
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
|
||||
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [ ] Wave 0 covers all MISSING references
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Feedback latency < 120s
|
||||
- [ ] AIUI repo test command confirmed
|
||||
- [ ] `nyquist_compliant: true` set in frontmatter
|
||||
|
||||
**Approval:** pending
|
||||
Reference in New Issue
Block a user