From 589cbb030f0a8dc127ea55ca77a14bbd62b47b97 Mon Sep 17 00:00:00 2001 From: archipelago Date: Tue, 4 Aug 2026 07:06:10 -0400 Subject: [PATCH] =?UTF-8?q?docs(13-04):=20complete=20music=20model=20+=20t?= =?UTF-8?q?ag=20extraction=20plan=20=E2=80=94=20summary,=20state,=20config?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 13-04-SUMMARY.md: all 3 tasks, broken-pipe recovery (verbatim wip checkpoint be8f24b4), MP3 fixture off-by-one deviation, 7/7 tests green - STATE.md: 13-04 complete (7/15 phase-13 plans), D-13 decision + lofty gate recorded in accumulated context, next = waves 3+ - config.json: fold in pre-existing use_worktrees=false from the broken session (wave-continue bookkeeping, intentionally kept) Co-Authored-By: Claude Fable 5 --- .planning/STATE.md | 14 +- .planning/config.json | 3 +- .../13-04-SUMMARY.md | 179 ++++++++++++++++++ 3 files changed, 189 insertions(+), 7 deletions(-) create mode 100644 .planning/phases/13-aiui-functional-conversational-node-control-and-content-surf/13-04-SUMMARY.md diff --git a/.planning/STATE.md b/.planning/STATE.md index 5851c2f2..dedbb5f6 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,15 +5,15 @@ milestone_name: milestone current_phase: 13 current_phase_name: aiui-functional-conversational-node-control-and-content-surf status: executing -stopped_at: "13-09 COMPLETE 2026-08-04 (session broke on a pipe right after Task 4 finished; summary recovered+committed on resume). Task 4 resolved with operator approval: live nginx config MERGED (repo base, :443 block kept removed, /api/peer-content/ + /ext/ demo proxies preserved; rollback at /etc/nginx/archipelago.conf.bak-pre-1302-sync; sites-enabled/archipelago is a SYMLINK — back up the TARGET). Browser steps 5-7 verified via playwright Chromium on the real node: AIUI renders in the Chat iframe, fetch('/rpc/v1') CSP-BLOCKED from inside the frame but 200 from top-level (T-13-54 mitigated), embedded chat answered via the node's Claude key. CSP header re-confirmed live on resume. NEXT: execute 13-04, then waves 3+. Window 20 (NEW): repo/ISO nginx config lacks /api/peer-content/ — fresh installs silently lose peer media streaming; decide add-or-retire. Still open for operator: merge chore/aiui-monorepo-migration to main before next OTA; Claude key on archy-x250-dev3; window 18 (nginx self-heal reverting /etc/nginx deploys) at next OTA. Stale-but-harmless: p13-01/p13-02 worktrees (merged, safe to remove); stale phase-09 HANDOFF.json in .planning/ predates this and is NOT a resume source. Prior 13-09 detail preserved in git history of this file (b3b580aa)." -last_updated: "2026-08-04T08:24:00.000Z" +stopped_at: "13-04 COMPLETE 2026-08-04 (session broke on a pipe mid-Task-3; recovered work was checkpoint-committed VERBATIM as be8f24b4, then completed on top — no history rewrite). All 3 tasks done: Task 1 operator decision bca4660a (13-MUSIC-MODEL.md: hybrid-identity, derived-albums, index-format-json, sources=both, MUSIC_SCHEMA_VERSION=1); Task 2 operator-approved lofty 0.24.0 legitimacy gate 61564440; Task 3 tag extraction be8f24b4+4b577493 (7/7 music:: tests green; deviation: MP3 fixture frame length 210→209 — lofty truncates the division BEFORE adding padding; fixture-only fix verified against lofty's vendored source). NEXT: waves 3+ per ROADMAP/plan dependencies (13-04 was the last wave-2 plan). Window 20 still open: repo/ISO nginx config lacks /api/peer-content/ — fresh installs silently lose peer media streaming; decide add-or-retire. Still open for operator: merge chore/aiui-monorepo-migration to main before next OTA; Claude key on archy-x250-dev3; window 18 (nginx self-heal reverting /etc/nginx deploys) at next OTA. Stale-but-harmless: p13-01/p13-02 worktrees (merged, safe to remove); stale phase-09 HANDOFF.json in .planning/ predates this and is NOT a resume source. Prior 13-09 detail preserved in git history of this file (bf9b4e6a)." +last_updated: "2026-08-04T10:50:00.000Z" last_activity: 2026-08-04 -last_activity_desc: "13-09 COMPLETE — Task 4 nginx sync + browser verification done; summary recovered after broken-pipe and committed. Next: 13-04" +last_activity_desc: "13-04 complete — D-13 music model decided, lofty gated in, tag extraction 7/7 green. Next: waves 3+" progress: total_phases: 13 completed_phases: 2 total_plans: 60 - completed_plans: 40 + completed_plans: 45 percent: 15 --- @@ -29,9 +29,9 @@ See: .planning/PROJECT.md (updated 2026-07-29) ## Current Position Phase: 13 (aiui-functional-conversational-node-control-and-content-surf) — EXECUTING -Plan: 6 of 15 complete (13-01, 13-02, 13-03, 13-05, 13-06, 13-09) — next: 13-04, then waves 3+ +Plan: 7 of 15 complete (13-01, 13-02, 13-03, 13-04, 13-05, 13-06, 13-09) — next: waves 3+ Status: Executing Phase 13 -Last activity: 2026-08-04 — 13-09 complete (nginx sync + CSP browser verification on archi-dev-box) +Last activity: 2026-08-04 — 13-04 complete (D-13 music model, lofty gate, tag extraction 7/7 green) Progress: [█████░░░░░] 54% @@ -82,6 +82,8 @@ Progress: [█████░░░░░] 54% - [Phase 13, D-19 (2026-08-03)]: **AIUI's source migrated into this repo at `aiui/`**, via `git subtree` with its full 230-commit history (`7ba3109b`, from AIUI `development` @ `e30ac1d`). Supersedes D-15's two-repo premise and voids D-18 entirely. Retires the recurring "AIUI commit stranded local-only" failure (windows 4 and 17) — `e30ac1d` came across in the import and is now pushed. D-17 (standalone mode) is unaffected: archy has no root `package.json`, so AIUI's pnpm/turbo workspace does not collide. Consequence: plans **13-06, 13-09, 13-11** still target `/home/archipelago/Projects/AIUI/` absolute paths and must be re-planned before wave 2; 13-09's `scripts/aiui.pin` deliverable is now meaningless for an in-repo directory. - [Phase 13, wave 1]: Recovered from a broken-pipe interruption that left 13-01 and 13-02 executor work uncommitted in orphaned worktrees. Both were checkpointed verbatim before any agent touched them, then re-committed atomically per task by continuation executors (both chose `reset --soft` + recommit). 13-03 was already complete and was fast-forwarded in. - [Phase 13, execution mode]: `parallelization: false` — this 4-core box thrashed (load 35-55, 15G/23G swap) running two worktree cargo builds concurrently alongside a live node (bitcoind/electrumx/lnd). Serial execution is strictly faster here. +- [Phase 13, 13-04 / D-13 one-way half (2026-08-04, operator)]: Music entity model recorded in 13-MUSIC-MODEL.md BEFORE any node indexes a library — hybrid-identity ((source, canonical path) row key + lazily-backfilled content-hash dedupe column), derived-albums (albums/artists computed at read time, never stored rows), index-format-json (data_dir/music/index.json, content_server.rs precedent), sources = both OwnLibrary and Peer. MUSIC_SCHEMA_VERSION starts at 1; an older binary treats a newer-versioned index as absent (log + empty in-memory index, never overwrite) until an explicit reindex. +- [Phase 13, 13-04 Task 2 (2026-08-04, operator)]: lofty 0.24.0 approved through the blocking-human package-legitimacy gate ([ASSUMED] in 13-RESEARCH.md's audit); dep tree reviewed, no networking crates. Media-root confinement in extract_tags is a parameter (media_roots), checked via canonicalize before any file open (T-13-20). Decisions are logged in PROJECT.md (10 locked ADRs in the `` block + milestone decisions table). Recent decisions affecting current work: diff --git a/.planning/config.json b/.planning/config.json index 4477e5ce..b953ae4a 100644 --- a/.planning/config.json +++ b/.planning/config.json @@ -1,6 +1,7 @@ { "workflow": { - "_auto_chain_active": false + "_auto_chain_active": false, + "use_worktrees": false }, "parallelization": false } diff --git a/.planning/phases/13-aiui-functional-conversational-node-control-and-content-surf/13-04-SUMMARY.md b/.planning/phases/13-aiui-functional-conversational-node-control-and-content-surf/13-04-SUMMARY.md new file mode 100644 index 00000000..db9bf198 --- /dev/null +++ b/.planning/phases/13-aiui-functional-conversational-node-control-and-content-surf/13-04-SUMMARY.md @@ -0,0 +1,179 @@ +--- +phase: 13-aiui-functional-conversational-node-control-and-content-surf +plan: 04 +subsystem: music +tags: [lofty, id3v2, vorbis, mp4, ogg, rust, tag-extraction, music-library] + +# Dependency graph +requires: + - phase: 13-01 + provides: "mod-declaration serialization in core/archipelago/src/main.rs (binary-only crate, single declaration site) — no logical coupling" +provides: + - "13-MUSIC-MODEL.md — the recorded one-way D-13 decision: hybrid-identity, derived-albums, index-format-json, sources=both" + - "core/archipelago/src/music/mod.rs — Track/Album/Artist entity types, TrackId/AlbumId/ArtistId, MusicSource, MUSIC_SCHEMA_VERSION=1" + - "core/archipelago/src/music/tags.rs — extract_tags(path, media_roots) with root confinement, filename-stem fallback, distinct NotAudio error" + - "lofty 0.24.0 in core/archipelago/Cargo.toml (entered via human legitimacy gate)" +affects: [13-11, 13-12, music-index, song-grid, peer-audio] + +# Tech tracking +tech-stack: + added: [lofty 0.24.0] + patterns: + - "Media-root confinement as a parameter (media_roots: &[PathBuf]) checked via canonicalize + starts_with BEFORE any file open" + - "Programmatic byte-level audio fixtures built into tempdirs at test time — no binary fixtures in the repo" + - "Untagged-but-readable audio is Ok(has_tags=false) with filename-stem title; not-audio is a distinguishable Err variant" + +key-files: + created: + - .planning/phases/13-aiui-functional-conversational-node-control-and-content-surf/13-MUSIC-MODEL.md + - core/archipelago/src/music/mod.rs + - core/archipelago/src/music/tags.rs + modified: + - core/archipelago/src/main.rs + - core/archipelago/Cargo.toml + - core/Cargo.lock + +key-decisions: + - "D-13 one-way half decided by the operator at the Task 1 checkpoint: hybrid-identity (path row key + lazily-backfilled content-hash dedupe column), derived-albums (no stored album/artist rows), index-format-json (data_dir/music/index.json, content_server.rs precedent), sources = both OwnLibrary and Peer" + - "MUSIC_SCHEMA_VERSION starts at 1; a newer-versioned on-disk index is treated as absent (never reinterpreted or overwritten) by an older binary" + - "lofty 0.24.0 approved by the operator at the Task 2 blocking-human gate (crates.io downloads + Serial-ATA/lofty-rs repo + dep tree reviewed, no networking crates)" + +patterns-established: + - "T-13-20 mitigation shape: filesystem confinement is a function parameter, not a constant — callers cannot bypass it by construction" + - "Audio test fixtures are Rust functions emitting spec-correct container bytes (ID3v2.4 synchsafe frames, FLAC STREAMINFO blocks, ISO-BMFF atoms, OGG pages) — keeps licensing-clean, review-friendly tests" + +requirements-completed: [] # AIUI-03 spans multiple plans (13-11 SongGrid wiring still pending); not complete yet + +coverage: + - id: D1 + description: "One-way D-13 music entity model recorded in 13-MUSIC-MODEL.md before any code (identity scheme, derived albums, index path/format, sources, reindex path, newer-version handling)" + verification: [] + human_judgment: true + rationale: "checkpoint:decision — the operator made and approved the decision at the gate; document content is a human artifact" + - id: D2 + description: "lofty entered the tree only after human registry-legitimacy verification (Task 2 blocking-human gate)" + verification: [] + human_judgment: true + rationale: "Package legitimacy is exactly the judgment the [ASSUMED] audit fallback rule reserves for a human" + - id: D3 + description: "Tag extraction returns a full typed record for MP3/FLAC/M4A/OGG, filename-stem fallback for untagged audio, a distinct error for non-audio, and refuses paths outside media roots before opening" + verification: + - kind: unit + ref: "core/archipelago/src/music/tags.rs#mp3_id3v24_yields_full_record" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#flac_vorbis_yields_full_record" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#m4a_yields_full_record" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#ogg_yields_full_record" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#untagged_file_falls_back_to_filename_stem" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#non_audio_returns_err_distinct_from_untagged" + status: pass + - kind: unit + ref: "core/archipelago/src/music/tags.rs#path_outside_media_roots_is_refused" + status: pass + human_judgment: false + +# Metrics +duration: ~95min (wall clock, including a broken-pipe interruption and two full non-incremental builds on the 4-core box) +completed: 2026-08-04 +status: complete +--- + +# Phase 13 Plan 04: Music Entity Model, lofty Gate, and Tag Extraction Summary + +**One-way D-13 music model decided and recorded (hybrid-identity / derived-albums / JSON index), lofty 0.24.0 admitted through a human legitimacy gate, and lofty-based tag extraction landed for MP3/FLAC/M4A/OGG with media-root confinement — 7/7 tests green.** + +## Performance + +- **Duration:** ~95 min wall clock (dominated by two non-incremental cargo builds and a broken-pipe recovery) +- **Started:** 2026-08-04T09:00Z (approx — first session) +- **Completed:** 2026-08-04T10:45Z (approx) +- **Tasks:** 3 (1 decision checkpoint, 1 human-verify gate, 1 auto/TDD) +- **Files modified:** 6 + +## Accomplishments + +- The irreversible half of D-13 is now a written, operator-made decision (`13-MUSIC-MODEL.md`), not an emergent property of the first implementation: hybrid-identity (path row key + lazily-backfilled content-hash dedupe column), derived albums/artists, `data_dir/music/index.json` JSON index, both `OwnLibrary` and `Peer` sources, `MUSIC_SCHEMA_VERSION = 1` with an explicit newer-version-on-older-binary contract. +- `lofty 0.24.0` entered the tree through the Task 2 `blocking-human` legitimacy gate (its 13-RESEARCH.md audit entry was `[ASSUMED]`), with the dependency tree reviewed for networking crates. +- `music/mod.rs` implements the decided entity model exactly; `music/tags.rs::extract_tags` extracts title/artist/album/albumartist/track/disc/year/duration across all four formats, falls back to the filename stem for untagged audio (`Ok`, `has_tags: false`), returns a distinct `NotAudio` error for non-audio, and refuses paths outside caller-supplied `media_roots` before any file is opened (T-13-20). +- All test fixtures are generated programmatically (byte-level ID3v2.4 / FLAC / ISO-BMFF / OGG builders into tempdirs) — zero binary audio files committed. + +## Task Commits + +1. **Task 1: Decide the music entity model (checkpoint:decision)** — `bca4660a` (docs) — operator selected `hybrid-identity, derived-albums, index-format-json, sources=both` +2. **Task 2: lofty legitimacy gate (checkpoint:human-verify, blocking-human)** — `61564440` (chore) — operator approved (crates.io history + Serial-ATA/lofty-rs repo + `cargo tree -i lofty` reviewed); `cargo add lofty` → 0.24.0 +3. **Task 3: Tag extraction across four formats (auto, tdd)** — `be8f24b4` (wip checkpoint, recovered work committed verbatim) + `4b577493` (feat, fixture fix completing the task) + +_Note: Task 3's split into `be8f24b4` + `4b577493` is a broken-pipe recovery artifact — see Issues Encountered. `be8f24b4` was already pushed and stays in history; the fix was committed on top, no rewrite._ + +## Files Created/Modified + +- `.planning/phases/13-.../13-MUSIC-MODEL.md` — the recorded one-way decision (identity, albums, index, sources, reindex path) +- `core/archipelago/src/music/mod.rs` — `Track`, `Album`, `Artist`, `TrackId`, `AlbumId`, `ArtistId`, `MusicSource`, `MUSIC_SCHEMA_VERSION = 1` +- `core/archipelago/src/music/tags.rs` — `extract_tags`, `RawTags`, `TagExtractionError`, `fallback_from_filename`, 7 tests + programmatic fixture builders +- `core/archipelago/src/main.rs` — `mod music;` in the alphabetical block (between `mod monitoring;` and `mod names;`) +- `core/archipelago/Cargo.toml` / `core/Cargo.lock` — `lofty 0.24.0` + +## Decisions Made + +- All D-13 sub-decisions were made by the operator at the Task 1 gate and are recorded in `13-MUSIC-MODEL.md`; `mod.rs` implements them verbatim (spot-checked field-by-field against the document). +- `AlbumId` is `(album_artist: Option, album: String)` — a derived grouping key, never persisted as a row, per derived-albums. +- `Track::content_hash` is `#[serde(default)] Option` so the lazily-backfilled dedupe column can appear later without a schema bump. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] MP3 test fixture frame length off by one (210 vs 209 bytes)** +- **Found during:** Task 3 (`mp3_id3v24_yields_full_record` failing: `NotAudio(FileDecoding(Mpeg: "File contains an invalid frame"))`) +- **Issue:** The programmatic MP3 fixture computed the MPEG-1 Layer III frame length for header `FF FB 52 C4` as ⌈1152·64·125/44100⌉+1 = 210, but lofty's `Header::read` truncates the integer division *before* adding the padding byte: `1152*64*125/44100 = 208`, `+1` = **209**. lofty confirms a frame sync by comparing the candidate header against the bytes exactly `len` later (`find_next_frame`/`cmp_header`), so the off-by-one made the second sync check land one byte short, exhausting the search and rejecting the file. +- **Fix:** Fixture-only — `MP3_FRAME_LEN` 210 → 209 with a corrected doc comment citing lofty's exact formula. Production code untouched: it is a straight `lofty::read_from_path` call and is correct for real-world MP3s, whose encoders emit correctly-sized frames. +- **Files modified:** `core/archipelago/src/music/tags.rs` +- **Verification:** `CARGO_INCREMENTAL=0 cargo test --package archipelago music::` → `ok. 7 passed; 0 failed` +- **Committed in:** `4b577493` + +--- + +**Total deviations:** 1 auto-fixed (Rule 1 bug, test-fixture-only) +**Impact on plan:** None on scope — the fixture math was verified against lofty 0.24.0's vendored source (`mpeg/header.rs`, `mpeg/read.rs`) rather than guessed. + +## Issues Encountered + +- **Broken-pipe interruption mid-Task-3.** The original executor session died after writing `mod.rs`/`tags.rs` and running the first test pass (6/7). The recovered working tree was checkpoint-committed **verbatim** as `wip(13-04)` `be8f24b4` before any continuation work, per the track-progress process rule; this continuation session then fixed the one failing fixture on top (`4b577493`). `be8f24b4` was already pushed, so history was not rewritten. +- **Background cargo runs on this box are unreliable to observe:** the 600s foreground timeout moved the first test run to background where its piped `tail` output was lost; re-running in foreground after the dependency rlibs were cached completed quickly. Serial cargo only (live node on the same 4-core host). + +## Verification (plan-level) + +- `cargo test --package archipelago music::` → **`ok. 7 passed; 0 failed`** (test invocation also compiled the package — build exit 0) +- `grep -c '^mod music;' core/archipelago/src/main.rs` → 1 (line 64) +- `pub struct Track` + `MUSIC_SCHEMA_VERSION` in `mod.rs`; `pub fn extract_tags` + `media_roots` parameter in `tags.rs` +- `git ls-files core/archipelago | grep -ciE '\.(mp3|flac|m4a|ogg)$'` → 0 (no binary fixtures) +- `grep -c '^lofty' core/archipelago/Cargo.toml` → 1 +- Entity fields in `mod.rs` match `13-MUSIC-MODEL.md` (hybrid `TrackId`, `content_hash` column, derived `AlbumId`/`ArtistId`, `MusicSource::{OwnLibrary, Peer{onion}}`, version 1) + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- The music domain root and tag extraction exist; `music/index.rs` (scan/persist per `content_server.rs::load_catalog`'s shape) and SongGrid wiring (13-11) can now build on a decided, written schema. +- T-13-22 (peer tag text reaching model context) remains deliberately deferred to 13-12's `wrap_untrusted` boundary — `RawTags` strings carry no trust and nothing here puts them in a model context. +- Waves 3+ per ROADMAP/plan dependencies are unblocked. + +## Self-Check: PASSED + +- FOUND: `.planning/phases/13-.../13-MUSIC-MODEL.md`, `core/archipelago/src/music/mod.rs`, `core/archipelago/src/music/tags.rs` +- FOUND commits: `bca4660a`, `61564440`, `be8f24b4`, `4b577493` + +--- +*Phase: 13-aiui-functional-conversational-node-control-and-content-surf* +*Completed: 2026-08-04*