docs(02-10): complete timing-regression verdict plan
This commit is contained in:
archipelago
@@ -0,0 +1,185 @@
|
||||
---
|
||||
phase: 02-ui-performance
|
||||
plan: 10
|
||||
subsystem: ui
|
||||
tags: [playwright, performance-profiling, keepalive, dispersion-analysis, gap-closure]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 02-ui-performance/02-09
|
||||
provides: "Confirmation that Server's/Web5's KeepAlive 'remount' was a probe-measurement artifact — 02-10's re-measure runs against the build that includes this finding (no source change, so trivially satisfied)"
|
||||
- phase: 02-ui-performance/02-01
|
||||
provides: "The frozen perf harness and 02-PERF-BASELINE.json"
|
||||
- phase: 02-ui-performance/02-08
|
||||
provides: "02-PERF-AFTER.json, the second data point in this plan's three-way comparison"
|
||||
provides:
|
||||
- "02-PERF-REMEASURE.json — third on-device measurement (5 runs/surface) against archi-dev-box"
|
||||
- "02-FINDINGS.md § Re-measurement (gap closure) — three-way dispersion table and a verdict per named surface"
|
||||
- "REQUIREMENTS.md PERF-02/PERF-03 traceability rows updated to point at the resolved verdicts"
|
||||
affects: []
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Three-way dispersion comparison (min/median/max across all samples, not bare medians) to distinguish a real, monotonically-worsening regression from run-to-run noise — a regression that shrinks toward baseline as conditions improve is noise; one that grows despite improving conditions is real"
|
||||
- "Git-log bisection against the pre-phase-2 baseline commit (a75b6709) to confirm a regression traces to a phase-2/gap-closure commit before naming it 'phase-2-caused' rather than pre-existing"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- .planning/phases/02-ui-performance/02-PERF-REMEASURE.json
|
||||
modified:
|
||||
- .planning/phases/02-ui-performance/02-FINDINGS.md
|
||||
- .planning/REQUIREMENTS.md
|
||||
|
||||
key-decisions:
|
||||
- "Deployed archi-dev-box to 3e3159fa (frontend-only) before measuring — the previously-deployed 8fe6217b predated the 02-review commits touching 4 of the 6 named surfaces (Web5/Discover/Server/OpenWrtGateway); a stale-bundle re-measure would have been worthless for this plan's purpose"
|
||||
- "No further deploy performed after the coordinator flagged a shared-tree hazard mid-plan (concurrent security-follow-up and BotFights sessions leaving uncommitted edits in neode-ui/src) — every confirmed regression is recorded as an accepted deviation rather than a landed fix, since the plan's own 'fixed' branch requires a deploy-and-re-measure step that was unavailable this round"
|
||||
- "First ARCHY_PERF_RUNS=5 attempt hit the harness's own hardcoded 20-minute test.setTimeout under concurrent node load (an unrelated podman build) and aborted after 9/15 surfaces; re-ran once per the plan's own contingency rather than hand-editing the partial artifact — the second attempt completed cleanly in 7.9 minutes"
|
||||
- "Wallet/send-flow cleared as noise (re-measure median and 3/5 samples at or below the baseline's own minimum); Discover/Server/Web5/AppDetails/OpenWrtGateway confirmed as real, phase-2-caused regressions because their medians climbed monotonically across all three independent runs despite disk pressure genuinely easing between runs — the opposite of what the noise theory predicts"
|
||||
- "Named cause for all five confirmed regressions is the same client-side render/reactivation 'split-signal' class 02-08 already identified for Web5/Fleet: RPC count flat-or-improved and (for Server/Web5) instance survival independently confirmed by 02-09, yet wall-clock revisit time keeps climbing — pointing at Vue-reactivity/re-render cost from the useCachedResource/KeepAlive conversion itself, not a network wait or a remount"
|
||||
|
||||
requirements-completed: []
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "Third measurement (02-PERF-REMEASURE.json) taken against archi-dev-box with the frozen harness (git diff --stat 3ee20430 -- neode-ui/e2e/perf/ shows only 02-09's separate probe spec), runs=5, 15/15 rows, run conditions (clock, df, load, concurrent activity) recorded before and after"
|
||||
requirement: "PERF-02"
|
||||
verification:
|
||||
- kind: e2e
|
||||
ref: "neode-ui/e2e/perf/surface-perf.spec.ts run against http://archi-dev-box with ARCHY_PERF_RUNS=5 — first attempt timed out under concurrent load (harness's own 20-min test.setTimeout, not a harness modification), re-run once, second attempt exit code 0, 15/15 rows"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "Every one of the six surfaces 02-VERIFICATION.md named for gap 2 carries an explicit verdict backed by three-way dispersion data and git-log bisection: 5 confirmed regressions (accepted deviations, named cause + destination), 1 cleared as noise"
|
||||
requirement: "PERF-02"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "02-FINDINGS.md § Re-measurement (gap closure) — three-way min/median/max table + per-surface verdict + Accepted Deviations table"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "The environmental-noise theory is settled with data, not left as a caveat: current run conditions (79% disk, load, concurrent podman/vitest/vite/typeorm activity) recorded quantitatively; prior runs' known conditions restated for comparison"
|
||||
requirement: "PERF-03"
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "02-FINDINGS.md § Re-measurement (gap closure), 'Run header and recorded conditions' table"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "Full vitest suite and type-check confirmed green (required whether or not a fix landed); no fix landed this session (deploy blocked), so this is a sanity check against the tree as it stood"
|
||||
requirement: "PERF-02"
|
||||
verification:
|
||||
- kind: automated
|
||||
ref: "npm test -- --run: 95 test files / 785 tests passed; npm run type-check: clean"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
duration: ~55min (includes one ~20min harness timeout-and-retry cycle and an 8min successful harness run)
|
||||
completed: 2026-07-31
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 02 Plan 10: Timing-Regression Verdict — Three-Way Re-Measure Summary
|
||||
|
||||
**Re-measured all 15 surfaces a third time (5 runs each) against a freshly-deployed archi-dev-box and, using full sample dispersion rather than bare medians, found the environmental-noise theory only half holds: Wallet/send-flow's apparent regression clears as noise, but Discover/Server/Web5/AppDetails/OpenWrtGateway all show a real, monotonically-worsening revisit-ms regression across three independent runs even as disk pressure genuinely eased — confirmed phase-2-caused via git-log bisection, named as the same client-side render/reactivation cost 02-08 already flagged for Web5/Fleet, and recorded as accepted deviations rather than fixed because a mid-plan shared-tree hazard blocked any further deploy this session.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~55 min (Task 1: deploy ~3min, first harness attempt timed out at 20min under concurrent podman-build load, second attempt completed in 7.9min; Task 2: dispersion analysis + findings write-up + requirements update + test-suite sanity check)
|
||||
- **Completed:** 2026-07-31
|
||||
- **Tasks:** 2/2 completed, no checkpoints (autonomous plan)
|
||||
- **Files modified:** 1 created (`02-PERF-REMEASURE.json`), 2 modified (`02-FINDINGS.md`, `.planning/REQUIREMENTS.md` — the latter is a scope deviation from `files_modified`, see below)
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Deployed archi-dev-box (`--frontend-only`) to commit `3e3159fa` before measuring, since the previously-deployed `8fe6217b` predated the `02-review` commits touching Web5/Discover/Server/OpenWrtGateway and 02-09's own investigation — confirmed via `scripts/deploy-history.log` and a clean-tree (`dirty=false`) deploy.
|
||||
- Confirmed the harness is byte-for-byte frozen since the after-run (`git diff --stat 3ee20430 -- neode-ui/e2e/perf/` shows only 02-09's separately-added `keepalive-remount-probe.spec.ts`, zero changes to the three measured files).
|
||||
- Ran the harness at `ARCHY_PERF_RUNS=5`. First attempt hit the harness's own hardcoded 20-minute `test.setTimeout` while an unrelated `podman build` consumed CPU on the shared box, aborting after 9/15 surfaces with the remainder failing "browser has been closed" — recognized as a harness-timeout artifact, not measurement data, and not written into the artifact as such. Re-ran once per the plan's own contingency; the second attempt completed cleanly in 7.9 minutes once the concurrent build finished. `02-PERF-REMEASURE.json` committed: 15/15 rows, `runs: 5`, `baseUrl: http://archi-dev-box`.
|
||||
- Recorded run conditions quantitatively for the first time in this three-run series: 79% disk (1.4T/1.8T, down from the qualitative "85%" both prior runs cited — confirming the ~118G disk reclaim actually shows up in the numbers), load averages 8.70–11.88 across pre/post checks, and named concurrent activity (a BotFights `podman build` pre-run; `npm run build`/`vitest`/`typeorm migration:run` from other sessions post-run).
|
||||
- Built the three-way min/median/max dispersion table (not bare medians) for all six of 02-VERIFICATION.md's named surfaces plus two bonus data points (Fleet, Chat), then wrote a verdict per surface into a new `## Re-measurement (gap closure)` section in `02-FINDINGS.md`:
|
||||
- **Wallet/send-flow: cleared as noise.** Re-measure spread (1729–2551ms) sits at or below the baseline's own spread (2562–2619ms); 3 of 5 samples are below the baseline minimum entirely. The separate, pre-existing revisit-slower-than-first-visit anomaly (unrelated to phase 2) is unchanged and stays in Outstanding.
|
||||
- **Discover, Server, Web5, AppDetails, OpenWrtGateway: confirmed real regressions**, each with a monotonically-climbing median across all three independent runs (e.g., Web5: 566→709→1329ms with zero sample overlap between any two runs) — the opposite of what shrinking disk pressure should produce if this were noise. Each traced via `git log a75b6709..HEAD` to specific phase-2/`02-review` commits confirming phase-2 origin, not pre-existing behavior.
|
||||
- Named cause for all five: the same client-side render/reactivation "split-signal" class 02-08 already identified for Web5/Fleet — RPC count flat or improved, and for Server/Web5 instance survival independently proven by 02-09, yet wall-clock revisit time keeps growing. Most plausibly Vue-reactivity/watcher/re-render cost from the `useCachedResource`/KeepAlive conversion itself.
|
||||
- Recorded as **accepted deviations**, not fixes: the plan's own "fixed" branch requires a deploy-and-re-measure step to prove a fix moved the number, and that step became unavailable mid-plan (see Deviations below).
|
||||
- Fleet (out-of-scope bonus, same mechanism, most severe magnitude: 330→1054→2631ms) and Chat (measured for the first time this phase, no baseline counterpart) recorded as data points, not verdicts.
|
||||
- Updated `.planning/REQUIREMENTS.md`'s PERF-02/PERF-03 traceability rows to point at this section, per the plan's own Task 2 instruction to reconcile the record.
|
||||
- Full `npm test` (95 files / 785 tests) and `npm run type-check` confirmed green.
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically, staging only the exact paths this plan owns (never `git add -A`, given a confirmed shared-tree hazard mid-plan — see Deviations):
|
||||
|
||||
1. **Task 1: Re-run the frozen harness against archi-dev-box under recorded conditions** - `db629f6f` (feat)
|
||||
2. **Task 2: Three-way comparison, a verdict per named surface, and the resulting action** - `1d6b6c22` (docs)
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `.planning/phases/02-ui-performance/02-PERF-REMEASURE.json` - Third on-device measurement, 15 rows, `runs: 5`, `baseUrl: http://archi-dev-box`
|
||||
- `.planning/phases/02-ui-performance/02-FINDINGS.md` - New `## Re-measurement (gap closure)` section: run conditions for all three runs, three-way dispersion table, per-surface verdict, Accepted Deviations table
|
||||
- `.planning/REQUIREMENTS.md` - PERF-02/PERF-03 traceability rows updated to cite the resolved verdicts (scope deviation — not in this plan's `files_modified`; called out here per the plan checker's hygiene note, and directly instructed by this plan's own Task 2 action text)
|
||||
|
||||
## Decisions Made
|
||||
|
||||
See `key-decisions` in frontmatter for the full list. Highlights:
|
||||
- **Deploy-then-lock:** deployed once at the start of Task 1 (clean tree, needed for measurement validity against 4 of 6 named surfaces), then performed **no further deploy** after a mid-plan coordinator directive identified concurrent uncommitted work from other sessions sharing the tree — every confirmed regression is an accepted deviation rather than a fix for exactly this reason.
|
||||
- **Dispersion over medians:** the whole verdict rests on full min/median/max ranges across all samples in all three artifacts, not the single bare medians 02-08 compared. This is what let Wallet/send-flow clear (its re-measure spread undercuts the baseline's own minimum) while catching that the other five aren't noise (their spreads never shrink back toward baseline — they climb monotonically across three independent, differently-loaded runs).
|
||||
- **Git-bisection-confirmed phase-2 origin:** every confirmed regression's implicated view file was checked against `a75b6709` (the pre-phase-2 baseline commit) to name the specific phase-2/`02-review` commit responsible, rather than asserting "must be phase 2" from timing alone.
|
||||
- **Harness-timeout handling:** treated the first attempt's 20-minute Playwright `test.setTimeout` cutoff (hardcoded in the frozen `surface-perf.spec.ts`, sized for 3 runs) as a legitimate execution constraint under `ARCHY_PERF_RUNS=5` plus concurrent node load — not a reason to edit the frozen harness, and not a reason to hand-edit the partial artifact. Re-ran once per the plan's own contingency instead.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
None — no bugs found in this plan's own scope requiring a Rule 1/2/3 fix. (The five confirmed timing regressions are pre-existing phase-2 defects this plan's own scope explicitly limits to "fix only if small and deployable this session" — see below for why that branch wasn't available.)
|
||||
|
||||
### Scope deviations (documented, not auto-fixed)
|
||||
|
||||
**1. Confirmed regressions recorded as accepted deviations instead of fixed, due to a mid-plan deploy block**
|
||||
- **Found during:** Task 2 analysis
|
||||
- **Issue:** Discover/Server/Web5/AppDetails/OpenWrtGateway all confirmed as real, phase-2-caused regressions with a plausible, well-evidenced cause (client-side render/reactivation cost). The plan's own "fix it" branch requires: a small, single-file, test-covered fix, followed by `npm test`/type-check/build, an `ARCHIPELAGO_TARGET=archi-dev-box` deploy, and a targeted harness re-run proving the number moved.
|
||||
- **Why not fixed:** Partway through this plan's execution, the coordinator identified that a concurrent security-follow-up session and a BotFights session had left uncommitted edits in shared `neode-ui/src` files (`resources.ts`, `useCachedResource.ts`, `Cloud.vue`, `AppDetails.vue`, and others) — files this plan never touches. A second deploy this session would have shipped that in-progress, unreviewed work to the node alongside anything this plan fixed, which the coordinator explicitly instructed against. Without the deploy-and-re-measure step, "fixed" cannot be proven, so all five are recorded as accepted deviations with a named cause, why they're out of bounds, and a destination (carried forward in `STATE.md`'s Blockers/Concerns for a future dedicated Vue-render-profiling / phase-2 gap-closure plan).
|
||||
- **Files modified:** None (no source change attempted).
|
||||
- **Committed in:** `1d6b6c22` (the verdict itself, not a fix).
|
||||
|
||||
**2. `.planning/REQUIREMENTS.md` modified outside this plan's `files_modified`**
|
||||
- **Found during:** Task 2, per the plan's own action text ("update the status note for those rows... so the next verification pass reads the resolved state")
|
||||
- **Issue:** The plan's frontmatter `files_modified` lists only `02-PERF-REMEASURE.json` and `02-FINDINGS.md`, but Task 2's own instructions explicitly direct a `REQUIREMENTS.md` coverage-table update.
|
||||
- **Resolution:** Updated only the PERF-02/PERF-03 traceability rows (not the requirement checkboxes, which are unrelated to this plan's evidence-note scope). Called out here per the plan checker's own hygiene note.
|
||||
- **Files modified:** `.planning/REQUIREMENTS.md`.
|
||||
- **Committed in:** `1d6b6c22`.
|
||||
|
||||
### Harness-execution deviation (not a scope deviation — anticipated by the plan's own contingency)
|
||||
|
||||
**3. First `ARCHY_PERF_RUNS=5` attempt aborted at the harness's hardcoded 20-minute test timeout**
|
||||
- **Found during:** Task 1
|
||||
- **Issue:** `surface-perf.spec.ts`'s `test.setTimeout(20 * 60 * 1000)` is sized for "15 surfaces x 3 runs" per its own inline comment; at `runs=5` under concurrent podman-build load, the first attempt reached only 9/15 surfaces before timing out, force-closing the browser and failing every remaining surface with "Target page, context or browser has been closed."
|
||||
- **Fix:** Did not edit the frozen harness file (would violate this plan's own freeze gate) and did not hand-edit the partial artifact. Re-ran the identical command once per the plan's explicit contingency; the concurrent build had finished by then and the second attempt completed cleanly in 7.9 minutes.
|
||||
- **Files modified:** None.
|
||||
- **Verification:** `02-PERF-REMEASURE.json` from the second run has 15/15 rows, `runs: 5`, no `null`/error rows except Mesh (unmeasured for the same device-not-connected reason as both prior runs).
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 2 scope deviations (both documented, neither a code bug), 1 harness-execution retry anticipated by the plan itself.
|
||||
**Impact on plan:** The plan's own success criteria are still met — every one of the six named surfaces has an explicit, data-backed verdict (5 confirmed + named cause + destination, 1 cleared with data) rather than being left in the "regressed or noisy, unresolved" state that blocked the phase's clean pass. No regression was silently dropped; the deploy block only changed which branch ("fixed" vs. "accepted deviation") each confirmed regression landed in.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
- **Shared working tree, mid-plan coordinator directive:** a concurrent security-follow-up session and a BotFights session were both actively committing to this same tree during this plan's execution. Handled per instruction: staged only exact paths this plan owns for every commit (never `git add -A`), unstaged one file (`02-REVIEW.md`) that another session had staged into the shared index before my own commit (using `git restore --staged`, which does not alter their working-tree content), and performed no deploy after the coordinator's directive. Both other sessions' commits (`5bfe6088`, `b8391115`, `16102558`) landed cleanly on top of/alongside this plan's commits with no conflicts, confirmed via `git log`.
|
||||
- **Harness timeout under real concurrent load** — see Deviations #3 above; resolved by a single re-run, no artifact corruption.
|
||||
- **`02-PERF-REMEASURE.json`'s `commit` header field reads `7c063a20`** (the local working-tree HEAD at the moment the spec ran) rather than `3e3159fa` (the commit actually deployed and served) — the two docs-only/unrelated-app commits between them (`d7fbba98`, `058d7609`, `002de661`) touch no `neode-ui/src` file, confirmed via `git diff --stat 3e3159fa..7c063a20 -- neode-ui/src` (empty), so this is a cosmetic artifact-header discrepancy, not a measurement-validity problem, and is explicitly called out in `02-FINDINGS.md`.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None for this plan's own execution — `ARCHY_PASSWORD` was supplied out-of-band by the coordinator per the plan's `user_setup` block (as in 02-01/02-08/02-09), passed only as a runtime environment variable, never written to any file, commit, or this summary.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- Verification gap 2 is closed: every one of the six surfaces 02-VERIFICATION.md named now carries an explicit, data-backed verdict — no surface remains in the "regressed or noisy, unresolved" state.
|
||||
- **Carried forward, not resolved:** Discover/Server/Web5/AppDetails/OpenWrtGateway/Fleet's confirmed client-side render/reactivation regressions need (a) DevTools Performance-panel profiling to pinpoint the exact reactivation cost per surface, and (b) a clean deploy window once the shared tree's concurrent security-follow-up and BotFights work land — tracked in `STATE.md`'s Blockers/Concerns, not silently dropped.
|
||||
- Combined with 02-09 (gap 1, Server/Web5 remount proven to be a probe artifact, no fix needed), both of `02-VERIFICATION.md`'s gaps now have committed, data-backed resolutions ready for a `gsd-verifier` re-run on phase 02, per the phase's own `.continue-here.md` handoff note.
|
||||
- No blockers for the milestone's next phase from this plan specifically; the five accepted-deviation regressions are a known, quantified, non-blocking follow-up item, not a new blocker.
|
||||
|
||||
---
|
||||
*Phase: 02-ui-performance*
|
||||
*Completed: 2026-07-31*
|
||||
Reference in New Issue
Block a user