Repair and release execution — 2026-09-29

Status: IN PROGRESS. Do not publish an OTA or ISO until the release gates pass.

User requires all tasks completed and tested on the development box before the next OTA and raw ISO. Passing unit tests alone does not establish live correctness.

Confirmed evidence

Dev-to-Shorty 100-sat Cashu file purchases failed twice. Both sellers' and buyers' accepted mints match. Shorty's mint swap returned HTTP 422; both attempted purchases were refunded 100 sats. The old message guessed a mint mismatch without evidence.
Wallet import repaired truncated V2 keyset IDs, while paid-content redemption bypassed that repair. Central swap repair and protocol-level regression tests now pass.
Core installation on dev reused existing chain data. At 17:42 UTC it was advancing through block replay with no Core container restarts. At 17:49 UTC it had connected to peers and started transaction-index synchronization.
LND exited repeatedly with bitcoind start timeout while Core loaded. After Core became available LND stayed running and reported waiting for backend sync.
Framework source fix 4237fb5e is already an ancestor of main. Existing live reboot/native balance evidence is in the incident document. Final display confirmation remains pending.

Changes under validation

Cashu V4/V2 ID expansion at every swap; fee-aware underpayment rejection; single-mint/sat-only/cryptographic paid tokens; no false mint-mismatch or unconditional refund claims. Missing content checked before redemption.
mempool.space default; migrate old tx1138 default with fresh consent, retain local explorer priority and custom preferences.
Core/Knots optional pruning on the version modal and app detail install path; persist choice across runtime restarts; use identical 50,000 MiB automatic pruning entrypoint behavior on large and small disks.
Plain Bitcoin block-index startup message; defer LND wallet initialization or unlock until Bitcoin RPC is usable; authenticated dependency status and LND UI waiting states; no partial total displayed as a complete balance.

Validation and release gates

Final backend regression suite passes (including mock mint HTTP and real curve signatures, v1/full-v2/truncated-v2, fees, errors, duplicate redemption).
Initial explorer and pruning modal tests pass: 15 tests.
Both actual manifest entrypoints tested with isolated fake bitcoind across 6 disk/choice combinations each. No existing chain pruned for this test.
Initial LND UI install/start/sync/recovery and invalid-balance tests pass.
Frontend production build and relevant existing wallet tests pass (34 focused tests, including 12 Home failure/recovery checks). Final UI suite: 1,117 passed; production build passed. Updated gate rerun pending.
Fault tests and final source review complete.
Candidate deployed with rollback to dev and Shorty; hashes verified.
Live paid-file purchase succeeds; failed purchase/refund behavior verified.
Live waiting/recovery and UI state verified on dev.
Framework final confirmation recorded.
Release version/changelog, catalog/image implications, signing prepared.
Signed OTA built, tested, published to git and ngit.
Raw ISO built, boot-tested, signed and published; download command supplied.

Tests must not wipe/recreate wallets, prune the operator's existing full chain, or claim that arbitrary failures can never happen. Record material gaps before release. Signing keys remain with the user; prepare concrete artifacts first.

Further startup findings

Live dev /v1/state returned RPC_ACTIVE while /v1/getinfo timed out during Bitcoin initial sync. Candidate startup now recognizes the already-unlocked state instead of repeating unlock attempts for ten minutes. The health watchdog also now excludes Bitcoin initial sync, warmup, unavailable/stale status and LND height progress from its restart criteria. A later observed podman restart was externally initiated; its precise caller has not yet been established, so the watchdog defect is a source finding rather than a confirmed attribution.

Framework SSH rejected the previously provided login on 2026-09-29. No password was saved and no wallet changes were attempted. The human display-confirmation question remains pending. Do not repeat a Framework reboot to reconfirm old work.

LND UI waiting-state, stale-balance, partial-failure/recovery and prompt-render tests pass (4 Node tests). Waiting states avoid calls to LND endpoints that block until sync, and prevent overlapping refreshes.

Final source validation

The final backend suite passed: 1,548 passed, zero failed, four existing ignored live/hardware tests. Includes saved pruning preference, rejecting an old catalog that cannot honor explicit pruning, and all nine paid-Cashu protocol tests. Unsigned candidate catalog passes strict drift and fleet registry trust checks. The release gate caught a missing What's New entry; generated it from the curated changelog and reran the frontend gate/build. No public release has been changed.

At 18:23 UTC dev Bitcoin exited with status 137 and restarted; current container is not marked OOM-killed and no kernel/oomd record identified the cause. Bitcoin is replaying blocks again (height 482071 at 18:31 UTC). Installed old LND continues to time out while Bitcoin RPC warms up. Candidate is not deployed yet; verify its readiness deferral live before declaring this fixed. Do not attribute the Bitcoin exit to a specific actor without evidence.

Doctor restart cause established and repaired

Full system journal identifies container-doctor at 18:23:21 UTC issuing raw podman restart bitcoin-core for an allegedly missing 8333 listener. The same script restarted LND at 17:57:48 and 18:23:35 UTC. The port was actually listening. Reproduced the original ss | awk | grep -q pipeline returning 0 141 0: grep exits after its match, awk gets SIGPIPE, and pipefail falsely reports no listener. The raw restart also enforces a short stop timeout and races Quadlet cleanup.

The repaired check consumes the entire socket snapshot, distinguishes inspection failure from a missing port, and leaves containers running when inspection fails. Necessary restarts use their managed systemd units and shutdown timeouts; unmanaged Bitcoin/LND fallback receives 600/330-second grace respectively. Regression uses 20,000 socket rows plus mocked service/container commands and passes. Thirty read-only checks of the actual Bitcoin listener pass. Script deployed to dev and Shorty with root-only rollback copies. OTA runtime payload includes scripts/. This evidence supersedes the earlier unknown-caller/unknown-exit attribution.

Initial candidate live validation — 18:48 UTC

Source 0f85f588, optimized backend SHA256 84434c495c5f8472cf6bfcb6c65e762502c74718ad88271619373335c0054bb6, deployed to dev and Shorty with matching hashes and rollback copies. Both management services restarted; wallets/channels were not reset. Old embedded runtime assets restored the old doctor on backend startup; updated the live script AND embedded runtime copy on both nodes. Final OTA will contain the new script directly.

Authenticated dev readiness transitioned from waiting_start to waiting_sync. Real Chromium at 1440px and 390px showed Waiting for Bitcoin to sync, an unknown balance, and no blocked native LND calls. Screenshot review also caught invented zero capacity/channel counts during waiting: corrected them and the empty-channel recommendation; five UI regression tests now pass.

Real Minibits Cashu purchase from dev to Shorty succeeded for one sat and returned the expected 44 bytes. A rejected one-sat underpayment was refunded exactly, and two cached downloads charged zero. Temporary seller files/catalog entries removed. The first test runner expected data_base64 while the first-purchase API returns data; cached responses use data_base64. Existing purchase clients only consume data, so a follow-up normalizes both response variants to both fields.

The optional Files copy failed because FileBrowser owns host paths as mapped UID 100000. Follow-up uses its authenticated API with override=false and collision suffixes. A live API probe succeeded, refused overwrite with HTTP409, preserved original bytes, and cleaned up. New protocol tests cover folder creation, escaped names, collisions, authentication failure, disk-full, and unavailable service. Full backend suite for these follow-ups is running; do not package the earlier backend as final.

Follow-up validation and OTA delivery check

Paid-response and Files API regressions passed in the full backend run: 1,552 passed, zero failed, four existing ignored tests. Live browser waiting checks passed again after removing invented zero capacity and channel counts.

OTA inspection found that companion image :local (created by old installers and used on dev) bypassed both source-staleness detection and rebuilding. The earlier assumption that build-context detection covered these nodes was incorrect. Follow-up applies the existing source-mtime/stamp checks to both :local and :latest, preserving the existing tag and rebuilding only stale source. Existing image-ID comparison then restarts the UI companion onto the new image. This does not restart LND itself. Regression covers every companion's two local tags; final backend suite is running. Verify the resulting live rebuilt image before release.

Test isolation finding — release remains blocked

The next full run passed 1,552 tests but one existing boot-loop timing test failed. Its output and node logs exposed an independent test defect: MockRuntime tests still invoked real Quadlet service operations and Podman socket recovery. These caused further LND/companion restarts during unrestricted unit runs. They were not a recurrence of the repaired doctor port check. Stopped unrestricted testing; LND has remained running since 19:02:46 UTC during isolated test execution.

New isolated runner hides live wallets, service buses, container storage and host process IDs, supplies a private network and temporary writable fixture paths, and keeps host filesystems read-only. An independent boundary probe passed. Test-only service helpers use a temporary Quadlet directory and simulated service results; mocked runtimes skip real Podman socket/network provisioning. Host file helpers require the isolated-runner marker and execute inside the namespace instead of escaping through sudo/systemd-run. Release harness and AGENTS now require this runner. Initial isolation trials correctly blocked host operations and exposed fixture permission assumptions; final runner compiles and executes the full suite with those fixture paths isolated. No final pass claimed yet.

Main dashboard candidate and AIUI build at b634f41a are now deployed on dev; served index SHA matches the build. Live package.versions returns bitcoinPrune=false for Core and Knots, preserving current automatic mode. Existing full chain stays unpruned. Final backend (Files/cached response/legacy UI delivery follow-ups) is not yet deployed; earlier 0f85f588 backend remains live on both nodes.

Final isolated backend run: 1,553 passed, zero failed, four existing ignored in 13 seconds after compilation. Boundary probe confirms no host service buses, live wallet data, host process IDs, or external network. Bitcoin/LND start times remained unchanged during isolated execution. Production helpers are unchanged; the namespace-specific command behavior is compiled only into unit tests. Release and ISO gates now use the isolated runner.

12 KiB Raw Blame History