archy/docs/repair-release-20260929.md

Repair and release execution — 2026-09-29

Status: IN PROGRESS. Do not publish an OTA or ISO until the release gates pass.

User requires all tasks completed and tested on the development box before the next OTA and raw ISO. Passing unit tests alone does not establish live correctness.

Confirmed evidence

• Dev-to-Shorty 100-sat Cashu file purchases failed twice. Both sellers' and buyers' accepted mints match. Shorty's mint swap returned HTTP 422; both attempted purchases were refunded 100 sats. The old message guessed a mint mismatch without evidence.
• Wallet import repaired truncated V2 keyset IDs, while paid-content redemption bypassed that repair. Central swap repair and protocol-level regression tests now pass.
• Core installation on dev reused existing chain data. At 17:42 UTC it was advancing through block replay with no Core container restarts. At 17:49 UTC it had connected to peers and started transaction-index synchronization.
• LND exited repeatedly with bitcoind start timeout while Core loaded. After Core became available LND stayed running and reported waiting for backend sync.
• Framework source fix 4237fb5e is already an ancestor of main. Existing live reboot/native balance evidence is in the incident document. Final display confirmation remains pending.

Changes under validation

• Cashu V4/V2 ID expansion at every swap; fee-aware underpayment rejection; single-mint/sat-only/cryptographic paid tokens; no false mint-mismatch or unconditional refund claims. Missing content checked before redemption.
• mempool.space default; migrate old tx1138 default with fresh consent, retain local explorer priority and custom preferences.
• Core/Knots optional pruning on the version modal and app detail install path; persist choice across runtime restarts; use identical 50,000 MiB automatic pruning entrypoint behavior on large and small disks.
• Plain Bitcoin block-index startup message; defer LND wallet initialization or unlock until Bitcoin RPC is usable; authenticated dependency status and LND UI waiting states; no partial total displayed as a complete balance.

Validation and release gates

• Final backend regression suite passes (including mock mint HTTP and real curve signatures, v1/full-v2/truncated-v2, fees, errors, duplicate redemption).
• Initial explorer and pruning modal tests pass: 15 tests.
• Both actual manifest entrypoints tested with isolated fake bitcoind across 6 disk/choice combinations each. No existing chain pruned for this test.
• Initial LND UI install/start/sync/recovery and invalid-balance tests pass.
• Frontend production build and relevant existing wallet tests pass (34 focused tests, including 12 Home failure/recovery checks). Final UI suite: 1,117 passed; production build passed. Updated gate rerun pending.
• Fault tests and final source review complete.
• Candidate deployed with rollback to dev and Shorty; hashes verified.
• Live paid-file purchase succeeds; failed purchase/refund behavior verified.
• Live waiting/recovery and UI state verified on dev.
• Framework final confirmation recorded.
• Release version/changelog, catalog/image implications, signing prepared.
• Signed OTA built, tested, published to git and ngit.
• Raw ISO built, boot-tested, signed and published; download command supplied.

Tests must not wipe/recreate wallets, prune the operator's existing full chain, or claim that arbitrary failures can never happen. Record material gaps before release. Signing keys remain with the user; prepare concrete artifacts first.

Further startup findings

Live dev /v1/state returned RPC_ACTIVE while /v1/getinfo timed out during Bitcoin initial sync. Candidate startup now recognizes the already-unlocked state instead of repeating unlock attempts for ten minutes. The health watchdog also now excludes Bitcoin initial sync, warmup, unavailable/stale status and LND height progress from its restart criteria. A later observed podman restart was externally initiated; its precise caller has not yet been established, so the watchdog defect is a source finding rather than a confirmed attribution.

Framework SSH rejected the previously provided login on 2026-09-29. No password was saved and no wallet changes were attempted. The human display-confirmation question remains pending. Do not repeat a Framework reboot to reconfirm old work.

LND UI waiting-state, stale-balance, partial-failure/recovery and prompt-render tests pass (4 Node tests). Waiting states avoid calls to LND endpoints that block until sync, and prevent overlapping refreshes.

Final source validation

The final backend suite passed: 1,548 passed, zero failed, four existing ignored live/hardware tests. Includes saved pruning preference, rejecting an old catalog that cannot honor explicit pruning, and all nine paid-Cashu protocol tests. Unsigned candidate catalog passes strict drift and fleet registry trust checks. The release gate caught a missing What's New entry; generated it from the curated changelog and reran the frontend gate/build. No public release has been changed.

At 18:23 UTC dev Bitcoin exited with status 137 and restarted; current container is not marked OOM-killed and no kernel/oomd record identified the cause. Bitcoin is replaying blocks again (height 482071 at 18:31 UTC). Installed old LND continues to time out while Bitcoin RPC warms up. Candidate is not deployed yet; verify its readiness deferral live before declaring this fixed. Do not attribute the Bitcoin exit to a specific actor without evidence.

Doctor restart cause established and repaired

Full system journal identifies container-doctor at 18:23:21 UTC issuing raw podman restart bitcoin-core for an allegedly missing 8333 listener. The same script restarted LND at 17:57:48 and 18:23:35 UTC. The port was actually listening. Reproduced the original ss | awk | grep -q pipeline returning 0 141 0: grep exits after its match, awk gets SIGPIPE, and pipefail falsely reports no listener. The raw restart also enforces a short stop timeout and races Quadlet cleanup.

The repaired check consumes the entire socket snapshot, distinguishes inspection failure from a missing port, and leaves containers running when inspection fails. Necessary restarts use their managed systemd units and shutdown timeouts; unmanaged Bitcoin/LND fallback receives 600/330-second grace respectively. Regression uses 20,000 socket rows plus mocked service/container commands and passes. Thirty read-only checks of the actual Bitcoin listener pass. Script deployed to dev and Shorty with root-only rollback copies. OTA runtime payload includes scripts/. This evidence supersedes the earlier unknown-caller/unknown-exit attribution.