From 4302138b4fda1fa793e9f59d3f0782e21935f7b2 Mon Sep 17 00:00:00 2001 From: archipelago Date: Tue, 15 Sep 2026 14:56:01 -0400 Subject: [PATCH] docs: make Framework LND incident a persistent investigation blocker --- AGENTS.md | 23 +++++ docs/TODO.md | 8 ++ docs/incident-framework-lnd-startup.md | 133 +++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 AGENTS.md create mode 100644 docs/incident-framework-lnd-startup.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..6b75350c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,23 @@ +# Blocking incident: Framework LND startup and false zero balance + +Before starting or resuming work in this repository, read +`docs/incident-framework-lnd-startup.md` and check its status. + +The user explicitly required this incident to take priority over later work. +While its status is OPEN: + +- Surface this blocker at session start, including when the user asks for unrelated work. +- Prioritize investigation and repair on the actual Framework node. If access is + unavailable, ask for it and continue useful offline investigation; defer unrelated + work unless the user explicitly overrides this priority. +- A manual LND restart, a source patch, passing local tests, or publishing an OTA + does not resolve the incident. Do not mark it fixed until the Framework's startup, + Receive flow, and balance behavior are verified on the node, including a controlled + reboot with access and recovery arrangements in place. +- Preserve wallet identity, wallet/channel databases, credentials, and backups. + Never run wallet wipe/recreation as an automatic investigation or recovery step. +- Record evidence, changes, validation, and remaining work in the incident document. + +This priority comes from the user's explicit instruction on 2026-09-15. It remains +in effect across sessions until the documented acceptance criteria are met or the +user explicitly changes it. diff --git a/docs/TODO.md b/docs/TODO.md index 1f937c50..1db93d91 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -3,6 +3,14 @@ Working backlog of forward-looking items not yet scoped into a dedicated plan doc. See [`ROADMAP.md`](ROADMAP.md) for the curated, public-facing direction. +## Blocking incident — before unrelated work + +- **OPEN: Framework LND startup / missing Receive address / false zero balance.** + User requires investigation and a verified fix on the actual node before later + unrelated work. Access is pending; a manual LND restart is only a workaround. + See [incident evidence and closure criteria](incident-framework-lnd-startup.md) + and the repository `AGENTS.md` session-start instructions. + ## Dev & build process (priority) - Formalize the contributor workflow: releases, CI, maintainers, automated diff --git a/docs/incident-framework-lnd-startup.md b/docs/incident-framework-lnd-startup.md new file mode 100644 index 00000000..d72695f3 --- /dev/null +++ b/docs/incident-framework-lnd-startup.md @@ -0,0 +1,133 @@ +# Framework: LND startup, missing Receive address, false zero balance + +**Status: OPEN — live Framework access required; not fixed.** + +Reported: 2026-09-15. Source inspected: main at `3b9b74da` (v1.8.17-alpha publication). +The Framework's installed version and exact incident time have not been verified. + +## Mandatory priority across sessions + +The user explicitly requested that this be investigated and fixed on the node +before resuming unrelated work in later sessions. `AGENTS.md` in the repository +and `/home/archipelago/.codex/AGENTS.md` carry this session-start priority. +Only live verification below, or an explicit user change of priority, clears it. + +## Reported observations + +- Framework stopped showing its Lightning address in Receive. +- After a restart, LND did not initialize and the UI displayed a balance of zero. +- Manually restarting LND restored operation. +- Node access will be supplied later. No Framework connection, restart, wallet + operation, or deployment was performed during this offline investigation. +- Still clarify whether the restart was a full reboot or management-service + restart, and which Receive item vanished: a Lightning invoice, an on-chain + address, or the Cashu tab's `@minibits.cash` address. + +A successful manual restart is a workaround, not a root cause or durable fix. +The zero display does not establish that any funds were lost. Its relation to +v1.8.17-alpha is unknown; do not infer a release regression from timing alone. + +## Confirmed source findings + +### 1. LND errors can be presented as successful zero balances + +`core/archipelago/src/api/rpc/lnd/info.rs`, `handle_lnd_getinfo`: + +- `/v1/getinfo` is decoded without checking HTTP success. Its response fields are + optional, so an error object such as `{"code":14,"message":"wallet not ready"}` + can deserialize with every expected field absent instead of rejecting the call. +- Channel and blockchain balance requests suppress connection/JSON failures and + substitute responses with absent balances. HTTP status is not checked here either. +- Missing or unparsable balances become `0` through `unwrap_or(0)`. +- `neode-ui/src/views/Home.vue`, `loadWeb5Status`, treats this RPC response as + success, sets the wallet connected flag, overwrites prior balances, and can + persist the false zero in the wallet snapshot. Its existing failure handling + preserves prior balances only when the RPC actually rejects. + +This is a confirmed code defect and a plausible explanation for the reported +display. It is not proof of the Framework's failure sequence. + +Required fix: reject unsuccessful/incomplete LND balance responses or model +availability explicitly end to end. Never translate unavailable data into a +verified zero. Preserve known balances with a clear unavailable/stale indication; +show an unknown state when no valid balance is known. Genuine successful zeros +must still render as zero. Cover outage, partial failure, cold load, and recovery. + +### 2. Startup readiness and wallet unlock need live evidence + +- `main.rs` runs crash/container boot recovery before starting the reconciler. +- `crash_recovery.rs` can start existing containers directly. +- `container/prod_orchestrator.rs` runs LND post-start hooks on explicit restart + and on normal reconciliation of already-running containers. Therefore it is + incorrect to conclude that running containers categorically skip unlock. +- `container/lnd.rs::ensure_wallet_initialized` checks wallet existence and + `/v1/getinfo`, then attempts unlock. Its unlock wait budget is approximately ten + minutes; per-request timeouts can extend elapsed time. Historical comments + describe slow database startup and restart loops, but that is not Framework evidence. +- `health_monitor.rs` models LND's Bitcoin dependency. Container-running state + alone is not proof of wallet readiness, Bitcoin connectivity, or invoice readiness. + +Investigate boot ordering, Bitcoin readiness, listener/port mapping, wallet unlock, +mount availability, stopped markers, restart counters, and actual reconcile logs. + +### 3. Destructive automatic recovery exists; exclude it from diagnosis + +`container/lnd.rs::ensure_wallet_initialized` calls +`recreate_wallet_destructively` when all candidate passwords are rejected. That +function can delete the LND chain and graph data directories. Its comment assumes +alpha wallets hold no real funds; that assumption must not guide this investigation. + +No evidence establishes that it ran on Framework. Preserve the original wallet +and channels; rejected passwords must lead to a recoverable error, not automatic +wallet deletion. Review and disable this destructive fallback before using a +modified initialization path as a repair. The existing +`unlock_existing_wallet_no_wipe` demonstrates the non-destructive error behavior. + +### 4. The missing address must be identified precisely + +`ReceiveBitcoinModal.vue` generates Lightning invoices using `lnd.createinvoice` +after a readiness check, and Bitcoin addresses using `lnd.newaddress`. Its Cashu +Lightning address uses `wallet.ecash-lnaddress` and the Minibits service separately. +Do not assume the Minibits address disappears because LND is down. Trace the actual +tab and response once the user clarifies and the node can be inspected. + +## Next session: live investigation order + +1. Request Framework access and verify node identity without publishing its hostname, + address, credentials, or wallet identifiers. Do not substitute the development box. +2. Record installed backend/image versions, boot and incident timestamps, and exact + restart/action sequence. Capture current and previous-boot management/LND logs + before another restart can obscure evidence. Keep raw logs private and redact + secrets, invoices, wallet identifiers, and personally identifying data in summaries. +3. Read container/service state, restart counters, mounts, stopped markers, listener + mappings, Bitcoin readiness, LND wallet state, and authenticated API results. + Never dump container environments, macaroons, passwords, seeds, or wallet databases. +4. Compare HTTP status and data from LND getinfo/balance endpoints with the RPC and + visible Receive/balance state. Distinguish unavailable data, locked wallet, + syncing wallet, and genuine zero. Preserve last-known balance evidence privately. +5. Establish whether the manual restart ran a missing/failed hook, waited out a + dependency, refreshed networking/credentials, or masked another failure. +6. Implement the evidenced startup repair and unavailable-balance handling with + regressions. Preserve wallet/channel state and arrange recovery access before + deploying or deliberately rebooting the node. + +## Acceptance criteria — all required to close + +- [ ] Root cause of Framework startup failure supported by node evidence. +- [ ] Fix implemented and focused regression tests pass. +- [ ] Failed, locked, delayed, and partial LND responses never masquerade as a + fresh zero balance; genuine zero remains correct. +- [ ] Existing wallet identity and channel state preserved through the repair. +- [ ] Framework starts LND and reaches usable wallet readiness after a controlled + full reboot, without manually restarting LND. +- [ ] The originally affected Receive flow works after boot and after recovery; + outages show an actionable state and recover without requiring a page reload. +- [ ] Balance display verified against authenticated LND data on the node. +- [ ] Logs show no restart loop, repeated unlock failure, or wallet-recreation path. +- [ ] Evidence, tested versions, deployment, and limitations recorded here; user + informed of live results. Only then set status RESOLVED and clear the blockers. + +## Work completed so far + +2026-09-15: source investigation and persistent session-start instructions only. +No code fix, release, node deployment, or live reproduction for this incident yet.