docs: make Framework LND incident a persistent investigation blocker

2026-09-15 14:56:01 -04:00
parent 3b9b74dae5
commit 4302138b4f
3 changed files with 164 additions and 0 deletions
@@ -0,0 +1,23 @@
 # Blocking incident: Framework LND startup and false zero balance
 Before starting or resuming work in this repository, read
 `docs/incident-framework-lnd-startup.md` and check its status.
 The user explicitly required this incident to take priority over later work.
 While its status is OPEN:
 - Surface this blocker at session start, including when the user asks for unrelated work.
 - Prioritize investigation and repair on the actual Framework node. If access is
  unavailable, ask for it and continue useful offline investigation; defer unrelated
  work unless the user explicitly overrides this priority.
 - A manual LND restart, a source patch, passing local tests, or publishing an OTA
  does not resolve the incident. Do not mark it fixed until the Framework's startup,
  Receive flow, and balance behavior are verified on the node, including a controlled
  reboot with access and recovery arrangements in place.
 - Preserve wallet identity, wallet/channel databases, credentials, and backups.
  Never run wallet wipe/recreation as an automatic investigation or recovery step.
 - Record evidence, changes, validation, and remaining work in the incident document.
 This priority comes from the user's explicit instruction on 2026-09-15. It remains
 in effect across sessions until the documented acceptance criteria are met or the
 user explicitly changes it.
@@ -3,6 +3,14 @@
 Working backlog of forward-looking items not yet scoped into a dedicated plan
 doc. See [`ROADMAP.md`](ROADMAP.md) for the curated, public-facing direction.
 ## Blocking incident — before unrelated work
 - **OPEN: Framework LND startup / missing Receive address / false zero balance.**
  User requires investigation and a verified fix on the actual node before later
  unrelated work. Access is pending; a manual LND restart is only a workaround.
  See [incident evidence and closure criteria](incident-framework-lnd-startup.md)
  and the repository `AGENTS.md` session-start instructions.
 ## Dev & build process (priority)
 - Formalize the contributor workflow: releases, CI, maintainers, automated
@@ -0,0 +1,133 @@
 # Framework: LND startup, missing Receive address, false zero balance
 **Status: OPEN — live Framework access required; not fixed.**
 Reported: 2026-09-15. Source inspected: main at `3b9b74da` (v1.8.17-alpha publication).
 The Framework's installed version and exact incident time have not been verified.
 ## Mandatory priority across sessions
 The user explicitly requested that this be investigated and fixed on the node
 before resuming unrelated work in later sessions. `AGENTS.md` in the repository
 and `/home/archipelago/.codex/AGENTS.md` carry this session-start priority.
 Only live verification below, or an explicit user change of priority, clears it.
 ## Reported observations
 - Framework stopped showing its Lightning address in Receive.
 - After a restart, LND did not initialize and the UI displayed a balance of zero.
 - Manually restarting LND restored operation.
 - Node access will be supplied later. No Framework connection, restart, wallet
  operation, or deployment was performed during this offline investigation.
 - Still clarify whether the restart was a full reboot or management-service
  restart, and which Receive item vanished: a Lightning invoice, an on-chain
  address, or the Cashu tab's `@minibits.cash` address.
 A successful manual restart is a workaround, not a root cause or durable fix.
 The zero display does not establish that any funds were lost. Its relation to
 v1.8.17-alpha is unknown; do not infer a release regression from timing alone.
 ## Confirmed source findings
 ### 1. LND errors can be presented as successful zero balances
 `core/archipelago/src/api/rpc/lnd/info.rs`, `handle_lnd_getinfo`:
 - `/v1/getinfo` is decoded without checking HTTP success. Its response fields are
  optional, so an error object such as `{"code":14,"message":"wallet not ready"}`
  can deserialize with every expected field absent instead of rejecting the call.
 - Channel and blockchain balance requests suppress connection/JSON failures and
  substitute responses with absent balances. HTTP status is not checked here either.
 - Missing or unparsable balances become `0` through `unwrap_or(0)`.
 - `neode-ui/src/views/Home.vue`, `loadWeb5Status`, treats this RPC response as
  success, sets the wallet connected flag, overwrites prior balances, and can
  persist the false zero in the wallet snapshot. Its existing failure handling
  preserves prior balances only when the RPC actually rejects.
 This is a confirmed code defect and a plausible explanation for the reported
 display. It is not proof of the Framework's failure sequence.
 Required fix: reject unsuccessful/incomplete LND balance responses or model
 availability explicitly end to end. Never translate unavailable data into a
 verified zero. Preserve known balances with a clear unavailable/stale indication;
 show an unknown state when no valid balance is known. Genuine successful zeros
 must still render as zero. Cover outage, partial failure, cold load, and recovery.
 ### 2. Startup readiness and wallet unlock need live evidence
 - `main.rs` runs crash/container boot recovery before starting the reconciler.
 - `crash_recovery.rs` can start existing containers directly.
 - `container/prod_orchestrator.rs` runs LND post-start hooks on explicit restart
  and on normal reconciliation of already-running containers. Therefore it is
  incorrect to conclude that running containers categorically skip unlock.
 - `container/lnd.rs::ensure_wallet_initialized` checks wallet existence and
  `/v1/getinfo`, then attempts unlock. Its unlock wait budget is approximately ten
  minutes; per-request timeouts can extend elapsed time. Historical comments
  describe slow database startup and restart loops, but that is not Framework evidence.
 - `health_monitor.rs` models LND's Bitcoin dependency. Container-running state
  alone is not proof of wallet readiness, Bitcoin connectivity, or invoice readiness.
 Investigate boot ordering, Bitcoin readiness, listener/port mapping, wallet unlock,
 mount availability, stopped markers, restart counters, and actual reconcile logs.
 ### 3. Destructive automatic recovery exists; exclude it from diagnosis
 `container/lnd.rs::ensure_wallet_initialized` calls
 `recreate_wallet_destructively` when all candidate passwords are rejected. That
 function can delete the LND chain and graph data directories. Its comment assumes
 alpha wallets hold no real funds; that assumption must not guide this investigation.
 No evidence establishes that it ran on Framework. Preserve the original wallet
 and channels; rejected passwords must lead to a recoverable error, not automatic
 wallet deletion. Review and disable this destructive fallback before using a
 modified initialization path as a repair. The existing
 `unlock_existing_wallet_no_wipe` demonstrates the non-destructive error behavior.
 ### 4. The missing address must be identified precisely
 `ReceiveBitcoinModal.vue` generates Lightning invoices using `lnd.createinvoice`
 after a readiness check, and Bitcoin addresses using `lnd.newaddress`. Its Cashu
 Lightning address uses `wallet.ecash-lnaddress` and the Minibits service separately.
 Do not assume the Minibits address disappears because LND is down. Trace the actual
 tab and response once the user clarifies and the node can be inspected.
 ## Next session: live investigation order
 1. Request Framework access and verify node identity without publishing its hostname,
   address, credentials, or wallet identifiers. Do not substitute the development box.
 2. Record installed backend/image versions, boot and incident timestamps, and exact
   restart/action sequence. Capture current and previous-boot management/LND logs
   before another restart can obscure evidence. Keep raw logs private and redact
   secrets, invoices, wallet identifiers, and personally identifying data in summaries.
 3. Read container/service state, restart counters, mounts, stopped markers, listener
   mappings, Bitcoin readiness, LND wallet state, and authenticated API results.
   Never dump container environments, macaroons, passwords, seeds, or wallet databases.
 4. Compare HTTP status and data from LND getinfo/balance endpoints with the RPC and
   visible Receive/balance state. Distinguish unavailable data, locked wallet,
   syncing wallet, and genuine zero. Preserve last-known balance evidence privately.
 5. Establish whether the manual restart ran a missing/failed hook, waited out a
   dependency, refreshed networking/credentials, or masked another failure.
 6. Implement the evidenced startup repair and unavailable-balance handling with
   regressions. Preserve wallet/channel state and arrange recovery access before
   deploying or deliberately rebooting the node.
 ## Acceptance criteria — all required to close
 - [ ] Root cause of Framework startup failure supported by node evidence.
 - [ ] Fix implemented and focused regression tests pass.
 - [ ] Failed, locked, delayed, and partial LND responses never masquerade as a
      fresh zero balance; genuine zero remains correct.
 - [ ] Existing wallet identity and channel state preserved through the repair.
 - [ ] Framework starts LND and reaches usable wallet readiness after a controlled
      full reboot, without manually restarting LND.
 - [ ] The originally affected Receive flow works after boot and after recovery;
      outages show an actionable state and recover without requiring a page reload.
 - [ ] Balance display verified against authenticated LND data on the node.
 - [ ] Logs show no restart loop, repeated unlock failure, or wallet-recreation path.
 - [ ] Evidence, tested versions, deployment, and limitations recorded here; user
      informed of live results. Only then set status RESOLVED and clear the blockers.
 ## Work completed so far
 2026-09-15: source investigation and persistent session-start instructions only.
 No code fix, release, node deployment, or live reproduction for this incident yet.