archy/docs/incident-framework-lnd-startup.md

Framework: LND startup, missing Receive address, false zero balance

Status: OPEN — live Framework access required; not fixed.

Reported: 2026-09-15. Source inspected: main at 3b9b74da (v1.8.17-alpha publication). The Framework's installed version and exact incident time have not been verified.

Mandatory priority across sessions

The user explicitly requested that this be investigated and fixed on the node before resuming unrelated work in later sessions. AGENTS.md in the repository and /home/archipelago/.codex/AGENTS.md carry this session-start priority. Only live verification below, or an explicit user change of priority, clears it.

Reported observations

• Framework stopped showing its Lightning address in Receive.
• After a restart, LND did not initialize and the UI displayed a balance of zero.
• Manually restarting LND restored operation.
• Node access will be supplied later. No Framework connection, restart, wallet operation, or deployment was performed during this offline investigation.
• Still clarify whether the restart was a full reboot or management-service restart, and which Receive item vanished: a Lightning invoice, an on-chain address, or the Cashu tab's @minibits.cash address.

A successful manual restart is a workaround, not a root cause or durable fix. The zero display does not establish that any funds were lost. Its relation to v1.8.17-alpha is unknown; do not infer a release regression from timing alone.

Confirmed source findings

1. LND errors can be presented as successful zero balances

core/archipelago/src/api/rpc/lnd/info.rs, handle_lnd_getinfo:

• /v1/getinfo is decoded without checking HTTP success. Its response fields are optional, so an error object such as {"code":14,"message":"wallet not ready"} can deserialize with every expected field absent instead of rejecting the call.
• Channel and blockchain balance requests suppress connection/JSON failures and substitute responses with absent balances. HTTP status is not checked here either.
• Missing or unparsable balances become 0 through unwrap_or(0).
• neode-ui/src/views/Home.vue, loadWeb5Status, treats this RPC response as success, sets the wallet connected flag, overwrites prior balances, and can persist the false zero in the wallet snapshot. Its existing failure handling preserves prior balances only when the RPC actually rejects.

This is a confirmed code defect and a plausible explanation for the reported display. It is not proof of the Framework's failure sequence.

Required fix: reject unsuccessful/incomplete LND balance responses or model availability explicitly end to end. Never translate unavailable data into a verified zero. Preserve known balances with a clear unavailable/stale indication; show an unknown state when no valid balance is known. Genuine successful zeros must still render as zero. Cover outage, partial failure, cold load, and recovery.

2. Startup readiness and wallet unlock need live evidence

• main.rs runs crash/container boot recovery before starting the reconciler.
• crash_recovery.rs can start existing containers directly.
• container/prod_orchestrator.rs runs LND post-start hooks on explicit restart and on normal reconciliation of already-running containers. Therefore it is incorrect to conclude that running containers categorically skip unlock.
• container/lnd.rs::ensure_wallet_initialized checks wallet existence and /v1/getinfo, then attempts unlock. Its unlock wait budget is approximately ten minutes; per-request timeouts can extend elapsed time. Historical comments describe slow database startup and restart loops, but that is not Framework evidence.
• health_monitor.rs models LND's Bitcoin dependency. Container-running state alone is not proof of wallet readiness, Bitcoin connectivity, or invoice readiness.

Investigate boot ordering, Bitcoin readiness, listener/port mapping, wallet unlock, mount availability, stopped markers, restart counters, and actual reconcile logs.

3. Destructive automatic recovery exists; exclude it from diagnosis

container/lnd.rs::ensure_wallet_initialized calls recreate_wallet_destructively when all candidate passwords are rejected. That function can delete the LND chain and graph data directories. Its comment assumes alpha wallets hold no real funds; that assumption must not guide this investigation.

No evidence establishes that it ran on Framework. Preserve the original wallet and channels; rejected passwords must lead to a recoverable error, not automatic wallet deletion. Review and disable this destructive fallback before using a modified initialization path as a repair. The existing unlock_existing_wallet_no_wipe demonstrates the non-destructive error behavior.

4. The missing address must be identified precisely

ReceiveBitcoinModal.vue generates Lightning invoices using lnd.createinvoice after a readiness check, and Bitcoin addresses using lnd.newaddress. Its Cashu Lightning address uses wallet.ecash-lnaddress and the Minibits service separately. Do not assume the Minibits address disappears because LND is down. Trace the actual tab and response once the user clarifies and the node can be inspected.

Next session: live investigation order

1. Request Framework access and verify node identity without publishing its hostname, address, credentials, or wallet identifiers. Do not substitute the development box.
2. Record installed backend/image versions, boot and incident timestamps, and exact restart/action sequence. Capture current and previous-boot management/LND logs before another restart can obscure evidence. Keep raw logs private and redact secrets, invoices, wallet identifiers, and personally identifying data in summaries.
3. Read container/service state, restart counters, mounts, stopped markers, listener mappings, Bitcoin readiness, LND wallet state, and authenticated API results. Never dump container environments, macaroons, passwords, seeds, or wallet databases.
4. Compare HTTP status and data from LND getinfo/balance endpoints with the RPC and visible Receive/balance state. Distinguish unavailable data, locked wallet, syncing wallet, and genuine zero. Preserve last-known balance evidence privately.
5. Establish whether the manual restart ran a missing/failed hook, waited out a dependency, refreshed networking/credentials, or masked another failure.
6. Implement the evidenced startup repair and unavailable-balance handling with regressions. Preserve wallet/channel state and arrange recovery access before deploying or deliberately rebooting the node.

Acceptance criteria — all required to close

• Root cause of Framework startup failure supported by node evidence.
• Fix implemented and focused regression tests pass.
• Failed, locked, delayed, and partial LND responses never masquerade as a fresh zero balance; genuine zero remains correct.
• Existing wallet identity and channel state preserved through the repair.
• Framework starts LND and reaches usable wallet readiness after a controlled full reboot, without manually restarting LND.
• The originally affected Receive flow works after boot and after recovery; outages show an actionable state and recover without requiring a page reload.
• Balance display verified against authenticated LND data on the node.
• Logs show no restart loop, repeated unlock failure, or wallet-recreation path.
• Evidence, tested versions, deployment, and limitations recorded here; user informed of live results. Only then set status RESOLVED and clear the blockers.

Work completed so far

2026-09-15: source investigation and persistent session-start instructions only. No code fix, release, node deployment, or live reproduction for this incident yet.