archy/docs/incident-framework-lnd-startup.md

# Framework: LND startup, missing Receive address, false zero balance

**Status: OPEN — Framework startup and Cashu address verified live; source integration and final dashboard balance confirmation remain.**

Reported: 2026-09-15. Source inspected: main at `3b9b74da` (v1.8.17-alpha publication).
The Framework's installed version and exact incident time have not been verified.

## Mandatory priority across sessions

The user explicitly requested that this be investigated and fixed on the node
before resuming unrelated work in later sessions. `AGENTS.md` in the repository
and `/home/archipelago/.codex/AGENTS.md` carry this session-start priority.
Only live verification below, or an explicit user change of priority, clears it.

## Reported observations

- Framework stopped showing its Lightning address in Receive.
- After a restart, LND did not initialize and the UI displayed a balance of zero.
- Manually restarting LND restored operation.
- Node access will be supplied later. No Framework connection, restart, wallet
  operation, or deployment was performed during this offline investigation.
- Still clarify whether the restart was a full reboot or management-service
  restart, and which Receive item vanished: a Lightning invoice, an on-chain
  address, or the Cashu tab's `@minibits.cash` address.

A successful manual restart is a workaround, not a root cause or durable fix.
The zero display does not establish that any funds were lost. Its relation to
v1.8.17-alpha is unknown; do not infer a release regression from timing alone.

## Confirmed source findings

### 1. LND errors can be presented as successful zero balances

`core/archipelago/src/api/rpc/lnd/info.rs`, `handle_lnd_getinfo`:

- `/v1/getinfo` is decoded without checking HTTP success. Its response fields are
  optional, so an error object such as `{"code":14,"message":"wallet not ready"}`
  can deserialize with every expected field absent instead of rejecting the call.
- Channel and blockchain balance requests suppress connection/JSON failures and
  substitute responses with absent balances. HTTP status is not checked here either.
- Missing or unparsable balances become `0` through `unwrap_or(0)`.
- `neode-ui/src/views/Home.vue`, `loadWeb5Status`, treats this RPC response as
  success, sets the wallet connected flag, overwrites prior balances, and can
  persist the false zero in the wallet snapshot. Its existing failure handling
  preserves prior balances only when the RPC actually rejects.

This is a confirmed code defect and a plausible explanation for the reported
display. It is not proof of the Framework's failure sequence.

Required fix: reject unsuccessful/incomplete LND balance responses or model
availability explicitly end to end. Never translate unavailable data into a
verified zero. Preserve known balances with a clear unavailable/stale indication;
show an unknown state when no valid balance is known. Genuine successful zeros
must still render as zero. Cover outage, partial failure, cold load, and recovery.

### 2. Startup readiness and wallet unlock need live evidence

- `main.rs` runs crash/container boot recovery before starting the reconciler.
- `crash_recovery.rs` can start existing containers directly.
- `container/prod_orchestrator.rs` runs LND post-start hooks on explicit restart
  and on normal reconciliation of already-running containers. Therefore it is
  incorrect to conclude that running containers categorically skip unlock.
- `container/lnd.rs::ensure_wallet_initialized` checks wallet existence and
  `/v1/getinfo`, then attempts unlock. Its unlock wait budget is approximately ten
  minutes; per-request timeouts can extend elapsed time. Historical comments
  describe slow database startup and restart loops, but that is not Framework evidence.
- `health_monitor.rs` models LND's Bitcoin dependency. Container-running state
  alone is not proof of wallet readiness, Bitcoin connectivity, or invoice readiness.

Investigate boot ordering, Bitcoin readiness, listener/port mapping, wallet unlock,
mount availability, stopped markers, restart counters, and actual reconcile logs.

### 3. Destructive automatic recovery exists; exclude it from diagnosis

`container/lnd.rs::ensure_wallet_initialized` calls
`recreate_wallet_destructively` when all candidate passwords are rejected. That
function can delete the LND chain and graph data directories. Its comment assumes
alpha wallets hold no real funds; that assumption must not guide this investigation.

No evidence establishes that it ran on Framework. Preserve the original wallet
and channels; rejected passwords must lead to a recoverable error, not automatic
wallet deletion. Review and disable this destructive fallback before using a
modified initialization path as a repair. The existing
`unlock_existing_wallet_no_wipe` demonstrates the non-destructive error behavior.

### 4. The missing address must be identified precisely

`ReceiveBitcoinModal.vue` generates Lightning invoices using `lnd.createinvoice`
after a readiness check, and Bitcoin addresses using `lnd.newaddress`. Its Cashu
Lightning address uses `wallet.ecash-lnaddress` and the Minibits service separately.
Do not assume the Minibits address disappears because LND is down. Trace the actual
tab and response once the user clarifies and the node can be inspected.

## Next session: live investigation order

1. Request Framework access and verify node identity without publishing its hostname,
   address, credentials, or wallet identifiers. Do not substitute the development box.
2. Record installed backend/image versions, boot and incident timestamps, and exact
   restart/action sequence. Capture current and previous-boot management/LND logs
   before another restart can obscure evidence. Keep raw logs private and redact
   secrets, invoices, wallet identifiers, and personally identifying data in summaries.
3. Read container/service state, restart counters, mounts, stopped markers, listener
   mappings, Bitcoin readiness, LND wallet state, and authenticated API results.
   Never dump container environments, macaroons, passwords, seeds, or wallet databases.
4. Compare HTTP status and data from LND getinfo/balance endpoints with the RPC and
   visible Receive/balance state. Distinguish unavailable data, locked wallet,
   syncing wallet, and genuine zero. Preserve last-known balance evidence privately.
5. Establish whether the manual restart ran a missing/failed hook, waited out a
   dependency, refreshed networking/credentials, or masked another failure.
6. Implement the evidenced startup repair and unavailable-balance handling with
   regressions. Preserve wallet/channel state and arrange recovery access before
   deploying or deliberately rebooting the node.

## Acceptance criteria — all required to close

- [x] Root cause of Framework startup failure supported by node evidence.
- [x] Fix implemented and focused regression tests pass.
- [ ] Failed, locked, delayed, and partial LND responses never masquerade as a
      fresh zero balance; genuine zero remains correct.
- [x] Existing wallet identity and channel state preserved through the repair.
- [x] Framework starts LND and reaches usable wallet readiness after a controlled
      full reboot, without manually restarting LND.
- [ ] The originally affected Receive flow works after boot and after recovery;
      outages show an actionable state and recover without requiring a page reload.
- [ ] Display confirmation pending; authenticated LND balances match pre-reboot values.
- [x] LND logs show no restart loop, repeated unlock failure, or wallet-recreation path.
- [ ] Evidence, tested versions, deployment, and limitations recorded here; user
      informed of live results. Only then set status RESOLVED and clear the blockers.

## Work completed so far

2026-09-15: source investigation and persistent session-start instructions only.
No code fix, release, node deployment, or live reproduction for this incident yet.

## Live evidence captured 2026-09-15

Access was provided during the same session. Read-only inspection confirmed:

- Framework runs `1.8.17-alpha-dev`; the current full boot began at 18:40:09 UTC.
- LND opened its databases in 6.7 seconds and requested its wallet password at
  18:40:20. It then rejected GetInfo/ChannelBalance/WalletBalance as wallet locked.
- The management service's first sequential reconcile pass was occupied by
  unrelated image recovery, including a missing voice image from 18:40:24 and
  later a missing Core Lightning image. Manifests are iterated from a HashMap;
  wallet readiness has no initial priority. Boot recovery itself completed at
  18:40:18; the first full app-reconcile report appeared at 18:44:34.
- The user's manual LND restart was recorded at 18:42:33. The replacement LND
  process started at 18:42:40, requested its password at 18:43:05, and unlocked
  at 18:43:07 through the explicit restart hook. This supports delayed unlock
  behind unrelated recovery, rather than a missing wallet or bad password.
- At inspection, `/v1/state` reports SERVER_ACTIVE; getinfo reports chain and
  graph sync and two active channels. Both authenticated balance endpoints
  report nonzero balances. No wallet-recreation event was found in captured logs.
- The Minibits RPC separately fails with “The ecash wallet has no seed yet”.
  `wallet/cashu_seed.json` and `wallet/minibits.json` are absent. The existing
  ecash wallet is present with proofs and an August modification timestamp.
  Do not overwrite it or generate an unrelated recovery identity. Still identify
  which Receive item the user meant before declaring this part repaired.

Private raw evidence: `/home/archipelago/.local/state/archy-incidents/framework-lnd-20260915/`.
Files have mode 0600 and the directory 0700. Do not commit or publish raw logs.

Candidate changes on `investigate/framework-lnd-startup`:

- Run Bitcoin and LND reconciliation before unrelated image pulls/builds.
- Reject failed/incomplete LND balance responses instead of manufacturing zeros.
- Preserve known Home balances on invalid responses, visibly label unavailable
  balances, and clear the warning after a successful refresh.
- Remove automatic destructive wallet recreation; failed unlock preserves data.
- Add backend outage/zero/ordering regressions and UI failure/recovery coverage.

These changes are not yet deployed or verified through a Framework reboot.

### Candidate validation and staging

Source fix commit: `4237fb5e` on `investigate/framework-lnd-startup`.

- 44 focused backend tests passed (including LND errors, genuine zero, startup ordering).
- 58 additional reconciliation/update tests passed.
- 12 Home UI tests passed, including outage/partial response/cold-load/recovery cases.
- Rust formatting, frontend type checking and production build passed.
- Optimized backend build passed in 8m02s.
- Both candidate artifacts were copied to Framework and SHA-256 matched locally.
- Private on-node baseline and static channel backup are under
  `/var/lib/archipelago/support/framework-lnd-20260915/`, along with the previous
  backend, dashboard, and `rollback.sh`. This directory is root-only.
- Candidate staged at `/tmp/archy-framework-candidate/`; not applied yet.
- A timing confirmation for the maintenance restart/full reboot was requested
  because it interrupts all node services. Do not reboot while that is pending.
- SSH works through the temporary control socket
  `/tmp/archy-framework-connection/control`. No SSH password was saved to disk.
- The supplied SSH password did not authenticate to the dashboard. Do not guess
  additional passwords or alter dashboard authentication. Native LND diagnostics
  are authenticated using its existing local macaroon without printing it.

Status remains OPEN until deployment and live boot/Receive/balance verification.

### Authorized deployment and full reboot — 2026-09-15

The user answered “yes please” to applying the staged fix and rebooting. Timing
approval is no longer pending. Applied the staged backend and dashboard after
rechecking both checksums and rollback copies. There were no pending channel
HTLCs at reboot. No wallet data, secrets, or recovery identities were replaced.

Live results:

- A different boot ID confirms a full reboot occurred.
- Running backend on disk matches candidate SHA-256
  `5a354f76ebe619561eef0d318e4f41f177d04004682504d7434d632733f8e298`.
- Management service started around 19:23:57 UTC; LND asked for its wallet
  password at 19:24:10 and logged automatic unlock at 19:24:18. No manual LND
  restart or interactive unlock was used after this reboot.
- LND reports SERVER_ACTIVE and chain sync. Its identity and channel-point set
  are identical to the private pre-reboot baseline; both channels are active.
- On-chain and Lightning balances exactly equal the pre-reboot values.
- LND container and systemd restart counts are zero after recovery.
- Public HTTP checks on the node returned 200 for the dashboard index and new
  Home bundle; their bytes match the installed candidate, including the new
  unavailable-balance notice.
- Captured post-reboot management and LND journals in the private local evidence
  directory. Detailed before/after identity, channel, and balance records remain
  in the root-only support directory on Framework.

The user was asked to refresh the dashboard and confirm the originally missing
Receive item and displayed balances. Keep OPEN until that reply is assessed;
Minibits seed absence was a separate finding and must not be mistaken for an
LND startup failure. Candidate is a direct node deployment, not a newly signed
fleet release. The source branch must be integrated before a subsequent release
can preserve this fix across the fleet.

### Cashu Receive follow-up

The user confirmed that the remaining error is specifically on the Ecash tab:
“Lightning address unavailable — you can still paste a token below.”

Read-only checks confirm Framework has an encrypted node master seed, existing
Cashu proofs, and neither `wallet/cashu_seed.json` nor `wallet/minibits.json`.
The existing Minibits handler requires an ecash seed, but setup was available
only through the Settings backup screen; Receive hid the actionable cause.

UI fix commit: `a3b64670`.

- Receive checks the non-secret seed status when registration fails.
- Unseeded wallets get the existing password/TOTP/backup-passphrase-verified setup
  component directly in Receive, with import/restore controls excluded from this
  focused setup screen. Setup derives from the saved node seed when present.
- The recovery words stay in the existing local reveal UI, are cleared on Done,
  and are never emitted to Receive. Receive retries registration after Done.
- Seeded wallets with service outages get Retry, without offering a new identity.
- Ten focused Receive/backup tests and the production UI build passed.
- Deployed the dashboard change without restarting services; live HTTP index and
  setup bundle returned 200 and byte-matched the candidate.
- Backed up original Cashu proofs to the root-only support directory as
  `ecash-before-address-setup.json`. No seed or proof mutation was performed by
  the assistant. Prior LND-fixed dashboard is also backed up there.

The user was asked to refresh Receive → Ecash → Set up address, authenticate in
that node UI, and click Done. Dashboard password is required to decrypt the node
seed; the SSH password did not authenticate to the dashboard. Do not request or
print recovery words, bypass authentication, or create an unrelated random seed.
After completion, verify saved seed/profile presence, registration success,
address display, and unchanged original proofs before closing the incident.

### Cashu setup completed and verified — 2026-09-15

The user initially reported a forgotten passphrase, then said “did it now”. No
independent-seed fallback was implemented or used. The user completed the existing
password-verified setup themselves; the assistant did not receive recovery words.

Read-only node verification confirmed:

- `wallet/cashu_seed.json` exists, is nonempty, and records source `node-seed`.
- `wallet/minibits.json` exists with a `@minibits.cash` address and no pending claims.
- The original ecash wallet file is byte-for-byte unchanged from the protected
  pre-setup copy; every original proof is preserved.
- The registered address's public LNURL-pay metadata returns HTTP 200, tag
  `payRequest`, an HTTPS callback, and a valid amount range. No invoice was paid
  and no funded payment test was performed.

LND automatic startup and native balances were already verified after the full
reboot. Cashu setup and address registration are now also verified on Framework.
Do not ask for the forgotten passphrase again or propose a replacement Cashu seed.

Remaining: integrate the tested source branch before the next fleet release;
record final human confirmation of the rendered dashboard balance (native balances
match exactly, and UI failure/recovery regressions pass). Keep this follow-up
visible across sessions; do not rebuild/reboot/reinitialize a working wallet just
to repeat already completed checks.