archy/docs/incident-framework-lnd-startup.md

# Framework: LND startup, missing Receive address, false zero balance

**Status: OPEN — Framework startup and Cashu address verified live; source integration and final dashboard balance confirmation remain.**

Reported: 2026-09-15. Source inspected: main at `3b9b74da` (v1.8.17-alpha publication).
The Framework's installed version and exact incident time have not been verified.

## Mandatory priority across sessions

The user explicitly requested that this be investigated and fixed on the node
before resuming unrelated work in later sessions. `AGENTS.md` in the repository
and `/home/archipelago/.codex/AGENTS.md` carry this session-start priority.
Only live verification below, or an explicit user change of priority, clears it.

## Reported observations

- Framework stopped showing its Lightning address in Receive.
- After a restart, LND did not initialize and the UI displayed a balance of zero.
- Manually restarting LND restored operation.
- Node access will be supplied later. No Framework connection, restart, wallet
  operation, or deployment was performed during this offline investigation.
- Still clarify whether the restart was a full reboot or management-service
  restart, and which Receive item vanished: a Lightning invoice, an on-chain
  address, or the Cashu tab's `@minibits.cash` address.

A successful manual restart is a workaround, not a root cause or durable fix.
The zero display does not establish that any funds were lost. Its relation to
v1.8.17-alpha is unknown; do not infer a release regression from timing alone.

## Confirmed source findings

### 1. LND errors can be presented as successful zero balances

`core/archipelago/src/api/rpc/lnd/info.rs`, `handle_lnd_getinfo`:

- `/v1/getinfo` is decoded without checking HTTP success. Its response fields are
  optional, so an error object such as `{"code":14,"message":"wallet not ready"}`
  can deserialize with every expected field absent instead of rejecting the call.
- Channel and blockchain balance requests suppress connection/JSON failures and
  substitute responses with absent balances. HTTP status is not checked here either.
- Missing or unparsable balances become `0` through `unwrap_or(0)`.
- `neode-ui/src/views/Home.vue`, `loadWeb5Status`, treats this RPC response as
  success, sets the wallet connected flag, overwrites prior balances, and can
  persist the false zero in the wallet snapshot. Its existing failure handling
  preserves prior balances only when the RPC actually rejects.

This is a confirmed code defect and a plausible explanation for the reported
display. It is not proof of the Framework's failure sequence.

Required fix: reject unsuccessful/incomplete LND balance responses or model
availability explicitly end to end. Never translate unavailable data into a
verified zero. Preserve known balances with a clear unavailable/stale indication;
show an unknown state when no valid balance is known. Genuine successful zeros
must still render as zero. Cover outage, partial failure, cold load, and recovery.

### 2. Startup readiness and wallet unlock need live evidence

- `main.rs` runs crash/container boot recovery before starting the reconciler.
- `crash_recovery.rs` can start existing containers directly.
- `container/prod_orchestrator.rs` runs LND post-start hooks on explicit restart
  and on normal reconciliation of already-running containers. Therefore it is
  incorrect to conclude that running containers categorically skip unlock.
- `container/lnd.rs::ensure_wallet_initialized` checks wallet existence and
  `/v1/getinfo`, then attempts unlock. Its unlock wait budget is approximately ten
  minutes; per-request timeouts can extend elapsed time. Historical comments
  describe slow database startup and restart loops, but that is not Framework evidence.
- `health_monitor.rs` models LND's Bitcoin dependency. Container-running state
  alone is not proof of wallet readiness, Bitcoin connectivity, or invoice readiness.

Investigate boot ordering, Bitcoin readiness, listener/port mapping, wallet unlock,
mount availability, stopped markers, restart counters, and actual reconcile logs.

### 3. Destructive automatic recovery exists; exclude it from diagnosis

`container/lnd.rs::ensure_wallet_initialized` calls
`recreate_wallet_destructively` when all candidate passwords are rejected. That
function can delete the LND chain and graph data directories. Its comment assumes
alpha wallets hold no real funds; that assumption must not guide this investigation.

No evidence establishes that it ran on Framework. Preserve the original wallet
and channels; rejected passwords must lead to a recoverable error, not automatic
wallet deletion. Review and disable this destructive fallback before using a
modified initialization path as a repair. The existing
`unlock_existing_wallet_no_wipe` demonstrates the non-destructive error behavior.

### 4. The missing address must be identified precisely

`ReceiveBitcoinModal.vue` generates Lightning invoices using `lnd.createinvoice`
after a readiness check, and Bitcoin addresses using `lnd.newaddress`. Its Cashu
Lightning address uses `wallet.ecash-lnaddress` and the Minibits service separately.
Do not assume the Minibits address disappears because LND is down. Trace the actual
tab and response once the user clarifies and the node can be inspected.

## Next session: live investigation order

1. Request Framework access and verify node identity without publishing its hostname,
   address, credentials, or wallet identifiers. Do not substitute the development box.
2. Record installed backend/image versions, boot and incident timestamps, and exact
   restart/action sequence. Capture current and previous-boot management/LND logs
   before another restart can obscure evidence. Keep raw logs private and redact
   secrets, invoices, wallet identifiers, and personally identifying data in summaries.
3. Read container/service state, restart counters, mounts, stopped markers, listener
   mappings, Bitcoin readiness, LND wallet state, and authenticated API results.
   Never dump container environments, macaroons, passwords, seeds, or wallet databases.
4. Compare HTTP status and data from LND getinfo/balance endpoints with the RPC and
   visible Receive/balance state. Distinguish unavailable data, locked wallet,
   syncing wallet, and genuine zero. Preserve last-known balance evidence privately.
5. Establish whether the manual restart ran a missing/failed hook, waited out a
   dependency, refreshed networking/credentials, or masked another failure.
6. Implement the evidenced startup repair and unavailable-balance handling with
   regressions. Preserve wallet/channel state and arrange recovery access before
   deploying or deliberately rebooting the node.

## Acceptance criteria — all required to close

- [x] Root cause of Framework startup failure supported by node evidence.
- [x] Fix implemented and focused regression tests pass.
- [ ] Failed, locked, delayed, and partial LND responses never masquerade as a
      fresh zero balance; genuine zero remains correct.
- [x] Existing wallet identity and channel state preserved through the repair.
- [x] Framework starts LND and reaches usable wallet readiness after a controlled
      full reboot, without manually restarting LND.
- [ ] The originally affected Receive flow works after boot and after recovery;
      outages show an actionable state and recover without requiring a page reload.
- [ ] Display confirmation pending; authenticated LND balances match pre-reboot values.
- [x] LND logs show no restart loop, repeated unlock failure, or wallet-recreation path.
- [ ] Evidence, tested versions, deployment, and limitations recorded here; user
      informed of live results. Only then set status RESOLVED and clear the blockers.

## Work completed so far

2026-09-15: source investigation and persistent session-start instructions only.
No code fix, release, node deployment, or live reproduction for this incident yet.

## Live evidence captured 2026-09-15

Access was provided during the same session. Read-only inspection confirmed:

- Framework runs `1.8.17-alpha-dev`; the current full boot began at 18:40:09 UTC.
- LND opened its databases in 6.7 seconds and requested its wallet password at
  18:40:20. It then rejected GetInfo/ChannelBalance/WalletBalance as wallet locked.
- The management service's first sequential reconcile pass was occupied by
  unrelated image recovery, including a missing voice image from 18:40:24 and
  later a missing Core Lightning image. Manifests are iterated from a HashMap;
  wallet readiness has no initial priority. Boot recovery itself completed at
  18:40:18; the first full app-reconcile report appeared at 18:44:34.
- The user's manual LND restart was recorded at 18:42:33. The replacement LND
  process started at 18:42:40, requested its password at 18:43:05, and unlocked
  at 18:43:07 through the explicit restart hook. This supports delayed unlock
  behind unrelated recovery, rather than a missing wallet or bad password.
- At inspection, `/v1/state` reports SERVER_ACTIVE; getinfo reports chain and
  graph sync and two active channels. Both authenticated balance endpoints
  report nonzero balances. No wallet-recreation event was found in captured logs.
- The Minibits RPC separately fails with “The ecash wallet has no seed yet”.
  `wallet/cashu_seed.json` and `wallet/minibits.json` are absent. The existing
  ecash wallet is present with proofs and an August modification timestamp.
  Do not overwrite it or generate an unrelated recovery identity. Still identify
  which Receive item the user meant before declaring this part repaired.

Private raw evidence: `/home/archipelago/.local/state/archy-incidents/framework-lnd-20260915/`.
Files have mode 0600 and the directory 0700. Do not commit or publish raw logs.

Candidate changes on `investigate/framework-lnd-startup`:

- Run Bitcoin and LND reconciliation before unrelated image pulls/builds.
- Reject failed/incomplete LND balance responses instead of manufacturing zeros.
- Preserve known Home balances on invalid responses, visibly label unavailable
  balances, and clear the warning after a successful refresh.
- Remove automatic destructive wallet recreation; failed unlock preserves data.
- Add backend outage/zero/ordering regressions and UI failure/recovery coverage.

These changes are not yet deployed or verified through a Framework reboot.

### Candidate validation and staging

Source fix commit: `4237fb5e` on `investigate/framework-lnd-startup`.

- 44 focused backend tests passed (including LND errors, genuine zero, startup ordering).
- 58 additional reconciliation/update tests passed.
- 12 Home UI tests passed, including outage/partial response/cold-load/recovery cases.
- Rust formatting, frontend type checking and production build passed.
- Optimized backend build passed in 8m02s.
- Both candidate artifacts were copied to Framework and SHA-256 matched locally.
- Private on-node baseline and static channel backup are under
  `/var/lib/archipelago/support/framework-lnd-20260915/`, along with the previous
  backend, dashboard, and `rollback.sh`. This directory is root-only.
- Candidate staged at `/tmp/archy-framework-candidate/`; not applied yet.
- A timing confirmation for the maintenance restart/full reboot was requested
  because it interrupts all node services. Do not reboot while that is pending.
- SSH works through the temporary control socket
  `/tmp/archy-framework-connection/control`. No SSH password was saved to disk.
- The supplied SSH password did not authenticate to the dashboard. Do not guess
  additional passwords or alter dashboard authentication. Native LND diagnostics
  are authenticated using its existing local macaroon without printing it.

Status remains OPEN until deployment and live boot/Receive/balance verification.

### Authorized deployment and full reboot — 2026-09-15

The user answered “yes please” to applying the staged fix and rebooting. Timing
approval is no longer pending. Applied the staged backend and dashboard after
rechecking both checksums and rollback copies. There were no pending channel
HTLCs at reboot. No wallet data, secrets, or recovery identities were replaced.

Live results:

- A different boot ID confirms a full reboot occurred.
- Running backend on disk matches candidate SHA-256
  `5a354f76ebe619561eef0d318e4f41f177d04004682504d7434d632733f8e298`.
- Management service started around 19:23:57 UTC; LND asked for its wallet
  password at 19:24:10 and logged automatic unlock at 19:24:18. No manual LND
  restart or interactive unlock was used after this reboot.
- LND reports SERVER_ACTIVE and chain sync. Its identity and channel-point set
  are identical to the private pre-reboot baseline; both channels are active.
- On-chain and Lightning balances exactly equal the pre-reboot values.
- LND container and systemd restart counts are zero after recovery.
- Public HTTP checks on the node returned 200 for the dashboard index and new
  Home bundle; their bytes match the installed candidate, including the new
  unavailable-balance notice.
- Captured post-reboot management and LND journals in the private local evidence
  directory. Detailed before/after identity, channel, and balance records remain
  in the root-only support directory on Framework.

The user was asked to refresh the dashboard and confirm the originally missing
Receive item and displayed balances. Keep OPEN until that reply is assessed;
Minibits seed absence was a separate finding and must not be mistaken for an
LND startup failure. Candidate is a direct node deployment, not a newly signed
fleet release. The source branch must be integrated before a subsequent release
can preserve this fix across the fleet.

### Cashu Receive follow-up

The user confirmed that the remaining error is specifically on the Ecash tab:
“Lightning address unavailable — you can still paste a token below.”

Read-only checks confirm Framework has an encrypted node master seed, existing
Cashu proofs, and neither `wallet/cashu_seed.json` nor `wallet/minibits.json`.
The existing Minibits handler requires an ecash seed, but setup was available
only through the Settings backup screen; Receive hid the actionable cause.

UI fix commit: `a3b64670`.

- Receive checks the non-secret seed status when registration fails.
- Unseeded wallets get the existing password/TOTP/backup-passphrase-verified setup
  component directly in Receive, with import/restore controls excluded from this
  focused setup screen. Setup derives from the saved node seed when present.
- The recovery words stay in the existing local reveal UI, are cleared on Done,
  and are never emitted to Receive. Receive retries registration after Done.
- Seeded wallets with service outages get Retry, without offering a new identity.
- Ten focused Receive/backup tests and the production UI build passed.
- Deployed the dashboard change without restarting services; live HTTP index and
  setup bundle returned 200 and byte-matched the candidate.
- Backed up original Cashu proofs to the root-only support directory as
  `ecash-before-address-setup.json`. No seed or proof mutation was performed by
  the assistant. Prior LND-fixed dashboard is also backed up there.

The user was asked to refresh Receive → Ecash → Set up address, authenticate in
that node UI, and click Done. Dashboard password is required to decrypt the node
seed; the SSH password did not authenticate to the dashboard. Do not request or
print recovery words, bypass authentication, or create an unrelated random seed.
After completion, verify saved seed/profile presence, registration success,
address display, and unchanged original proofs before closing the incident.

### Cashu setup completed and verified — 2026-09-15

The user initially reported a forgotten passphrase, then said “did it now”. No
independent-seed fallback was implemented or used. The user completed the existing
password-verified setup themselves; the assistant did not receive recovery words.

Read-only node verification confirmed:

- `wallet/cashu_seed.json` exists, is nonempty, and records source `node-seed`.
- `wallet/minibits.json` exists with a `@minibits.cash` address and no pending claims.
- The original ecash wallet file is byte-for-byte unchanged from the protected
  pre-setup copy; every original proof is preserved.
- The registered address's public LNURL-pay metadata returns HTTP 200, tag
  `payRequest`, an HTTPS callback, and a valid amount range. No invoice was paid
  and no funded payment test was performed.

LND automatic startup and native balances were already verified after the full
reboot. Cashu setup and address registration are now also verified on Framework.
Do not ask for the forgotten passphrase again or propose a replacement Cashu seed.

Remaining: integrate the tested source branch before the next fleet release;
record final human confirmation of the rendered dashboard balance (native balances
match exactly, and UI failure/recovery regressions pass). Keep this follow-up
visible across sessions; do not rebuild/reboot/reinitialize a working wallet just
to repeat already completed checks.

### Backup copy and layout — 2026-09-15

At the user's request, shortened the ecash backup explanations and stacked each
card section's text and full-width action vertically. Kept the distinction
between node-derived and separate phrases, and the warning that a newly created
phrase covers future coins rather than existing legacy coins.

All 10 Receive/backup tests and the production UI build pass. Deployed the UI to
Framework without a restart; served index and backup-component bundle match the
build byte-for-byte. The prior UI is saved as `web-ui-before-backup-copy` in the
protected incident directory. Source integration and final rendered dashboard
balance confirmation remain pending as above.

### LNURL comment-length report — 2026-09-15

User reports a maximum-comment-length error in some sending wallets. Live
Framework address metadata advertises integer `commentAllowed: 100`. The QR
contains the address only; Archy's Receive UI does not add a comment. The
Minibits-hosted callback returned invoices for omitted/empty comments, 100 ASCII
characters, 101 ASCII characters, and 100 accented characters. These were unpaid
invoice requests at the advertised minimum amount; no funds were sent.

The callback did not reproduce the error, including beyond its advertised limit.
Sending-wallet validation against the advertised 100-character limit is therefore
a hypothesis, not a confirmed root cause. Asked which wallets fail and whether
an empty comment also fails. Need that result before selecting a code fix.
The service controls the advertised limit; changing local Receive text or QR
cannot raise it for other wallets.

### Primal Spark: automatic recipient note exceeds the address limit

User clarified that no comment was entered and the sender is Primal Spark.
Checked Framework's management journal over the preceding 20 minutes: no
comment-length errors, service active, and zero pending Minibits claims. Recent
claim polling connected to and disconnected from the relay normally. Historical
seed-authentication failures preceded the successful setup already documented.

The live address's Minibits `text/plain` description is **101 ASCII characters**,
while `commentAllowed` is **100**. Description template (address redacted):
`Pay to [ADDRESS] with Lightning. Receiver will receive ecash into Minibits Wallet.`

Primal Android source at `36939db97213e7f8eeefaa4adaf125d839fc662e`:
- `WalletTextParserImpl.handleLnUrlText` assigns the parsed description to
  `DraftTx.noteRecipient`, including for Lightning-address input.
- `TransactionEditor` initializes its editable recipient note from that value.
- `SparkWalletServiceImpl` passes it untrimmed to `PrepareLnurlPayRequest.comment`.
- Breez Spark source at `8bb38ec292a590907360c4e7f2a4134b8f09de9e`,
  `common/src/lnurl/pay.rs::validate_user_input`, rejects a comment exceeding the
  limit with the exact reported error before requesting the callback.

This identifies a concrete compatibility failure: the address description can
become an automatic over-limit comment without the sender typing anything.
The user confirmed that explicitly clearing the prefilled recipient note made
the payment work, and supplied the same description observed in live metadata.
This confirms the automatic-comment compatibility failure. The installed Primal
platform/version was not captured. Node logs alone cannot show sender-side
validation or requests to the external Minibits callback.

Durable upstream correction: Primal should keep receiver metadata separate from
the sender's comment and enforce the limit on actual user comments. Minibits can
also shorten its description or raise its advertised comment limit. Archy does
not serve this external LNURL metadata; do not rename an existing wallet address,
rotate its seed, or claim that a local dashboard edit fixes this sender behavior.

### Primal workaround confirmed by user

The user confirmed successful payment after removing the automatic description.
The permanent sender-side correction is to leave the recipient comment empty by
default and retain receiver metadata only as display text. In Primal Android,
remove the assignment of the LNURL description to the draft recipient note in
`WalletTextParserImpl.handleLnUrlText`; also validate explicitly entered comments
against the endpoint's limit. No upstream change has been submitted or deployed.
Existing Framework addresses and wallet identities remain unchanged.

### Can Archy shorten the current address description?

Inspected Minibits' public wallet client (`src/services/minibitsService.ts`,
`updateWalletProfile`) and `WalletProfileRecord`. The supported profile update
fields are name, lud16, and avatar; there is no exposed LNURL description or
comment-limit setting. Its public web repository also contains no implementation
of the LNURL metadata endpoint or description template.

For the existing `@minibits.cash` address, no supported client-side mechanism
to shorten this text was found. Do not send guessed profile-update fields or
rename the address to disguise the problem. A Minibits server change could use
`Pay to [ADDRESS]`, well below the current limit. Controlling this metadata in
Archy would instead require an Archy-hosted LNURL service/address and correct
invoice metadata binding; rewriting the QR label or only proxying edited metadata
is insufficient. No wallet/profile mutations were made during this investigation.