v1.8.9's move to Router.SendPaymentV2 shipped without fee_limit_sat,
and the v2 route treats an ABSENT fee limit as zero allowed fees.
Every real route carries a routing fee (the 2-hop route here: 1.5
sats), so the pathfinder rejected them all and the wallet answered
"No route to the recipient" on EVERY send — all day, on healthy
channels with plenty of liquidity both ways.
The router debug log makes it unambiguous:
wallet payment (v1.8.9 backend): fee_limit=0 mSAT -> no route
same payment by hand (lncli --fee_limit=100): fee_limit=100000 mSAT -> settles in 0.65s
My earlier "pipeline verified" claim was wrong — the manual lncli
verification set a fee limit by hand and masked this exact bug. The
400k that succeeded this morning went through the pre-update backend
on the pre-update LND.
Payments now carry lncli's own default budget — the payment amount
(100%), preferring the payer-supplied amount for zero-value invoices
and the invoice's own amount otherwise, with a nominal floor so the
limit can never be zero. Unit-pinned so it cannot regress.
LND normally reconnects channel peers after a restart, but not reliably:
after long or repeated downtime (an app update, a node reboot,
reconciler churn) the peer link can stay down for hours while BOTH
endpoints keep the channel flagged disabled in the routing graph. The
node looks perfectly healthy, the wallet shows balance, and every
payment in either direction fails "no route to the recipient" —
observed live on framework-pt (2026-09-01): its only channel sat
disabled on both policy sides for ~17 hours after the LND 0.21.2
update, while shorty had 583k spendable and the user was told, by a
mis-mapped modal, that they had 'no payment channel'.
The channel graph is desired state — every open channel should have a
live peer connection. A daemon-side watchdog now enforces it:
- every 2 minutes, list channels + peers over LND REST
- for each channel whose remote peer is not connected, look the peer's
advertised addresses up in the public graph and dial one
- per-peer retries throttled to 10 minutes so an unreachable peer is
not hammered; 'already connected' counts as done; a peer with no
advertised address is logged once per pass (cannot be dialed)
- no-ops quietly on nodes without LND (missing macaroon) and while a
wallet is locked (503 body has no channels)
Unit tests pin the selection against the live REST shapes
(remote_pubkey in /v1/channels vs pub_key in /v1/peers).
v1.8.10 CHANGELOG + What's New entries staged so the next release run
is clean first time.
"LND thinks I do not have a channel" while the wallet showed plenty of
liquidity (framework-pt, 2026-09-01): the send gate sums outbound over
FULLY-OPEN channels only, which is correct — a just-opened channel
sits in LND's pending list until it has ~3 confirmations, and an
open channel can have all its balance on the far side — but the modal
then claimed the node had NO channel at all, in every one of those
states, and pointed the user at opening another one.
The gate already fetched the full channel list; it now records WHY
liquidity is zero and the modal says the truth per state:
- pending channels -> "your new channel is waiting for on-chain
confirmations, it unlocks automatically, nothing is needed from you"
(and no "Open a channel" button — that would send the user to fix
a problem they don't have, possibly opening a second channel)
- open channels, zero on the needed side -> "balance is on the far
side — you can receive but there's nothing to send right now"
- payment refused with a routing/liquidity error -> says so, instead
of claiming no channels
- only a genuinely channel-less node keeps the open-one guidance
Eleven unit tests pin the state machine, including the regression
case (pending-only -> 'pending', not 'none') and fail-open on RPC
errors.
The release gate requires the freshly-built bundle to embed the new
version, and the version reaches the bundle through the What's New
modal in AccountInfoSection — there was no v1.8.9-alpha block yet, so
create-release.sh correctly refused to ship a bundle that looked stale.
This adds the block (the user-facing summary of today's LND/HTTPS/
launcher/NPM/Portainer fixes) and carries the version bumps the
aborted run had already written (Cargo.toml, Cargo.lock, package.json,
package-lock), so the re-run starts from a clean tree.
Verified: npm run build now produces assets containing 1.8.9-alpha
(Settings chunk), i.e. the exact check the script runs passes.
Regenerated from the fixed apps/nginx-proxy-manager/manifest.yml (the
only semantic change vs the previous signed catalog) and signed with
the release-root key. Catalog-covered nodes pick this up on their next
hourly fetch and the NPM start/die loop ends: s6 gets its /etc/letsencrypt
mount back and the internal nginx can bind 80/443/81 again under
--cap-drop=ALL.
Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.
Three launcher/bridge defects combined to make HTTPS dashboards look
broken while HTTP ones worked:
1. portAuth() looked the launch port up under the name the user clicks
('mempool-web', 'lnd', 'bitcoin-knots'…), but the signed catalog
declares those ports under the manifest id that owns them
(archy-mempool-web, lnd-ui, bitcoin-ui). The lookup missed,
portIsGateFronted answered false, and an HTTPS dashboard handed app
frames http:// URLs — blocked as mixed content: mempool and IndeeHub
'did not connect', bitcoin knots/core opened http:// in a new tab.
Resolution now follows launch aliases, then a port-wide catalog scan
that only answers when every declarer of that port agrees (a port
any app publishes as plain HTTP is never upgraded to https).
2. The signed-catalog cache was only warmed by the Store/Discover
views, so a user who went straight to My Apps launched apps with an
empty cache. Warmed at dashboard mount now — fetchAppCatalog()
already memoizes with a 1h TTL.
3. The NIP-07 bridge compared event.origin for strict equality with the
recorded (http) app URL and replied to the recorded URL as the
postMessage targetOrigin — both break the moment a frame is scheme-
upgraded (cached HSTS did exactly that): every nostr request was
silently dropped and replies to the stale origin threw. The bridge
now matches host+port (scheme deliberately ignored) and always
replies to event.origin — the frame's real origin.
Unit tests cover alias resolution (incl. bitcoin-knots→8334→https),
the conservative port-scan, and scheme-agnostic sender matching.
Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
Portainer >=2.21 no longer lets whoever loads the page first claim the
admin account: on a fresh install it mints a one-time setup token and
prints it ONLY to the server logs. On an appliance that is a dead end —
'check the Portainer server logs' is exactly what a user cannot follow,
and after the 2.45.0 update it made a freshly restarted Portainer look
broken ('disappeared', then demands a token nobody can find).
package.credentials — the same RPC that powers the login-credentials
card on the app page — now extracts the setup_token line from
portainer's recent container logs and hands it over with the existing
copy-button treatment, titled and explained for a first-time user. The
token stops being printed once setup completes, and any container
recreate drops the log line, so the card disappears on its own and no
dead token lingers. Parsing is a pure, unit-tested scan against the
live-captured 2.45.0 log shape (64 hex chars after setup_token=).
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
A Claude worktree under aiui/ was committed as a submodule gitlink
(160000) with no .gitmodules entry, so actions/checkout's recursive
submodule pass exited 128 and every 'Build & push demo images' run since
v1.8.6 failed. Removing the index entry — the worktree is local state
and was never meant to be tracked.
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
Settings gains the SSH-over-mesh card (danger-zone confirmation for the
any-peer scope, sshd preflights, fipssh copy hint). The signed-catalog
merge filters components via the shared serviceNames module; Discover
grids get the standard icon container; install no longer yanks the user
to My Apps; v1.8.8 release notes.
fips/ssh_mesh.rs owns the 90-ssh.nft drop-in lifecycle: off by default,
any-peer scope behind the UI's danger confirmation or an explicit mesh
address list, reconciled on every daemon config install. The scanner now
takes installed apps' icons from their real manifest metadata (Cuprate's
Services tile) and classifies manifest-declared UI apps as launchable
even when the address probe misses (Alby Hub).
Store-listing components are filtered via the shared serviceNames canon;
these four never earn a tile: MorphOS server is old, the Web5 DID wallet
and CryptPad are untested, Lightning Stack is an untracked upstream
bundle (LND covers it).
archipelago