v1.8.9's move to Router.SendPaymentV2 shipped without fee_limit_sat,
and the v2 route treats an ABSENT fee limit as zero allowed fees.
Every real route carries a routing fee (the 2-hop route here: 1.5
sats), so the pathfinder rejected them all and the wallet answered
"No route to the recipient" on EVERY send — all day, on healthy
channels with plenty of liquidity both ways.
The router debug log makes it unambiguous:
wallet payment (v1.8.9 backend): fee_limit=0 mSAT -> no route
same payment by hand (lncli --fee_limit=100): fee_limit=100000 mSAT -> settles in 0.65s
My earlier "pipeline verified" claim was wrong — the manual lncli
verification set a fee limit by hand and masked this exact bug. The
400k that succeeded this morning went through the pre-update backend
on the pre-update LND.
Payments now carry lncli's own default budget — the payment amount
(100%), preferring the payer-supplied amount for zero-value invoices
and the invoice's own amount otherwise, with a nominal floor so the
limit can never be zero. Unit-pinned so it cannot regress.
LND normally reconnects channel peers after a restart, but not reliably:
after long or repeated downtime (an app update, a node reboot,
reconciler churn) the peer link can stay down for hours while BOTH
endpoints keep the channel flagged disabled in the routing graph. The
node looks perfectly healthy, the wallet shows balance, and every
payment in either direction fails "no route to the recipient" —
observed live on framework-pt (2026-09-01): its only channel sat
disabled on both policy sides for ~17 hours after the LND 0.21.2
update, while shorty had 583k spendable and the user was told, by a
mis-mapped modal, that they had 'no payment channel'.
The channel graph is desired state — every open channel should have a
live peer connection. A daemon-side watchdog now enforces it:
- every 2 minutes, list channels + peers over LND REST
- for each channel whose remote peer is not connected, look the peer's
advertised addresses up in the public graph and dial one
- per-peer retries throttled to 10 minutes so an unreachable peer is
not hammered; 'already connected' counts as done; a peer with no
advertised address is logged once per pass (cannot be dialed)
- no-ops quietly on nodes without LND (missing macaroon) and while a
wallet is locked (503 body has no channels)
Unit tests pin the selection against the live REST shapes
(remote_pubkey in /v1/channels vs pub_key in /v1/peers).
v1.8.10 CHANGELOG + What's New entries staged so the next release run
is clean first time.
"LND thinks I do not have a channel" while the wallet showed plenty of
liquidity (framework-pt, 2026-09-01): the send gate sums outbound over
FULLY-OPEN channels only, which is correct — a just-opened channel
sits in LND's pending list until it has ~3 confirmations, and an
open channel can have all its balance on the far side — but the modal
then claimed the node had NO channel at all, in every one of those
states, and pointed the user at opening another one.
The gate already fetched the full channel list; it now records WHY
liquidity is zero and the modal says the truth per state:
- pending channels -> "your new channel is waiting for on-chain
confirmations, it unlocks automatically, nothing is needed from you"
(and no "Open a channel" button — that would send the user to fix
a problem they don't have, possibly opening a second channel)
- open channels, zero on the needed side -> "balance is on the far
side — you can receive but there's nothing to send right now"
- payment refused with a routing/liquidity error -> says so, instead
of claiming no channels
- only a genuinely channel-less node keeps the open-one guidance
Eleven unit tests pin the state machine, including the regression
case (pending-only -> 'pending', not 'none') and fail-open on RPC
errors.
The release gate requires the freshly-built bundle to embed the new
version, and the version reaches the bundle through the What's New
modal in AccountInfoSection — there was no v1.8.9-alpha block yet, so
create-release.sh correctly refused to ship a bundle that looked stale.
This adds the block (the user-facing summary of today's LND/HTTPS/
launcher/NPM/Portainer fixes) and carries the version bumps the
aborted run had already written (Cargo.toml, Cargo.lock, package.json,
package-lock), so the re-run starts from a clean tree.
Verified: npm run build now produces assets containing 1.8.9-alpha
(Settings chunk), i.e. the exact check the script runs passes.
Regenerated from the fixed apps/nginx-proxy-manager/manifest.yml (the
only semantic change vs the previous signed catalog) and signed with
the release-root key. Catalog-covered nodes pick this up on their next
hourly fetch and the NPM start/die loop ends: s6 gets its /etc/letsencrypt
mount back and the internal nginx can bind 80/443/81 again under
--cap-drop=ALL.
Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.
Three launcher/bridge defects combined to make HTTPS dashboards look
broken while HTTP ones worked:
1. portAuth() looked the launch port up under the name the user clicks
('mempool-web', 'lnd', 'bitcoin-knots'…), but the signed catalog
declares those ports under the manifest id that owns them
(archy-mempool-web, lnd-ui, bitcoin-ui). The lookup missed,
portIsGateFronted answered false, and an HTTPS dashboard handed app
frames http:// URLs — blocked as mixed content: mempool and IndeeHub
'did not connect', bitcoin knots/core opened http:// in a new tab.
Resolution now follows launch aliases, then a port-wide catalog scan
that only answers when every declarer of that port agrees (a port
any app publishes as plain HTTP is never upgraded to https).
2. The signed-catalog cache was only warmed by the Store/Discover
views, so a user who went straight to My Apps launched apps with an
empty cache. Warmed at dashboard mount now — fetchAppCatalog()
already memoizes with a 1h TTL.
3. The NIP-07 bridge compared event.origin for strict equality with the
recorded (http) app URL and replied to the recorded URL as the
postMessage targetOrigin — both break the moment a frame is scheme-
upgraded (cached HSTS did exactly that): every nostr request was
silently dropped and replies to the stale origin threw. The bridge
now matches host+port (scheme deliberately ignored) and always
replies to event.origin — the frame's real origin.
Unit tests cover alias resolution (incl. bitcoin-knots→8334→https),
the conservative port-scan, and scheme-agnostic sender matching.
Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
Portainer >=2.21 no longer lets whoever loads the page first claim the
admin account: on a fresh install it mints a one-time setup token and
prints it ONLY to the server logs. On an appliance that is a dead end —
'check the Portainer server logs' is exactly what a user cannot follow,
and after the 2.45.0 update it made a freshly restarted Portainer look
broken ('disappeared', then demands a token nobody can find).
package.credentials — the same RPC that powers the login-credentials
card on the app page — now extracts the setup_token line from
portainer's recent container logs and hands it over with the existing
copy-button treatment, titled and explained for a first-time user. The
token stops being printed once setup completes, and any container
recreate drops the log line, so the card disappears on its own and no
dead token lingers. Parsing is a pure, unit-tested scan against the
live-captured 2.45.0 log shape (64 hex chars after setup_token=).
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
A Claude worktree under aiui/ was committed as a submodule gitlink
(160000) with no .gitmodules entry, so actions/checkout's recursive
submodule pass exited 128 and every 'Build & push demo images' run since
v1.8.6 failed. Removing the index entry — the worktree is local state
and was never meant to be tracked.
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
Settings gains the SSH-over-mesh card (danger-zone confirmation for the
any-peer scope, sshd preflights, fipssh copy hint). The signed-catalog
merge filters components via the shared serviceNames module; Discover
grids get the standard icon container; install no longer yanks the user
to My Apps; v1.8.8 release notes.
fips/ssh_mesh.rs owns the 90-ssh.nft drop-in lifecycle: off by default,
any-peer scope behind the UI's danger confirmation or an explicit mesh
address list, reconciled on every daemon config install. The scanner now
takes installed apps' icons from their real manifest metadata (Cuprate's
Services tile) and classifies manifest-declared UI apps as launchable
even when the address probe misses (Alby Hub).
Store-listing components are filtered via the shared serviceNames canon;
these four never earn a tile: MorphOS server is old, the Web5 DID wallet
and CryptPad are untested, Lightning Stack is an untracked upstream
bundle (LND covers it).
directAppUrl(), the legacy open() path, and resolveRuntimeLaunchUrl()
now upgrade to https only for ports the app gate fronts — decided from
the signed catalog's embedded manifest ports (auth gated/open), so
plain-HTTP publishes (legacy installs, auth:none API ports like
Cuprate's RPC) keep http instead of failing outright. fetchAppCatalog()
merges the daemon-verified signed catalog into the App Store listing
(signed entries appear immediately; community copy supplies featured
and curated metadata), and Marketplace.vue uses the same dynamic fetcher
as Discover so the grid sees signed-new apps too.
Nginx Proxy Manager, Tailscale, Ollama, CryptPad, and AdGuard Home now
carry full manifests: the app gate fronts their web ports (TLS on the
same port, node login where appropriate), installs run through the
orchestrator, and pins live in the signed catalog. Tailscale mirrors its
legacy shape exactly (userspace networking, web console on 8240, plain
HTTP for the gate to front). Ollama stays loopback-only — the
assistant's local model backend, not a web app.
Retires the four already-removed apps for good (FIPS, Nostr VPN,
Routstr, Penpot pins dropped from image-versions.sh, the generator map,
and image_versions.rs), fixes Cuprate's duplicated metadata block that
strict YAML parsers reject, and updates the port-inventory review gates
for the new open (3 own-login consoles) and exempt (2 DNS) ports.
New-tab apps and the companion WebView got hardcoded http:// URLs, so a
node reached over HTTPS opened Vaultwarden, BTCPay, Grafana et al in
cleartext. Every app port is gate-owned and serves TLS on the same port
(appgate/tls.rs), so directAppUrl(), the legacy open() path, and
resolveRuntimeLaunchUrl() now follow the page's scheme. HTTP pages (the
kiosk, LAN) are unchanged; netbird keeps its unconditional https.
The RELEASE_NOTES_BACKLOG gate for cutting the next release, closed out:
- Eight sections backfilled to the curated standard, from the Settings
What's New blocks, the old-lineage release commits, and the hotfix
diffs: v1.7.44 (was four raw commit-hash lines), v1.7.47/48/64/65
(thin), and v1.7.50/51/107 (real tagged releases whose sections were
missing entirely — v1.7.107 restored verbatim from the curated copy
at 35e9c624 that later went missing).
- Mechanical inventory across all 92 sections in range: every section
now has ≥3 curated bullets, zero raw-hash entries.
- What's New modal regenerated for the three restored versions
(sync-whats-new --check passes, 92 versions present).
- Manifest-notes-only confirmed by construction: the manifest reads its
changelog from CHANGELOG.md and check-release-manifest.sh rejects raw
or thin entries before publishing.
Evidence trail for the backfills is recorded in
docs/RELEASE_NOTES_BACKLOG.md.
generate-app-catalog.py only updates entries that already exist in each
catalog file, so the hand-curated cuprate entry (added to
app-catalog/catalog.json with 7b88ba59) never propagated to
neode-ui/public/catalog.json — the sync's field-bumps did, the new entry
did not. Both catalogs now carry identical 31-entry lists (verified
content-equal), so the browser-side store copy and the curated one agree.
Two reports from a fresh install without a cable:
(a) No way to see the WiFi password being typed. Every password field in
the app was a bare type=password input. PasswordRevealInput is the
reusable fix — masked by default, one-tap eye toggle, v-model and
enter pass-through — first applied to the WiFi prompt in ServerModals
so a long key typed from across the room can be verified.
(b) WiFi settings are undiscoverable with no wired internet. New
OnboardingNetworkCallout floats over every onboarding step when the
node has NO physical link at all (no ethernet up, no WiFi associated
— polled from network.list-interfaces, self-dismissing the moment a
link exists) and deep-links 'Connect to WiFi' to
/dashboard/server?open=wifi, which Server.vue consumes by popping the
WiFi picker on arrival. Deliberately scoped the other way too:
Archipelago is offline-first, so 'no internet' never nags — only 'no
link at all', only during onboarding (the wrapper hosts /login too;
the callout is restricted to /onboarding/* routes), and a failed probe
stays silent. The query is consumed via history.replaceState so a
KeepAlive tab-return never re-pops the modal, and Server.vue keeps
reading it from the real URL rather than vue-router — its
KeepAlive-mounted tests have no router context to give.
Verification: full frontend suite 1023/1023; type-check clean; production
build clean with both new strings confirmed in the emitted bundles
(OnboardingWrapper + Server chunks).
Hands the companion-owned work to the companion agent with precise
pointers (Android/ source, served APK at 0.5.27/vc47 + the deploy
pipeline from the 2026-07-23 handoff, the ArchipelagoNative bridge and
isCompanionApp gating pattern) and the queue: the ungated
CompanionBanner/intro-trigger residual of #61 (Discover.vue:156,
useCompanionIntro's openCompanionIntro), GrapheneOS backup/restore (#128,
reusing the node's ADR-005 backup envelope), and the NIP-46 remote-signer
phone side (#139, with the signer-login research doc as background).
Tracker labels applied earlier: #128 and #139 carry 'companion-agent'.
Curated release notes for the pending v1.8.5-alpha: Cuprate (with the
two review catches), kdump/rasdaemon + the host-fixup OTA channel, the
uninstall-abort fix, federation inline-picture routing, honest disk
usage, the three lying-screens fixes (#143/#127/#129), durable mesh
notifications + router recovery (#57/#103), and upstream-release tracking
with the first-sweep safe bumps.
What's New modal synced via scripts/sync-whats-new.py (--check passes;
89 versions, all present). Per docs/RELEASE_NOTES_BACKLOG.md the
v1.7.44-alpha -> current section audit remains the open item before the
tag.
The ISO's Dockerfile.rootfs gains kdump-tools/kexec-tools/rasdaemon with
USE_KDUMP=1, dumps to /var/crash and a compressed core collector, the
hang/panic sysctl drop-in, and rasdaemon + kdump-tools enabled — and the
installed target's GRUB cmdline gains crashkernel=256M next to the
existing quiet/splash line.
Source of truth note: the edit lands in
image-recipe/_archived/build-auto-installer-iso.sh — the builder that
generates the (git-ignored) image-recipe/build/auto-installer/ workspace,
which a cache-hit can reuse. The workspace copy was updated to match so
even a cached build ships the same state. Host fixups (previous commit)
converge already-deployed nodes to exactly this end state, so fresh and
old installs agree.
bash -n clean on the builder.
archipelago