v1.8.9's move to Router.SendPaymentV2 shipped without fee_limit_sat,
and the v2 route treats an ABSENT fee limit as zero allowed fees.
Every real route carries a routing fee (the 2-hop route here: 1.5
sats), so the pathfinder rejected them all and the wallet answered
"No route to the recipient" on EVERY send — all day, on healthy
channels with plenty of liquidity both ways.
The router debug log makes it unambiguous:
wallet payment (v1.8.9 backend): fee_limit=0 mSAT -> no route
same payment by hand (lncli --fee_limit=100): fee_limit=100000 mSAT -> settles in 0.65s
My earlier "pipeline verified" claim was wrong — the manual lncli
verification set a fee limit by hand and masked this exact bug. The
400k that succeeded this morning went through the pre-update backend
on the pre-update LND.
Payments now carry lncli's own default budget — the payment amount
(100%), preferring the payer-supplied amount for zero-value invoices
and the invoice's own amount otherwise, with a nominal floor so the
limit can never be zero. Unit-pinned so it cannot regress.
LND normally reconnects channel peers after a restart, but not reliably:
after long or repeated downtime (an app update, a node reboot,
reconciler churn) the peer link can stay down for hours while BOTH
endpoints keep the channel flagged disabled in the routing graph. The
node looks perfectly healthy, the wallet shows balance, and every
payment in either direction fails "no route to the recipient" —
observed live on framework-pt (2026-09-01): its only channel sat
disabled on both policy sides for ~17 hours after the LND 0.21.2
update, while shorty had 583k spendable and the user was told, by a
mis-mapped modal, that they had 'no payment channel'.
The channel graph is desired state — every open channel should have a
live peer connection. A daemon-side watchdog now enforces it:
- every 2 minutes, list channels + peers over LND REST
- for each channel whose remote peer is not connected, look the peer's
advertised addresses up in the public graph and dial one
- per-peer retries throttled to 10 minutes so an unreachable peer is
not hammered; 'already connected' counts as done; a peer with no
advertised address is logged once per pass (cannot be dialed)
- no-ops quietly on nodes without LND (missing macaroon) and while a
wallet is locked (503 body has no channels)
Unit tests pin the selection against the live REST shapes
(remote_pubkey in /v1/channels vs pub_key in /v1/peers).
v1.8.10 CHANGELOG + What's New entries staged so the next release run
is clean first time.
"LND thinks I do not have a channel" while the wallet showed plenty of
liquidity (framework-pt, 2026-09-01): the send gate sums outbound over
FULLY-OPEN channels only, which is correct — a just-opened channel
sits in LND's pending list until it has ~3 confirmations, and an
open channel can have all its balance on the far side — but the modal
then claimed the node had NO channel at all, in every one of those
states, and pointed the user at opening another one.
The gate already fetched the full channel list; it now records WHY
liquidity is zero and the modal says the truth per state:
- pending channels -> "your new channel is waiting for on-chain
confirmations, it unlocks automatically, nothing is needed from you"
(and no "Open a channel" button — that would send the user to fix
a problem they don't have, possibly opening a second channel)
- open channels, zero on the needed side -> "balance is on the far
side — you can receive but there's nothing to send right now"
- payment refused with a routing/liquidity error -> says so, instead
of claiming no channels
- only a genuinely channel-less node keeps the open-one guidance
Eleven unit tests pin the state machine, including the regression
case (pending-only -> 'pending', not 'none') and fail-open on RPC
errors.
The release gate requires the freshly-built bundle to embed the new
version, and the version reaches the bundle through the What's New
modal in AccountInfoSection — there was no v1.8.9-alpha block yet, so
create-release.sh correctly refused to ship a bundle that looked stale.
This adds the block (the user-facing summary of today's LND/HTTPS/
launcher/NPM/Portainer fixes) and carries the version bumps the
aborted run had already written (Cargo.toml, Cargo.lock, package.json,
package-lock), so the re-run starts from a clean tree.
Verified: npm run build now produces assets containing 1.8.9-alpha
(Settings chunk), i.e. the exact check the script runs passes.
Regenerated from the fixed apps/nginx-proxy-manager/manifest.yml (the
only semantic change vs the previous signed catalog) and signed with
the release-root key. Catalog-covered nodes pick this up on their next
hourly fetch and the NPM start/die loop ends: s6 gets its /etc/letsencrypt
mount back and the internal nginx can bind 80/443/81 again under
--cap-drop=ALL.
Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.
Three launcher/bridge defects combined to make HTTPS dashboards look
broken while HTTP ones worked:
1. portAuth() looked the launch port up under the name the user clicks
('mempool-web', 'lnd', 'bitcoin-knots'…), but the signed catalog
declares those ports under the manifest id that owns them
(archy-mempool-web, lnd-ui, bitcoin-ui). The lookup missed,
portIsGateFronted answered false, and an HTTPS dashboard handed app
frames http:// URLs — blocked as mixed content: mempool and IndeeHub
'did not connect', bitcoin knots/core opened http:// in a new tab.
Resolution now follows launch aliases, then a port-wide catalog scan
that only answers when every declarer of that port agrees (a port
any app publishes as plain HTTP is never upgraded to https).
2. The signed-catalog cache was only warmed by the Store/Discover
views, so a user who went straight to My Apps launched apps with an
empty cache. Warmed at dashboard mount now — fetchAppCatalog()
already memoizes with a 1h TTL.
3. The NIP-07 bridge compared event.origin for strict equality with the
recorded (http) app URL and replied to the recorded URL as the
postMessage targetOrigin — both break the moment a frame is scheme-
upgraded (cached HSTS did exactly that): every nostr request was
silently dropped and replies to the stale origin threw. The bridge
now matches host+port (scheme deliberately ignored) and always
replies to event.origin — the frame's real origin.
Unit tests cover alias resolution (incl. bitcoin-knots→8334→https),
the conservative port-scan, and scheme-agnostic sender matching.
Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
Portainer >=2.21 no longer lets whoever loads the page first claim the
admin account: on a fresh install it mints a one-time setup token and
prints it ONLY to the server logs. On an appliance that is a dead end —
'check the Portainer server logs' is exactly what a user cannot follow,
and after the 2.45.0 update it made a freshly restarted Portainer look
broken ('disappeared', then demands a token nobody can find).
package.credentials — the same RPC that powers the login-credentials
card on the app page — now extracts the setup_token line from
portainer's recent container logs and hands it over with the existing
copy-button treatment, titled and explained for a first-time user. The
token stops being printed once setup completes, and any container
recreate drops the log line, so the card disappears on its own and no
dead token lingers. Parsing is a pure, unit-tested scan against the
live-captured 2.45.0 log shape (64 hex chars after setup_token=).
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
A Claude worktree under aiui/ was committed as a submodule gitlink
(160000) with no .gitmodules entry, so actions/checkout's recursive
submodule pass exited 128 and every 'Build & push demo images' run since
v1.8.6 failed. Removing the index entry — the worktree is local state
and was never meant to be tracked.
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
Settings gains the SSH-over-mesh card (danger-zone confirmation for the
any-peer scope, sshd preflights, fipssh copy hint). The signed-catalog
merge filters components via the shared serviceNames module; Discover
grids get the standard icon container; install no longer yanks the user
to My Apps; v1.8.8 release notes.
fips/ssh_mesh.rs owns the 90-ssh.nft drop-in lifecycle: off by default,
any-peer scope behind the UI's danger confirmation or an explicit mesh
address list, reconciled on every daemon config install. The scanner now
takes installed apps' icons from their real manifest metadata (Cuprate's
Services tile) and classifies manifest-declared UI apps as launchable
even when the address probe misses (Alby Hub).
Store-listing components are filtered via the shared serviceNames canon;
these four never earn a tile: MorphOS server is old, the Web5 DID wallet
and CryptPad are untested, Lightning Stack is an untracked upstream
bundle (LND covers it).
directAppUrl(), the legacy open() path, and resolveRuntimeLaunchUrl()
now upgrade to https only for ports the app gate fronts — decided from
the signed catalog's embedded manifest ports (auth gated/open), so
plain-HTTP publishes (legacy installs, auth:none API ports like
Cuprate's RPC) keep http instead of failing outright. fetchAppCatalog()
merges the daemon-verified signed catalog into the App Store listing
(signed entries appear immediately; community copy supplies featured
and curated metadata), and Marketplace.vue uses the same dynamic fetcher
as Discover so the grid sees signed-new apps too.
Nginx Proxy Manager, Tailscale, Ollama, CryptPad, and AdGuard Home now
carry full manifests: the app gate fronts their web ports (TLS on the
same port, node login where appropriate), installs run through the
orchestrator, and pins live in the signed catalog. Tailscale mirrors its
legacy shape exactly (userspace networking, web console on 8240, plain
HTTP for the gate to front). Ollama stays loopback-only — the
assistant's local model backend, not a web app.
Retires the four already-removed apps for good (FIPS, Nostr VPN,
Routstr, Penpot pins dropped from image-versions.sh, the generator map,
and image_versions.rs), fixes Cuprate's duplicated metadata block that
strict YAML parsers reject, and updates the port-inventory review gates
for the new open (3 own-login consoles) and exempt (2 DNS) ports.
New-tab apps and the companion WebView got hardcoded http:// URLs, so a
node reached over HTTPS opened Vaultwarden, BTCPay, Grafana et al in
cleartext. Every app port is gate-owned and serves TLS on the same port
(appgate/tls.rs), so directAppUrl(), the legacy open() path, and
resolveRuntimeLaunchUrl() now follow the page's scheme. HTTP pages (the
kiosk, LAN) are unchanged; netbird keeps its unconditional https.
The RELEASE_NOTES_BACKLOG gate for cutting the next release, closed out:
- Eight sections backfilled to the curated standard, from the Settings
What's New blocks, the old-lineage release commits, and the hotfix
diffs: v1.7.44 (was four raw commit-hash lines), v1.7.47/48/64/65
(thin), and v1.7.50/51/107 (real tagged releases whose sections were
missing entirely — v1.7.107 restored verbatim from the curated copy
at 35e9c624 that later went missing).
- Mechanical inventory across all 92 sections in range: every section
now has ≥3 curated bullets, zero raw-hash entries.
- What's New modal regenerated for the three restored versions
(sync-whats-new --check passes, 92 versions present).
- Manifest-notes-only confirmed by construction: the manifest reads its
changelog from CHANGELOG.md and check-release-manifest.sh rejects raw
or thin entries before publishing.
Evidence trail for the backfills is recorded in
docs/RELEASE_NOTES_BACKLOG.md.
generate-app-catalog.py only updates entries that already exist in each
catalog file, so the hand-curated cuprate entry (added to
app-catalog/catalog.json with 7b88ba59) never propagated to
neode-ui/public/catalog.json — the sync's field-bumps did, the new entry
did not. Both catalogs now carry identical 31-entry lists (verified
content-equal), so the browser-side store copy and the curated one agree.
Two reports from a fresh install without a cable:
(a) No way to see the WiFi password being typed. Every password field in
the app was a bare type=password input. PasswordRevealInput is the
reusable fix — masked by default, one-tap eye toggle, v-model and
enter pass-through — first applied to the WiFi prompt in ServerModals
so a long key typed from across the room can be verified.
(b) WiFi settings are undiscoverable with no wired internet. New
OnboardingNetworkCallout floats over every onboarding step when the
node has NO physical link at all (no ethernet up, no WiFi associated
— polled from network.list-interfaces, self-dismissing the moment a
link exists) and deep-links 'Connect to WiFi' to
/dashboard/server?open=wifi, which Server.vue consumes by popping the
WiFi picker on arrival. Deliberately scoped the other way too:
Archipelago is offline-first, so 'no internet' never nags — only 'no
link at all', only during onboarding (the wrapper hosts /login too;
the callout is restricted to /onboarding/* routes), and a failed probe
stays silent. The query is consumed via history.replaceState so a
KeepAlive tab-return never re-pops the modal, and Server.vue keeps
reading it from the real URL rather than vue-router — its
KeepAlive-mounted tests have no router context to give.
Verification: full frontend suite 1023/1023; type-check clean; production
build clean with both new strings confirmed in the emitted bundles
(OnboardingWrapper + Server chunks).
Hands the companion-owned work to the companion agent with precise
pointers (Android/ source, served APK at 0.5.27/vc47 + the deploy
pipeline from the 2026-07-23 handoff, the ArchipelagoNative bridge and
isCompanionApp gating pattern) and the queue: the ungated
CompanionBanner/intro-trigger residual of #61 (Discover.vue:156,
useCompanionIntro's openCompanionIntro), GrapheneOS backup/restore (#128,
reusing the node's ADR-005 backup envelope), and the NIP-46 remote-signer
phone side (#139, with the signer-login research doc as background).
Tracker labels applied earlier: #128 and #139 carry 'companion-agent'.
Curated release notes for the pending v1.8.5-alpha: Cuprate (with the
two review catches), kdump/rasdaemon + the host-fixup OTA channel, the
uninstall-abort fix, federation inline-picture routing, honest disk
usage, the three lying-screens fixes (#143/#127/#129), durable mesh
notifications + router recovery (#57/#103), and upstream-release tracking
with the first-sweep safe bumps.
What's New modal synced via scripts/sync-whats-new.py (--check passes;
89 versions, all present). Per docs/RELEASE_NOTES_BACKLOG.md the
v1.7.44-alpha -> current section audit remains the open item before the
tag.
The ISO's Dockerfile.rootfs gains kdump-tools/kexec-tools/rasdaemon with
USE_KDUMP=1, dumps to /var/crash and a compressed core collector, the
hang/panic sysctl drop-in, and rasdaemon + kdump-tools enabled — and the
installed target's GRUB cmdline gains crashkernel=256M next to the
existing quiet/splash line.
Source of truth note: the edit lands in
image-recipe/_archived/build-auto-installer-iso.sh — the builder that
generates the (git-ignored) image-recipe/build/auto-installer/ workspace,
which a cache-hit can reuse. The workspace copy was updated to match so
even a cached build ships the same state. Host fixups (previous commit)
converge already-deployed nodes to exactly this end state, so fresh and
old installs agree.
bash -n clean on the builder.
kdump + rasdaemon on every node, per docs/kdump-rasdaemon-design.md with
the approved decisions: hang capture ON (a wedged kiosk dumps and reboots
itself instead of sitting dead), crashkernel=256M, backfill ships with
this release, phase-2 UI surfacing deferred.
Host fixups (docs/system-level-ota-design.md) are the general answer to
'deliver system-level updates OTA': curated OS packages, sysctl drop-ins,
service enablement and the GRUB crashkernel line, carried by the signed
binary and applied idempotently at startup — non-fatal by construction
(offline/locked-dpkg nodes converge on a later boot), skipped on dev
boxes and non-Debian hosts. This formalizes the polkit/audio repair
precedents into a channel with a stated policy: pinned packages and
parameter intent only, never dist-upgrade automation; the ISO bakes the
identical end state into fresh installs (next commit).
The one runtime limitation is honest: crashkernel memory can only be
reserved at boot, so the fixup writes GRUB, runs update-grub, and logs
that it takes effect on the next reboot.
tests/lifecycle/os-audit.sh gains section D — a graded baseline check:
FAIL if capture never landed, WARN if written but awaiting reboot, PASS
when reserved, policy live and rasdaemon recording. Section D runs
independently of RPC health: a wedged backend must not mask that the
node also stopped capturing evidence.
Verification: host_fixups unit tests 4/4; cargo fmt clean; full suite
runs in the release gate (create-release) and the archi-dev-box
lifecycle gate before the tag.
Gives stable names to what issue #134 showed gets conflated: Trusted peer
(invite-verified, operator decision), Discovered peer (learned from a
Trusted peer's advertisement, hard-capped at Observer — TRUST IS NOT
TRANSITIVE), Routing hint (what a Discovered peer actually contributes:
reachability, not trust), and Peer advertisement (the mechanism itself,
a feature not a leak).
Records the two rules that make the model sound (trust requires a
traceable operator decision; discovery is transitive, trust is not), why
advertisement exists (one invite makes a node reachable to the trusted
set without granting anything), and the deferred open questions: the
'don't advertise my peers' privacy toggle and UI tier vocabulary.
Design for capturing post-mortem and hardware-error evidence on fleet
nodes: kdump (crashkernel=256M, dump to /var/crash on the unencrypted
root — never the LUKS data partition, so the crash kernel never handles
key material; makedumpfile-compressed, keep-2 retention) and rasdaemon
(EDAC/ECC events into sqlite on the same root).
Deliberately phased: phase 1 = capture on the image + bootstrap backfill
for existing nodes (kernel cmdline can't travel by OTA; takes effect on
next reboot); phase 2 = a read-only system.diagnostics surface in the
UI, only after a fleet node has produced a real dump.
Four decisions flagged in the doc: hang-capture on/off (recommended ON
— a wedged kiosk is useless anyway, and this turns every freeze into
evidence + self-reboot), crashkernel size, backfill timing, and phase-2
scope. Implementation touchpoints listed (Dockerfile.rootfs,
auto-install.sh:1810 cmdline, kdump-tools config, bootstrap, lifecycle
gate assertions).
Signed by the release root (ceremony verify passed locally before push).
Contents of this catalog over the previous one:
NEW cuprate 0.1.0-preview-18-g618ff14 — alternative Monero
node (Rust); image verified present in the mirror
registry; manifest embedded; store entry curated
(money / optional)
BUMP strfry 1.1.1 -> 1.1.2
BUMP btcpay-server 2.4.2 -> 2.4.3
BUMP netbird (nginx) 1.31.3-alpine -> 1.31.4-alpine
BUMP pine (nginx) 1.31.3-alpine -> 1.31.4-alpine
All bump targets verified pullable from their public registries before
editing. The three mirror-backed bumps (vaultwarden 1.37.2-alpine,
archy-nbxplorer 2.6.11, home-assistant 2026.8.3) remain parked on
app-bumps-mirror-pending until a live registry-push token exists for the
lfg2025 namespace.
Drift gate clean: check-app-catalog-drift.py --release --strict
(31 store entries, 0 drift, 0 missing). 69 catalog entries total.
Nodes pick this up on their next hourly catalog refresh (or at startup)
— signature verified against the release-root key before application.
Pin bumps (all verified pullable from their public registries before
editing, so none can become an image-not-found on a node):
strfry 1.1.1 -> 1.1.2 (dockurr/strfry, direct pull)
btcpay-server 2.4.2 -> 2.4.3 (docker.io/btcpayserver, direct pull)
netbird (nginx) 1.31.3-alpine -> 1.31.4-alpine
pine (nginx) 1.31.3-alpine -> 1.31.4-alpine
image-versions.sh moved in lockstep for BTCPAY_IMAGE — it is the baseline
the update badge compares against. Held back deliberately, per the risk
policy from the Aug-17 pass: gitea (four minors of DB migrations),
portainer (six minors), filebrowser (2.27 -> 2.63), fedimint/gateway
(0.8 -> 0.12, real migrations), lnd (money-critical), netbird-server/
netbird-dashboard (0.x, must move in lockstep), and everything with a
major jump or a data migration.
Cuprate also gets its curated store entry (category money, tier optional,
icon, repo) — same shape as the Alby Hub / phoenixd entries — synced
through generate-app-catalog.py into both store catalogs and the
app-session config. The fips launch-port list is unchanged (Cuprate has
no UI port; the generated file round-trips to the committed bytes after
cargo fmt).
Three further bumps are prepared and parked on the
app-bumps-mirror-pending branch, blocked only on a registry-push token:
vaultwarden 1.37.2-alpine, archy-nbxplorer 2.6.11, home-assistant
2026.8.3 — all mirror-backed, and the push credential on record for the
lfg2025 namespace is dead.
Drift gate: check-app-catalog-drift.py --release --strict clean
(31 store entries, 0 drift, 0 missing). appSessionConfig tests 7/7.
Five apps had no app.upstream block, so nothing could ever tell us
when their pins fell behind upstream:
barkd gitlab ark-bitcoin/bark (GitLab-only project)
immich-postgres ghcr immich-app/postgres (image exists only on ghcr.io)
indeedhub-minio github minio/minio
pine-whisper dockerhub rhasspy/wyoming-whisper
lightning-stack manual — no public listing exists for
lightninglabs/lightning-stack anywhere (docker.io,
ghcr.io, github.com all checked), so it is tracked by hand
This adds two fetchers to scripts/check-upstream-releases.py to reach the
first two: latest_gitlab (GitLab releases API; strips the project-name
tag prefix, e.g. bark-0.6.2 -> 0.6.2) and latest_ghcr (anonymous pull
token + tags/list, the same handshake a docker pull performs).
Live-verified after the change:
barkd 0.3.0 -> 0.6.2 (bump gated on ark_client.rs REST compat)
immich-postgres 14-vectorchord0.4.3-pgvectors0.2.0 -> 17-vectorchord0.4.3-pgvector0.8.0
indeedhub-minio RELEASE.2024-11-07T00-52-20Z -> latest (date-opaque: UNCOMPARABLE, shown for hand comparison)
pine-whisper 3.4.1 -> 3.6.0 (tuned-args revision needs re-basing, not just a pin move)
Offline coverage check: 59 apps, 0 untracked.
When Bitcoin's IBD completed mid-Lightning-goal, the watcher toasted
"you can now fund your wallet" — but the on-chain wallet lives in LND,
not Bitcoin Core. The watcher only checked that the goal had pending
manual steps, never that the install-LND step had completed, so a user
whose LND wasn't installed yet was pointed at a flow that could not
work: the fund modal's address comes from lnd.newaddress and does not
exist until LND is installed (issue #143).
The toast now checks LND's install state at fire time. With LND
installed the message is unchanged; without it, the toast says the
actual next step — install Lightning (LND) — and the Finish setup
button lands on the goal wizard, whose active step is the pending
install-LND one (the wizard itself was already correctly sequenced).
The watcher had no tests; added four pinning its contract: the two
message branches, silence with no in-progress goal, and silence when
the chain was already synced at page load.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Disk usage was computed as used/size, where size is the raw device size.
ext4 reserves 5% of the filesystem for root — 92.4 GiB of this node's
1.8 TiB — which size includes but nothing can allocate. Two consequences,
both live on archi-dev-box today:
The dashboard advertised 251 GiB free when only 159 GiB could actually be
written, and reported 86.2% usage against df's 90.8%.
Worse, disk_monitor triggers automatic cleanup (podman image prune) at
90%. The disk has been genuinely above that threshold while this returned
86.2%, so the cleanup never once fired — which is exactly how ~72 GB of
dangling images accumulated unnoticed, and why deleting apps appeared to
free nothing.
Both call sites now ask df for avail and use used/(used+avail): the same
figure df itself prints, and the space an operator can actually spend.
Callers deriving free as total - used now get avail.
Note this shifts disk_total_bytes in the analytics series down by the
reserve; historical samples are not comparable across this change.
Tests updated for the three-column output, plus a regression test built
from this box's real numbers asserting the corrected math crosses the 90%
threshold the old math missed. 15/15 disk_monitor tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The manifest bound cuprated's unrestricted RPC (full node control) to
0.0.0.0 inside the container with
i_know_what_im_doing_allow_public_unrestricted_rpc = true, relying on
ports[].bind: 127.0.0.1 to keep it private. That only restricts the HOST
side. Verified live on archi-dev-box 2026-08-22: a peer container got a
valid unauthenticated get_info off container port 18081 — and still did
after cuprate was moved to its own network, because podman bridges route
to each other unless created with --opt isolate=true, which the
orchestrator's auto-create does not pass. Every app on the node could
therefore drive full node control with no credential.
The PR justified this as the pattern bitcoin-knots already uses, but
knots writes rpcuser/rpcpassword from generated secrets, so a 0.0.0.0
bind there still is not control without credentials. cuprated has no RPC
authentication at all, so the two are not equivalent.
Unrestricted RPC is now left at cuprated's own default — container
loopback only, published nowhere, reachable by nothing — which is what
upstream intends by refusing a non-local bind without an explicit
override. Restricted RPC (the safe-for-public subset wallets use) and p2p
are unchanged, and health_check moves to 18089 since 18184 is gone.
Re-verified after the change: peer container gets connection refused on
18081 (exit 7), restricted RPC and the health endpoint still answer, the
node still syncs, validator APPROVED, 76/76 container tests pass
including the unauthenticated-port canary (still 28 — an auth: local
port was removed, not an auth: none one).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The signer reads stdin to EOF, so pressing Enter submits nothing and a
second paste simply appends to the first. Step [6b/8] said only "paste the
release master mnemonic when prompted", which gives no hint that Ctrl-D is
what ends the input — a 24-word phrase arrived today as "invalid word
count: 89", about four pastes concatenated by someone reasonably assuming
Enter had not worked.
sign-manifest.sh already explains this properly; create-release.sh now says
the same thing, including that pasting twice is itself a failure mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`npm run build 2>&1 | tail -3` threw away npm's exit status, so a failed
build was indistinguishable from a good one. The run continued and blamed
the next check instead — "the frontend build no-opped or its output is
stale" — which points at a stale dist rather than at the build error that
actually happened, and cost a diagnosis cycle today.
Success still prints the same quiet 3 lines; a failure now prints the real
error, keeps the full log, and aborts on the spot.
Verified both branches with a stubbed npm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
create-release aborted at [4/8] with "web/dist/neode-ui does not contain
v1.8.4-alpha — the frontend build no-opped or its output is stale". The
build had not no-opped: it was fresh, and simply had no 1.8.4 string to
embed.
sync-whats-new.py only matches '## vX.Y.Z (YYYY-MM-DD)'. The entry read
'## v1.8.4-alpha (draft — date set at cut)', so the version was invisible
to it: the gate's whats-new-sync stage reported "87 versions, all present"
while the release being cut had no What's New block. That modal is the
only place a version string appears in the frontend, so the bundle carried
none and the freshness check — correctly — refused it, while naming the
wrong cause. Step [5/8] only greps for '^## v1.8.4-alpha (' so it passed
the draft too.
Three changes: date the v1.8.4-alpha entry, insert the modal block it was
owed, and make the sync tool refuse any version header without a real date
instead of skipping it. Skipping is what let a wrong "all present" through.
Verified: the draft header now fails the check with an explicit message,
the dated one passes (88 versions, up from 87), and a rebuilt bundle
contains 1.8.4-alpha where it did not before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four unrelated tests failed the release gate at once today — every one of
them "Test timed out in 5000ms", none an assertion. Wall times were 6.3s,
16.5s, 5.5s and 36.2s for tests that normally finish in milliseconds
(useModalKeyboard's takes 349ms on an idle box), and the whole suite took
405s against its usual ~70s. The cause was CPU starvation from a
concurrent cargo build, not anything in the code.
The 5s default says nothing about these tests and everything about the
machine: this box also runs a live node, so a gate run can always collide
with a build or container churn. 20s survives that while still bounding a
genuine hang.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cargo-test-weekly failed twice today with exit 124 at unit 427/429 — the
non-incremental test-profile build running out of wall clock mid-compile,
before a single test executed. The summary said only "FAIL: cargo-test-
weekly", which reads as a broken test and sends you hunting for one that
does not exist.
Two changes: the ceiling goes 1500s -> 3600s (580s was already found too
short; 1500s now dies on the biggest link on a loaded, swapping box), and
stage() names exit 124 as a timeout rather than printing a bare code.
Verified both reporting branches: a timed-out stage and an ordinary
non-zero exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Publishing the ISO was a manual step printed as a reminder at the end of
build-iso-release.sh: upload the ISO, its .sha256 and the signed checksum
JSON by hand. Only the OTA binary and frontend tarball were automated.
publish-release-assets.sh now uploads all three when an ISO for the
version exists in image-recipe/results/, with the same supply-chain rules
the OTA manifest already gets: the checksum JSON must be signed by the
pinned release root, the signature must cryptographically verify, and the
image must still match its own .sha256 (a truncated or half-copied ISO is
exactly what a signed checksum exists to expose). After upload it
confirms every asset landed at its exact local size.
The stage runs AFTER main is pushed, deliberately. The ISO is not
referenced by releases/manifest.json, so no node's OTA path depends on
it — running it last means a slow or failed multi-GB upload can never
delay or strand an OTA release that has already been verified. When no
ISO exists yet (the usual case, since the ISO build needs the tag this
script pushes) it explains how to build and attach one, and exits clean.
Uploads take a max-time argument: 4h and a progress bar for the ISO,
where the previous fixed 15-minute silent ceiling would have killed a
multi-GB transfer partway through.
Verified with a stubbed harness: no-ISO skip, missing .sha256, unsigned
checksum, wrong signing key, corrupted image, happy path, and a truncated
upload caught by the size check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Left uncommitted by an aborted create-release run on 2026-08-19: the
version bump landed in the tree but the release never reached its tag or
manifest. Committing it so the tree is clean before the release is re-cut.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ui-unit-tests stage piped vitest through `tail -4`, which cut off the
failure block. A red gate reported "1 failed | 999 passed" and nothing
else — no file, no test name, no assertion — so the failure could not be
diagnosed after the run.
Success still prints the quiet 4-line summary; failure now dumps the full
log and keeps it on disk so a scrolled-off terminal isn't the end of it.
Verified both paths: green run unchanged, and a deliberately failing spec
now surfaces its file, test name, assertion and line number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate's first real stage is `cargo fmt --check`, and it had
44 diffs across 15 files — enough to abort `create-release.sh` at step 0
before it touched a version number. Some of that drift is mine from the
last two days, some predates it in files I never opened
(bootstrap.rs, ghost_reaper.rs, openwrt/router.rs), and one is the
regenerated fips/app_ports.rs.
No behaviour change — rustfmt only.
Gate now: 8 of 9 green. The remaining red is cargo-test-weekly exiting
124, which is the 25-minute `timeout` expiring during a cold
CARGO_INCREMENTAL=0 rebuild on a loaded node — the tests never started.
Not a test failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Regenerates both catalogs from the manifests so the 15 pin bumps become
real. The catalog overrides on-disk manifests on every node, so until
now those bumps were edited but inert.
There are two catalogs and regenerating one is not enough:
generate-app-catalog.sh writes releases/app-catalog.json (the signed one
nodes fetch), while generate-app-catalog.py writes app-catalog/catalog.json
and neode-ui/public/catalog.json (the source pair, the second baked into
the frontend app store). check-app-catalog-drift.py --release --strict
reads the *source* catalog, so regenerating only the release one left it
failing and would have aborted the ISO gate at stage 1 — after the
signing and tagging were already done. Drift is now 0.
The regeneration also rewrote fips/app_ports.rs, which had not been
regenerated since the initial open-source import. Diffing the port values
rather than the reformat: 36 -> 37, a single addition, **8187 — Alby
Hub**. Its port has never been in the FIPS firewall allow-list, and by
the same token neither has any app onboarded since that import. Nothing
else changed.
Catalog signed by the pinned release root and verified with
`ceremony verify`; registry trust floor checked before signing, both
hosts trusted by the deployed fleet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both import refusals reached the operator with runs of ~18 spaces mid
sentence — "Importing a different one means coins
minted…". The string literals had been written as single long lines with
the line-continuation whitespace baked in rather than escaped, so Rust
preserved it verbatim.
Only visible once the sanitizer stopped swallowing these messages, which
is its own small lesson: the text had been wrong since it was written and
nothing could show it.
Cosmetic, but not trivially so — this is the warning that stops someone
replacing the phrase their balance was minted under, and text that looks
broken is text people stop reading.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With registry push access, the 24 mirror-backed apps stopped being
blocked. Ten images are now mirrored (single-platform amd64, matching
the existing convention) and their pins moved:
alby-hub v1.23.0 -> v1.24.0
mempool-frontend v3.0.1 -> v3.3.1 (mempool, archy-mempool-web)
mempool-backend v3.0.0 -> v3.3.1
fedimintd v0.10.0 -> v0.10.1
gatewayd v0.10.0 -> v0.10.1
nostr-rs-relay 0.9.0 -> 0.10.0
portainer 2.39.1 -> 2.39.6
vaultwarden 1.30.0-alpine -> 1.37.1-alpine
jellyfin 10.8.13 -> 10.11.11
home-assistant 2026.7.3 -> 2026.8.2
Every one verified pullable from our mirror after copying, so none can
become an image-not-found on a node. image-versions.sh moved in lockstep
— it is the baseline the update badge compares against when the catalog
does not cover an app, and leaving it behind would have kept advertising
an update that had already been applied.
Chosen by risk, not by count: these are patch/minor bumps with no data
migration. The ones held back are held for a reason each — Postgres
15->18 and 16->18 refuse to start on an older cluster, Redis 7->8,
Valkey 7->9, Nextcloud 29->32 must go one major at a time, plus
uptime-kuma 1->2, grafana 10->13, electrumx 1->2, photoprism, and
core-lightning's three years of schema migrations. Those are each a
migration plan, not a pin edit. LND (v0.18.4 -> v0.21.2) is held
separately: it is only a minor bump by version but it migrates its
channel database irreversibly, and this box holds real funds.
Note the checker still reports several of these as behind, and that is
correct: it reads the *catalog* pin, which is what nodes actually act on.
These land when the catalog is regenerated and re-signed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 33 apps behind upstream, these five pull straight from a public
registry, so their targets exist already and the bump is real work rather
than a promise:
strfry 1.0.4 -> 1.1.1
netbird (nginx) 1.27-alpine -> 1.31.3-alpine
pine (nginx) 1.27-alpine -> 1.31.3-alpine
pine-piper 2.2.2 -> 2.4.2
nostr-rs-relay 0.8.9 -> 0.10.0
All five targets verified present upstream with skopeo before editing, so
none of these can turn into an image-not-found on a node.
Deliberately NOT bumped here, though they are also direct-pull:
core-lightning (v23.08 -> v26.06, ~3 years of schema migrations), gitea
(four minors of DB migrations), and netbird-server/netbird-dashboard —
which have to move in lockstep and carry their own migrations. Those are
each a piece of work, not a line edit.
The other 24 are blocked on something else entirely: their images live in
our mirror and none of the upgrade targets have been mirrored yet, so a
pin bump alone would break every install. That needs registry push
credentials.
These take effect when the catalog is regenerated and re-signed — the
catalog overrides on-disk manifests, so editing here changes nothing on a
node until the signing ceremony.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archipelago