Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.
Three launcher/bridge defects combined to make HTTPS dashboards look
broken while HTTP ones worked:
1. portAuth() looked the launch port up under the name the user clicks
('mempool-web', 'lnd', 'bitcoin-knots'…), but the signed catalog
declares those ports under the manifest id that owns them
(archy-mempool-web, lnd-ui, bitcoin-ui). The lookup missed,
portIsGateFronted answered false, and an HTTPS dashboard handed app
frames http:// URLs — blocked as mixed content: mempool and IndeeHub
'did not connect', bitcoin knots/core opened http:// in a new tab.
Resolution now follows launch aliases, then a port-wide catalog scan
that only answers when every declarer of that port agrees (a port
any app publishes as plain HTTP is never upgraded to https).
2. The signed-catalog cache was only warmed by the Store/Discover
views, so a user who went straight to My Apps launched apps with an
empty cache. Warmed at dashboard mount now — fetchAppCatalog()
already memoizes with a 1h TTL.
3. The NIP-07 bridge compared event.origin for strict equality with the
recorded (http) app URL and replied to the recorded URL as the
postMessage targetOrigin — both break the moment a frame is scheme-
upgraded (cached HSTS did exactly that): every nostr request was
silently dropped and replies to the stale origin threw. The bridge
now matches host+port (scheme deliberately ignored) and always
replies to event.origin — the frame's real origin.
Unit tests cover alias resolution (incl. bitcoin-knots→8334→https),
the conservative port-scan, and scheme-agnostic sender matching.
Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
Portainer >=2.21 no longer lets whoever loads the page first claim the
admin account: on a fresh install it mints a one-time setup token and
prints it ONLY to the server logs. On an appliance that is a dead end —
'check the Portainer server logs' is exactly what a user cannot follow,
and after the 2.45.0 update it made a freshly restarted Portainer look
broken ('disappeared', then demands a token nobody can find).
package.credentials — the same RPC that powers the login-credentials
card on the app page — now extracts the setup_token line from
portainer's recent container logs and hands it over with the existing
copy-button treatment, titled and explained for a first-time user. The
token stops being printed once setup completes, and any container
recreate drops the log line, so the card disappears on its own and no
dead token lingers. Parsing is a pure, unit-tested scan against the
live-captured 2.45.0 log shape (64 hex chars after setup_token=).
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
The previous commit on this branch copied cuprated's raw
--generate-config defaults (stdout=info, file=debug, max_log_files=7)
verbatim. Turns out that's the wrong reference: compared against
ssmithx@archy-dev-pa:/home/ssmithx/cuprate/Cuprated.toml — the actual
dev config this app was built and tested against — file logging is
meant to run at "info" with 14 rotated files, not the binary's raw
"debug"/7. Confirmed live on amishparadise: podman logs (stdout) was
already clean at info, but the on-disk file log
(.local/share/cuprate/logs/<date>) was flooding with per-peer DEBUG
gossip (~400KB in 2 minutes) because it inherited the binary default
instead of the intended one.
Left the resource-tuning knobs in the reference config (8GB
target_max_memory, tokio/rayon thread counts, P2P connection counts,
explicit reader_threads) out of this file — those were sized for
ssmithx's dev box and don't apply here; this manifest's
target_max_memory already stays deliberately under the container's
4Gi memory_limit.
Regenerated releases/app-catalog.json (still unsigned).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
apps/cuprate/manifest.yml only ever wrote network/target_max_memory/
rpc.restricted.enable into Cuprated.toml, so the [tracing.stdout] and
[tracing.file] tables were silently absent — cuprated still applied
its built-in info/debug/7 defaults, but nothing on disk showed it.
Verified live on amishparadise 2026-09-01: the deployed 5-line file
had no [tracing] section at all, and the level was only discoverable
by running `cuprated --generate-config` and diffing.
Add both tables to the manifest's files[].content with the same
values cuprated already defaults to, so every new install ships a
Cuprated.toml an operator can actually read and tune. overwrite:false
means already-deployed nodes (amishparadise included) keep their
existing file untouched — this only changes what fresh installs get.
Regenerated releases/app-catalog.json (unsigned) to embed the updated
manifest; needs scripts/sign-catalog.sh before it's authoritative for
the fleet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
A Claude worktree under aiui/ was committed as a submodule gitlink
(160000) with no .gitmodules entry, so actions/checkout's recursive
submodule pass exited 128 and every 'Build & push demo images' run since
v1.8.6 failed. Removing the index entry — the worktree is local state
and was never meant to be tracked.
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
Settings gains the SSH-over-mesh card (danger-zone confirmation for the
any-peer scope, sshd preflights, fipssh copy hint). The signed-catalog
merge filters components via the shared serviceNames module; Discover
grids get the standard icon container; install no longer yanks the user
to My Apps; v1.8.8 release notes.
fips/ssh_mesh.rs owns the 90-ssh.nft drop-in lifecycle: off by default,
any-peer scope behind the UI's danger confirmation or an explicit mesh
address list, reconciled on every daemon config install. The scanner now
takes installed apps' icons from their real manifest metadata (Cuprate's
Services tile) and classifies manifest-declared UI apps as launchable
even when the address probe misses (Alby Hub).
Store-listing components are filtered via the shared serviceNames canon;
these four never earn a tile: MorphOS server is old, the Web5 DID wallet
and CryptPad are untested, Lightning Stack is an untracked upstream
bundle (LND covers it).
directAppUrl(), the legacy open() path, and resolveRuntimeLaunchUrl()
now upgrade to https only for ports the app gate fronts — decided from
the signed catalog's embedded manifest ports (auth gated/open), so
plain-HTTP publishes (legacy installs, auth:none API ports like
Cuprate's RPC) keep http instead of failing outright. fetchAppCatalog()
merges the daemon-verified signed catalog into the App Store listing
(signed entries appear immediately; community copy supplies featured
and curated metadata), and Marketplace.vue uses the same dynamic fetcher
as Discover so the grid sees signed-new apps too.
Nginx Proxy Manager, Tailscale, Ollama, CryptPad, and AdGuard Home now
carry full manifests: the app gate fronts their web ports (TLS on the
same port, node login where appropriate), installs run through the
orchestrator, and pins live in the signed catalog. Tailscale mirrors its
legacy shape exactly (userspace networking, web console on 8240, plain
HTTP for the gate to front). Ollama stays loopback-only — the
assistant's local model backend, not a web app.
Retires the four already-removed apps for good (FIPS, Nostr VPN,
Routstr, Penpot pins dropped from image-versions.sh, the generator map,
and image_versions.rs), fixes Cuprate's duplicated metadata block that
strict YAML parsers reject, and updates the port-inventory review gates
for the new open (3 own-login consoles) and exempt (2 DNS) ports.
New-tab apps and the companion WebView got hardcoded http:// URLs, so a
node reached over HTTPS opened Vaultwarden, BTCPay, Grafana et al in
cleartext. Every app port is gate-owned and serves TLS on the same port
(appgate/tls.rs), so directAppUrl(), the legacy open() path, and
resolveRuntimeLaunchUrl() now follow the page's scheme. HTTP pages (the
kiosk, LAN) are unchanged; netbird keeps its unconditional https.
Full state at session end (all public surfaces verified byte-identical
at 0.5.28/vc48; only node web-bundle redeploys outstanding), the feature
map, and the operational playbook next sessions need: Tor SOCKS proxy
for Gitea API/curl (the 'unreachable API' was a missing proxy flag),
token scopes, protected-main ship flow via -ship branch + PR + API merge,
stale local main lineage, the foundation server's two surfaces, demo CI
auto-redeploy, build/test commands, and the open items.
For the archi-dev-box agent: companion 0.5.28 is on main (PR #149) and
Gitea raw serves it (verified byte-identical, v1+v2+v3). Remaining: the
foundation server's static /packages mirror (the real-node QR download
URL — currently 0.5.27), node web-bundle redeploys (same as the
2026-07-23 flow), and confirming the demo stack flipped after CI's
webhook redeploy. Exact commands, expected shasum, and final verify
block included.
archipelago
ssmithxandClaude Sonnet 5
lfg2025