The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
kdump + rasdaemon on every node, per docs/kdump-rasdaemon-design.md with
the approved decisions: hang capture ON (a wedged kiosk dumps and reboots
itself instead of sitting dead), crashkernel=256M, backfill ships with
this release, phase-2 UI surfacing deferred.
Host fixups (docs/system-level-ota-design.md) are the general answer to
'deliver system-level updates OTA': curated OS packages, sysctl drop-ins,
service enablement and the GRUB crashkernel line, carried by the signed
binary and applied idempotently at startup — non-fatal by construction
(offline/locked-dpkg nodes converge on a later boot), skipped on dev
boxes and non-Debian hosts. This formalizes the polkit/audio repair
precedents into a channel with a stated policy: pinned packages and
parameter intent only, never dist-upgrade automation; the ISO bakes the
identical end state into fresh installs (next commit).
The one runtime limitation is honest: crashkernel memory can only be
reserved at boot, so the fixup writes GRUB, runs update-grub, and logs
that it takes effect on the next reboot.
tests/lifecycle/os-audit.sh gains section D — a graded baseline check:
FAIL if capture never landed, WARN if written but awaiting reboot, PASS
when reserved, policy live and rasdaemon recording. Section D runs
independently of RPC health: a wedged backend must not mask that the
node also stopped capturing evidence.
Verification: host_fixups unit tests 4/4; cargo fmt clean; full suite
runs in the release gate (create-release) and the archi-dev-box
lifecycle gate before the tag.
cargo-test-weekly failed twice today with exit 124 at unit 427/429 — the
non-incremental test-profile build running out of wall clock mid-compile,
before a single test executed. The summary said only "FAIL: cargo-test-
weekly", which reads as a broken test and sends you hunting for one that
does not exist.
Two changes: the ceiling goes 1500s -> 3600s (580s was already found too
short; 1500s now dies on the biggest link on a loaded, swapping box), and
stage() names exit 124 as a timeout rather than printing a bare code.
Verified both reporting branches: a timed-out stage and an ordinary
non-zero exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ui-unit-tests stage piped vitest through `tail -4`, which cut off the
failure block. A red gate reported "1 failed | 999 passed" and nothing
else — no file, no test name, no assertion — so the failure could not be
diagnosed after the run.
Success still prints the quiet 4-line summary; failure now dumps the full
log and keeps it on disk so a scrolled-off terminal isn't the end of it.
Verified both paths: green run unchanged, and a deliberately failing spec
now surfaces its file, test name, assertion and line number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archipelago