Taking over a parked item from the app-UI work. AIUI's built index.html
emits ABSOLUTE /assets/<hashed> paths, so the browser asks for
/assets/index-BC2fBBaW.js. That lands in the MAIN UI's assets dir, where
it does not exist — the real files are in aiui/assets/. Both of AIUI's
two entry assets 404'd, so the embedded sidebar loaded nothing.
The config already contained a /aiui-assets/ location whose comment names
this exact problem ("AIUI may reference /assets/ without /aiui/ prefix"),
but it only catches requests to /aiui-assets/, a path AIUI never asks
for. It described the bug without fixing it.
/assets/ now falls back to a named location that rewrites into
aiui/assets/ and 404s from there. A fallback rather than copying the two
files up one level, because a frontend deploy replaces web-ui wholesale —
update.rs preserves the aiui/ DIRECTORY, not copies made into assets/ —
so a copy is erased by the very next deploy while this survives one.
Both server blocks (HTTP and HTTPS) are patched; named locations are
per-server, so each needs its own.
Verified on archi-dev-box after reload:
/assets/index-BC2fBBaW.js 200, 305256 bytes, application/javascript
/assets/index-BJkaQ2c4.css 200, 150716 bytes, text/css
/assets/does-not-exist.js 404 (the fallback is not over-broad)
/assets/index--lyLAgu1.js 200 (real main-UI chunks still come from
/assets/vendor-CmYeCqL_.js 200 the main dir — try_files hits them
/assets/index-CiMaoNII.css 200 before the fallback is consulted)
/ /aiui/ /health 200
Hash collision between the two builds is not a concern: Vite hashes are
content-derived, and any main-UI asset that exists is served by try_files
before the fallback runs.
Noted while doing this, not fixed here: the node's own
/etc/nginx/sites-enabled/archipelago is 378 lines BEHIND this repo file
(984 vs 1362) — it predates the IPv6 listener and the @asset_missing
no-store handling, among others. The node was patched minimally in its
own shape rather than overwritten, since a wholesale copy of a config
this diverged is not a safe unattended action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parsing each half back proves each is well-formed; it never proves they
belong together. A key from one generation beside a cert from another
passes both individual parse checks, gets blessed, and then nginx refuses
to start at the exact moment the marker claims first boot succeeded.
gen_tls() now extracts the public key from each half and compares them
before the swap, and needs_tls() applies the same check to what is already
installed, so a mismatched pair that reached disk some other way (an older
build, a half-finished manual edit) is repaired instead of quietly
breaking nginx. Extraction subsumes parsing, so this replaces the separate
-noout parse checks rather than adding to them.
Kept deliberately in step with regenerate_tls_cert() in
core/archipelago/src/api/rpc/system/handlers.rs, which does the same
comparison on the running node after a rename.
Test harness: the openssl stub keypair now carries the generation it came
from, and STUB_OPENSSL_MISMATCH emits a cert from a different one — the
pair that passes both parse checks and still breaks nginx. New case 9
covers both directions: fail closed when the mismatch arises during
generation, repair exactly once when found already on disk, and no spin
on the run after either. 9/9 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure fail-closed cannot catch, because generation SUCCEEDS.
This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long
before time has synced. `openssl req -x509` stamps notBefore from whatever the
clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be
years out: clock ahead -> clients reject it as "not yet valid", a harder failure
than the usual self-signed warning; clock behind -> notAfter is already in the
past once time syncs. The completion marker was then set and never revisited —
a node permanently serving a cert nothing accepts.
Finding 1, reported rather than assumed: this image does NOT use
systemd-timesyncd. It installs and enables chrony, and chrony-wait.service —
the unit that is Before=time-sync.target — is not enabled. So time-sync.target
is inert here and ordering After= it would buy nothing. Enabling chrony-wait to
make it meaningful would stall boot behind NTP on a node with no network, and
these nodes are routinely offline at first boot. Not deadlocking boot outranks
cert-date elegance, so the ordering is deliberately left alone.
Fixed locally instead, in two parts:
1. Backdate notBefore by 24h so ordinary skew between node and client cannot
invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and
the rootfs is debian:trixie which ships it, but the capability is PROBED,
not assumed — guessing wrong would fail every attempt and brick the node,
the exact outcome all of this exists to prevent. Without the flags we simply
do not backdate and rule 2 still covers the dangerous case.
2. Refuse to bless a cert dated by a clock outside a plausible window
(2026-01-01 .. 2056-01-01). The material stays installed so the node is
usable and sshd comes up, but the bad dates are recorded as
failed=cert-dates and the cert is regenerated automatically once time syncs.
Generation is now driven by need rather than by "is the marker absent", and
ConditionPathExists=! is removed from the unit so a node that already completed
can still be re-examined — skipping the unit is precisely how such a node stays
broken forever. The script exits in milliseconds when everything is fine.
Anti-spin is one condition: a date-driven regeneration happens ONLY when the
clock is currently plausible. A node whose clock is still wrong re-checks and
mints nothing.
Regression caught while writing this: driving generation purely by content made
needs_ssh() false whenever any host key existed, which would have left an
image-baked fleet-shared key in place forever — F-03 reopened. The marker check
is back in both needs_ functions and case 1 (which prestages a baked key and
asserts it was replaced) is what caught it.
Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin
directions. Controls: blessing regardless of clock reddens only case 8
(run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8
(SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially
passed against a broken guard because the assertion compared certificate dates,
and a re-mint under a frozen clock produces a byte-identical notBefore — the
assertion now counts mints, which is the only thing that distinguishes "left
alone" from "regenerated again".
Not covered here: nodes already deployed from earlier ISOs never receive this
script (it is installed by the installer, not by OTA), so fleet remediation for
them remains 10-04/OTA work in core/**, which is held by other executors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was unquoted, so the build
shell performed command substitution on the Dockerfile body. Any backtick in a
Dockerfile COMMENT was executed on the build host and its output spliced into
the generated file. Six comments did this. One of them ran
`systemctl start archipelago-fips.service` against the build machine on every
ISO build; the others were harmless only by accident of being command-not-found.
Fixes the class, not the six instances. The delimiter is now quoted, so the
body is emitted verbatim and a future backticked comment is inert. Verified the
boundary by line range first: the other backticked comments in this file
(:264, :809, :1188, :1289, :1506, :1605, :3597, :3651) are ordinary shell
comments outside any unquoted heredoc and were never at risk — they are
untouched.
The body needs exactly four build-time values and they are all package names
(LINUX_IMAGE_PKG, GRUB_EFI_PKG, GRUB_EFI_SIGNED_PKG, GRUB_PC_PKG), on four
consecutive lines. So quoting was practical: the heredoc is split into
DOCKERFILE_HEAD and DOCKERFILE_TAIL, both quoted, with a single explicit printf
interpolating those four names between them. Escapes that existed only because
the heredoc was unquoted are undone in the same pass: six trailing `\\` become
`\` (Docker line continuations) and four `\$` become `$` (RUN arguments reach
the shell verbatim — Docker does not substitute variables in RUN).
Verified by rendering the generated Dockerfile before and after with the same
inputs and diffing them normalised (continuations joined, whitespace
collapsed). Both are 190 normalised lines and the ONLY differences are the six
comments regaining their text — every instruction is byte-identical. Before:
"# the archipelago backend calls" / after: "# the archipelago backend calls
`systemctl start archipelago-fips.service`".
Test: case 7 asserts every heredoc writing Dockerfile.rootfs has a quoted
delimiter, and when one is not, reports which body lines would execute. The
assertion is on the delimiter, not on backticks — with quoting a backticked
comment is legal and six of them are back in the body on purpose, so flagging
backticks would flag a non-bug and fail on the very comments this restored.
This bug is invisible to `bash -n`; an instance of it introduced earlier in
this plan hung a syntactically-clean build for two minutes before being caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.
Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.
Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.
Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.
Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.
Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
marker set, failed units restarted. Control: dead-end on a node that already
failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
-> only case 6 red, naming the line.
Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rootfs is a container image exported to a tar and extracted verbatim onto
every disk flashed from the ISO, and the ISO is published. It baked two things
nobody asked for: Debian's openssh-server postinst generates /etc/ssh/ssh_host_*
during the container build, and the `openssl req` layer writes the TLS keypair.
Both were therefore identical on every node and known to every downloader.
Add a final RUN layer to Dockerfile.rootfs that removes /etc/ssh/ssh_host_*,
removes the archipelago TLS keypair (keeping the ssl directory so the first-boot
staging swap has somewhere to land), truncates /etc/machine-id to systemd's
documented "regenerate on next boot" state, and drops a non-shared
/var/lib/dbus/machine-id if one exists as a real file rather than a symlink.
It also writes /opt/archipelago/rootfs-identity-stripped so a node can answer
after the fact whether its rootfs came from a stripped build; no timestamp,
so the RECIPE_HASH cache stays reproducible.
This is what makes 10-03's fail-closed regeneration structural instead of
procedural: with the material gone, a regeneration failure degrades to
"no key, service refuses to start" rather than "fleet-shared key, silently".
The `openssl req` layer is deliberately left in place — it keeps proving
openssl is present and keeps the SAN template next to its consumer; the strip
layer is what makes the output non-shared.
Two comment corrections that follow from the strip:
- The installer's TLS block is no longer a rarely-taken safety net; it now
fires on every install. It is per-install and never image-wide, so it does
not reopen F-03, but it does mean a first-boot failure still leaves the web
UI with a cert while SSH has nothing. Comment updated to say so.
- The first-boot script header overstated the fail-closed cost for TLS for the
same reason; corrected to claim certainty only for SSH.
This edit is inside the RECIPE_HASH region, so the next build is forced to
rebuild the rootfs tar — required for the C-4 evidence to mean anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first-boot per-device secret regeneration was fail-open: both branches
logged a warning and continued, and `touch "$MARKER"` ran unconditionally
outside both `if` blocks. Combined with the unit's ConditionPathExists=! and
the script's own marker fast-path, one transient failure left that node on the
image-wide shared SSH host key and TLS private key permanently and silently —
and the ISO is a published artefact, so every downloader holds those keys.
- Retry each generator 3 times with backoff (D-05), so a transient first-boot
condition recovers inside the same boot instead of being terminal.
- Write the completion marker ONLY when both TLS and SSH succeeded, so a
failed boot leaves the unit eligible to run again on the next boot.
- On terminal failure: durable record at
/var/lib/archipelago/first-boot-secrets.failed naming which generator
failed, plus console + logger + stderr, and exit 1 so the unit lands in
`failed` rather than `active`. The record is cleared on a later success.
- Add FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF seams. Unset in
production the behaviour is byte-identical; set, they let the fail-closed
property be asserted rather than claimed.
- Order the unit After=systemd-random-seed.service (no-op today, correct if a
seed file is ever baked).
- State the operational trade in the script header: after the rootfs strip, a
terminal failure means no SSH and no TLS and needs the physical console.
That was chosen deliberately over running on fleet-shared keys.
tests/first-boot-secrets/run-tests.sh extracts the shipped heredoc body from
the builder and drives it against a temp root with stubbed generators: both
succeed, openssl fails every attempt, ssh-keygen fails twice then succeeds.
Moving the marker touch back outside the success branch makes case 2 fail with
MARKER-SET-ON-FAILURE, which is the regression this pins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both the 20G sparse qcow2 test disk and the serial console log were
hardcoded to /tmp regardless of $TMPDIR, so pointing the ISO release
build at a disk-backed scratch dir (to avoid tmpfs space pressure)
would not have covered this stage. Falls back to /tmp when TMPDIR is
unset — no behavior change for existing callers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v1.7.117 broke Reticulum mesh on OTA-only fleet nodes two ways: the
update swaps only the backend binary and frontend tarball, so nodes
kept a stale archy-reticulum-daemon whose argparse exits on the new
--enable-transport flag (mesh session died on every spawn — confirmed
on framework-pt), and they never had archy-rnodeconf at all, so the
in-app Flash LoRa flow failed with a bare "No such file or directory".
Four-part fix:
- The Rust supervisor probes `daemon --help` and only passes
--enable-transport when the daemon advertises it; unsupported daemons
run edge-only exactly as pre-1.7.117 (tested against stub daemons
both ways + missing-binary fail-safe).
- Both PyInstaller tools ride the frontend tarball's runtime payload
(radio-tools/) and bootstrap.rs promotes them to /usr/local/bin on
startup when bytes differ — the first OTA path that ever updates
them. create-release.sh now rebuilds them every release and the
manifest script hard-fails if they're missing.
- The ISO bundles archy-rnodeconf alongside the daemon (it never did).
- Flash LoRa reports "tool not installed — update the node" instead of
the bare spawn error when rnodeconf is absent everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both sides independently fixed the serial-alias dedup and the ESP32
boot-reset races; kept the branch's defer-to-auto-detect for unpinned
preferred paths (single probe pass per cycle) on top of main's
advert-name threading, Reticulum name propagation and radio-first
routing. Modal keeps main's 'Set Recommended' naming + probe progress
bar alongside the branch's in-app firmware flasher step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.
Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
the unit reaches 'active' in seconds; recovery/reconcile/listener
continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
the daemon back. Manual is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
instead of a tokio-sleep child of the process being stopped, whose
start-half was being lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- scripts/build-iso-release.sh: single gated command from signed release
to tested ISO (preflight version/signature parity, release gate harness,
strict catalog drift, full Rust suite, artifact version checks, build,
mount smoke, best-effort QEMU boot)
- scripts/iso-smoke-test.sh: standalone mount-level ISO verification incl.
stale-binary version assertion inside the payload
- .gitea/workflows/build-iso.yml: resurrected ISO CI as dispatch-only job
calling the gated orchestrator (old one was stranded in _archived/)
- tests/release/run.sh: catalog drift now runs --strict (was silently
always-pass)
- test-iso-qemu.sh: headless mode used -append without -kernel, which
QEMU rejects; use -display none so -serial file: capture works
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phones on the FIPS mesh reach the node at http://[<fips0 ULA>], but every
shipped nginx config listened on IPv4 only — nothing answered [::]:80/443,
so mesh HTTP failed with the tunnel fully healthy (found live on
framework-pt during the vc24 5G session, 2026-07-23). Canonical conf gains
listen [::] lines and patch_nginx_conf self-heals deployed nodes (covers
the sites-enabled -> archipelago-http dev variant via the existing
canonicalize pass); validated by the existing nginx -t + rollback flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--enable-features=OverlayScrollbar gives the thin auto-hiding scrollbars a
remote browser shows (Peers view had a permanent classic scrollbar on the
TV). Rides the existing kiosk self-heal + ISO splice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh listener spawns /usr/local/bin/archy-reticulum-daemon for
Reticulum sticks, but the ISO never shipped it (framework-pt connected
its RNode only after a hand-copy, 2026-07-22). Bundled from the build
host with a loud warning when absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kiosk image ships no color-emoji font, so the wallet card's 🥜 and ⚓
drew as empty squares on TVs. Swap them for inline stroke SVGs matching
the neighbouring rows (wallet card + scan modal), and add
fonts-noto-color-emoji to the ISO package set so remaining emoji across
the UI (peer files, chat, content) render on future installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kiosk window was sized in physical pixels, but Chromium interprets
--window-size in DIPs: with any device-scale > 1 the window painted
scale-times larger than the panel, and with no window manager in the
kiosk X session --kiosk/--start-fullscreen could not snap it back. On a
4K TV at scale 3 that meant an 11520x6480 window — 9x the panel area —
which blew the compositor tile-memory budget ("tile memory limits
exceeded, some content may not draw") and froze the screen on the last
painted frame; smaller scales cropped content off the right/bottom.
- Size the window in DIPs (panel / scale) so DIPs x scale = the panel.
- Panels >=2560 wide now target a 1920-wide layout (4K -> scale 2.0,
validated on the Framework's 72" 4K TV: spacious desktop layout at 2x
sharpness); smaller panels keep the 1280 target (1080p TV -> 1.5).
- Optional ARCHIPELAGO_KIOSK_MAX_WIDTH cap: drop to the best <=N-wide
xrandr mode for weak hardware (TV upscales instead of us rasterizing).
- Source /etc/archipelago/kiosk-display.conf so the upcoming Settings
display-resolution picker can manage these knobs per node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fresh-install kiosk issues found on a Framework node:
- MOTD said 'Ctrl+Alt+F7 for kiosk' / 'F1 for terminal', but the kiosk's
Xorg runs on vt1 — so F7 was an empty black VT (the reported black screen
and the failed 'return to kiosk'). Corrected: kiosk=F1, terminal=F2.
- The UTF-8 block-glyph logo corrupted intermittently because returning from
the kiosk can leave the console VT out of UTF-8 mode; force it with ESC%G.
- A short power-button press shut the node down instantly. Ignore the short
press, keep long-press poweroff, so accidental brushes don't kill the node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AIUI themes via @media (prefers-color-scheme) and falls back to its light
variant (white panels) when the browser reports no preference — which the
minimal kiosk X session does, so AIUI came up light instead of its designed
dark. The main UI hardcodes dark and is unaffected. Push two independent
dark signals on the kiosk Chromium: GTK_THEME=Adwaita:dark (version-
independent, can never force light) and --blink-settings=preferredColorScheme=0
(0 = kDark in modern Chromium). Needs on-device kiosk verification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The relay image was rebuilt to the standard nostr-rs-relay layout — WORKDIR
/usr/src/app, execs ./nostr-rs-relay — but the build still cp'd from the old
/usr/local/bin path, so extraction silently produced nothing. Once the
missing-binary guard was added, that silent miss became a hard build
failure. Verified the binary is a valid amd64 ELF (GLIBC 2.34, under Debian
13's 2.40) at /usr/src/app/nostr-rs-relay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO build hard-failed at 'Required binaries not extracted: nvpn'
because it still pulled 146.59.87.168:3000/lfg2025/nostr-vpn:v0.3.7 — an
image that was deleted from the registry when NostrVPN was removed from
the product. The native nvpn daemon is already masked here (ln -sf
/dev/null nostr-vpn.service) and is never spawned at runtime (core vpn.rs
manages WireGuard/Tailscale, only reading a legacy nvpn config as
fallback), so extracting its binary was dead weight that bricked every
build once the image went away.
Drop the nvpn extraction and remove it from the missing-binary guard.
nostr-rs-relay stays required — its nostr-relay unit is still enabled and
the image still pulls. Change is in STEP 3, so the cached rootfs is
unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO built FIPS from an unpinned --depth 1 clone of upstream main, so
every build shipped whatever happened to be on main that day. Pin to
v0.4.1 via a FIPS_VERSION build arg — the version fips/config.rs renders
its typed config against, and the one validated in the field on .198 and
the thinkpad. The pin lands inside the STEP 1 recipe-hash range, so the
cached rootfs correctly invalidates on the next build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend: tor.list-services now hides services whose name maps to a
known-but-uninstalled catalog app (aliases covered: bitcoin/knots/core,
electrumx/electrs, btcpay, mempool). The node's own service, the relay,
and custom user services always show.
ISO: first-boot tor setup no longer pre-creates hidden services for six
apps that may never be installed — only the node's own onion is baked.
It also only seeds services.json/torrc on FIRST boot instead of
clobbering backend-managed services on every boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.7.104 shipped ISOs whose units crash-looped (nostr-relay 3s loop,
archipelago-diag 203/EXEC) because build-time extraction failures were mere
warnings and the cached rootfs never tracked its recipe:
- hash the rootfs-defining region of the build script and rebuild when it
changes (stale cache shipped ISOs without wpasupplicant/iw/rfkill)
- refuse to build without nvpn / nostr-rs-relay unless
ALLOW_MISSING_VPN_BINARIES=1
- ConditionPathExists on nostr-relay/nostr-vpn/diag units so a stripped
image skips them instead of crash-looping
- post-install test: every enabled archipelago*/nostr* unit must have an
existing ExecStart payload
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.
Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundling fmcd for offline installs (6dcdada3) shipped the raw
uncompressed podman save tar and grew the ISO ~220MB (2.3G -> 2.5G in
RC6-RC9). Save the core bundle as .tar.zst instead — podman load
auto-detects compression, verified locally against the RC9 fmcd.tar
(228M -> 65M, loads cleanly into rootless storage). The first-boot
loader and installer copy globs now pick up .tar.zst too.
Also trims the installer's USB->disk image copy and first-boot load
time on fresh installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chromium's SingletonLock is a symlink encoding <hostname>-<pid>. After a
node rename changes the OS hostname, the lock left from the previous
boot reads as "another computer" holding the profile; Chromium refuses
to start, --noerrdialogs hides the only dialog, and the kiosk
black-screens forever.
Two layers:
- kiosk launcher removes Singleton{Lock,Cookie,Socket} before every
Chromium start (any prior owner is dead at that point — pkill at
session start / loop respawn). Ships via ISO splice AND the binary's
include_str! self-heal, so existing nodes get it with the next OTA.
- the rename handler clears the lock immediately as a hostname
side-effect, covering nodes still running a pre-fix launcher.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 454c4bb2 kiosk flattening only covered the Dashboard's background
stack, but the kiosk boots into the login/onboarding screen —
OnboardingWrapper.vue — whose scoped 3D stack (perspective +
preserve-3d bg layers + mix-blend-mode glitch overlays) has the exact
compositing pattern that fails to repaint under the kiosk's software
compositor, leaving the black body fill visible. Extend the same
html.kiosk-mode flattening to this screen via :global() rules: 2D
opacity crossfades stay, 3D transforms / blur filters / blend-mode
glitch layers go.
Also raise the kiosk chromium JS heap cap 128→256MB: the intro
cinematic (video + audio + app) inside a 128MB V8 old-space risks
renderer OOM, and a crashed kiosk tab auto-reloads — indistinguishable
from the reported "flickers and refreshes".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuilds of the same version overwrote the previous ISO under an
identical filename, making a flashed stick indistinguishable from a
newer build. The builder now keeps a per-version RC counter (next to
the build counter) and names outputs like
archipelago-installer-1.7.101-alpha-unbundled-x86_64_RC2.iso.
Override with RC=n env when re-cutting a specific candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh offline installs came up without fedimint-clientd and filebrowser
(they only appeared after connecting internet). Three root causes:
- archipelago-load-images.service ran 'podman load' as root, but every
container runs rootless as archipelago — bundled images landed in
root's storage where the rootless runtime can't see them, so all
container creation silently depended on registry pulls. The loader now
loads into the archipelago user's storage (with linger + runtime-dir
wait + system migrate).
- The unbundled ISO bundled only filebrowser.tar; fmcd (fedimint-clientd)
is a baseline first-boot app too and is now part of the unbundled core
bundle.
- first-boot's pull_with_fallback always hit the network; it now uses an
already-loaded local image first and skips the pull entirely.
Also: fedimint-clientd added to the UI's hardcoded curated-app fallback
list (it was missing when all catalog fetches fail offline), plus its
INSTALLED_ALIASES entry, and the stale fmcd bundling comment in
image-versions.sh corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unit's ExecStartPre baked the first hostname -I token into
host-ip.env, which config.rs prefers over its own detection — once a
VPN/bridge interface existed (netbird's wg tunnel), every host_ip
consumer (install configs, VPN, mint URLs) got the tunnel IP. Read the
main-table default route's src address instead, falling back to
hostname -I on routeless hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO's rootfs tar is a cached build artifact (currently 2026-04-20) and
its baked-in archipelago.service predates the B17 fix — no
RequiresMountsFor=/var/lib/archipelago — while auto-install.sh only copied
the configs/ unit when the rootfs had none. Every fresh install therefore
raced the LUKS data mount on cold boot and looped
'[FAILED] Failed to start archipelago.service' for minutes (seen on the
1.7.99-alpha ISO).
Ship configs/archipelago.service on the ISO at archipelago/configs/ and
make the installer overwrite the rootfs copy unconditionally. Also add
RequiresMountsFor to the legacy install-to-disk.sh inline unit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The X250 symptom was the UI rendered at a hardcoded 1920x1080 and clipped
on the 1366x768 panel (fixed in 7e996d6), not eDP corruption. Keep RC2
minimal: no speculative kernel params.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThinkPad X250 (Broadwell HD 5500) internal panels show horizontal corruption
bands at the kiosk; known i915 eDP power-feature issue. Boot installed and
live systems with i915.enable_ips=0 enable_psr=0 enable_fbc=0 (no-ops off
Intel, nothing lost on a kiosk). Debug boot entries stay stock for A/B.
Also size the Chromium kiosk window from the detected mode instead of a
hardcoded 1920x1080 that clips on 1366x768 panels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The insecure-registry config was written to /etc unconditionally; as the
archipelago build user the redirection fails and set -e aborts check_tools
before Step 1. Write to the rootless podman per-user config dir when not
root, tolerate failure (nodes may already carry the config), and drop the
duplicated 146.59.87.168:3000 entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b8002885 moved the archived builder's repo-root refs to ../../ (correct for
direct _archived/ execution) but the wrapper pins SCRIPT_DIR to image-recipe/
and only rewrote the ../../scripts form. core, neode-ui, docker, apps, demo,
web and the $(cd "$SCRIPT_DIR/../..") expressions all resolved one level
above the repo, so wrapper builds ran with an empty BUILD_VERSION and would
have failed at the backend/web-UI steps. Rewrite the ../../ prefix generically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BUILD_VERSION/EXPECTED_VERSION/LOCAL_RELEASE, the backend container build
context, and the neode-ui/web-dist/apps/demo/docker copies all resolved
one level short (image-recipe/ instead of the repo root), so the builder
never found the prebuilt binary and its container fallback died on
'COPY core ./core'. configs/ references were already correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ssmithx