Build #189 proved the IPv4 fix works (downloads all succeeded) and proved
my own edit wrong: appending inet4_only to /etc/wgetrc before wget's
package landed made dpkg's conffile prompt hit EOF, leaving wget and
debootstrap unconfigured. Moved the pin below the apt install.
Co-Authored-By: Claude <noreply@anthropic.com>
This box (and its containers) blackhole IPv6: deb.debian.org answers AAAA
first, wget tries v6 until debootstrap's per-package timeout, and the
installer-env stage died twice today with 'Couldn't download packages'.
Probed in a debian:trixie container: v4 OK, v6 hangs. inet4_only for wget
(covers debootstrap) + Acquire::ForceIPv4 for every apt-get, including the
chroot and the rootfs Dockerfile stages.
Co-Authored-By: Claude <noreply@anthropic.com>
RC1's aiui/index.html pointed at the stale checked-in demo bundle while
today's dist sat beside it unreferenced: demo/aiui was tried first, and the
rsync without --delete merged it over the fresh capture from /opt. Now the
newest index.html across all candidates wins (demo/aiui remains the
fresh-clone fallback) and the copy deletes before writing.
Found by mounting the RC1 ISO and diffing bundle hashes against the tree —
exactly the 'verify the frontend INSIDE the ISO' rule.
Co-Authored-By: Claude <noreply@anthropic.com>
S4: /aiui/api/web-search proxied straight to SearXNG with no auth — anyone
reaching the web port ran searches attributed to the node's IP. Now routed
through the daemon's session-gated model proxy like the claude/ollama legs
(both nginx server blocks), forcing format=json upstream (the client never
sent it — search could 200 with HTML that parsed as nothing).
S3: the forwarder also serves the STANDALONE frontend, whose bodies carry
full history/images with no assistant loop behind them — a pasted seed
phrase went to Anthropic unscreened. The forwarder now runs the egress
secret-shape scan (G-B1) with the node's own secrets dir as deny corpus on
Claude bodies and search queries; blocked requests get a plain-language 400.
Also fixes a REAL gap in the egress tokenizer found by these tests: a JSON
key glued to a string value's first word ('content":"abandon...') dropped
that word, so an exactly-12-word seed pasted as a bare message yielded an
11-member run — checksum misses, backstop misses. Non-member words now
rescan within the token. egress 15/15 + model_proxy 10/10 green.
Co-Authored-By: Claude <noreply@anthropic.com>
The node served a bare self-signed leaf, so a browser exception had to be
granted per ORIGIN — scheme + host + port. The dashboard on :443 and an app
on :8334 are different origins, and a certificate interstitial CANNOT be
accepted inside an iframe, so a gated app embedded over HTTPS could never
render no matter how many warnings the user clicked through. (Mixed content
blocks the plain-HTTP variant first, before the SameSite cookie question the
symptom was originally filed under.)
A CA fixes it structurally: ports are not part of a certificate's identity, so
one leaf with the right SANs covers every port on the host, and one installed
CA trusts them all.
- scripts/setup-node-ca.sh generates the CA (4096-bit, pathlen:0, keyCertSign
only) and issues a 397-day leaf covering archipelago.local, the hostname, the
Tailscale MagicDNS name and every global address the host holds. Idempotent —
re-running reuses the CA and only reissues the leaf, so gaining an address
does not invalidate copies users already installed. --force-ca is the
deliberate escape hatch and says what it costs.
- nginx serves the public CA at /ca.crt on both schemes, unauthenticated by
design: a device fetches it before it can validate the node, so gating it
behind HTTPS or a login would be a chicken-and-egg.
- Settings → System shows the fingerprint and per-platform install steps.
crypto.subtle does not exist outside a secure context — precisely the case
this feature exists to fix — so an HTTP dashboard gets the openssl command
to verify by hand instead of a blank field.
Verified locally: chain validates, key pairs with the leaf, CA:TRUE/CA:FALSE
are correct, keys are 0600. Two TLS servers on different ports both verify
(ssl_verify_result=0) against the CA alone and are rejected without it — the
one-CA-covers-every-port claim, tested rather than assumed.
Not yet wired: app ports still serve plain HTTP. Putting TLS on them is the
next step and is what actually closes the iframe-login bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a Content-Security-Policy header to both nginx `location /aiui/`
blocks whose connect-src is scoped to the AIUI path prefix, so AIUI's
own JavaScript is browser-prevented from issuing a same-origin fetch
to /rpc/v1 with the ambient session cookie. Explicitly rejects the
`sandbox` iframe attribute (allow-scripts + allow-same-origin is the
known escape; dropping allow-same-origin breaks AIUI's storage and
its origin-checked bridge) and records why in both the nginx comment
and a new comment above the Chat.vue iframe. Adds
referrerpolicy="no-referrer" to the iframe so a media URL or page path
never leaks upstream via Referer.
Also adds an explicit `location /aiui/api/openrouter/ { return 404; }`
to both server blocks, closing 13-02's Task 3 checkpoint finding
(operator-accepted deviation 2026-08-03): the relay was already
structurally gone but the SPA catch-all served 200/405 instead of 404.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the live production exposure this plan targets: /aiui/api/claude/
and /aiui/api/ollama/ proxied straight through with "no session gate
needed", and /aiui/api/openrouter/ was a plain unauthenticated relay to a
paid third-party API the node holds no key for (T-13-08/T-13-09/T-13-10).
image-recipe/configs/nginx-archipelago.conf (BOTH server blocks, ~line 49
and ~line 961 — a fix applied to only one leaves the exposure live on
whichever block serves the request, T-13-15):
- /aiui/api/claude/ and /aiui/api/ollama/ proxy_pass re-pointed from
127.0.0.1:3142 / 127.0.0.1:11434 to the Rust daemon at 127.0.0.1:5678
(no trailing path component, so the daemon's own prefix match sees the
full request URI)
- Forward the session Cookie header to the daemon so it can re-derive auth
- location /aiui/api/openrouter/ deleted outright in both blocks
- Old comment "API key managed by proxy, no session gate needed" (the
reasoning error that produced the exposure) replaced with rationale
scripts/deploy-to-target.sh: deleted the embedded claude-api-proxy.py
heredoc, its systemd unit creation/enable/restart, the ANTHROPIC_API_KEY
extraction, and the 3141->3142 sed fixups. Added an unconditional step that
stops/disables/removes any pre-existing claude-api-proxy unit and deletes
/opt/archipelago/claude-api-proxy.py and
<data_dir>/secrets/claude-api-proxy.env on every deploy — so
already-provisioned nodes actually lose the old unauthenticated listener,
not just newly-deployed ones.
scripts/setup-aiui-server.sh: dropped the hard ANTHROPIC_API_KEY
requirement and the patch-nginx-claude.py step; the script's remaining job
is the AIUI dist rsync. (Also drops the FileBrowser-fix step that lived
here — that logic already exists, and is kept, in deploy-to-target.sh; this
script narrows to exactly what its rewritten header now says it does.)
core/archipelago/src/api/rpc/system/handlers.rs: `claude_api_key` setting
branch no longer writes a second key copy to secrets/claude-api-proxy.env
or restarts claude-api-proxy. secrets/claude-api-key (0600) remains the
single ledger, with a comment naming it as such.
`cargo build --package archipelago` succeeds. Verified via grep against
every acceptance criterion in 13-02-PLAN.md's Task 2 (openrouter count 0,
3142 gone from nginx, both location blocks present, PORT=3142 gone,
claude-api-proxy gone from handlers.rs, secrets/claude-api-key present).
Task 3 (real-node curl/systemd verification, S-15) is NOT done in this
commit — see 13-02-SUMMARY.md.
Continues WIP checkpoint 13b576da (reset --soft, recommitted atomically
per task per plan protocol).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Taking over a parked item from the app-UI work. AIUI's built index.html
emits ABSOLUTE /assets/<hashed> paths, so the browser asks for
/assets/index-BC2fBBaW.js. That lands in the MAIN UI's assets dir, where
it does not exist — the real files are in aiui/assets/. Both of AIUI's
two entry assets 404'd, so the embedded sidebar loaded nothing.
The config already contained a /aiui-assets/ location whose comment names
this exact problem ("AIUI may reference /assets/ without /aiui/ prefix"),
but it only catches requests to /aiui-assets/, a path AIUI never asks
for. It described the bug without fixing it.
/assets/ now falls back to a named location that rewrites into
aiui/assets/ and 404s from there. A fallback rather than copying the two
files up one level, because a frontend deploy replaces web-ui wholesale —
update.rs preserves the aiui/ DIRECTORY, not copies made into assets/ —
so a copy is erased by the very next deploy while this survives one.
Both server blocks (HTTP and HTTPS) are patched; named locations are
per-server, so each needs its own.
Verified on archi-dev-box after reload:
/assets/index-BC2fBBaW.js 200, 305256 bytes, application/javascript
/assets/index-BJkaQ2c4.css 200, 150716 bytes, text/css
/assets/does-not-exist.js 404 (the fallback is not over-broad)
/assets/index--lyLAgu1.js 200 (real main-UI chunks still come from
/assets/vendor-CmYeCqL_.js 200 the main dir — try_files hits them
/assets/index-CiMaoNII.css 200 before the fallback is consulted)
/ /aiui/ /health 200
Hash collision between the two builds is not a concern: Vite hashes are
content-derived, and any main-UI asset that exists is served by try_files
before the fallback runs.
Noted while doing this, not fixed here: the node's own
/etc/nginx/sites-enabled/archipelago is 378 lines BEHIND this repo file
(984 vs 1362) — it predates the IPv6 listener and the @asset_missing
no-store handling, among others. The node was patched minimally in its
own shape rather than overwritten, since a wholesale copy of a config
this diverged is not a safe unattended action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parsing each half back proves each is well-formed; it never proves they
belong together. A key from one generation beside a cert from another
passes both individual parse checks, gets blessed, and then nginx refuses
to start at the exact moment the marker claims first boot succeeded.
gen_tls() now extracts the public key from each half and compares them
before the swap, and needs_tls() applies the same check to what is already
installed, so a mismatched pair that reached disk some other way (an older
build, a half-finished manual edit) is repaired instead of quietly
breaking nginx. Extraction subsumes parsing, so this replaces the separate
-noout parse checks rather than adding to them.
Kept deliberately in step with regenerate_tls_cert() in
core/archipelago/src/api/rpc/system/handlers.rs, which does the same
comparison on the running node after a rename.
Test harness: the openssl stub keypair now carries the generation it came
from, and STUB_OPENSSL_MISMATCH emits a cert from a different one — the
pair that passes both parse checks and still breaks nginx. New case 9
covers both directions: fail closed when the mismatch arises during
generation, repair exactly once when found already on disk, and no spin
on the run after either. 9/9 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure fail-closed cannot catch, because generation SUCCEEDS.
This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long
before time has synced. `openssl req -x509` stamps notBefore from whatever the
clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be
years out: clock ahead -> clients reject it as "not yet valid", a harder failure
than the usual self-signed warning; clock behind -> notAfter is already in the
past once time syncs. The completion marker was then set and never revisited —
a node permanently serving a cert nothing accepts.
Finding 1, reported rather than assumed: this image does NOT use
systemd-timesyncd. It installs and enables chrony, and chrony-wait.service —
the unit that is Before=time-sync.target — is not enabled. So time-sync.target
is inert here and ordering After= it would buy nothing. Enabling chrony-wait to
make it meaningful would stall boot behind NTP on a node with no network, and
these nodes are routinely offline at first boot. Not deadlocking boot outranks
cert-date elegance, so the ordering is deliberately left alone.
Fixed locally instead, in two parts:
1. Backdate notBefore by 24h so ordinary skew between node and client cannot
invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and
the rootfs is debian:trixie which ships it, but the capability is PROBED,
not assumed — guessing wrong would fail every attempt and brick the node,
the exact outcome all of this exists to prevent. Without the flags we simply
do not backdate and rule 2 still covers the dangerous case.
2. Refuse to bless a cert dated by a clock outside a plausible window
(2026-01-01 .. 2056-01-01). The material stays installed so the node is
usable and sshd comes up, but the bad dates are recorded as
failed=cert-dates and the cert is regenerated automatically once time syncs.
Generation is now driven by need rather than by "is the marker absent", and
ConditionPathExists=! is removed from the unit so a node that already completed
can still be re-examined — skipping the unit is precisely how such a node stays
broken forever. The script exits in milliseconds when everything is fine.
Anti-spin is one condition: a date-driven regeneration happens ONLY when the
clock is currently plausible. A node whose clock is still wrong re-checks and
mints nothing.
Regression caught while writing this: driving generation purely by content made
needs_ssh() false whenever any host key existed, which would have left an
image-baked fleet-shared key in place forever — F-03 reopened. The marker check
is back in both needs_ functions and case 1 (which prestages a baked key and
asserts it was replaced) is what caught it.
Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin
directions. Controls: blessing regardless of clock reddens only case 8
(run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8
(SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially
passed against a broken guard because the assertion compared certificate dates,
and a re-mint under a frozen clock produces a byte-identical notBefore — the
assertion now counts mints, which is the only thing that distinguishes "left
alone" from "regenerated again".
Not covered here: nodes already deployed from earlier ISOs never receive this
script (it is installed by the installer, not by OTA), so fleet remediation for
them remains 10-04/OTA work in core/**, which is held by other executors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was unquoted, so the build
shell performed command substitution on the Dockerfile body. Any backtick in a
Dockerfile COMMENT was executed on the build host and its output spliced into
the generated file. Six comments did this. One of them ran
`systemctl start archipelago-fips.service` against the build machine on every
ISO build; the others were harmless only by accident of being command-not-found.
Fixes the class, not the six instances. The delimiter is now quoted, so the
body is emitted verbatim and a future backticked comment is inert. Verified the
boundary by line range first: the other backticked comments in this file
(:264, :809, :1188, :1289, :1506, :1605, :3597, :3651) are ordinary shell
comments outside any unquoted heredoc and were never at risk — they are
untouched.
The body needs exactly four build-time values and they are all package names
(LINUX_IMAGE_PKG, GRUB_EFI_PKG, GRUB_EFI_SIGNED_PKG, GRUB_PC_PKG), on four
consecutive lines. So quoting was practical: the heredoc is split into
DOCKERFILE_HEAD and DOCKERFILE_TAIL, both quoted, with a single explicit printf
interpolating those four names between them. Escapes that existed only because
the heredoc was unquoted are undone in the same pass: six trailing `\\` become
`\` (Docker line continuations) and four `\$` become `$` (RUN arguments reach
the shell verbatim — Docker does not substitute variables in RUN).
Verified by rendering the generated Dockerfile before and after with the same
inputs and diffing them normalised (continuations joined, whitespace
collapsed). Both are 190 normalised lines and the ONLY differences are the six
comments regaining their text — every instruction is byte-identical. Before:
"# the archipelago backend calls" / after: "# the archipelago backend calls
`systemctl start archipelago-fips.service`".
Test: case 7 asserts every heredoc writing Dockerfile.rootfs has a quoted
delimiter, and when one is not, reports which body lines would execute. The
assertion is on the delimiter, not on backticks — with quoting a backticked
comment is legal and six of them are back in the body on purpose, so flagging
backticks would flag a non-bug and fail on the very comments this restored.
This bug is invisible to `bash -n`; an instance of it introduced earlier in
this plan hung a syntactically-clean build for two minutes before being caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.
Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.
Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.
Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.
Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.
Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
marker set, failed units restarted. Control: dead-end on a node that already
failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
-> only case 6 red, naming the line.
Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rootfs is a container image exported to a tar and extracted verbatim onto
every disk flashed from the ISO, and the ISO is published. It baked two things
nobody asked for: Debian's openssh-server postinst generates /etc/ssh/ssh_host_*
during the container build, and the `openssl req` layer writes the TLS keypair.
Both were therefore identical on every node and known to every downloader.
Add a final RUN layer to Dockerfile.rootfs that removes /etc/ssh/ssh_host_*,
removes the archipelago TLS keypair (keeping the ssl directory so the first-boot
staging swap has somewhere to land), truncates /etc/machine-id to systemd's
documented "regenerate on next boot" state, and drops a non-shared
/var/lib/dbus/machine-id if one exists as a real file rather than a symlink.
It also writes /opt/archipelago/rootfs-identity-stripped so a node can answer
after the fact whether its rootfs came from a stripped build; no timestamp,
so the RECIPE_HASH cache stays reproducible.
This is what makes 10-03's fail-closed regeneration structural instead of
procedural: with the material gone, a regeneration failure degrades to
"no key, service refuses to start" rather than "fleet-shared key, silently".
The `openssl req` layer is deliberately left in place — it keeps proving
openssl is present and keeps the SAN template next to its consumer; the strip
layer is what makes the output non-shared.
Two comment corrections that follow from the strip:
- The installer's TLS block is no longer a rarely-taken safety net; it now
fires on every install. It is per-install and never image-wide, so it does
not reopen F-03, but it does mean a first-boot failure still leaves the web
UI with a cert while SSH has nothing. Comment updated to say so.
- The first-boot script header overstated the fail-closed cost for TLS for the
same reason; corrected to claim certainty only for SSH.
This edit is inside the RECIPE_HASH region, so the next build is forced to
rebuild the rootfs tar — required for the C-4 evidence to mean anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first-boot per-device secret regeneration was fail-open: both branches
logged a warning and continued, and `touch "$MARKER"` ran unconditionally
outside both `if` blocks. Combined with the unit's ConditionPathExists=! and
the script's own marker fast-path, one transient failure left that node on the
image-wide shared SSH host key and TLS private key permanently and silently —
and the ISO is a published artefact, so every downloader holds those keys.
- Retry each generator 3 times with backoff (D-05), so a transient first-boot
condition recovers inside the same boot instead of being terminal.
- Write the completion marker ONLY when both TLS and SSH succeeded, so a
failed boot leaves the unit eligible to run again on the next boot.
- On terminal failure: durable record at
/var/lib/archipelago/first-boot-secrets.failed naming which generator
failed, plus console + logger + stderr, and exit 1 so the unit lands in
`failed` rather than `active`. The record is cleared on a later success.
- Add FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF seams. Unset in
production the behaviour is byte-identical; set, they let the fail-closed
property be asserted rather than claimed.
- Order the unit After=systemd-random-seed.service (no-op today, correct if a
seed file is ever baked).
- State the operational trade in the script header: after the rootfs strip, a
terminal failure means no SSH and no TLS and needs the physical console.
That was chosen deliberately over running on fleet-shared keys.
tests/first-boot-secrets/run-tests.sh extracts the shipped heredoc body from
the builder and drives it against a temp root with stubbed generators: both
succeed, openssl fails every attempt, ssh-keygen fails twice then succeeds.
Moving the marker touch back outside the success branch makes case 2 fail with
MARKER-SET-ON-FAILURE, which is the regression this pins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both the 20G sparse qcow2 test disk and the serial console log were
hardcoded to /tmp regardless of $TMPDIR, so pointing the ISO release
build at a disk-backed scratch dir (to avoid tmpfs space pressure)
would not have covered this stage. Falls back to /tmp when TMPDIR is
unset — no behavior change for existing callers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v1.7.117 broke Reticulum mesh on OTA-only fleet nodes two ways: the
update swaps only the backend binary and frontend tarball, so nodes
kept a stale archy-reticulum-daemon whose argparse exits on the new
--enable-transport flag (mesh session died on every spawn — confirmed
on framework-pt), and they never had archy-rnodeconf at all, so the
in-app Flash LoRa flow failed with a bare "No such file or directory".
Four-part fix:
- The Rust supervisor probes `daemon --help` and only passes
--enable-transport when the daemon advertises it; unsupported daemons
run edge-only exactly as pre-1.7.117 (tested against stub daemons
both ways + missing-binary fail-safe).
- Both PyInstaller tools ride the frontend tarball's runtime payload
(radio-tools/) and bootstrap.rs promotes them to /usr/local/bin on
startup when bytes differ — the first OTA path that ever updates
them. create-release.sh now rebuilds them every release and the
manifest script hard-fails if they're missing.
- The ISO bundles archy-rnodeconf alongside the daemon (it never did).
- Flash LoRa reports "tool not installed — update the node" instead of
the bare spawn error when rnodeconf is absent everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both sides independently fixed the serial-alias dedup and the ESP32
boot-reset races; kept the branch's defer-to-auto-detect for unpinned
preferred paths (single probe pass per cycle) on top of main's
advert-name threading, Reticulum name propagation and radio-first
routing. Modal keeps main's 'Set Recommended' naming + probe progress
bar alongside the branch's in-app firmware flasher step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.
Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
the unit reaches 'active' in seconds; recovery/reconcile/listener
continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
the daemon back. Manual is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
instead of a tokio-sleep child of the process being stopped, whose
start-half was being lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- scripts/build-iso-release.sh: single gated command from signed release
to tested ISO (preflight version/signature parity, release gate harness,
strict catalog drift, full Rust suite, artifact version checks, build,
mount smoke, best-effort QEMU boot)
- scripts/iso-smoke-test.sh: standalone mount-level ISO verification incl.
stale-binary version assertion inside the payload
- .gitea/workflows/build-iso.yml: resurrected ISO CI as dispatch-only job
calling the gated orchestrator (old one was stranded in _archived/)
- tests/release/run.sh: catalog drift now runs --strict (was silently
always-pass)
- test-iso-qemu.sh: headless mode used -append without -kernel, which
QEMU rejects; use -display none so -serial file: capture works
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phones on the FIPS mesh reach the node at http://[<fips0 ULA>], but every
shipped nginx config listened on IPv4 only — nothing answered [::]:80/443,
so mesh HTTP failed with the tunnel fully healthy (found live on
framework-pt during the vc24 5G session, 2026-07-23). Canonical conf gains
listen [::] lines and patch_nginx_conf self-heals deployed nodes (covers
the sites-enabled -> archipelago-http dev variant via the existing
canonicalize pass); validated by the existing nginx -t + rollback flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--enable-features=OverlayScrollbar gives the thin auto-hiding scrollbars a
remote browser shows (Peers view had a permanent classic scrollbar on the
TV). Rides the existing kiosk self-heal + ISO splice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh listener spawns /usr/local/bin/archy-reticulum-daemon for
Reticulum sticks, but the ISO never shipped it (framework-pt connected
its RNode only after a hand-copy, 2026-07-22). Bundled from the build
host with a loud warning when absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kiosk image ships no color-emoji font, so the wallet card's 🥜 and ⚓
drew as empty squares on TVs. Swap them for inline stroke SVGs matching
the neighbouring rows (wallet card + scan modal), and add
fonts-noto-color-emoji to the ISO package set so remaining emoji across
the UI (peer files, chat, content) render on future installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kiosk window was sized in physical pixels, but Chromium interprets
--window-size in DIPs: with any device-scale > 1 the window painted
scale-times larger than the panel, and with no window manager in the
kiosk X session --kiosk/--start-fullscreen could not snap it back. On a
4K TV at scale 3 that meant an 11520x6480 window — 9x the panel area —
which blew the compositor tile-memory budget ("tile memory limits
exceeded, some content may not draw") and froze the screen on the last
painted frame; smaller scales cropped content off the right/bottom.
- Size the window in DIPs (panel / scale) so DIPs x scale = the panel.
- Panels >=2560 wide now target a 1920-wide layout (4K -> scale 2.0,
validated on the Framework's 72" 4K TV: spacious desktop layout at 2x
sharpness); smaller panels keep the 1280 target (1080p TV -> 1.5).
- Optional ARCHIPELAGO_KIOSK_MAX_WIDTH cap: drop to the best <=N-wide
xrandr mode for weak hardware (TV upscales instead of us rasterizing).
- Source /etc/archipelago/kiosk-display.conf so the upcoming Settings
display-resolution picker can manage these knobs per node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fresh-install kiosk issues found on a Framework node:
- MOTD said 'Ctrl+Alt+F7 for kiosk' / 'F1 for terminal', but the kiosk's
Xorg runs on vt1 — so F7 was an empty black VT (the reported black screen
and the failed 'return to kiosk'). Corrected: kiosk=F1, terminal=F2.
- The UTF-8 block-glyph logo corrupted intermittently because returning from
the kiosk can leave the console VT out of UTF-8 mode; force it with ESC%G.
- A short power-button press shut the node down instantly. Ignore the short
press, keep long-press poweroff, so accidental brushes don't kill the node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AIUI themes via @media (prefers-color-scheme) and falls back to its light
variant (white panels) when the browser reports no preference — which the
minimal kiosk X session does, so AIUI came up light instead of its designed
dark. The main UI hardcodes dark and is unaffected. Push two independent
dark signals on the kiosk Chromium: GTK_THEME=Adwaita:dark (version-
independent, can never force light) and --blink-settings=preferredColorScheme=0
(0 = kDark in modern Chromium). Needs on-device kiosk verification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The relay image was rebuilt to the standard nostr-rs-relay layout — WORKDIR
/usr/src/app, execs ./nostr-rs-relay — but the build still cp'd from the old
/usr/local/bin path, so extraction silently produced nothing. Once the
missing-binary guard was added, that silent miss became a hard build
failure. Verified the binary is a valid amd64 ELF (GLIBC 2.34, under Debian
13's 2.40) at /usr/src/app/nostr-rs-relay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO build hard-failed at 'Required binaries not extracted: nvpn'
because it still pulled 146.59.87.168:3000/lfg2025/nostr-vpn:v0.3.7 — an
image that was deleted from the registry when NostrVPN was removed from
the product. The native nvpn daemon is already masked here (ln -sf
/dev/null nostr-vpn.service) and is never spawned at runtime (core vpn.rs
manages WireGuard/Tailscale, only reading a legacy nvpn config as
fallback), so extracting its binary was dead weight that bricked every
build once the image went away.
Drop the nvpn extraction and remove it from the missing-binary guard.
nostr-rs-relay stays required — its nostr-relay unit is still enabled and
the image still pulls. Change is in STEP 3, so the cached rootfs is
unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO built FIPS from an unpinned --depth 1 clone of upstream main, so
every build shipped whatever happened to be on main that day. Pin to
v0.4.1 via a FIPS_VERSION build arg — the version fips/config.rs renders
its typed config against, and the one validated in the field on .198 and
the thinkpad. The pin lands inside the STEP 1 recipe-hash range, so the
cached rootfs correctly invalidates on the next build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend: tor.list-services now hides services whose name maps to a
known-but-uninstalled catalog app (aliases covered: bitcoin/knots/core,
electrumx/electrs, btcpay, mempool). The node's own service, the relay,
and custom user services always show.
ISO: first-boot tor setup no longer pre-creates hidden services for six
apps that may never be installed — only the node's own onion is baked.
It also only seeds services.json/torrc on FIRST boot instead of
clobbering backend-managed services on every boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.7.104 shipped ISOs whose units crash-looped (nostr-relay 3s loop,
archipelago-diag 203/EXEC) because build-time extraction failures were mere
warnings and the cached rootfs never tracked its recipe:
- hash the rootfs-defining region of the build script and rebuild when it
changes (stale cache shipped ISOs without wpasupplicant/iw/rfkill)
- refuse to build without nvpn / nostr-rs-relay unless
ALLOW_MISSING_VPN_BINARIES=1
- ConditionPathExists on nostr-relay/nostr-vpn/diag units so a stripped
image skips them instead of crash-looping
- post-install test: every enabled archipelago*/nostr* unit must have an
existing ExecStart payload
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.
Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundling fmcd for offline installs (6dcdada3) shipped the raw
uncompressed podman save tar and grew the ISO ~220MB (2.3G -> 2.5G in
RC6-RC9). Save the core bundle as .tar.zst instead — podman load
auto-detects compression, verified locally against the RC9 fmcd.tar
(228M -> 65M, loads cleanly into rootless storage). The first-boot
loader and installer copy globs now pick up .tar.zst too.
Also trims the installer's USB->disk image copy and first-boot load
time on fresh installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chromium's SingletonLock is a symlink encoding <hostname>-<pid>. After a
node rename changes the OS hostname, the lock left from the previous
boot reads as "another computer" holding the profile; Chromium refuses
to start, --noerrdialogs hides the only dialog, and the kiosk
black-screens forever.
Two layers:
- kiosk launcher removes Singleton{Lock,Cookie,Socket} before every
Chromium start (any prior owner is dead at that point — pkill at
session start / loop respawn). Ships via ISO splice AND the binary's
include_str! self-heal, so existing nodes get it with the next OTA.
- the rename handler clears the lock immediately as a hostname
side-effect, covering nodes still running a pre-fix launcher.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 454c4bb2 kiosk flattening only covered the Dashboard's background
stack, but the kiosk boots into the login/onboarding screen —
OnboardingWrapper.vue — whose scoped 3D stack (perspective +
preserve-3d bg layers + mix-blend-mode glitch overlays) has the exact
compositing pattern that fails to repaint under the kiosk's software
compositor, leaving the black body fill visible. Extend the same
html.kiosk-mode flattening to this screen via :global() rules: 2D
opacity crossfades stay, 3D transforms / blur filters / blend-mode
glitch layers go.
Also raise the kiosk chromium JS heap cap 128→256MB: the intro
cinematic (video + audio + app) inside a 128MB V8 old-space risks
renderer OOM, and a crashed kiosk tab auto-reloads — indistinguishable
from the reported "flickers and refreshes".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuilds of the same version overwrote the previous ISO under an
identical filename, making a flashed stick indistinguishable from a
newer build. The builder now keeps a per-version RC counter (next to
the build counter) and names outputs like
archipelago-installer-1.7.101-alpha-unbundled-x86_64_RC2.iso.
Override with RC=n env when re-cutting a specific candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh offline installs came up without fedimint-clientd and filebrowser
(they only appeared after connecting internet). Three root causes:
- archipelago-load-images.service ran 'podman load' as root, but every
container runs rootless as archipelago — bundled images landed in
root's storage where the rootless runtime can't see them, so all
container creation silently depended on registry pulls. The loader now
loads into the archipelago user's storage (with linger + runtime-dir
wait + system migrate).
- The unbundled ISO bundled only filebrowser.tar; fmcd (fedimint-clientd)
is a baseline first-boot app too and is now part of the unbundled core
bundle.
- first-boot's pull_with_fallback always hit the network; it now uses an
already-loaded local image first and skips the pull entirely.
Also: fedimint-clientd added to the UI's hardcoded curated-app fallback
list (it was missing when all catalog fetches fail offline), plus its
INSTALLED_ALIASES entry, and the stale fmcd bundling comment in
image-versions.sh corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unit's ExecStartPre baked the first hostname -I token into
host-ip.env, which config.rs prefers over its own detection — once a
VPN/bridge interface existed (netbird's wg tunnel), every host_ip
consumer (install configs, VPN, mint URLs) got the tunnel IP. Read the
main-table default route's src address instead, falling back to
hostname -I on routeless hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ssmithx