The dashboard's /app/mempool/ proxy forwarded plain HTTP only: no
Upgrade/Connection headers, so the browser's /api/v1/ws handshake reached
mempool-api as a bare GET and Express 404'd it. The mempool page loaded
fine and every REST probe was green while the user saw a UI that never
connects — the backend was fully healthy and completely unreachable in
the only way that matters to the page.
This hid behind the electrumx initial-sync outage: once sync finished and
the API answered, "mempool works" was declared from REST checks while the
websocket path stayed dead. Reported by the operator from the browser,
which is the only place it was visible.
Adds Upgrade/$http_upgrade + Connection "upgrade" to the /app/mempool/
location in both shipped sources (image-recipe snippet + scripts copy).
Live-verified on the dev node: ws through the proxy now answers 101; the
control probe without upgrade headers reproduces the 404 signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
S4: /aiui/api/web-search proxied straight to SearXNG with no auth — anyone
reaching the web port ran searches attributed to the node's IP. Now routed
through the daemon's session-gated model proxy like the claude/ollama legs
(both nginx server blocks), forcing format=json upstream (the client never
sent it — search could 200 with HTML that parsed as nothing).
S3: the forwarder also serves the STANDALONE frontend, whose bodies carry
full history/images with no assistant loop behind them — a pasted seed
phrase went to Anthropic unscreened. The forwarder now runs the egress
secret-shape scan (G-B1) with the node's own secrets dir as deny corpus on
Claude bodies and search queries; blocked requests get a plain-language 400.
Also fixes a REAL gap in the egress tokenizer found by these tests: a JSON
key glued to a string value's first word ('content":"abandon...') dropped
that word, so an exactly-12-word seed pasted as a bare message yielded an
11-member run — checksum misses, backstop misses. Non-member words now
rescan within the token. egress 15/15 + model_proxy 10/10 green.
Co-Authored-By: Claude <noreply@anthropic.com>
The node served a bare self-signed leaf, so a browser exception had to be
granted per ORIGIN — scheme + host + port. The dashboard on :443 and an app
on :8334 are different origins, and a certificate interstitial CANNOT be
accepted inside an iframe, so a gated app embedded over HTTPS could never
render no matter how many warnings the user clicked through. (Mixed content
blocks the plain-HTTP variant first, before the SameSite cookie question the
symptom was originally filed under.)
A CA fixes it structurally: ports are not part of a certificate's identity, so
one leaf with the right SANs covers every port on the host, and one installed
CA trusts them all.
- scripts/setup-node-ca.sh generates the CA (4096-bit, pathlen:0, keyCertSign
only) and issues a 397-day leaf covering archipelago.local, the hostname, the
Tailscale MagicDNS name and every global address the host holds. Idempotent —
re-running reuses the CA and only reissues the leaf, so gaining an address
does not invalidate copies users already installed. --force-ca is the
deliberate escape hatch and says what it costs.
- nginx serves the public CA at /ca.crt on both schemes, unauthenticated by
design: a device fetches it before it can validate the node, so gating it
behind HTTPS or a login would be a chicken-and-egg.
- Settings → System shows the fingerprint and per-platform install steps.
crypto.subtle does not exist outside a secure context — precisely the case
this feature exists to fix — so an HTTP dashboard gets the openssl command
to verify by hand instead of a blank field.
Verified locally: chain validates, key pairs with the leaf, CA:TRUE/CA:FALSE
are correct, keys are 0600. Two TLS servers on different ports both verify
(ssl_verify_result=0) against the CA alone and are rejected without it — the
one-CA-covers-every-port claim, tested rather than assumed.
Not yet wired: app ports still serve plain HTTP. Putting TLS on them is the
next step and is what actually closes the iframe-login bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a Content-Security-Policy header to both nginx `location /aiui/`
blocks whose connect-src is scoped to the AIUI path prefix, so AIUI's
own JavaScript is browser-prevented from issuing a same-origin fetch
to /rpc/v1 with the ambient session cookie. Explicitly rejects the
`sandbox` iframe attribute (allow-scripts + allow-same-origin is the
known escape; dropping allow-same-origin breaks AIUI's storage and
its origin-checked bridge) and records why in both the nginx comment
and a new comment above the Chat.vue iframe. Adds
referrerpolicy="no-referrer" to the iframe so a media URL or page path
never leaks upstream via Referer.
Also adds an explicit `location /aiui/api/openrouter/ { return 404; }`
to both server blocks, closing 13-02's Task 3 checkpoint finding
(operator-accepted deviation 2026-08-03): the relay was already
structurally gone but the SPA catch-all served 200/405 instead of 404.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the live production exposure this plan targets: /aiui/api/claude/
and /aiui/api/ollama/ proxied straight through with "no session gate
needed", and /aiui/api/openrouter/ was a plain unauthenticated relay to a
paid third-party API the node holds no key for (T-13-08/T-13-09/T-13-10).
image-recipe/configs/nginx-archipelago.conf (BOTH server blocks, ~line 49
and ~line 961 — a fix applied to only one leaves the exposure live on
whichever block serves the request, T-13-15):
- /aiui/api/claude/ and /aiui/api/ollama/ proxy_pass re-pointed from
127.0.0.1:3142 / 127.0.0.1:11434 to the Rust daemon at 127.0.0.1:5678
(no trailing path component, so the daemon's own prefix match sees the
full request URI)
- Forward the session Cookie header to the daemon so it can re-derive auth
- location /aiui/api/openrouter/ deleted outright in both blocks
- Old comment "API key managed by proxy, no session gate needed" (the
reasoning error that produced the exposure) replaced with rationale
scripts/deploy-to-target.sh: deleted the embedded claude-api-proxy.py
heredoc, its systemd unit creation/enable/restart, the ANTHROPIC_API_KEY
extraction, and the 3141->3142 sed fixups. Added an unconditional step that
stops/disables/removes any pre-existing claude-api-proxy unit and deletes
/opt/archipelago/claude-api-proxy.py and
<data_dir>/secrets/claude-api-proxy.env on every deploy — so
already-provisioned nodes actually lose the old unauthenticated listener,
not just newly-deployed ones.
scripts/setup-aiui-server.sh: dropped the hard ANTHROPIC_API_KEY
requirement and the patch-nginx-claude.py step; the script's remaining job
is the AIUI dist rsync. (Also drops the FileBrowser-fix step that lived
here — that logic already exists, and is kept, in deploy-to-target.sh; this
script narrows to exactly what its rewritten header now says it does.)
core/archipelago/src/api/rpc/system/handlers.rs: `claude_api_key` setting
branch no longer writes a second key copy to secrets/claude-api-proxy.env
or restarts claude-api-proxy. secrets/claude-api-key (0600) remains the
single ledger, with a comment naming it as such.
`cargo build --package archipelago` succeeds. Verified via grep against
every acceptance criterion in 13-02-PLAN.md's Task 2 (openrouter count 0,
3142 gone from nginx, both location blocks present, PORT=3142 gone,
claude-api-proxy gone from handlers.rs, secrets/claude-api-key present).
Task 3 (real-node curl/systemd verification, S-15) is NOT done in this
commit — see 13-02-SUMMARY.md.
Continues WIP checkpoint 13b576da (reset --soft, recommitted atomically
per task per plan protocol).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Taking over a parked item from the app-UI work. AIUI's built index.html
emits ABSOLUTE /assets/<hashed> paths, so the browser asks for
/assets/index-BC2fBBaW.js. That lands in the MAIN UI's assets dir, where
it does not exist — the real files are in aiui/assets/. Both of AIUI's
two entry assets 404'd, so the embedded sidebar loaded nothing.
The config already contained a /aiui-assets/ location whose comment names
this exact problem ("AIUI may reference /assets/ without /aiui/ prefix"),
but it only catches requests to /aiui-assets/, a path AIUI never asks
for. It described the bug without fixing it.
/assets/ now falls back to a named location that rewrites into
aiui/assets/ and 404s from there. A fallback rather than copying the two
files up one level, because a frontend deploy replaces web-ui wholesale —
update.rs preserves the aiui/ DIRECTORY, not copies made into assets/ —
so a copy is erased by the very next deploy while this survives one.
Both server blocks (HTTP and HTTPS) are patched; named locations are
per-server, so each needs its own.
Verified on archi-dev-box after reload:
/assets/index-BC2fBBaW.js 200, 305256 bytes, application/javascript
/assets/index-BJkaQ2c4.css 200, 150716 bytes, text/css
/assets/does-not-exist.js 404 (the fallback is not over-broad)
/assets/index--lyLAgu1.js 200 (real main-UI chunks still come from
/assets/vendor-CmYeCqL_.js 200 the main dir — try_files hits them
/assets/index-CiMaoNII.css 200 before the fallback is consulted)
/ /aiui/ /health 200
Hash collision between the two builds is not a concern: Vite hashes are
content-derived, and any main-UI asset that exists is served by try_files
before the fallback runs.
Noted while doing this, not fixed here: the node's own
/etc/nginx/sites-enabled/archipelago is 378 lines BEHIND this repo file
(984 vs 1362) — it predates the IPv6 listener and the @asset_missing
no-store handling, among others. The node was patched minimally in its
own shape rather than overwritten, since a wholesale copy of a config
this diverged is not a safe unattended action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.
Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
the unit reaches 'active' in seconds; recovery/reconcile/listener
continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
the daemon back. Manual is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
instead of a tokio-sleep child of the process being stopped, whose
start-half was being lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phones on the FIPS mesh reach the node at http://[<fips0 ULA>], but every
shipped nginx config listened on IPv4 only — nothing answered [::]:80/443,
so mesh HTTP failed with the tunnel fully healthy (found live on
framework-pt during the vc24 5G session, 2026-07-23). Canonical conf gains
listen [::] lines and patch_nginx_conf self-heals deployed nodes (covers
the sites-enabled -> archipelago-http dev variant via the existing
canonicalize pass); validated by the existing nginx -t + rollback flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--enable-features=OverlayScrollbar gives the thin auto-hiding scrollbars a
remote browser shows (Peers view had a permanent classic scrollbar on the
TV). Rides the existing kiosk self-heal + ISO splice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kiosk window was sized in physical pixels, but Chromium interprets
--window-size in DIPs: with any device-scale > 1 the window painted
scale-times larger than the panel, and with no window manager in the
kiosk X session --kiosk/--start-fullscreen could not snap it back. On a
4K TV at scale 3 that meant an 11520x6480 window — 9x the panel area —
which blew the compositor tile-memory budget ("tile memory limits
exceeded, some content may not draw") and froze the screen on the last
painted frame; smaller scales cropped content off the right/bottom.
- Size the window in DIPs (panel / scale) so DIPs x scale = the panel.
- Panels >=2560 wide now target a 1920-wide layout (4K -> scale 2.0,
validated on the Framework's 72" 4K TV: spacious desktop layout at 2x
sharpness); smaller panels keep the 1280 target (1080p TV -> 1.5).
- Optional ARCHIPELAGO_KIOSK_MAX_WIDTH cap: drop to the best <=N-wide
xrandr mode for weak hardware (TV upscales instead of us rasterizing).
- Source /etc/archipelago/kiosk-display.conf so the upcoming Settings
display-resolution picker can manage these knobs per node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fresh-install kiosk issues found on a Framework node:
- MOTD said 'Ctrl+Alt+F7 for kiosk' / 'F1 for terminal', but the kiosk's
Xorg runs on vt1 — so F7 was an empty black VT (the reported black screen
and the failed 'return to kiosk'). Corrected: kiosk=F1, terminal=F2.
- The UTF-8 block-glyph logo corrupted intermittently because returning from
the kiosk can leave the console VT out of UTF-8 mode; force it with ESC%G.
- A short power-button press shut the node down instantly. Ignore the short
press, keep long-press poweroff, so accidental brushes don't kill the node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AIUI themes via @media (prefers-color-scheme) and falls back to its light
variant (white panels) when the browser reports no preference — which the
minimal kiosk X session does, so AIUI came up light instead of its designed
dark. The main UI hardcodes dark and is unaffected. Push two independent
dark signals on the kiosk Chromium: GTK_THEME=Adwaita:dark (version-
independent, can never force light) and --blink-settings=preferredColorScheme=0
(0 = kDark in modern Chromium). Needs on-device kiosk verification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.7.104 shipped ISOs whose units crash-looped (nostr-relay 3s loop,
archipelago-diag 203/EXEC) because build-time extraction failures were mere
warnings and the cached rootfs never tracked its recipe:
- hash the rootfs-defining region of the build script and rebuild when it
changes (stale cache shipped ISOs without wpasupplicant/iw/rfkill)
- refuse to build without nvpn / nostr-rs-relay unless
ALLOW_MISSING_VPN_BINARIES=1
- ConditionPathExists on nostr-relay/nostr-vpn/diag units so a stripped
image skips them instead of crash-looping
- post-install test: every enabled archipelago*/nostr* unit must have an
existing ExecStart payload
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.
Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chromium's SingletonLock is a symlink encoding <hostname>-<pid>. After a
node rename changes the OS hostname, the lock left from the previous
boot reads as "another computer" holding the profile; Chromium refuses
to start, --noerrdialogs hides the only dialog, and the kiosk
black-screens forever.
Two layers:
- kiosk launcher removes Singleton{Lock,Cookie,Socket} before every
Chromium start (any prior owner is dead at that point — pkill at
session start / loop respawn). Ships via ISO splice AND the binary's
include_str! self-heal, so existing nodes get it with the next OTA.
- the rename handler clears the lock immediately as a hostname
side-effect, covering nodes still running a pre-fix launcher.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 454c4bb2 kiosk flattening only covered the Dashboard's background
stack, but the kiosk boots into the login/onboarding screen —
OnboardingWrapper.vue — whose scoped 3D stack (perspective +
preserve-3d bg layers + mix-blend-mode glitch overlays) has the exact
compositing pattern that fails to repaint under the kiosk's software
compositor, leaving the black body fill visible. Extend the same
html.kiosk-mode flattening to this screen via :global() rules: 2D
opacity crossfades stay, 3D transforms / blur filters / blend-mode
glitch layers go.
Also raise the kiosk chromium JS heap cap 128→256MB: the intro
cinematic (video + audio + app) inside a 128MB V8 old-space risks
renderer OOM, and a crashed kiosk tab auto-reloads — indistinguishable
from the reported "flickers and refreshes".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unit's ExecStartPre baked the first hostname -I token into
host-ip.env, which config.rs prefers over its own detection — once a
VPN/bridge interface existed (netbird's wg tunnel), every host_ip
consumer (install configs, VPN, mint URLs) got the tunnel IP. Read the
main-table default route's src address instead, falling back to
hostname -I on routeless hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThinkPad X250 (Broadwell HD 5500) internal panels show horizontal corruption
bands at the kiosk; known i915 eDP power-feature issue. Boot installed and
live systems with i915.enable_ips=0 enable_psr=0 enable_fbc=0 (no-ops off
Intel, nothing lost on a kiosk). Debug boot entries stay stock for A/B.
Also size the Chromium kiosk window from the detected mode instead of a
hardcoded 1920x1080 that clips on 1366x768 panels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflict resolution: keep the branch's CPUQuota=200% (75% was the proven
cause of choppy HDMI audio — the file's own comment says so; the 75% on main
was an accidental regression bundled into an unrelated UI commit) and keep
main's newer MemoryMax=2800M/MemoryHigh=2200M tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Routes ALSA's "default" device through PulseAudio/PipeWire. Was a manual,
untracked fix living only on archy-x250-exp — a reprovision or fresh image
would silently lose it, same failure mode that already bit the GPU-flags
and CPUQuota fixes. Wired into deploy-to-target.sh (both the primary --live
path and the .198/.253 secondary-copy path) and deploy-tailscale.sh, mirroring
the existing 99-mesh-radio.rules udev sync pattern (diff-check, copy if changed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both had regressed back to the pre-2026-06-28-incident state: GPU_FLAGS used
--enable-gpu-rasterization on any node with a GPU, and CPUQuota was 75%.
On archy-x250-exp (Intel HD 5500 under X11) this spins a dedicated GPU
process at 55-92% CPU (falls back to software compositing anyway), and the
75% cgroup quota throttled the kiosk unit ~81% of the time (nr_throttled
4856/6005) — the exact CPU-starvation signature documented as the choppy-
audio root cause. Restores the validated fix: --in-process-gpu,
--num-raster-threads=1, GpuRasterization disabled via --disable-features,
CPUQuota=200%. Verified on archy-x250-exp: no separate gpu-process, throttling
dropped to ~3% (nr_throttled 14/503).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher's sudo -u archipelago invocation set DISPLAY/HOME but not
XDG_RUNTIME_DIR, so Chromium's audio backend couldn't find the PipeWire-Pulse
socket at /run/user/<uid>/pulse/native. It silently fell back to raw ALSA
"default", which also failed ("Connection refused"), producing no HDMI audio
at all with no visible error since --noerrdialogs suppresses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
configure_display picked whichever mode was already "active" on the HDMI
output, so if X ever booted cloned to the laptop panel's resolution it
would keep re-confirming that wrong mode forever instead of self-healing
to the display's native mode.
The kiosk chromium pinned ~92% of a core (software-compositing spin from
--enable-gpu-rasterization on a GPU-less/headless node), saturating the machine
and starving the backend + container builds — it caused the .198 receive timeout
and the deploy storms.
- archipelago-kiosk.service: CPUQuota=75% + MemoryMax/High + Delegate, so a
runaway kiosk can never take the whole node down.
- archipelago-kiosk-launcher.sh: detect /dev/dri — use GPU rasterization only
when a GPU exists, else --disable-gpu (avoids the headless spin).
- bootstrap::ensure_kiosk_hardened: OTA self-heal that installs the updated
unit+launcher on already-deployed nodes, daemon-reloads, and only try-restarts
a *running* kiosk (never re-enables an operator-disabled one).
cargo check clean; launcher bash -n clean; unit syntax valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On production nodes /var/lib/archipelago (the app data dir AND podman's
graphroot=/var/lib/archipelago/containers/storage) is a separate
device-mapper volume. archipelago.service ordered only After=network-online
.target, so on cold boots it (and its ExecStartPre) could start BEFORE
var-lib-archipelago.mount, write to the bare mountpoint on rootfs, fail every
podman call, exit, and be restarted every 5s until the volume mounted — the
"~20x [FAILED] Failed to start over ~5min" boot flap. Proven live on .198:
"var-lib-archipelago.mount: Directory /var/lib/archipelago to mount over is
not empty, mounting anyway" — the service had written there pre-mount.
Fix: RequiresMountsFor=/var/lib/archipelago (adds Requires= + After= on the
mount unit).
- image-recipe/configs/archipelago.service: ships the directive on fresh ISOs.
- bootstrap::ensure_archipelago_mount_ordering(): self-heals already-deployed
nodes' installed unit + daemon-reload (boot-ordering only, effective next
reboot; never restarts the running service). Idempotent; harmless on rootfs
installs (maps to the always-mounted root).
Verified on .198: after applying, systemctl shows After=var-lib-archipelago
.mount and systemd-analyze verify is clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fedimint UI HTML/CSS reference absolute /assets/* paths; under /app/fedimint/
those hit the main SPA, not the fedimint container, so the UI renders
unstyled. Add the proven sub_filter asset-rewrite pattern (as indeedhub/
botfights use) to the /app/fedimint/ block in the nginx template + https
snippet (also rewrites url(...) for the CSS background image). Bootstrap
self-heal for already-deployed nodes is the documented resume point.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ssmithxandClaude Sonnet 5