The kiosk image ships no color-emoji font, so the wallet card's 🥜 and ⚓
drew as empty squares on TVs. Swap them for inline stroke SVGs matching
the neighbouring rows (wallet card + scan modal), and add
fonts-noto-color-emoji to the ISO package set so remaining emoji across
the UI (peer files, chat, content) render on future installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fresh-install kiosk issues found on a Framework node:
- MOTD said 'Ctrl+Alt+F7 for kiosk' / 'F1 for terminal', but the kiosk's
Xorg runs on vt1 — so F7 was an empty black VT (the reported black screen
and the failed 'return to kiosk'). Corrected: kiosk=F1, terminal=F2.
- The UTF-8 block-glyph logo corrupted intermittently because returning from
the kiosk can leave the console VT out of UTF-8 mode; force it with ESC%G.
- A short power-button press shut the node down instantly. Ignore the short
press, keep long-press poweroff, so accidental brushes don't kill the node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The relay image was rebuilt to the standard nostr-rs-relay layout — WORKDIR
/usr/src/app, execs ./nostr-rs-relay — but the build still cp'd from the old
/usr/local/bin path, so extraction silently produced nothing. Once the
missing-binary guard was added, that silent miss became a hard build
failure. Verified the binary is a valid amd64 ELF (GLIBC 2.34, under Debian
13's 2.40) at /usr/src/app/nostr-rs-relay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO build hard-failed at 'Required binaries not extracted: nvpn'
because it still pulled 146.59.87.168:3000/lfg2025/nostr-vpn:v0.3.7 — an
image that was deleted from the registry when NostrVPN was removed from
the product. The native nvpn daemon is already masked here (ln -sf
/dev/null nostr-vpn.service) and is never spawned at runtime (core vpn.rs
manages WireGuard/Tailscale, only reading a legacy nvpn config as
fallback), so extracting its binary was dead weight that bricked every
build once the image went away.
Drop the nvpn extraction and remove it from the missing-binary guard.
nostr-rs-relay stays required — its nostr-relay unit is still enabled and
the image still pulls. Change is in STEP 3, so the cached rootfs is
unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO built FIPS from an unpinned --depth 1 clone of upstream main, so
every build shipped whatever happened to be on main that day. Pin to
v0.4.1 via a FIPS_VERSION build arg — the version fips/config.rs renders
its typed config against, and the one validated in the field on .198 and
the thinkpad. The pin lands inside the STEP 1 recipe-hash range, so the
cached rootfs correctly invalidates on the next build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend: tor.list-services now hides services whose name maps to a
known-but-uninstalled catalog app (aliases covered: bitcoin/knots/core,
electrumx/electrs, btcpay, mempool). The node's own service, the relay,
and custom user services always show.
ISO: first-boot tor setup no longer pre-creates hidden services for six
apps that may never be installed — only the node's own onion is baked.
It also only seeds services.json/torrc on FIRST boot instead of
clobbering backend-managed services on every boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.7.104 shipped ISOs whose units crash-looped (nostr-relay 3s loop,
archipelago-diag 203/EXEC) because build-time extraction failures were mere
warnings and the cached rootfs never tracked its recipe:
- hash the rootfs-defining region of the build script and rebuild when it
changes (stale cache shipped ISOs without wpasupplicant/iw/rfkill)
- refuse to build without nvpn / nostr-rs-relay unless
ALLOW_MISSING_VPN_BINARIES=1
- ConditionPathExists on nostr-relay/nostr-vpn/diag units so a stripped
image skips them instead of crash-looping
- post-install test: every enabled archipelago*/nostr* unit must have an
existing ExecStart payload
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundling fmcd for offline installs (6dcdada3) shipped the raw
uncompressed podman save tar and grew the ISO ~220MB (2.3G -> 2.5G in
RC6-RC9). Save the core bundle as .tar.zst instead — podman load
auto-detects compression, verified locally against the RC9 fmcd.tar
(228M -> 65M, loads cleanly into rootless storage). The first-boot
loader and installer copy globs now pick up .tar.zst too.
Also trims the installer's USB->disk image copy and first-boot load
time on fresh installs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuilds of the same version overwrote the previous ISO under an
identical filename, making a flashed stick indistinguishable from a
newer build. The builder now keeps a per-version RC counter (next to
the build counter) and names outputs like
archipelago-installer-1.7.101-alpha-unbundled-x86_64_RC2.iso.
Override with RC=n env when re-cutting a specific candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh offline installs came up without fedimint-clientd and filebrowser
(they only appeared after connecting internet). Three root causes:
- archipelago-load-images.service ran 'podman load' as root, but every
container runs rootless as archipelago — bundled images landed in
root's storage where the rootless runtime can't see them, so all
container creation silently depended on registry pulls. The loader now
loads into the archipelago user's storage (with linger + runtime-dir
wait + system migrate).
- The unbundled ISO bundled only filebrowser.tar; fmcd (fedimint-clientd)
is a baseline first-boot app too and is now part of the unbundled core
bundle.
- first-boot's pull_with_fallback always hit the network; it now uses an
already-loaded local image first and skips the pull entirely.
Also: fedimint-clientd added to the UI's hardcoded curated-app fallback
list (it was missing when all catalog fetches fail offline), plus its
INSTALLED_ALIASES entry, and the stale fmcd bundling comment in
image-versions.sh corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO's rootfs tar is a cached build artifact (currently 2026-04-20) and
its baked-in archipelago.service predates the B17 fix — no
RequiresMountsFor=/var/lib/archipelago — while auto-install.sh only copied
the configs/ unit when the rootfs had none. Every fresh install therefore
raced the LUKS data mount on cold boot and looped
'[FAILED] Failed to start archipelago.service' for minutes (seen on the
1.7.99-alpha ISO).
Ship configs/archipelago.service on the ISO at archipelago/configs/ and
make the installer overwrite the rootfs copy unconditionally. Also add
RequiresMountsFor to the legacy install-to-disk.sh inline unit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The X250 symptom was the UI rendered at a hardcoded 1920x1080 and clipped
on the 1366x768 panel (fixed in 7e996d6), not eDP corruption. Keep RC2
minimal: no speculative kernel params.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThinkPad X250 (Broadwell HD 5500) internal panels show horizontal corruption
bands at the kiosk; known i915 eDP power-feature issue. Boot installed and
live systems with i915.enable_ips=0 enable_psr=0 enable_fbc=0 (no-ops off
Intel, nothing lost on a kiosk). Debug boot entries stay stock for A/B.
Also size the Chromium kiosk window from the detected mode instead of a
hardcoded 1920x1080 that clips on 1366x768 panels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The insecure-registry config was written to /etc unconditionally; as the
archipelago build user the redirection fails and set -e aborts check_tools
before Step 1. Write to the rootless podman per-user config dir when not
root, tolerate failure (nodes may already carry the config), and drop the
duplicated 146.59.87.168:3000 entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BUILD_VERSION/EXPECTED_VERSION/LOCAL_RELEASE, the backend container build
context, and the neode-ui/web-dist/apps/demo/docker copies all resolved
one level short (image-recipe/ instead of the repo root), so the builder
never found the prebuilt binary and its container fallback died on
'COPY core ./core'. configs/ references were already correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-first-boot-secrets.service regenerates the self-signed TLS
keypair (device hostname in SAN) and all SSH host keys on the installed
system's first boot, before ssh/nginx start — the squashfs bakes one
key set at build time, so every flashed device shared them (§F 🔴)
- staging-first swaps: a failed regeneration keeps the baked keys rather
than leaving the device keyless
- the builder now emits <iso>.sha256 next to the ISO, and
scripts/sign-iso-checksums.sh signs {artifact,sha256,size} as a JSON
doc with the release-root ceremony (verify: ceremony verify) (§F 🟠)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- reticulum.rs: send_text_msg was lossy-UTF8-mangling binary CBOR control
envelopes (ReadReceipt etc.) before sending as LXMF text; base64-encode
with a marker instead, decoded losslessly on receive.
- typed_messages.rs: mesh.send-read-receipt fired automatically on every
chat view with no is_archy_peer gate, so viewing a message from a stock
(non-archy) LXMF peer auto-sent it an undecodable control envelope,
surfacing as garbage text right after whatever it just sent. Now a no-op
for non-archy peers.
- mesh/listener/mod.rs: RX_STALL_TIMEOUT was 300s and forced a full
auto-detect reconnect on any otherwise-healthy but quiet mesh link
(visible as "Connecting..." flapping); this also wiped Reticulum's
in-memory peer-address table every cycle, breaking messaging with peers
who hadn't re-announced in the window. Bumped to 1800s.
- reticulum.rs: persist the peer prefix/dest-hash/display-name table to
disk so a restart doesn't force every peer back to "Anonymous Peer"
until they re-announce.
- decode.rs/frames.rs: Meshcore was discarding the SNR its wire format
carries; wire it onto the peer record. Mesh.vue's signalBars() now falls
back to SNR-based bars when RSSI is unavailable (always true for
Meshcore); Reticulum has neither and correctly stays at 0/"no data".
- system/handlers.rs, dispatcher.rs: new system.get-hostname RPC + cert
regeneration (with a proper SAN) whenever server.set-name changes the
hostname, so HTTPS doesn't add a mismatch warning on top of the
self-signed one after a rename.
- AccountInfoSection.vue: surface the mDNS hostname + http/https links in
Settings (HTTPS needed for mic/camera secure-context features) — never
forced, both keep working.
- build-auto-installer-iso.sh: ship avahi-daemon so .local names actually
resolve on the LAN, and give the self-signed cert a real SAN instead of
a bare CN, both at image-build and install-time-fallback.
- Mesh.vue/MediaLightbox.vue/mesh-styles.css: mic/attach-stack no longer
closes on a plain hover-past; mesh images open in the shared lightbox
and have a real download button; lightbox close button moves to
bottom-center on mobile instead of under the status bar; mesh device
panel gets the same height/padding as its sibling tabs.
Verified: 108/108 mesh unit tests, deployed + confirmed healthy on
.116/.198/.228 (matching binary hash across all three), live Reticulum
messaging confirmed working end-to-end post-deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PyInstaller-based daemons (e.g. the Reticulum mesh daemon) self-extract
to /tmp on every launch and leak their extraction dir on abnormal exit;
combined with release-build staging dirs this exhausted the default cap
on .116 and silently broke Reticulum ("no space left on device" during
self-extraction, masquerading as a connect failure). Ship a tmp.mount.d
override (75% of RAM) in the installer image so fresh installs don't
inherit the same ceiling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.
Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
replaces fragile post-start exec that failed under restricted-cap rootless
podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition
Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
tester, every app × every transition. Run before each release.
Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Releases no longer ship as bootable ISOs. Archipelago updates are
distributed as the backend binary plus a frontend tarball referenced by
releases/manifest.json. Nodes OTA-update via scripts/self-update.sh.
Filebrowser and AIUI remain bundled inside the frontend tarball and
deployed atomically, verified present in v1.7.43-alpha release artifact
(189 AIUI files, filebrowser-client bundle).
Archived under image-recipe/_archived/ (resurrectable if ISO distribution
is reintroduced):
- build-auto-installer-iso.sh
- build-unbundled-iso.sh
- test-iso-qemu.sh
- scripts/convert-iso-to-disk.sh
- BUILD-ISO-STATUS.md, ISO-BUILD-CHECKLIST.md
- branding/isohdpfx.bin
- .gitea/workflows/build-iso-dev.yml
Updated release process docs to drop ISO references:
- scripts/create-release.sh (next-steps text)
- docs/BETA-RELEASE-CHECKLIST.md
- docs/hotfix-process.md
- README.md
