Fresh offline installs came up without fedimint-clientd and filebrowser
(they only appeared after connecting internet). Three root causes:
- archipelago-load-images.service ran 'podman load' as root, but every
container runs rootless as archipelago — bundled images landed in
root's storage where the rootless runtime can't see them, so all
container creation silently depended on registry pulls. The loader now
loads into the archipelago user's storage (with linger + runtime-dir
wait + system migrate).
- The unbundled ISO bundled only filebrowser.tar; fmcd (fedimint-clientd)
is a baseline first-boot app too and is now part of the unbundled core
bundle.
- first-boot's pull_with_fallback always hit the network; it now uses an
already-loaded local image first and skips the pull entirely.
Also: fedimint-clientd added to the UI's hardcoded curated-app fallback
list (it was missing when all catalog fetches fail offline), plus its
INSTALLED_ALIASES entry, and the stale fmcd bundling comment in
image-versions.sh corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO's rootfs tar is a cached build artifact (currently 2026-04-20) and
its baked-in archipelago.service predates the B17 fix — no
RequiresMountsFor=/var/lib/archipelago — while auto-install.sh only copied
the configs/ unit when the rootfs had none. Every fresh install therefore
raced the LUKS data mount on cold boot and looped
'[FAILED] Failed to start archipelago.service' for minutes (seen on the
1.7.99-alpha ISO).
Ship configs/archipelago.service on the ISO at archipelago/configs/ and
make the installer overwrite the rootfs copy unconditionally. Also add
RequiresMountsFor to the legacy install-to-disk.sh inline unit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The X250 symptom was the UI rendered at a hardcoded 1920x1080 and clipped
on the 1366x768 panel (fixed in 7e996d6), not eDP corruption. Keep RC2
minimal: no speculative kernel params.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThinkPad X250 (Broadwell HD 5500) internal panels show horizontal corruption
bands at the kiosk; known i915 eDP power-feature issue. Boot installed and
live systems with i915.enable_ips=0 enable_psr=0 enable_fbc=0 (no-ops off
Intel, nothing lost on a kiosk). Debug boot entries stay stock for A/B.
Also size the Chromium kiosk window from the detected mode instead of a
hardcoded 1920x1080 that clips on 1366x768 panels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The insecure-registry config was written to /etc unconditionally; as the
archipelago build user the redirection fails and set -e aborts check_tools
before Step 1. Write to the rootless podman per-user config dir when not
root, tolerate failure (nodes may already carry the config), and drop the
duplicated 146.59.87.168:3000 entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BUILD_VERSION/EXPECTED_VERSION/LOCAL_RELEASE, the backend container build
context, and the neode-ui/web-dist/apps/demo/docker copies all resolved
one level short (image-recipe/ instead of the repo root), so the builder
never found the prebuilt binary and its container fallback died on
'COPY core ./core'. configs/ references were already correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-first-boot-secrets.service regenerates the self-signed TLS
keypair (device hostname in SAN) and all SSH host keys on the installed
system's first boot, before ssh/nginx start — the squashfs bakes one
key set at build time, so every flashed device shared them (§F 🔴)
- staging-first swaps: a failed regeneration keeps the baked keys rather
than leaving the device keyless
- the builder now emits <iso>.sha256 next to the ISO, and
scripts/sign-iso-checksums.sh signs {artifact,sha256,size} as a JSON
doc with the release-root ceremony (verify: ceremony verify) (§F 🟠)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- reticulum.rs: send_text_msg was lossy-UTF8-mangling binary CBOR control
envelopes (ReadReceipt etc.) before sending as LXMF text; base64-encode
with a marker instead, decoded losslessly on receive.
- typed_messages.rs: mesh.send-read-receipt fired automatically on every
chat view with no is_archy_peer gate, so viewing a message from a stock
(non-archy) LXMF peer auto-sent it an undecodable control envelope,
surfacing as garbage text right after whatever it just sent. Now a no-op
for non-archy peers.
- mesh/listener/mod.rs: RX_STALL_TIMEOUT was 300s and forced a full
auto-detect reconnect on any otherwise-healthy but quiet mesh link
(visible as "Connecting..." flapping); this also wiped Reticulum's
in-memory peer-address table every cycle, breaking messaging with peers
who hadn't re-announced in the window. Bumped to 1800s.
- reticulum.rs: persist the peer prefix/dest-hash/display-name table to
disk so a restart doesn't force every peer back to "Anonymous Peer"
until they re-announce.
- decode.rs/frames.rs: Meshcore was discarding the SNR its wire format
carries; wire it onto the peer record. Mesh.vue's signalBars() now falls
back to SNR-based bars when RSSI is unavailable (always true for
Meshcore); Reticulum has neither and correctly stays at 0/"no data".
- system/handlers.rs, dispatcher.rs: new system.get-hostname RPC + cert
regeneration (with a proper SAN) whenever server.set-name changes the
hostname, so HTTPS doesn't add a mismatch warning on top of the
self-signed one after a rename.
- AccountInfoSection.vue: surface the mDNS hostname + http/https links in
Settings (HTTPS needed for mic/camera secure-context features) — never
forced, both keep working.
- build-auto-installer-iso.sh: ship avahi-daemon so .local names actually
resolve on the LAN, and give the self-signed cert a real SAN instead of
a bare CN, both at image-build and install-time-fallback.
- Mesh.vue/MediaLightbox.vue/mesh-styles.css: mic/attach-stack no longer
closes on a plain hover-past; mesh images open in the shared lightbox
and have a real download button; lightbox close button moves to
bottom-center on mobile instead of under the status bar; mesh device
panel gets the same height/padding as its sibling tabs.
Verified: 108/108 mesh unit tests, deployed + confirmed healthy on
.116/.198/.228 (matching binary hash across all three), live Reticulum
messaging confirmed working end-to-end post-deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PyInstaller-based daemons (e.g. the Reticulum mesh daemon) self-extract
to /tmp on every launch and leak their extraction dir on abnormal exit;
combined with release-build staging dirs this exhausted the default cap
on .116 and silently broke Reticulum ("no space left on device" during
self-extraction, masquerading as a connect failure). Ship a tmp.mount.d
override (75% of RAM) in the installer image so fresh installs don't
inherit the same ceiling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.
Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
replaces fragile post-start exec that failed under restricted-cap rootless
podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition
Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
tester, every app × every transition. Run before each release.
Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Releases no longer ship as bootable ISOs. Archipelago updates are
distributed as the backend binary plus a frontend tarball referenced by
releases/manifest.json. Nodes OTA-update via scripts/self-update.sh.
Filebrowser and AIUI remain bundled inside the frontend tarball and
deployed atomically, verified present in v1.7.43-alpha release artifact
(189 AIUI files, filebrowser-client bundle).
Archived under image-recipe/_archived/ (resurrectable if ISO distribution
is reintroduced):
- build-auto-installer-iso.sh
- build-unbundled-iso.sh
- test-iso-qemu.sh
- scripts/convert-iso-to-disk.sh
- BUILD-ISO-STATUS.md, ISO-BUILD-CHECKLIST.md
- branding/isohdpfx.bin
- .gitea/workflows/build-iso-dev.yml
Updated release process docs to drop ISO references:
- scripts/create-release.sh (next-steps text)
- docs/BETA-RELEASE-CHECKLIST.md
- docs/hotfix-process.md
- README.md
