Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO's rootfs tar is a cached build artifact (currently 2026-04-20) and
its baked-in archipelago.service predates the B17 fix — no
RequiresMountsFor=/var/lib/archipelago — while auto-install.sh only copied
the configs/ unit when the rootfs had none. Every fresh install therefore
raced the LUKS data mount on cold boot and looped
'[FAILED] Failed to start archipelago.service' for minutes (seen on the
1.7.99-alpha ISO).
Ship configs/archipelago.service on the ISO at archipelago/configs/ and
make the installer overwrite the rootfs copy unconditionally. Also add
RequiresMountsFor to the legacy install-to-disk.sh inline unit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The X250 symptom was the UI rendered at a hardcoded 1920x1080 and clipped
on the 1366x768 panel (fixed in 7e996d6), not eDP corruption. Keep RC2
minimal: no speculative kernel params.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThinkPad X250 (Broadwell HD 5500) internal panels show horizontal corruption
bands at the kiosk; known i915 eDP power-feature issue. Boot installed and
live systems with i915.enable_ips=0 enable_psr=0 enable_fbc=0 (no-ops off
Intel, nothing lost on a kiosk). Debug boot entries stay stock for A/B.
Also size the Chromium kiosk window from the detected mode instead of a
hardcoded 1920x1080 that clips on 1366x768 panels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The insecure-registry config was written to /etc unconditionally; as the
archipelago build user the redirection fails and set -e aborts check_tools
before Step 1. Write to the rootless podman per-user config dir when not
root, tolerate failure (nodes may already carry the config), and drop the
duplicated 146.59.87.168:3000 entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BUILD_VERSION/EXPECTED_VERSION/LOCAL_RELEASE, the backend container build
context, and the neode-ui/web-dist/apps/demo/docker copies all resolved
one level short (image-recipe/ instead of the repo root), so the builder
never found the prebuilt binary and its container fallback died on
'COPY core ./core'. configs/ references were already correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-first-boot-secrets.service regenerates the self-signed TLS
keypair (device hostname in SAN) and all SSH host keys on the installed
system's first boot, before ssh/nginx start — the squashfs bakes one
key set at build time, so every flashed device shared them (§F 🔴)
- staging-first swaps: a failed regeneration keeps the baked keys rather
than leaving the device keyless
- the builder now emits <iso>.sha256 next to the ISO, and
scripts/sign-iso-checksums.sh signs {artifact,sha256,size} as a JSON
doc with the release-root ceremony (verify: ceremony verify) (§F 🟠)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- reticulum.rs: send_text_msg was lossy-UTF8-mangling binary CBOR control
envelopes (ReadReceipt etc.) before sending as LXMF text; base64-encode
with a marker instead, decoded losslessly on receive.
- typed_messages.rs: mesh.send-read-receipt fired automatically on every
chat view with no is_archy_peer gate, so viewing a message from a stock
(non-archy) LXMF peer auto-sent it an undecodable control envelope,
surfacing as garbage text right after whatever it just sent. Now a no-op
for non-archy peers.
- mesh/listener/mod.rs: RX_STALL_TIMEOUT was 300s and forced a full
auto-detect reconnect on any otherwise-healthy but quiet mesh link
(visible as "Connecting..." flapping); this also wiped Reticulum's
in-memory peer-address table every cycle, breaking messaging with peers
who hadn't re-announced in the window. Bumped to 1800s.
- reticulum.rs: persist the peer prefix/dest-hash/display-name table to
disk so a restart doesn't force every peer back to "Anonymous Peer"
until they re-announce.
- decode.rs/frames.rs: Meshcore was discarding the SNR its wire format
carries; wire it onto the peer record. Mesh.vue's signalBars() now falls
back to SNR-based bars when RSSI is unavailable (always true for
Meshcore); Reticulum has neither and correctly stays at 0/"no data".
- system/handlers.rs, dispatcher.rs: new system.get-hostname RPC + cert
regeneration (with a proper SAN) whenever server.set-name changes the
hostname, so HTTPS doesn't add a mismatch warning on top of the
self-signed one after a rename.
- AccountInfoSection.vue: surface the mDNS hostname + http/https links in
Settings (HTTPS needed for mic/camera secure-context features) — never
forced, both keep working.
- build-auto-installer-iso.sh: ship avahi-daemon so .local names actually
resolve on the LAN, and give the self-signed cert a real SAN instead of
a bare CN, both at image-build and install-time-fallback.
- Mesh.vue/MediaLightbox.vue/mesh-styles.css: mic/attach-stack no longer
closes on a plain hover-past; mesh images open in the shared lightbox
and have a real download button; lightbox close button moves to
bottom-center on mobile instead of under the status bar; mesh device
panel gets the same height/padding as its sibling tabs.
Verified: 108/108 mesh unit tests, deployed + confirmed healthy on
.116/.198/.228 (matching binary hash across all three), live Reticulum
messaging confirmed working end-to-end post-deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PyInstaller-based daemons (e.g. the Reticulum mesh daemon) self-extract
to /tmp on every launch and leak their extraction dir on abnormal exit;
combined with release-build staging dirs this exhausted the default cap
on .116 and silently broke Reticulum ("no space left on device" during
self-extraction, masquerading as a connect failure). Ship a tmp.mount.d
override (75% of RAM) in the installer image so fresh installs don't
inherit the same ceiling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.
Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
replaces fragile post-start exec that failed under restricted-cap rootless
podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition
Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
tester, every app × every transition. Run before each release.
Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Releases no longer ship as bootable ISOs. Archipelago updates are
distributed as the backend binary plus a frontend tarball referenced by
releases/manifest.json. Nodes OTA-update via scripts/self-update.sh.
Filebrowser and AIUI remain bundled inside the frontend tarball and
deployed atomically, verified present in v1.7.43-alpha release artifact
(189 AIUI files, filebrowser-client bundle).
Archived under image-recipe/_archived/ (resurrectable if ISO distribution
is reintroduced):
- build-auto-installer-iso.sh
- build-unbundled-iso.sh
- test-iso-qemu.sh
- scripts/convert-iso-to-disk.sh
- BUILD-ISO-STATUS.md, ISO-BUILD-CHECKLIST.md
- branding/isohdpfx.bin
- .gitea/workflows/build-iso-dev.yml
Updated release process docs to drop ISO references:
- scripts/create-release.sh (next-steps text)
- docs/BETA-RELEASE-CHECKLIST.md
- docs/hotfix-process.md
- README.md
