After an unclean shutdown, crash recovery walked the running-containers
snapshot and retried 'no such container' failures (2 attempts, 10s
backoff) for containers that no longer exist. Recovery runs before the
server binds :5678 and notifies systemd ready, so a stale snapshot
pushed startup past TimeoutStartSec=5min — systemd killed the daemon
mid-recovery, the next boot saw a crash again, and the node looped
forever (151 restarts on .116 after a hard poweroff).
- pre-filter the snapshot against 'podman ps -a' and skip vanished
containers outright (fail-open if the query fails)
- never retry a 'no such container' failure
- extend the systemd start timeout ahead of each container start so a
heavy node recovering dozens of real containers isn't killed while
making progress
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend: tor.list-services now hides services whose name maps to a
known-but-uninstalled catalog app (aliases covered: bitcoin/knots/core,
electrumx/electrs, btcpay, mempool). The node's own service, the relay,
and custom user services always show.
ISO: first-boot tor setup no longer pre-creates hidden services for six
apps that may never be installed — only the node's own onion is baked.
It also only seeds services.json/torrc on FIRST boot instead of
clobbering backend-managed services on every boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stray `archipelago --version` used to start a full second instance next
to the systemd one (issue #74). Handle -V/--version, -h/--help, and reject
unknown dashed options before any tracing/state init, mirroring the
ceremony subcommand's clean-stdout precedent. Version output matches the
health RPC format: <pkg-version>-<git-hash|dev>.
Fixes#74
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both tests take the global single-flight UPDATE_OP_LOCK; run concurrently
by the test harness, one saw the other's lock and failed with 'another
update operation is already running' instead of its expected refusal.
A shared test mutex makes them mutually exclusive deterministically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.
Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of .198 bricking on the 1.7.103 OTA: two concurrent
update.download RPCs shared one staging file, cancel_download wiped
staging mid-flight, a third download began re-filling it, and
apply_update mv'd the 3-second-old 17MB partial of the 49MB binary
into /usr/local/bin -> SEGV boot loop (236 restarts, no rollback).
- Single-flight UPDATE_OP_LOCK across download/apply; concurrent
callers get an explicit 'already running' error.
- apply_update now requires the .download-complete marker AND
re-verifies every staged component (size + SHA-256 + BLAKE3)
against the manifest before touching the system.
- cancel_download only wipes staging when no operation holds the
lock; otherwise it just flags the in-flight loop to bail.
- Fixed the 'file already complete' path in
download_component_resumable, which skipped verification and fell
through to the bogus 'download failed without a captured error'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chromium's SingletonLock is a symlink encoding <hostname>-<pid>. After a
node rename changes the OS hostname, the lock left from the previous
boot reads as "another computer" holding the profile; Chromium refuses
to start, --noerrdialogs hides the only dialog, and the kiosk
black-screens forever.
Two layers:
- kiosk launcher removes Singleton{Lock,Cookie,Socket} before every
Chromium start (any prior owner is dead at that point — pkill at
session start / loop respawn). Ships via ISO splice AND the binary's
include_str! self-heal, so existing nodes get it with the next OTA.
- the rename handler clears the lock immediately as a hostname
side-effect, covering nodes still running a pre-fix launcher.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
auth.setup only wrote the web password hash to user.json, so the
archipelago system user kept the image default password ("archipelago")
on console/SSH even after the user picked a real password during
onboarding. Reuse the existing usermod-based sync (already used by
auth.changePassword's alsoChangeSsh path) at setup time, best-effort so
a failure can never break onboarding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Lightning setup (Open a Shop, Accept Payments, Run a Lightning Node)
gains a guided path to a working channel:
- New "Fund Your Bitcoin Wallet" step, gated on the blockchain finishing
its initial sync: while syncing it shows a live progress bar with a
counting-down time-remaining estimate; once synced it shows the on-chain
balance and a "Fund Wallet" button that opens the receive modal with a
fresh address, QR, and the Zeus channel limits (min 150,000 / max
1,500,000 sats) noted.
- When the sync finishes while a Lightning setup is mid-flight, a toast
pops with a "Finish setup" link straight back to the wizard (toasts now
support action links). If several setups are in flight, one is chosen —
the shared steps complete the lightning part of any of them.
- Channel steps are Zeus-branded (logo + copy) and land on the Lightning
Channels screen, which now carries an "Open a channel with Zeus" card
that prefills the open-channel modal with the Olympus peer URI, 150k
sats, and private-channel checked. "Get Zeus" links to zeusln.com.
- Setup completion now actually requires walking the manual steps (fund,
open channel, configure) — previously any step whose app was installed
was silently auto-ticked and running apps marked the whole goal done.
- Completion CTAs now go to the app you just set up ("Go to my shop
(BTCPay)" etc.) instead of the generic services list; iframe apps open
in the on-top app overlay.
- On-chain send modal gains a "Send all funds" sweep toggle, backed by
LND's send_all on the backend (amount no longer required when sweeping).
- Demo/mock: bitcoin.getinfo now returns the block_height/sync_progress
contract the UI reads and simulates a ~90s IBD ramp per visitor so the
timer, toast, and fund flow can all be demoed live; channel list data
fixed (status/channel_point/liquidity totals — the panel previously
crashed on the missing status field); sendcoins supports send_all.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PodmanClient::pull_image had a single attempt with a hard 600s wall
clock — on a fresh node, any image that legitimately takes >10 min to
download got killed mid-pull and failed the install. Podman keeps the
completed layers, so the user's retry started half-done and succeeded:
the exact "first install fails, second works" pattern reported for
btcpay and indeedhub (and earlier for LND, whose fix in 72d7fa07 only
covered the legacy install path — this is the same cure ported to the
orchestrator path both fresh installs and upgrades use).
A pull is now only killed when nothing is observably happening — no
stderr output AND no staged-byte growth in TMPDIR for 180s — or at a
1800s absolute ceiling, with 3 attempts and backoff on top. Dead
registries still fail fast (no bytes ever land). Failures now surface
podman's actual stderr tail instead of a bare exit status, and the
pull stages via the user's containers tmp like every other pull path
(rootless /var/tmp is read-only).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Orchestrator-installed stacks (indeedhub, btcpay, netbird, immich,
mempool) sat at "Preparing… 5%" for the entire install — the
install_stack_via_orchestrator path never called set_install_phase /
set_install_progress after the initial Preparing, so a 7-image,
multi-gigabyte indeedhub pull gave the user zero feedback ("just says
preparing for ages"). The legacy installers already report phases and
the frontend already interpolates X-of-N across the PullingImage band;
the new path simply never fed it.
Now: PullingImage phase up front, an X-of-N counter plus a per-member
"Installing postgres (2 of 7)…" message as each member installs, and
the legacy installers' truthful PostInstall→Done settle at the end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full backup carried identity/lnd_aezeed.enc but NOT the per-node
wallet secret that encrypts it (secrets/lnd-wallet-password), so a
restored node could never decrypt its Lightning seed. The secrets dir
now rides the backup (archive v3; the .bak itself is
passphrase-encrypted). Restore gains a staging-presence guard so
restoring an older archive without a secrets dir can no longer displace
and delete the node's live secrets — covered by two new tests.
Backups can now also be downloaded through the browser: a session-gated
GET /api/blob/backup/<id> endpoint (under the existing nginx /api/blob
prefix, so no fleet nginx changes) plus a Download button next to
Verify / USB / Restore / Delete in Settings → Backup & Restore.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The health monitor opened a bare TCP connection to every published host
port each cycle. Against TLS listeners that connect+close logs noise on
the app side — LND printed 'http: TLS handshake error … EOF' endlessly.
Containers that define a podman healthcheck (LND uses lncli getinfo)
now rely on it alone; bare containers keep the port probe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two confirmed fresh-node failure modes made 'install LND' fail on the
first try and succeed on the second:
- The dependency gate failed fast when Bitcoin was still mid-install
(image pulling, container not created yet): DepProbe.existing only
covered 'podman ps -a', so an in-flight dependency looked identical to
'not installed at all'. The probe now also reports packages in
Installing/Updating state, and the gate waits up to 30 minutes for an
in-flight dependency install to finish (surfacing 'Waiting for X to
finish installing…' on the card) before its normal 3-minute
started-but-not-running window.
- Image pulls had a hard 300s per-URL wall clock, which killed
slow-but-progressing pulls of large images (LND on fresh-node WiFi);
the retry then succeeded off the warm layer cache. Pulls are now
stall-aware: killed only after 3 minutes with no podman output AND no
byte movement in the staging dir, with a 30-minute absolute ceiling.
Dead registries still fail in ~3 minutes.
Adds dep-gate unit tests for the in-flight-install wait and the
neither-installed-nor-installing fail-fast.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
server.set-name ran hostnamectl but left Debian's 127.0.1.1 line on the
old name (breaking sudo's 'unable to resolve host' and hostname -f) and
waited on avahi to notice the change by itself. Sync /etc/hosts and
avahi-set-host-name (fallback: try-restart avahi-daemon) right after a
successful rename, so http(s)://<hostname>.local works immediately —
best-effort, never blocks the rename.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Add Service' only worked for the 19 apps in the hardcoded
known_service_port map — everything else failed with 'Unknown app'.
tor.create-service and tor.toggle-app now fall back to the app's live
launch address from the scanner state, so any manifest-driven app gets
a .onion without a per-app entry (the static map still wins for
protocol services like bitcoin, which must expose 8333, not a UI port).
toggle-app also no longer writes a broken port-0 HiddenServicePort for
unknown apps.
Every successful package.install now auto-creates the app's hidden
service (detached, best-effort, retries while the scanner derives the
port; skips protocol services and existing entries).
The demo mock gains stateful tor.create-service/tor.delete-service so
the demo round-trips instead of erroring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a fresh ISO node, NetBird's own WireGuard tunnel (10.44.0.1) sorted
ahead of the real NIC in hostname -I, so the NetBird launch URL (and the
{{HOST_IP}} baked into its cert/config) pointed at the tunnel instead of
the LAN address. The main routing table's default route names the
physical uplink even while a VPN is active (NetBird/Tailscale steer
traffic via policy-routing rules, not the main table), so read src/dev
from 'ip -4 route show default' first, then fall back to a connected UDP
socket's source address, then to the old hostname -I scan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The optimistic package_data entry painted at install-spawn survived
failure as a 'Stopped' tile for an app with zero footprint (netbird on
the fresh ISO). On INSTALL FAIL with no container/user-stop marker for
the app, the entry and the pre-saved dynamic app config are removed and
an error notification is pushed; failures that left a real container
keep today's behavior (visible exited state for retry/diagnosis).
Install suite 39/39.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
write_bitcoin_conf silently produced an unpruned (~810GB and growing)
config regardless of disk: the framework node got a full-archive chain
on a 205GB volume. Volumes under 1.2TB now get prune = 25% of the
volume, clamped to [550MB, 100GB]; bigger volumes keep full archive;
unknown df results stay unpruned (don't surprise big iron). Framework
node switched live to prune=50000.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
936b4cca made ANY loaded manifest with an absent container self-heal via
install_fresh — but the signed catalog and the ISO ship manifests for
EVERY app, installed or not, so 'manifest loaded' is not installation
evidence. On a fresh node this mass-installed the catalog: the framework
node grew portainer, vaultwarden, searxng, strfry, mempool, immich and
fedimint uninvited within an hour (fleet nodes showed the same loop as
reconcile-failed noise for unpullable images).
ExistingOnly (boot) reconcile now creates containers from nothing only
for required-baseline apps (filebrowser, fedimint-clientd); every other
absent app is Left('absent') and recreated ONLY via desired-state
recovery when it was running at the last snapshot — which still covers
the vanished-container regressions (indeedhub/immich) that motivated
936b4cca. Reconcile suite 16/16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plugging in a LoRa radio now pops a global two-step setup modal on any
page: step 1 shows the ACTUAL detected board (25 board SVGs vendored
from the Meshtastic web-flasher, GPL-3.0; matched by USB product string
on native-USB boards, vid:pid heuristics for bridge chips, animated
fallback otherwise); step 2 configures with Archipelago presets — the
LoRa region pre-selected from the node's location (new lat/lon→region
mapping), channel 'archipelago', node name, and an advanced firmware
pin. Options adapt per protocol: region+channel program Meshtastic;
MeshCore shows its community frequency plan (firmware owns RF); RNode
defers to the Reticulum daemon config.
Backend: mesh.configure now accepts lora_region (validated against the
driver's region table) and device_kind (probe pin); mesh.status returns
the persisted values plus per-port USB identity (sysfs vid/pid/product)
for board identification. The Mesh Device tab gains a full settings
form (region with duty-cycle/legality hints, firmware pin, channel,
name, identity broadcast) replacing the read-only panel; the old
Mesh-page-only onboarding modal is superseded by the global flow.
Mock backend: stateful mesh config so the dev UI demos the full
detect→configure round-trip (disable mesh → modal fires with Heltec V3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the fresh-install triple failure (LND needing ~5 install
attempts, Bitcoin UI bitcoin-rpc 502, backend getblockchaininfo errors):
the legacy conf writer set rpcallowip but never rpcbind, so bitcoind
bound RPC to the container's loopback only — every dial over the
container network (LND, bitcoin-ui) and the rootless port-forward was
refused. LND exited instantly, the 60s post-start poll read that as
INSTALL CRASH, and the wallet (and its seed backup) never got created.
- write_bitcoin_conf + ensure_bitcoin_rpc_config now emit/ensure
rpcbind=0.0.0.0 (host exposure unchanged: publish stays 127.0.0.1)
- new install gate: bitcoin-dependent apps (lnd/electrs/electrumx/
mempool-electrs/btcpay) wait up to 3 min for an authenticated
getblockchaininfo to answer before their container starts, with
INSTALL WAIT progress lines; on timeout the install fails with an
actionable message instead of a crash-looping container
Verified live on the fresh-ISO framework node: after adding rpcbind and
restarting bitcoin-knots, LND connected, created its wallet, baked
macaroons, and settled into 'Waiting for chain backend to finish sync'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tests landed twice (interrupted edit applied both paths), breaking
the test build with E0428. Federation suite: 33/33 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Invites carried no trust level, and both accept_invite and the
peer-joined handler hardcoded TrustLevel::Trusted — so 'Invite a Peer'
(Observer) federated both sides as fully Trusted.
- invite codes now carry a 'trust' field (legacy codes parse as Trusted)
- federation.invite accepts trust_level: trusted|observer
- accept_invite assigns the invite's level on the acceptor side
- peer-joined resolves the granted level authoritatively by matching the
acceptor's echoed invite token against our stored outgoing invites;
the peer's own unsigned claim is honored only as a downgrade, so no
escalation is possible
- discovery/connection-request approvals mint Observer invites directly
(the post-hoc demotion in handshake.rs stays as a legacy safety net)
- asymmetry self-heal re-asserts at the locally-held level
Tests: observer threading + legacy default-trusted parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
telemetry.fleet-status only read the telemetry-fleet/ collector inbox —
an opt-in anonymous-telemetry pipeline that federated peers never write
to — so Fleet showed 'No nodes reporting' even with a healthy federation.
Build the fleet list from nodes.json instead: every TRUSTED federated
node is mapped from its last_state sync snapshot (cpu/mem/disk, uptime,
apps, peer count) into the FleetNode shape the UI expects, keyed by DID.
Observer ('peer') and untrusted nodes are excluded by design. Collector
reports are still merged in for non-federated telemetry nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Web5ConnectedNodes sent { request_id } but the backend only read
params.id, so accepting/rejecting a connection request always failed
with 'Missing required parameter: id'. Frontend now sends id; backend
accepts both spellings for older deployed UIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aezeed words are now captured at BOTH wallet-init paths and stored
encrypted (Argon2 + ChaCha20-Poly1305, per-node wallet secret) at
identity/lnd_aezeed.enc; ack marker cleared when a wallet is recreated
- lnd.init-wallet-from-seed was posting seed_entropy to /v1/initwallet,
which is a GenSeed field — the wallet was never actually derived from
the master seed; now GenSeed(entropy) → InitWallet(words)
- new RPCs: lnd.seed-backup-status / lnd.seed-reveal (password + 2FA
gated, same as seed.reveal via shared verify_reveal_auth) /
lnd.seed-backup-ack
- LND app detail page: Lightning-seed card with first-launch backup
prompt, tap-to-reveal modal, 'I've backed it up' acknowledgment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- new lnd.lightning-history RPC: settled invoices (incoming) + succeeded
payments (outgoing) normalized to the wallet-transaction shape; on-chain
history stays in lnd.gettransactions
- Home merges on-chain + lightning + ecash history into one sorted list;
every entry now carries kind (onchain/lightning/cashu/fedimint)
- modal renders a rail badge for non-onchain rows instead of a bogus
confirmation count, and only on-chain rows link into mempool
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same kiosk.service conflict resolution as the previous merge (branch carried
rebased duplicates of the kiosk commits): CPUQuota=200% + main's memory tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
