Root cause of 'server starting up' forever / crash-on-install
(framework-pt, v1.7.114->115, 2026-07-26): on a node with many stacks,
the synchronous boot recovery (recover + start_stopped_containers) runs
BEFORE sd_notify(Ready), so the unit sits in 'activating' for minutes.
Anything touching the service in that window — a superseding
start/restart, an install-time reconcile churn — killed a half-started
instance; it exits 0 on SIGTERM and Restart=on-failure then never
restarts it. Node dead behind 'server starting up'.
Fixes:
- signal READY (+ start the watchdog keepalive) BEFORE boot recovery, so
the unit reaches 'active' in seconds; recovery/reconcile/listener
continue after. No more minutes-long activating window.
- Restart=always (was on-failure): a clean-exit SIGTERM must still bring
the daemon back. Manual is still honored.
- OTA restart via a PID1-owned transient timer (systemd-run --on-active=2)
instead of a tokio-sleep child of the process being stopped, whose
start-half was being lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The companion reaches the node at its fips0 ULA, and the web UI builds app
links as http://[<ULA>]:<direct port> — but rootless-podman published ports
bind 0.0.0.0 only, so every app URL was refused over the mesh (:8334 first).
A reconcile loop in the backend mirrors public IPv4 listeners: any port
>=1024 on 0.0.0.0 without an IPv6 any-listener gets a v6-ONLY [::] forwarder
to 127.0.0.1, following /proc/net/tcp* so install/remove and hardcoded
companion ports are covered without touching a single container. Purely
additive: IPv4/LAN/Tor access paths are untouched, v6only listeners cannot
collide with or intercept v4 traffic, and foreign IPv6 listeners win.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- auth.createDeviceToken / listDeviceTokens / revokeDeviceToken RPCs; only
SHA-256 hashes persist in data_dir/device-tokens.json
- auth.login accepts {token} (same rate limiter, skips TOTP like remember-me)
- pairing QR now carries name (server name, fallback "My Archipelago"),
tok (instant login), and FIPS mesh params from new fips.pair-info RPC
(npub, fips0 ULA, transport ports)
- companion onboarding drops the WireGuard install/tunnel screens — remote
access moves to the FIPS mesh embedded in the companion app
- fix: fips daemon UDP bind now 2121, matching the published container port
and fleet rosters (was upstream's 8668 — inbound UDP was dead on bridged
installs, mesh silently rode TCP 8443)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stray `archipelago --version` used to start a full second instance next
to the systemd one (issue #74). Handle -V/--version, -h/--help, and reject
unknown dashed options before any tracing/state init, mirroring the
ceremony subcommand's clean-stdout precedent. Version output matches the
health RPC format: <pkg-version>-<git-hash|dev>.
Fixes#74
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a fresh ISO node, NetBird's own WireGuard tunnel (10.44.0.1) sorted
ahead of the real NIC in hostname -I, so the NetBird launch URL (and the
{{HOST_IP}} baked into its cert/config) pointed at the tunnel instead of
the LAN address. The main routing table's default route names the
physical uplink even while a VPN is active (NetBird/Tailscale steer
traffic via policy-routing rules, not the main table), so read src/dev
from 'ip -4 route show default' first, then fall back to a connected UDP
socket's source address, then to the old hostname -I scan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reconciler doesn't take the RPC layer's per-app FIFO op lock (known
limit of 891cbba4): between a restart worker's stop and start halves it saw
the mempool frontend "missing", repair-recreated it behind systemd's back,
killed the worker's fresh container 11s after start, and left the unit down
for ~3.5 min until the next heal — gate test 123 measured exactly that
window (.228 iteration 3, 2026-07-09).
New crate::app_ops module owns the op-lock registry + stack member table
(runtime.rs and dependencies.rs now delegate) so the reconciler can probe
lifecycle_op_in_flight(app_id) — covering both the app's own key and its
owning stack package — and skip that app for the cycle. The ownership-sweep
podman restart gets the same guard. Health monitor is a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tollgate-wrt keeps its own separate Cashu wallet on the router
(/etc/tollgate/wallet.db) — customer payments land there, never in
this node's own wallet. Its Lightning auto-payout is configured
independently in /etc/tollgate/identities.json, which is easy to
leave misconfigured or pointed at a placeholder address (found live:
both "owner" and "developer" identities on this deployment pointed at
the same unconfigured tollgate@minibits.cash default).
Rather than depend on getting that Lightning payout config right,
tollgate_sweep::sweep_once() periodically (every 5 min, via SSH)
checks the router's `tollgate wallet balance`, and if nonzero, runs
`tollgate wallet drain cashu` and receives the resulting token(s)
straight into the local wallet via the same path the "Receive ecash"
UI uses (wallet::ecash::receive_token) — bypassing Lightning payout
entirely. A no-op (Ok(0)) if no router is configured or it doesn't
have TollGate installed.
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turn on registry-distributed manifests for all apps: generate-app-catalog.sh now
embeds each apps/<id>/manifest.yml by default (EMBED_MANIFESTS opt-out), so nodes
install from the signed catalog (origin-wins overlay, disk = fallback) with no
OTA-shipped disk manifest. main.rs awaits a bounded (25s) refresh_catalog before
load_manifests so a fresh boot overlays the latest embedded catalog instead of a
restart later; offline/ISO boot falls through to disk and never hangs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Orchestrator-installed backends (immich, btcpay-db, …) run as plain podman
`--restart=unless-stopped` containers until the Phase-3 Quadlet rollout flips
use_quadlet_backends on. Nothing in the codebase enabled the user's
podman-restart.service, so those containers had NO reboot-survival mechanism.
Enable it (idempotent, best-effort) at orchestrator startup so unless-stopped
containers come back after a reboot. Already applied manually on .228 (covers
31 containers incl. immich + btcpay); this codifies it fleet-wide.
The deeper fix (render Quadlet for all orchestrator installs) remains the gated
Phase-3 Quadlet-everywhere rollout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workstream B phase 1 (node-side consume). The signed app-catalog can now carry a
full manifest per entry; the orchestrator overlays it over the disk manifest
(origin-wins) with disk as the migration fallback. Moves apps toward
registry-distributed manifests with no OTA-shipped disk file.
- app_catalog: `manifest: Option<Value>` on AppCatalogEntry (forward-compatible,
covered by the existing release-root signature over the raw JSON);
`catalog_manifest_values()` accessor.
- prod_orchestrator: `load_manifests` overlays catalog manifests after the disk
walk; `catalog_manifest_to_overlay()` returns None (→ disk fallback) on
unparseable value / app-id mismatch / failed validate() / build source
(build contexts aren't registry-distributed yet — phase 1 is image-only).
- manifest_dir stays PathBuf (build-only field); image-only apps never read it.
- 6 unit tests; compiles clean. No-op until a catalog embeds a manifest, so
existing nodes are unaffected.
See docs/registry-manifest-design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Buyer-side paid downloads now persist: purchases are cached on disk
(content_owned.rs) keyed by (seller onion, content_id), the gallery shows
an "Owned" badge unblurred, and items view/play in-app from the local
cache with no re-payment or reliance on a browser download (which
silently failed on the mobile companion). New RPCs content.owned-list /
content.owned-get. Validated e2e .116<-.198 (paid 100 sats via Fedimint,
166KB jpeg returns, survives restart).
fedimint-clientd manifest: restore the standard container capability set
(CHOWN/DAC_OVERRIDE/FOWNER/SETUID/SETGID) so fmcd's startup chown of an
existing-federation /data succeeds instead of dying EPERM (#7). Confirmed
the orchestrator applies these to the running container.
FIPS perf: tighten the supervisor warm-path keepalive 45s -> 25s so peer
paths stay inside the ~30-60s NAT cold window. Dials now reliably land on
FIPS instead of re-punching and falling back to Tor. Measured to the same
peer: cloud browse 18-22s -> 0.4s; full Fedimint paid download 29s -> 11s
(residual is the seller-side guardian reissue round-trip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lands the transport/swarm orchestration layer (the iroh engine attaches
later, behind a flag). The seam is fully exercised today with the origin
HTTP path; with no swarm providers registered the behaviour is byte-for-byte
identical to before.
- swarm/mod.rs: BlobProvider trait + fetch_content_addressed() — tries each
provider in order, VERIFIES peer-sourced bytes against the content digest
before accepting (untrusted seeds can't inject tampered bytes), falls back
to the origin closure if none serve. Returns Swarm|Origin.
- Cargo: iroh-swarm feature (off by default; heavy QUIC dep tree attaches
here). providers() is empty until enabled → every fetch hits origin.
- update.rs: components with a BLAKE3 digest route through the seam, using
the existing resumable HTTP downloader as the origin fallback; a swarm hit
is re-checked against the mandatory SHA-256 manifest gate (re-fetch from
origin on any disagreement). Components without blake3 take the original
path untouched.
44/44 swarm/update/content_hash/blobs tests pass (incl. swarm hit/miss,
tampered-bytes-rejected→origin, fall-through ordering).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the iroh-native, range-verifiable hash next to the incumbent SHA-256
so the swarm can later fetch/verify by BLAKE3 with the registry/origin as
fallback. Non-breaking: SHA-256 stays the mandatory gate; BLAKE3 is verified
only when present.
- content_hash.rs: HashAlg + ContentDigest (parse/verify '<alg>:<hex>'
multihash strings), blake3_hex/sha256_hex; BLAKE3 known-answer test
- update.rs: ComponentUpdate.blake3 (serde-default); verified ALONGSIDE
SHA-256 in the resumable download loop, re-download on mismatch
- blobs.rs: BlobMeta.blake3 computed on put (on-disk path stays
SHA-256-keyed for back-compat; advertises the future swarm address)
Drive-by: fix a pre-existing stale test (test_save_and_load_state_roundtrip)
that never wrote the .download-complete marker #26 requires, so load_state's
self-heal cleared update_in_progress. Unrelated to BLAKE3 — surfaced by
running the full update:: suite.
40/40 content_hash/update/blobs tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the parked trust module and wires it into the live build:
- main.rs: register `mod trust`
- app_catalog::fetch_one: verify the release-root detached signature when
present (verify against raw JSON so forward-compat fields stay in the
signed preimage); accept unsigned during the migration window, hard-reject
a present-but-bad signature so a tampering mirror can't pass altered bytes
- seed: pin release-root Ed25519 known-answer test (priv+pub) for the
signing ceremony / pinned-anchor / external-verifier cross-check
- signed_doc: drop unused import
20/20 Phase 0 unit tests pass (trust::canonical/did/signed_doc/anchor,
seed release-root, app_catalog). Crate compiles clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kiosk chromium pinned ~92% of a core (software-compositing spin from
--enable-gpu-rasterization on a GPU-less/headless node), saturating the machine
and starving the backend + container builds — it caused the .198 receive timeout
and the deploy storms.
- archipelago-kiosk.service: CPUQuota=75% + MemoryMax/High + Delegate, so a
runaway kiosk can never take the whole node down.
- archipelago-kiosk-launcher.sh: detect /dev/dri — use GPU rasterization only
when a GPU exists, else --disable-gpu (avoids the headless spin).
- bootstrap::ensure_kiosk_hardened: OTA self-heal that installs the updated
unit+launcher on already-deployed nodes, daemon-reloads, and only try-restarts
a *running* kiosk (never re-enables an operator-disabled one).
cargo check clean; launcher bash -n clean; unit syntax valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
immich_server/redis/postgres + indeedhub-* are multi-container stack members
whose sub-container app_ids are NOT in package_data, so the health monitor skips
them as "orphans" and never restarts them when they exit — Immich/IndeedHub stay
down until the next reboot (the boot-only start_stopped_stack_containers was the
only recovery). Spawn a 120s supervisor that reuses that same recovery at
runtime. It cheaply skips already-running containers and honours the user-stopped
list (set on every container by package.stop), so it only revives genuinely
crashed members and never fights a user stop.
cargo check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User: chat history (messages + mesh/Tor contacts) must persist and be
secure/encrypted per best practice. Root cause of the .198 loss was the B17
mount race writing empty stores over real data (B17 already fixes the trigger);
this hardens storage so it can never silently lose or expose data:
- storage_crypto: shared at-rest envelope mirroring credentials::store — key =
SHA-256(domain ‖ node identity key) (seed-derived, per-store domain
separation), ChaCha20-Poly1305 AEAD with a random 96-bit nonce, tamper-evident.
Transparent migration of legacy plaintext files. Unit-tested (round-trip,
wrong-key/tamper rejection, plaintext detection).
- messages.json: encrypted at rest + ATOMIC write (temp+rename) so a crash/
reboot mid-write cannot corrupt history; decrypt-with-migration on load; a
failed decrypt never overwrites the on-disk data.
- mesh contacts (alias/notes/pinned/blocked): were ONLY in memory and lost on
every restart — now persisted to mesh-contacts.json (encrypted, atomic),
loaded on MeshState startup, saved after contacts-save/contacts-block.
Explicit clear (mesh.clear-all) still wipes everything, as intended.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User priority: FIPS is the main transport but it was unreliable and needed a
manual "Activate" button. Improvements (all in the FIPS dial/supervisor):
- Auto-activate: ensure_activated() installs the daemon config + starts the
service on its own once seed onboarding has materialised the key — no Activate
button needed. Idempotent; runs from the supervisor every 45s so a node that
onboards after boot still comes up automatically.
- Dial retry: try_fips_get/post now retry ONCE on a connect/timeout error. The
first dial to a peer triggers NAT hole-punching and often times out before the
path is up; the retry lands on the now-warm path — the main reason calls were
dropping to Tor despite the peer being FIPS-reachable.
- More patient connect_timeout (5s→8s) so a reachable-but-cold peer isn't
abandoned to Tor while hole-punching completes.
- Path warmer: spawn_fips_supervisor() keeps hole-punched paths to known
federation peers warm (every 45s, concurrent), so on-demand dials are fast and
land on FIPS.
- Confirmed the daemon config already enables BOTH udp + tcp transports
(render_config_yaml), so FIPS already uses TCP where UDP is blocked; the Tor
fallback was path-establishment, addressed above.
cargo check + fmt clean. Backend — needs a binary rebuild+deploy to validate on
.116/.198 (watch last_transport flip fips, and FIPS coming up with no button).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On production nodes /var/lib/archipelago (the app data dir AND podman's
graphroot=/var/lib/archipelago/containers/storage) is a separate
device-mapper volume. archipelago.service ordered only After=network-online
.target, so on cold boots it (and its ExecStartPre) could start BEFORE
var-lib-archipelago.mount, write to the bare mountpoint on rootfs, fail every
podman call, exit, and be restarted every 5s until the volume mounted — the
"~20x [FAILED] Failed to start over ~5min" boot flap. Proven live on .198:
"var-lib-archipelago.mount: Directory /var/lib/archipelago to mount over is
not empty, mounting anyway" — the service had written there pre-mount.
Fix: RequiresMountsFor=/var/lib/archipelago (adds Requires= + After= on the
mount unit).
- image-recipe/configs/archipelago.service: ships the directive on fresh ISOs.
- bootstrap::ensure_archipelago_mount_ordering(): self-heals already-deployed
nodes' installed unit + daemon-reload (boot-ordering only, effective next
reboot; never restarts the running service). Idempotent; harmless on rootfs
installs (maps to the always-mounted root).
Verified on .198: after applying, systemctl shows After=var-lib-archipelago
.mount and systemd-analyze verify is clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 6 of the rust-orchestrator migration. Construct the container
orchestrator once in main.rs, call load_manifests + adopt_existing
immediately after Config::load, log the adoption report, and spawn
BootReconciler::run_forever with the 30s default interval. Thread the
orchestrator through Server::new -> ApiHandler::new -> RpcHandler::new
so the reconciler and RPC layer share one instance.
Wire a tokio::sync::Notify through the SIGTERM/SIGINT shutdown path so
the reconciler exits cleanly alongside the server drain. Uses notify_one
so the signal stores a permit if the reconciler is mid reconcile_all
when the signal fires.
Delete the commented-out run_boot_reconciliation block in main.rs that
documented the prior bash-script approach being unsafe on unbundled
installs — the new reconciler is manifest-driven and only touches apps
present in /opt/archipelago/apps, fixing that concern.
cargo check -p archipelago clean (6 pre-existing dead-code warnings on
trait methods not yet exercised until Step 9 hot-swap). Container test
suite 43/44 pass; the one failure (container::image_versions::
test_parse_image_versions) is pre-existing and unrelated.
Closes failure mode FM5 from docs/bulletproof-containers.md: the v1.7.38 +
v1.7.39 rollouts left every affected node on an unreachable UI (nginx 500)
with no recovery path short of SSH. This release adds a self-check
guardrail to the update flow.
What changed:
- apply_update() writes a pending-verify marker with old+new version and
a 150s deadline immediately before scheduling the service restart.
- verify_pending_update() runs from main.rs startup. If the marker is
present and within its freshness window, the new binary waits 15s for
nginx + backend to settle, then probes https://127.0.0.1/ every 5s for
up to 90s (self-signed certs accepted).
- On any probe success within the window, the marker is cleared and
nothing else happens.
- On window-exhaust, the new binary:
1. Moves the broken /opt/archipelago/web-ui to web-ui.failed.<ts>
(quarantined, not deleted, so we can post-mortem).
2. Restores web-ui.bak on top of web-ui.
3. Calls rollback_update() to restore the previous binary.
4. Updates state.current_version to reflect the rollback.
5. systemctl --no-block restart archipelago so the OLD binary boots.
- Markers older than 10 minutes are treated as stale and cleared without
probing, so a crashed-during-startup marker from weeks ago cannot
spontaneously roll back a healthy node on a later reboot.
- rollback_update() binary copy now goes through host_sudo instead of
tokio::fs::copy, so it escapes the service's ProtectSystem=strict
mount namespace. Without this, the rollback silently failed with
EROFS on /usr/local/bin and orphaned the rollback - the exact
opposite of what auto-rollback is for.
Tests: 4 new unit tests in update::tests covering marker round-trip,
absent-marker noop, no-panic on verify_pending_update with nothing to
verify, and an invariant assert that the 90s probe window stays below
the 600s stale threshold. All passing.
Side fix: scripts/create-release-manifest.sh was dying with exit 141
(SIGPIPE from tar tvzf pipe head pipe awk) under set -euo pipefail.
Replaced with a single awk NR==1 that doesn't short-circuit the upstream
pipe, so the release-build flow is idempotent again.
v1.7.38 shipped with an OTA bug: the tar-extracted staging dir inherited 700
perms and nginx (www-data) returned 500/403 on every request after the swap.
.116 hit this on rollout; had to chmod by hand to recover.
- update.rs: after extraction, explicitly chmod 755 dirs + 644 files on the
new staging dir before the mv into place, so nginx can stat/serve them.
- main.rs: self-heal on startup — if /opt/archipelago/web-ui is not
world-readable, run `sudo chmod -R u=rwX,go=rX` to repair. This is what
rescues nodes upgrading from v1.7.37/v1.7.38, since their extractor
(running on the old binary) doesn't have the chmod fix yet — the new
binary's first boot fixes the mess before nginx serves a single request.
Everything v1.7.38 shipped is still in this release:
- auth.rs auto-heals is_onboarding_complete() from setup_complete +
password_hash so nodes don't bounce back to /onboarding/intro after
browser clear / reboot / update
- useOnboarding tri-state: backend-unreachable no longer defaults to intro
- login sounds gated by isFirstInstallPhase() — silent after onboarding,
typing sounds unaffected
- FIPS app / Nostr Relay / Nostr VPN / Routstr / Penpot removed from
catalog + frontend + Rust + docker + icons; 15 image versions deleted
from tx1138, .168, gitea-local
- AIUI baked into release tarball via demo/aiui/
- prebuild hook syncs app-catalog/catalog.json → public/catalog.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- core/archipelago/src/bootstrap.rs (NEW): embed scripts/container-doctor.sh
and image-recipe/configs/archipelago-doctor.{service,timer} via
include_str! and sync to disk + enable the timer on every archipelago
startup. Idempotent (content-hash compare), dev-box symlink guard keeps
the git checkout untouched, best-effort (warn-only on failure) so
bootstrap never blocks server readiness. Wired in main.rs as a
background tokio task.
- scripts/container-doctor.sh: add fix_rootless_netns_egress(). Detects
when the rootless-netns has lost its pasta tap (container-to-container
still works but outbound DNS/TCP fails) via an nsenter probe into
aardvark-dns; with a two-probe 10s debounce to rule out transients and
a host-precheck that bails out if the host itself is offline. When the
rootless-netns is truly broken, does a graceful podman stop --all /
start --all so pasta + aardvark-dns rebuild the netns from scratch.
Bitcoin-knots and every other outbound container recover in one cycle.
- core/archipelago/src/update.rs: host_sudo → pub(crate) so bootstrap.rs
can reuse the existing systemd-run escape hatch.
- apps/bitcoin-core/manifest.yml: bump app version 24.0.0 → 28.4.0 and
image bitcoin/bitcoin:24.0 → bitcoin/bitcoin:28.4. Resources aligned
with the real container-specs.sh large-disk tune (4 GiB memory cap,
cpu_limit: 0 so bitcoind can run -par=auto across every core).
- neode-ui/src/views/apps/AppCard.vue + Apps.vue: add an Update button
+ Updating spinner to every app card that has available-update set.
Wires through serverStore.updatePackage(id) — the same RPC the detail
view already calls. common.update / common.updating i18n keys added in
en.json and es.json.
- core/archipelago/src/identity_manager.rs: add create_from_signing_key()
that mirrors an existing Ed25519 key as a manager-level identity with
a deterministic id (`node-<pubkey16>`). Idempotent across restarts,
gets the hex-SVG master avatar.
- core/archipelago/src/server.rs: the auto-create path on first boot now
mirrors the node's own signing_key (seed-derived on onboarded installs)
as a "Node" identity instead of generating a random "Default" keypair.
Once this ships, the DID on the Web5 DID Status card (via node.did
RPC), the Node entry on the Identities page (via identity.list), and
the DID used for peer-to-peer connects (via server_info.pubkey) all
resolve to the same seed-derived pubkey.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- HOTFIX: v1.7.31-alpha's frontend tarball was packaged with a
`neode-ui/` top-level directory instead of the flat layout v1.7.30
and earlier used. Nodes that applied v1.7.31 ended up with
`/opt/archipelago/web-ui/neode-ui/index.html` instead of
`/opt/archipelago/web-ui/index.html`, and nginx returned 403/500.
v1.7.32's tarball is built with `tar -C web/dist/neode-ui .` so
files land directly at web-ui root. Broken nodes auto-heal on this
update (web-ui dir is replaced).
- transport/lan.rs: add Drop impl that calls ServiceDaemon::shutdown()
on the mdns_sd daemon. Without this the OS thread it spawns, plus
the blocking `receiver.recv()` task, keep the tokio runtime alive
past SIGTERM — long enough for systemd's TimeoutStopSec to SIGKILL
the service and mark it Failed. Was visible on every update:
"shut down cleanly" logged, then 15s later systemd forcibly kills.
- main.rs: after logging "Archipelago shut down cleanly", call
`std::process::exit(0)` explicitly. Belt-and-suspenders against
any future non-daemon thread creeping in (reqwest resolver pool,
etc.) and causing the same SIGKILL regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 1fb71b4b on the same v1.7.0-alpha line.
Identity avatars
• New module `avatar.rs` generates two deterministic SVG styles keyed
off the pubkey: a 5×5 mirrored identicon for sub-identities and a
hexagonal-network motif for the master (seed index 0) identity.
Both returned as base64 data URLs, so a fresh identity has a
recognisable picture before the user uploads anything.
• `IdentityManager::create()` and `create_from_seed()` populate
`profile.picture` on creation. Index 0 gets the node SVG; all
other seed-derived + ad-hoc identities get the identicon.
Blob store — public flag for profile assets
• `BlobMeta.public` (default false) added; `BlobStore::put()` takes
a `public: bool`. Missing in legacy meta files = false.
• `POST /api/blob` now stores uploads with public=true and returns
`public_url` alongside `self_test_url`. public_url is
`http://<node-onion>/blob/<cid>` (no cap) if Tor has published the
archipelago hidden service, else falls back to the local path.
• `GET /blob/<cid>` bypasses the HMAC capability check when the
requested blob is flagged public — external Nostr clients fetching
a kind-0 `picture` URL can't hold a cap.
• Mesh callers (content_ref attachments, dispatch rehydration) pin
public=false explicitly so nothing leaks out of the mesh path.
Profile editor UX
• Collapsed Save + Save & Publish into one button — the Save action
now persists locally AND publishes the kind-0 metadata event in
one step. Uploads store `public_url` into `profile.picture` /
`profile.banner` so the published URL is reachable by external
clients.
Update client — the 15-second cliff
• Frontend `rpcClient.call` for `update.download` now has an
explicit 30-minute timeout (was falling back to the default 15 s).
`update.apply` gets 5 min, `update.git-apply` gets 15 min. Matches
what the backend is actually willing to wait for.
• Backend `load_state()` reconciles `state.current_version` with
`CARGO_PKG_VERSION` on every start. Sideloaded or reflashed nodes
were stuck advertising the old version even with a new binary in
place, which kept re-offering the same release as an update.
Manifest changelog rewritten for fleet readers per the saved feedback
(no function names, no file paths). Artefacts refreshed:
binary 12f838c5…5ba82d 40381864
frontend dc3b63af…e9a8370 76984288
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 8b7cb002 (no version bump — same v1.7.0-alpha manifest):
* WireGuard peer persistence. Kernel peer state is ephemeral; the add-peer
RPC wrote each peer to data_dir/nostr-vpn/peers/*.json but nothing
re-pushed them on reboot. Result on .198: wg0 came up listening with zero
peers after last night's reboot. Added vpn::restore_wg_peers() — reads
the peers dir, waits up to 30s for wg0 to exist, then replays each via
`archipelago-wg add-peer`. Spawned from main.rs alongside the other
startup tasks.
* Reconcile + filebrowser drift. scripts/container-specs.sh load_spec_
filebrowser now declares SPEC_NETWORK="archy-net" (to match what
first-boot-containers.sh creates) and pins the filebrowser-data volume
+ wget-style healthcheck so the reconciler stops reporting network
drift. Without this, reconcile would kill the healthy first-boot
filebrowser container and recreate it on bridge, breaking the archy-net
DNS name the backend proxies to.
Manifest binary sha/size refreshed:
6c178a76…3582cc, 40361912 bytes.
Rebuilt ISO at image-recipe/results/archipelago-installer-unbundled-x86_64.iso
(Apr 20 07:10) carries both fixes baked in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
need an archipelago restart
Previously the server checked `fips0` once at startup; if the
interface wasn't up (pre-onboarding, or post-onboarding before the
user clicked Activate FIPS), the peer listener never bound and stayed
unreachable until the next archipelago restart.
Replaced with a `peer_late_bind_loop` background task: polls every
30s for an fd00::/8 address on `fips0` and binds the listener the
moment one appears. First tick fires immediately so the hot path —
fips0 already up at startup — is still zero-cost. Cancellation
cascades through the same `tokio::sync::watch` channel the main
listener uses.
Side effects:
- main.rs no longer computes peer_addr eagerly; dropped the unused
param from serve_with_shutdown.
- FipsTransport::is_available already caches the service probe so
the 30s poll doesn't thrash systemctl.
Covers task #21. Unblocks the first-boot + onboarding flow for
fresh ISO installs on .253.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a user-configurable toggle for how each peer-to-peer service
reaches federated peers. Three options per service:
- Auto (default) — FIPS preferred, Tor fallback (current behavior).
- FIPS only — fail rather than fall through to Tor.
- Tor only — explicit opt-in to onion anonymity for that service.
Services covered (matching the UI rows):
- Federation — state sync, invites, peer notifications
- Peers — address/DID rotation broadcasts
- Peer Files — content catalog download/browse/preview
- Messaging — archipelago channel + mesh bridge
- Mesh File Sharing — content_ref blob fetches
Implementation:
- settings::transport — persisted struct + process-wide OnceLock handle
(so deep call sites don't need data_dir threaded through signatures).
On-disk file: <data_dir>/settings/transport_preferences.json; missing
or corrupt → defaults (Auto everywhere).
- settings::transport::init() called from main.rs after config load.
- fips::dial::PeerRequest gains a .service(kind) builder; send_* checks
the preference before choosing a transport. FIPS-only fails loudly
when FIPS is unavailable (so users who pick it know when something
falls back).
- Every FIPS-first migration site tags its PeerRequest with the
matching PeerService so the toggle actually applies.
- transport.preferences + transport.set-preference RPCs added; wired
into the dispatcher.
- neode-ui/src/views/settings/TransportPrefsCard.vue — standalone card
with a 5-row Auto/FIPS/Tor tri-state. Not wired into Settings.vue —
the user places components themselves (see feedback_ui_entry_points).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the FIPS transport end-to-end so peer-to-peer calls can reach
other nodes over the mesh without going through Tor:
- fips::dial — raw RFC 1035 DNS client (zero new deps) that queries the
FIPS daemon's local resolver at 127.0.0.1:5354 for `<npub>.fips` AAAA
records. Exposes peer_base_url(npub) → "http://[fd9d:…]:5679" plus a
reqwest client factory for call-site migrations.
- fips::iface — parses /proc/net/if_inet6 to find the ULA address on
`fips0`. Runs under the archipelago service user without extra caps.
- FipsTransport::is_available() — live probe of archipelago-fips and
upstream fips.service via `systemctl is-active`, cached 10s so the
send hot path doesn't thrash DBus.
- FipsTransport::send() — resolve npub, POST TransportMessage JSON to
the peer's /transport/inbox. Today /transport/inbox isn't wired on
the receive side, so call-site migrations use dial::peer_base_url
directly against the already-signed endpoints (/rpc/v1,
/archipelago/node-message, /content/*). The inbox handler lands as
part of the Settings/transport work.
- server::serve_with_shutdown — takes an optional peer_addr and spawns
a second listener bound specifically to the fips0 ULA on port 5679.
The peer listener applies is_peer_allowed_path() — a whitelist of
endpoints that already do per-request signature auth — and returns
404 for everything else. Shutdown cascades to both listeners via a
watch channel; 5s drain window preserved.
- main.rs — if fips0 has a ULA at startup, pass the peer SocketAddr to
serve_with_shutdown; otherwise run the main listener only.
Security: the peer listener is bound to the fips0 ULA directly, not
wildcard, so it's unreachable from WAN IPv6. The path whitelist limits
exposure to endpoints whose handlers verify ed25519 signatures or
federation DID headers server-side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bakes the FIPS (Free Internetworking Peering System) mesh daemon into
the node stack, supervised by archipelago alongside Tor. Runs as a
system service, identity derives from the same BIP-39 master seed, and
user-triggered updates track upstream main.
Identity
seed.rs: new HKDF label archipelago/fips/secp256k1/v1 → dedicated
secp256k1 key, distinct from the Nostr-node key for crypto isolation
but still seed-recoverable
identity.rs: writes fips_key[.pub] to /data/identity on onboarding,
chmod 0600; fips_key_exists / load_fips_keys / fips_npub accessors
Transport
TransportKind::Fips=3 inserted between LAN and Tor (Tor bumps to 4)
→ router prefers FIPS over Tor for all peer traffic
PeerRecord gains fips_npub + last_fips fields (serde(default) for
backward-compat with older nodes)
transport/fips.rs: NodeTransport stub, reports unavailable until the
daemon is live so router falls through to Tor cleanly
Federation invites
FederatedNode and FederationInvite carry optional fips_npub
create_invite / accept_invite / peer-joined callback thread it end
to end; signature domain deliberately unchanged — FIPS Noise does
its own session auth, so the unsigned hint only affects path
selection
crate::fips
config.rs: renders /etc/fips/fips.yaml and sudo-installs key material
service.rs: systemctl status/activate/restart/mask wrappers
update.rs: GitHub API check against upstream main; apply stubbed
until per-commit .deb artefact source is decided
RPC + dashboard
fips.status / fips.check-update / fips.apply-update / fips.install /
fips.restart registered in dispatcher
HomeNetworkCard.vue shipped standalone (unmounted — place in Home.vue
when ready); shows state pill, version, FIPS npub, update button,
activate button when key is present but service is down
ISO + systemd
archipelago-fips.service: conditional on key presence, masked by
default — backend unmasks after onboarding writes the key
build-auto-installer-iso.sh: multi-stage Dockerfile builds the FIPS
.deb from jmcorgan/fips main (fail-loud), COPYs it into rootfs, apt
installs it so trixie resolves deps; unit copied + masked
Version bump: 1.3.5 → 1.4.0
Tests: 33 new/updated passing (seed, identity, transport, federation,
fips module, transport::fips).
Known gaps: fips.apply-update returns a clear stub error until
upstream publishes per-commit .deb artefacts; HomeNetworkCard is not
mounted in Home.vue by default.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The .github/workflows/ci.yml Rust job runs cargo fmt --check, clippy
with -D warnings, and tests. All three were failing. This commit:
- Applies rustfmt across the tree (the bulk of the diff — untouched
since the last toolchain bump, so a wide sweep was unavoidable).
- Fixes the correctness-level clippy errors:
container/bitcoin_simulator.rs wildcard-in-or-pattern
container/manifest.rs from_str rename to parse (reserved name)
container/podman_client.rs .get(0) -> .first()
container/runtime.rs manual += collapse
archipelago/src/constants.rs doc-comment → module-doc
api/rpc/package/install.rs stray /// comment above a non-item
container/docker_packages.rs redundant field init
streaming/advertisement.rs missing Metric import in tests
tests/orchestration_tests.rs `vec!` in non-Vec contexts
mesh/listener/dispatch.rs unused store_plain_message import
api/rpc/tor/mod.rs and mesh/steganography.rs: push-after-new → vec!
- Quiets wide legacy surfaces with crate-level allows in main.rs for
stylistic lints (too_many_arguments, type_complexity, doc indent,
enum variant prefix, wildcard-in-or, assertions-on-constants,
drop_non_drop, unused_io_amount, ptr_arg) — these fired in dozens
of places with no correctness payoff and have been churning every
toolchain bump.
- Tags intentional-dead-code helpers: wallet/ and streaming/ modules
are WIP, mesh::send_chunked_payload and DM_V1_MARKER are kept for
rollback compatibility, vpn::get_nostr_vpn_status is surface-area
for a not-yet-landed RPC.
cargo fmt --check, cargo clippy --all-targets --all-features
-- -D warnings, and cargo test --all-features now all pass locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds core/archipelago/src/blobs.rs: a SHA-256 content-addressed store
that writes bytes to ${data_dir}/blobs/<cid> with a sibling <cid>.meta
JSON file (mime, filename, size, created_at, optional tiny thumbnail).
BlobStore::put is idempotent, max 64 MiB per blob, and issues HMAC-SHA256
capability tokens scoped to (cid, peer_pubkey_hex, expiry_epoch). Tokens
are verified in constant time and rejected on expiry. This is the
foundation piece for the mesh ContentRef typed envelope — the /blob/<cid>
HTTP route and ContentRef variant will land in a follow-up increment
once the HMAC key is plumbed from node identity.
No consumer yet, so the module compiles with dead_code warnings; these
will clear when the HTTP handler and ApiHandler state wiring land next.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cashu ecash protocol (BDHKE blind signatures, cashuA token format,
mint HTTP client) replacing the stub wallet. TollGate-inspired streaming
data payment system with step-based pricing (bytes/time/requests),
session management with incremental top-ups, usage metering, and
Nostr kind 10021 service advertisements.
13 new streaming.* RPC endpoints. Content server now verifies real
Cashu tokens. Profits tracking includes streaming revenue.
Frontend: GlobalAudioPlayer (persistent bottom bar across all pages),
video lightbox with full controls, audio in MediaLightbox, free file
previews (no blur), paid 10% audio/video previews, separated play
vs download buttons in PeerFiles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend: cache status in RwLock, refresh every 15s via background task.
Eliminates per-request TCP race to ElectrumX that caused volatile errors.
Fix error classification so "Failed to read" is transient, not hard error.
Frontend: keep last-known-good data across failed polls, persist Tor
onion once discovered, adaptive polling (5s active / 30s synced).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical flow fixes:
- Disable boot reconciliation that auto-created ALL containers on
unbundled installs (only FileBrowser should exist on first boot)
- Fix onboarding loop: RootRedirect no longer clears the
neode_onboarding_complete flag on boot screen completion
- Seed phrase persists when navigating back (no regeneration)
UI fixes:
- Boot screen: removed github and save icons from animation loop
- Seed screens: viewport height scaling with 100dvh
- Seed restore: removed outer card container from word input grid
- Seed screens use distinct background (bg-intro-1.jpg)
- Install progress simplified to "Installing" button style
- Uninstall state moved to global store (persists across navigation)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace fragmented random key generation with a single 24-word BIP-39
mnemonic that deterministically derives all node keys: Ed25519 (DID),
secp256k1 (Nostr/Bitcoin), BIP-84 xprv (Bitcoin Core), and LND aezeed
entropy. New onboarding flow: seed generate → word verification → identity
naming. Restore path enabled via 24-word entry. Includes seed RPC handlers,
mock backend support, LND/Bitcoin Core wallet-from-seed integration, and
UI polish across settings and discover views.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Critical fixes:
- Remove ensure_default_user() — no more auto-creating user with
password123. Login page now shows "Create Password" form on first
boot. User sets their own password during onboarding flow.
- CSRF 403: increased retry delay from 300ms to 500ms for stale
cookie recovery after remember-me session restore.
- Reboot: multiple fallback methods (/sbin/reboot, sysrq, kill init)
when USB is pulled and /usr/sbin isn't available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
