austin-sapien (100.70.96.88) sat dead for over two hours after taking
v1.7.122 — 'server starting' in the UI, service inactive, exit status
0/SUCCESS. It did not crash: the in-process updater replaces the binary and
exits cleanly for systemd to restart it, and that node's unit still carried
Restart=on-failure from an older install. systemd read the clean exit as
success and left it stopped. Every node with the old unit has this waiting
for it on the next update.
self-update.sh does refresh units, but the in-process update path never
runs it, so nothing was repairing them. The daemon now checks its own unit
at boot and rewrites only the Restart= line, so a node that starts even
once ends up with a policy that survives the next update.
Also carries the session-policy wiring: validate() now honours the
configured idle and absolute limits and the per-device class, instead of
the single hard-coded 24h constant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the Bitcoin crash-loop on 100.82.34.38: since a597c1d9
bitcoind launches with -conf=/tmp/rpc.conf and never reads the datadir
bitcoin.conf, but write_bitcoin_conf / ensure_bitcoin_rpc_config /
run_bitcoin_rpc_repair kept writing one on every install and restart.
Bitcoin Core's own datadir-conflict check then refuses to start at all.
Conflict resolved in favour of the PR: HEAD still carried
write_bitcoin_conf, whose deletion is the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Since a597c1d9 (bitcoind RPC creds off argv), bitcoin-core and bitcoin-knots
launch bitcoind with -conf=/tmp/rpc.conf and pass all other settings as CLI
args; bitcoind never reads /var/lib/archipelago/bitcoin/bitcoin.conf again.
write_bitcoin_conf, ensure_bitcoin_rpc_config, and bootstrap's
run_bitcoin_rpc_repair were never updated to match — they kept writing/
"repairing" server=/rpcbind=/rpcallowip=/listen= into that datadir file on
every install, reinstall, and service restart. Bitcoin Core's own
datadir-conflict safety check then refuses to start whenever that file
exists alongside an explicit -conf= arg, so the write and every repair
of it directly caused the crash it was trying to prevent.
Also drop the "restart already-running container after bitcoin.conf
repair" adoption-path branch: it assumed bind settings live in that file
and needs a restart to pick them up, which hasn't been true since
a597c1d9 — the running container's CLI args are already correct.
Replaces both writers with remove_stale_bitcoin_conf(), which renames
(not deletes) any leftover file so already-affected nodes self-heal on
next install/restart instead of staying permanently broken.
bitcoin_data_volume_gb is removed as dead code (it only fed the deleted
prune= line in write_bitcoin_conf, itself unused since a597c1d9 hardcoded
-prune=550 in the manifest's small-disk branch).
Investigated after a crash loop on archy-x250-beta; full incident
timeline and patch rationale in bitcoin-conf-crash-patch.md.
v1.7.117 broke Reticulum mesh on OTA-only fleet nodes two ways: the
update swaps only the backend binary and frontend tarball, so nodes
kept a stale archy-reticulum-daemon whose argparse exits on the new
--enable-transport flag (mesh session died on every spawn — confirmed
on framework-pt), and they never had archy-rnodeconf at all, so the
in-app Flash LoRa flow failed with a bare "No such file or directory".
Four-part fix:
- The Rust supervisor probes `daemon --help` and only passes
--enable-transport when the daemon advertises it; unsupported daemons
run edge-only exactly as pre-1.7.117 (tested against stub daemons
both ways + missing-binary fail-safe).
- Both PyInstaller tools ride the frontend tarball's runtime payload
(radio-tools/) and bootstrap.rs promotes them to /usr/local/bin on
startup when bytes differ — the first OTA path that ever updates
them. create-release.sh now rebuilds them every release and the
manifest script hard-fails if they're missing.
- The ISO bundles archy-rnodeconf alongside the daemon (it never did).
- Flash LoRa reports "tool not installed — update the node" instead of
the bare spawn error when rnodeconf is absent everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phones on the FIPS mesh reach the node at http://[<fips0 ULA>], but every
shipped nginx config listened on IPv4 only — nothing answered [::]:80/443,
so mesh HTTP failed with the tunnel fully healthy (found live on
framework-pt during the vc24 5G session, 2026-07-23). Canonical conf gains
listen [::] lines and patch_nginx_conf self-heals deployed nodes (covers
the sites-enabled -> archipelago-http dev variant via the existing
canonicalize pass); validated by the existing nginx -t + rollback flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-gamepad-keys: evdev->uinput daemon (pure python3 stdlib) that
mirrors any attached controller as a virtual keyboard — D-pad/stick to
arrows, A/B to Enter/Escape, X/Y to Space/f, shoulders to Tab/Shift+Tab.
The browser sees real keys, so controllers work inside EVERY app iframe
(IndeeHub, Jellyfin…) with zero per-app code. Ships via the ISO splice and
bootstrap self-heal (kiosk nodes only), hotplug via 5s rescans.
Implements docs/tv-input-iframe-apps.md.
- useControllerNav: an open [role=dialog][aria-modal] owns navigation —
focus is pulled inside, arrows move spatially between its controls, Enter
activates, Escape stays with the modal's own close handling. One standard
mapping for every modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of silent HDMI on kiosk TVs (framework-pt, LG TV):
1) fresh ISOs installed NO audio stack even though the kiosk launcher
expects PipeWire-Pulse — add pipewire/pipewire-pulse/pipewire-alsa/
wireplumber/alsa-utils to the rootfs and put the archipelago user in
'audio' (linger user manager gets no logind seat ACLs on /dev/snd);
2) the kiosk's boot-time Xorg modeset races the i915→HDA audio-component
bind, the ELD notify is lost, and every HDMI profile stays
'available: no' forever. Verified live: one off/on modeset re-delivers
the ELD and WirePlumber immediately switches to HDMI.
New archipelago-audio-router daemon (same splice-from-configs pattern as
the kiosk, ISO + include_str! self-heal): routes the card to the best
available HDMI profile, keeps the default sink on it, migrates live
streams on hot-plug, unmutes (HDMI forced 100% — the TV owns volume), and
re-modesets a connected external output once when no ELD reports a
monitor. bootstrap::ensure_audio_stack() installs packages + router on
already-deployed kiosk nodes via OTA (apt through systemd-run — the
service sandbox keeps /usr and dpkg read-only). main.rs also spawns the
pine satellite keeper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fleet fixes that OTA can deliver (unlike the ISO-only polkit rule):
- WiFi (issue #99): nodes drive NetworkManager from a system-level service
with no logind seat, so the stock polkit rule denies them and Wi-Fi setup
fails with 'Insufficient privileges'. Fresh ISOs since 2026-05 ship the
scoped rule, but OTA'd nodes never got it (OTA replaces binary + web UI,
not host system config). Add a startup self-heal that installs the rule if
missing and best-effort ensures polkitd — both wrapped so an offline/locked
apt can never fail startup; the rule is written regardless so it activates
once polkitd lands. Idempotent on the rule's unique subject.user marker.
- FIPS: regenerating fips.yaml (this build does, once, on first boot after
the OTA) restarts the fips daemon; for a few seconds /run/fips/control.sock
is gone and every seed-anchor dial fails, islanding the node until the next
5-min tick. Detect that exact failure (all dials fail on control.sock) and
retry in 15s instead — bounded to 8 fast retries/episode so a node with no
fips daemon falls back to the steady cadence rather than busy-looping.
FIPS unit suite: 33/33 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Large peer downloads (~178MB) failed with a generic 'Operation failed', and
the download path had three stacked problems:
- The FIPS reqwest client used a hard-coded 20s total timeout regardless of the
caller's .timeout(), so a big transfer over the mesh aborted at 20s before
the Tor fallback could help. Honor the per-request timeout (client_with_timeout).
- The peer-content proxy buffered the whole file into node memory via
resp.bytes() before sending a byte, and capped the transfer at 60s. Stream
the body through with hyper::Body::wrap_stream (constant memory) and raise the
timeout to 900s; bump the nginx peer-content read timeout to match.
- Free downloads pulled the file as base64 over RPC, doubling it in node memory
and the browser — fatal for large files. Download free files by streaming
from /api/peer-content straight to disk, after a 1-byte Range probe that
surfaces the real reason (peer offline on mesh and Tor) instead of a generic
failure. Paid downloads now return the real error through the {error} channel
the UI already displays.
Adds the reqwest 'stream' feature for bytes_stream().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kiosk chromium pinned ~92% of a core (software-compositing spin from
--enable-gpu-rasterization on a GPU-less/headless node), saturating the machine
and starving the backend + container builds — it caused the .198 receive timeout
and the deploy storms.
- archipelago-kiosk.service: CPUQuota=75% + MemoryMax/High + Delegate, so a
runaway kiosk can never take the whole node down.
- archipelago-kiosk-launcher.sh: detect /dev/dri — use GPU rasterization only
when a GPU exists, else --disable-gpu (avoids the headless spin).
- bootstrap::ensure_kiosk_hardened: OTA self-heal that installs the updated
unit+launcher on already-deployed nodes, daemon-reloads, and only try-restarts
a *running* kiosk (never re-enables an operator-disabled one).
cargo check clean; launcher bash -n clean; unit syntax valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On production nodes /var/lib/archipelago (the app data dir AND podman's
graphroot=/var/lib/archipelago/containers/storage) is a separate
device-mapper volume. archipelago.service ordered only After=network-online
.target, so on cold boots it (and its ExecStartPre) could start BEFORE
var-lib-archipelago.mount, write to the bare mountpoint on rootfs, fail every
podman call, exit, and be restarted every 5s until the volume mounted — the
"~20x [FAILED] Failed to start over ~5min" boot flap. Proven live on .198:
"var-lib-archipelago.mount: Directory /var/lib/archipelago to mount over is
not empty, mounting anyway" — the service had written there pre-mount.
Fix: RequiresMountsFor=/var/lib/archipelago (adds Requires= + After= on the
mount unit).
- image-recipe/configs/archipelago.service: ships the directive on fresh ISOs.
- bootstrap::ensure_archipelago_mount_ordering(): self-heals already-deployed
nodes' installed unit + daemon-reload (boot-ordering only, effective next
reboot; never restarts the running service). Idempotent; harmless on rootfs
installs (maps to the always-mounted root).
Verified on .198: after applying, systemctl shows After=var-lib-archipelago
.mount and systemd-analyze verify is clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B13 template fix only fixed fresh ISOs. Already-deployed nodes keep their
old nginx config, where /app/fedimint/ proxies to :8175 without rewriting the
Guardian UI's root-rooted asset URLs (src="/assets/...", url("/assets/...")).
Those resolve against the SPA root: bg-network.jpg exists there by luck, but
app-icons/fedimint.jpg 404s (location /assets/ uses try_files =404) — the
visibly-broken icon.
bootstrap.rs::patch_nginx_conf now heals both paths on startup:
- Style A (main conf, HTTP): swaps the old single nostr-provider sub_filter tail
for the full reroot set; byte-matches the shipped template.
- Style B (HTTPS app-proxy snippet): the snippet's fedimint block has no
sub_filter and a per-node-varying trailing directive, so anchor on the unique
:8175 proxy_pass and insert the reroot set after it (nginx ignores directive
order). Snippet added to the bootstrap nginx loop (skipped on HTTP-only nodes).
missing_* flags are now gated on their splice anchors so the included snippet
neither attempts the main-conf-only patches nor logs warn-skips every boot.
Idempotent via the 'href="/' 'href="/app/fedimint/' marker.
Verified on .198 (both paths): fedimint app-icon 404 -> 200 image/jpeg; nginx -t
OK; containers survived restart (Quadlet); idempotent steady state, no warn spam.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B3 streaming proxy endpoint existed in the backend but nginx had no
location for /api/peer-content/*, so the browser's requests fell through to
the SPA (200 text/html) and media still wouldn't play. Add an
NGINX_PEER_CONTENT_BLOCK that bootstrap patches into every server block
(forwards Cookie for session auth + Range, proxy_buffering off). Idempotent;
covers fresh-ISO nodes too since bootstrap runs on every startup.
Verified on .198: after restart the async nginx patch lands and
/api/peer-content/<onion>/<id> returns 401 (reaches backend, auth-gated)
instead of the SPA; nginx block present in both server blocks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The LND wallet UI (served on its own app port) fetches /lnd-connect-info
and /proxy/lnd/* cross-origin, so both need correct CORS headers.
(a) Older nginx configs add their own Access-Control-Allow-Origin in the
/lnd-connect-info location on top of the one the backend sets, yielding
a DUPLICATE header that browsers reject ("multiple values"). bootstrap
now strips that redundant nginx add_header (backend owns CORS).
(b) /proxy/lnd/* returned a 401 with no CORS headers when the session
check failed, so the browser saw an opaque CORS error instead of a
readable 401. Add unauthorized_cors() and use it on that path.
Adds tests/production-quality/ (bug tracker + lnd-cors-test.sh harness).
Verified: harness 4/4 on .116, .198, .103.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The whole fleet was silently never reaching the FIPS mesh: the default
public anchor was configured as fips.v0l.io:8668/udp, but the anchor only
answers on TCP/8443. Fix the default to 185.18.221.160:8443/tcp (IPv4
literal — the hostname resolves IPv6-first and the daemon binds v4-only,
which fails the handshake with EAFNOSUPPORT), and auto-seed it in
anchors::load() so every node dials it without operator action (removal
still persists). Proven live on .116: cold start → anchor_connected in
~400ms, anchor became mesh parent.
Wire fips::update::apply() against upstream GitHub releases (stable
channel only): resolve /releases/latest → SHA256-verify the .deb against
checksums-linux.txt → install → restart. dpkg runs via `systemd-run` to
escape archipelago's ProtectSystem=strict sandbox (else /var/lib/dpkg is
read-only), with --force-confold (archipelago manages /etc/fips conffiles)
and --force-downgrade (dev builds sort newer than the stable tag).
Validated live: .116 upgraded 0.3.0-dev -> stable v0.3.0.
Also: standalone fips-ui dashboard app (apps/fips-ui + docker/fips-ui,
static nginx proxying /rpc/v1 same-origin, copiable own-anchor address);
reserve UI port 8336; register fips/fips-ui as platform-managed. Includes
the Lightning wallet cross-origin (CORS) + LND proxy auth + nginx
self-healer fix so the wallet screen connects instead of "failed to fetch".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extracted the heal_podman_state cleanup list as a module-level
HEAL_RUNTIME_SUBDIRS const so a unit test can structurally enforce
the invariant: the list must contain "containers" + "libpod" but
must NOT contain "podman" (which holds systemd's podman.sock
listener and was the bug fixed in commit bb421803).
If anyone re-adds "podman" — accidentally, by reverting, or by
copy-paste from old plan memory — this test fires before we ship,
not on the next deploy when it nukes the orchestrator's HTTP path.
Total tests: 614 → 615.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed live on .198: heal_podman_state was removing
$XDG_RUNTIME_DIR/podman/ alongside containers/ and libpod/. That dir
holds the systemd-bound podman.sock — the listener systemd creates for
socket-activated podman.service. Removing it broke every libpod HTTP
call from the orchestrator until `systemctl --user restart
podman.socket` ran. Far worse than any wedge it was trying to repair.
Drop podman/ from the cleanup list. The runtime state we actually want
to clean for FM6 (bolt_state.db drift) lives in containers/ and
libpod/ only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes FM6 (podman bolt_state.db / runtime drift) — observed live on
.198 today: bitcoind was running for several minutes, but podman's
state DB reported the container as Exited. The reconciler then tried
to "restart" it, racing the still-bound port 8332 and failing in a
loop.
heal_podman_state() runs as the last bootstrap stage, BEFORE the
orchestrator's reconcile loop ticks. It probes `podman info` with a
5s timeout; on failure it removes the runtime-state dirs under
$XDG_RUNTIME_DIR and re-probes. Persistent storage under
~/.local/share/containers/storage/ is never touched, so containers
re-discover from manifests on next call.
Cleanup never includes `podman system reset` or `system renumber` —
those are destructive and must stay operator-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small, focused tightenings:
- core/container/src/podman_client.rs: drop the legacy Hetzner
23.182.128.160:3000 mirror from image_uses_insecure_registry().
It was decommissioned in v1.7.x and is stripped from active
registry config at load time; leaving it in the bypass list let
a stale config still skip TLS. Replace the inline match with a
named INSECURE_REGISTRY_HOSTS slice so future entries are one
line. Test now also pins the spoofing-immune semantics
("evil.example/146.59.87.168:3000/x" must NOT match).
- core/archipelago/src/api/rpc/package/config.rs: split bitcoin
from lnd in get_app_capabilities(). bitcoind never opens raw
sockets — drop CAP_NET_RAW from bitcoin/bitcoin-core/bitcoin-knots.
lnd/fedimint/fedimint-gateway keep it because they enumerate
network interfaces during cert generation.
- core/archipelago/src/bootstrap.rs: tighten_secrets_dir()
enforces 0700 on /var/lib/archipelago/secrets and 0600 on every
file inside on each startup. The dir-mode is the load-bearing
isolation boundary against rootless container escapes (their UID
maps to >=100000, can't traverse uid=1000/0700). The per-file
sweep is defense-in-depth against any installer that wrote 0644.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Snapshots the in-flight hardening work so subsequent reconcile/Quadlet
phases land on a clean before/after diff.
Changes:
- core/container/src/podman_client.rs: image_uses_insecure_registry()
whitelist for the OVH (146.59.87.168:3000) and legacy Hetzner
(23.182.128.160:3000) HTTP mirrors; podman_network_settings() lifts
custom networks into the Networks map so containers can join them.
- core/archipelago/src/container/prod_orchestrator.rs:
ensure_container_network() creates per-manifest networks on demand;
apply_data_uid() now goes through host_sudo for mkdir -p + chown so
bind-mount roots get created and chowned without password prompts.
- core/archipelago/src/api/rpc/package/{install,update,stacks}.rs:
podman pull adds --tls-verify=false only for whitelisted registries.
- core/archipelago/src/bootstrap.rs: removes stale dev-mode systemd
override on startup (live nodes carried it from old installers).
- core/archipelago/src/config.rs: ignore ARCHIPELAGO_DEV_MODE in prod
binaries — it had been silently rerouting volumes to /tmp.
- apps/bitcoin-{core,knots}/manifest.yml: locate bitcoind at runtime
so image-layout differences don't break entrypoint.
- scripts/app-catalog-image-smoke-test.py: production catalog/image
smoke test that probes a target node before users click Install.
- .gitignore: cover .codex, .pnpm-store, __pycache__, *.bak.
Removes filebrowser.rs.bak and two stale catalog.json.bak files
(verified identical to live counterparts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install flow
- api/rpc/package/install.rs: always append the literal image URL as a
last-resort pull candidate in do_pull_image, so images not carried by
any configured mirror (docker.io/bitcoin/bitcoin:28.4) still install
instead of masquerading as a generic pull failure across every mirror.
- api/rpc/package/install.rs: write_bitcoin_conf now skips on any stat
error, not just "file exists". Once bitcoin-knots' first-boot chowns
/var/lib/archipelago/bitcoin into the container's user namespace (700
perms, UID 100100/100101), the archipelago daemon can't even traverse
in — try_exists returns Err which unwrap_or(false) treated as "not
present" and drove a doomed write. Now errors out of the directory
traversal are treated as "conf already owned by container user" and
the write is skipped. Mirrors the lnd.conf pattern.
- api/rpc/package/install.rs: drop the hardcoded `prune=550` from the
conf default. Operators with multi-TB drives shouldn't be silently
pruned; users who want a pruned node can set it in bitcoin.conf
themselves. Full archive is the only honest default.
- api/rpc/package/config.rs: bitcoin-core now passes explicit
-server/-rpcbind/-rpcallowip/-rpcport/-printtoconsole/-datadir CLI
args. Vanilla bitcoin/bitcoin:28.4 has no entrypoint wrapper and
reads conf + argv only; without these the RPC listens on 127.0.0.1
inside the container and rootlessport can't reach it, so the
bitcoin-ui companion gets 502 on every /bitcoin-rpc/ call.
Bitcoin Knots keeps its own entrypoint-driven defaults.
- container/docker_packages.rs: split bitcoin-core out of the shared
AppMetadata arm. bitcoin-core now surfaces as "Bitcoin Core" with
bitcoin-core.svg and a Reference-implementation description; the
bitcoin + bitcoin-knots ids keep the Knots branding. Fixes the home
card showing "Bitcoin Knots" for a Core install.
Bitcoin node UI (docker/bitcoin-ui)
- index.html: impl name/tagline/logo now dynamic. applyImplBranding()
reads subversion from getnetworkinfo — /Satoshi:X/Knots:Y/ resolves
to Bitcoin Knots, plain /Satoshi:X/ resolves to Bitcoin Core. Both
get their own icon and subtitle. Settings modal replaced its
hardcoded Regtest/txindex=1/port-18443 placeholders with live values
from getblockchaininfo + getindexinfo + getzmqnotifications.
- index.html: new Storage info card (Full Archive · X GB /
Pruned · X GB from blockchainInfo.pruned + size_on_disk) visible on
the main dashboard, same level as Network. Settings modal mirrors it
with the prune height when applicable.
- Dockerfile + assets/: bitcoin-core.svg, bitcoin-knots.webp, and the
bg-network.jpg used by the dashboard are now COPY'd into the image
under /usr/share/nginx/html/assets. Previously the <img src> pointed
at paths that 404'd into the SPA fallback and the onerror handler
hid the broken logo silently.
Frontend
- appSession/appSessionConfig.ts: add bitcoin-core to APP_PORTS (8334),
HTTPS_PROXY_PATHS (/app/bitcoin-ui/), and APP_TITLES (Bitcoin Core).
Without these the AppSessionFrame showed "No URL found for
bitcoin-core" and the home/app-list title fell through to the raw id.
- settings/AccountInfoSection.vue: backfill What's New entries for
v1.7.31 through v1.7.37 that had been missed in earlier cuts.
Release plumbing
- releases/v1.7.37-alpha/: binary + frontend tarball.
- releases/manifest.json: v1.7.37-alpha, sha256/size refreshed.
- Cargo.toml / package.json: version bumps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- core/archipelago/src/bootstrap.rs (NEW): embed scripts/container-doctor.sh
and image-recipe/configs/archipelago-doctor.{service,timer} via
include_str! and sync to disk + enable the timer on every archipelago
startup. Idempotent (content-hash compare), dev-box symlink guard keeps
the git checkout untouched, best-effort (warn-only on failure) so
bootstrap never blocks server readiness. Wired in main.rs as a
background tokio task.
- scripts/container-doctor.sh: add fix_rootless_netns_egress(). Detects
when the rootless-netns has lost its pasta tap (container-to-container
still works but outbound DNS/TCP fails) via an nsenter probe into
aardvark-dns; with a two-probe 10s debounce to rule out transients and
a host-precheck that bails out if the host itself is offline. When the
rootless-netns is truly broken, does a graceful podman stop --all /
start --all so pasta + aardvark-dns rebuild the netns from scratch.
Bitcoin-knots and every other outbound container recover in one cycle.
- core/archipelago/src/update.rs: host_sudo → pub(crate) so bootstrap.rs
can reuse the existing systemd-run escape hatch.
- apps/bitcoin-core/manifest.yml: bump app version 24.0.0 → 28.4.0 and
image bitcoin/bitcoin:24.0 → bitcoin/bitcoin:28.4. Resources aligned
with the real container-specs.sh large-disk tune (4 GiB memory cap,
cpu_limit: 0 so bitcoind can run -par=auto across every core).
- neode-ui/src/views/apps/AppCard.vue + Apps.vue: add an Update button
+ Updating spinner to every app card that has available-update set.
Wires through serverStore.updatePackage(id) — the same RPC the detail
view already calls. common.update / common.updating i18n keys added in
en.json and es.json.
- core/archipelago/src/identity_manager.rs: add create_from_signing_key()
that mirrors an existing Ed25519 key as a manager-level identity with
a deterministic id (`node-<pubkey16>`). Idempotent across restarts,
gets the hex-SVG master avatar.
- core/archipelago/src/server.rs: the auto-create path on first boot now
mirrors the node's own signing_key (seed-derived on onboarded installs)
as a "Node" identity instead of generating a random "Default" keypair.
Once this ships, the DID on the Web5 DID Status card (via node.did
RPC), the Node entry on the Identities page (via identity.list), and
the DID used for peer-to-peer connects (via server_info.pubkey) all
resolve to the same seed-derived pubkey.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ssmithx