The v1.8.00-alpha changelog section never shipped as an OTA (no tag, live
manifest stayed 1.7.99-alpha); its content ships now as v1.7.100-alpha.
First 10 bullets are the OTA manifest changelog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docker login and buildkit both refused the HTTP registry. Host daemon on vps2
now lists it under insecure-registries (SIGHUP reload, no downtime); the
workflow configures buildkit with http=true for the registry host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mempool placeholder page (777d54ea) is served by the mock backend, but
the demo web nginx still had a location block reverse-proxying /app/mempool/
to mempool.guide, so it always won and the placeholder was unreachable.
Drop the block so mempool falls through to the generic /app/ backend proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Companion QR overlay: demo builds encode the demo's own origin
(/packages/archipelago-companion.apk ships in the web image) instead of
the vps2 raw URL.
- Dockerfile.web/.backend: build-time scrub replaces every remaining
occurrence of the release-server address with registry.demo.internal and
fails the build if any survives.
- Mock backend update-mirror list, sideload help text, and changelog prose
no longer name the server address.
- Copy demo-images workflow into .gitea/workflows/ — Gitea ignores
.github/workflows when .gitea/workflows exists, so the CI never ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also serves as the first push that exercises the workflow now that the
vps2 runner and registry vars/secrets are configured.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The public demo has no synced chain/electrs, so opening Mempool 502'd.
Serve a self-contained branded placeholder from the mock instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Bitcoin UI loaded the Tailwind Play CDN at runtime: unstyled on
offline nodes + a production console warning. Replaced with a
hand-extracted static stylesheet (~125 utilities actually used, same
approach as lnd-ui), served from the container. podman build + runtime
smoke test verified (GET /tailwind.css 200, zero cdn references).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The optimistic package_data entry painted at install-spawn survived
failure as a 'Stopped' tile for an app with zero footprint (netbird on
the fresh ISO). On INSTALL FAIL with no container/user-stop marker for
the app, the entry and the pre-saved dynamic app config are removed and
an error notification is pushed; failures that left a real container
keep today's behavior (visible exited state for retry/diagnosis).
Install suite 39/39.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
write_bitcoin_conf silently produced an unpruned (~810GB and growing)
config regardless of disk: the framework node got a full-archive chain
on a 205GB volume. Volumes under 1.2TB now get prune = 25% of the
volume, clamped to [550MB, 100GB]; bigger volumes keep full archive;
unknown df results stay unpruned (don't surprise big iron). Framework
node switched live to prune=50000.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Open/Close Channel modals sat at z-[60], BEHIND the wallet settings
BaseModal (z-3000) — clicking 'Open Channel' looked like nothing
happened while the form changed in the background. Raised to z-3100.
- TransactionsModal gains rail filter chips (All / On-chain / Lightning
/ Ecash with counts): instant ecash micro-payments were burying the
standard transactions.
- Global LND seed-backup banner: once the wallet exists and the seed is
un-acknowledged, a dismissible glass banner (24h snooze) prompts and
routes to the LND backup card; polls only when authenticated, stops
permanently once acked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Request-to-Peer (Federation Discover modal + Web5 Node Visibility list)
no longer fires instantly: a BaseModal confirm shows the target, an
optional 280-char message (handshake.connect already carried it), and
Send/Cancel with the standard orange CTA.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
936b4cca made ANY loaded manifest with an absent container self-heal via
install_fresh — but the signed catalog and the ISO ship manifests for
EVERY app, installed or not, so 'manifest loaded' is not installation
evidence. On a fresh node this mass-installed the catalog: the framework
node grew portainer, vaultwarden, searxng, strfry, mempool, immich and
fedimint uninvited within an hour (fleet nodes showed the same loop as
reconcile-failed noise for unpullable images).
ExistingOnly (boot) reconcile now creates containers from nothing only
for required-baseline apps (filebrowser, fedimint-clientd); every other
absent app is Left('absent') and recreated ONLY via desired-state
recovery when it was running at the last snapshot — which still covers
the vanished-container regressions (indeedhub/immich) that motivated
936b4cca. Reconcile suite 16/16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The global mesh-detection poll (added for the device-detected modal)
fired on the login page, 401'd, and the session-expired handler
redirected to /login — reloading the page we were already on, forever.
Background polls now run only when authenticated, and a 401 never
redirects when the path is already /login.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity with the backend's invite trust threading so the demo's Invite a
Peer / Link Your Nodes flows show the right level end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Native select arrows rendered with inconsistent right padding across
views and long labels could overflow into the icon. Global select rule:
appearance-none + a single SVG chevron at a fixed inset, uniform
padding-right (overrides ad-hoc pr-* utilities), ellipsis truncation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plugging in a LoRa radio now pops a global two-step setup modal on any
page: step 1 shows the ACTUAL detected board (25 board SVGs vendored
from the Meshtastic web-flasher, GPL-3.0; matched by USB product string
on native-USB boards, vid:pid heuristics for bridge chips, animated
fallback otherwise); step 2 configures with Archipelago presets — the
LoRa region pre-selected from the node's location (new lat/lon→region
mapping), channel 'archipelago', node name, and an advanced firmware
pin. Options adapt per protocol: region+channel program Meshtastic;
MeshCore shows its community frequency plan (firmware owns RF); RNode
defers to the Reticulum daemon config.
Backend: mesh.configure now accepts lora_region (validated against the
driver's region table) and device_kind (probe pin); mesh.status returns
the persisted values plus per-port USB identity (sysfs vid/pid/product)
for board identification. The Mesh Device tab gains a full settings
form (region with duty-cycle/legality hints, firmware pin, channel,
name, identity broadcast) replacing the read-only panel; the old
Mesh-page-only onboarding modal is superseded by the global flow.
Mock backend: stateful mesh config so the dev UI demos the full
detect→configure round-trip (disable mesh → modal fires with Heltec V3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh ISO installs had ZERO disk manifests: auto-install.sh copied
apps/ to /etc/archipelago/apps but the backend/orchestrator loads
/opt/archipelago/apps — so any app not overlaid by the signed catalog
(netbird: 'manifests not available on this node') could never install.
- installer now also populates /mnt/target/opt/archipelago/apps
- bootstrap gains run_apps_dir_repair(): when /opt/archipelago/apps is
empty and the installer copy exists, populate it at startup — heals
already-deployed ISO nodes via OTA without reinstalling
- bootstrap's bitcoin.conf repair now also ensures rpcbind=0.0.0.0
(same fix as install.rs; duplicate rpcbind with a manifest command
line is harmless — bitcoind binds the first and logs the rest)
Verified live on the framework node: /opt populated (56 dirs), backend
restart loaded '50 app manifest(s) (disk + registry catalog)', all
containers kept running (they live in user.slice, not the service
cgroup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the fresh-install triple failure (LND needing ~5 install
attempts, Bitcoin UI bitcoin-rpc 502, backend getblockchaininfo errors):
the legacy conf writer set rpcallowip but never rpcbind, so bitcoind
bound RPC to the container's loopback only — every dial over the
container network (LND, bitcoin-ui) and the rootless port-forward was
refused. LND exited instantly, the 60s post-start poll read that as
INSTALL CRASH, and the wallet (and its seed backup) never got created.
- write_bitcoin_conf + ensure_bitcoin_rpc_config now emit/ensure
rpcbind=0.0.0.0 (host exposure unchanged: publish stays 127.0.0.1)
- new install gate: bitcoin-dependent apps (lnd/electrs/electrumx/
mempool-electrs/btcpay) wait up to 3 min for an authenticated
getblockchaininfo to answer before their container starts, with
INSTALL WAIT progress lines; on timeout the install fails with an
actionable message instead of a crash-looping container
Verified live on the fresh-ISO framework node: after adding rpcbind and
restarting bitcoin-knots, LND connected, created its wallet, baked
macaroons, and settled into 'Waiting for chain backend to finish sync'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The old tri-state (hidden/discoverable/public) was confusing — its
'discoverable' level implied discovery by already-connected federated
peers, which is meaningless. The card is now a single Enable switch
(global ToggleSwitch style) that drives nostr presence publishing:
ON = public — anyone can find the node and request a connection
(requests join as Peer/observer on approval, never trusted).
The card also lists nostr-discoverable nodes inline with Request-to-Peer
buttons (same handshake.connect flow as the Federation Discover modal,
which stays available in Federation & Peers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Federation List/Map tabs now use the Home mode-switcher tablist
(full-width on mobile, shrink-wrapped on desktop, proper tablist a11y)
- Network Interfaces WLAN adapter toggle replaced with the shared
ToggleSwitch used by AI data access in Settings; ToggleSwitch gains
optional disabled/ariaLabel props (non-breaking)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- openwrt.get-status/scan/scan-wifi/provision-tollgate/configure-wan
mocked: 19-day-uptime GL-MT3000 TollGate gateway, two open TollGate
SSIDs + private SSID, live upstream WAN, 21 sat/min pricing (edits
via the reconfigure form persist in mock state)
- handshake.discover returns 4 discoverable nodes; nostr.set-discovery/
discovery-status stateful; handshake.connect + pending-request
approve/reject round-trip against mock state (incl. one inbound
request for the demo)
- node-nostr-discover (Web5) list grown to 5 nodes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The app-details credentials/version sidebar and the container-app Launch
button used solid bg-blue-600 CTAs and blue links/focus rings instead of
the global glass style; InstallVersionModal used an off-token solid
orange. All primary CTAs now use glass-button glass-button-warning (the
Create Identity orange) and links/focus accents use the orange tokens.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The invite type chosen in QuickActions was only used for the modal
title — the RPC never carried it, so every invite was minted Trusted.
Pass it through so 'Invite a Peer' mints observer invites and 'Link
Your Nodes' keeps trusted ones.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tests landed twice (interrupted edit applied both paths), breaking
the test build with E0428. Federation suite: 33/33 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Invites carried no trust level, and both accept_invite and the
peer-joined handler hardcoded TrustLevel::Trusted — so 'Invite a Peer'
(Observer) federated both sides as fully Trusted.
- invite codes now carry a 'trust' field (legacy codes parse as Trusted)
- federation.invite accepts trust_level: trusted|observer
- accept_invite assigns the invite's level on the acceptor side
- peer-joined resolves the granted level authoritatively by matching the
acceptor's echoed invite token against our stored outgoing invites;
the peer's own unsigned claim is honored only as a downgrade, so no
escalation is possible
- discovery/connection-request approvals mint Observer invites directly
(the post-hoc demotion in handshake.rs stays as a legacy safety net)
- asymmetry self-heal re-asserts at the locally-held level
Tests: observer threading + legacy default-trusted parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
telemetry.fleet-status only read the telemetry-fleet/ collector inbox —
an opt-in anonymous-telemetry pipeline that federated peers never write
to — so Fleet showed 'No nodes reporting' even with a healthy federation.
Build the fleet list from nodes.json instead: every TRUSTED federated
node is mapped from its last_state sync snapshot (cpu/mem/disk, uptime,
apps, peer count) into the FleetNode shape the UI expects, keyed by DID.
Observer ('peer') and untrusted nodes are excluded by design. Collector
reports are still merged in for non-federated telemetry nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ISO's rootfs tar is a cached build artifact (currently 2026-04-20) and
its baked-in archipelago.service predates the B17 fix — no
RequiresMountsFor=/var/lib/archipelago — while auto-install.sh only copied
the configs/ unit when the rootfs had none. Every fresh install therefore
raced the LUKS data mount on cold boot and looped
'[FAILED] Failed to start archipelago.service' for minutes (seen on the
1.7.99-alpha ISO).
Ship configs/archipelago.service on the ISO at archipelago/configs/ and
make the installer overwrite the rootfs copy unconditionally. Also add
RequiresMountsFor to the legacy install-to-disk.sh inline unit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Jul 10 asset swap (75ed3041) replaced the progressive JPEG with a
baseline re-encode that trips the known Chromium compositor black-layer
bug in the 3D perspective background stack — the image paints for a
second, then the layer drops and the #000 container fill shows through.
Restore the pre-swap progressive encoding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
btcpay-server started and reported healthy while its declared
dependencies (archy-btcpay-db, archy-nbxplorer) failed every boot
reconcile (live-testing report 2026-07-10, Fix 3). After each
reconcile pass, flag every app the pass left up whose manifest
dependency landed in the failure list — a loud DEGRADED journal error
instead of silence. True start-gating on dependency health is a
larger design change (boot-ordering/deadlock risk) and stays open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strfry/vaultwarden 8082 collision shipped in a signed catalog
because nothing checks cross-app host ports. Fix the remaining latent
pairs from the cfb8d953 audit — did-wallet 8083→8088 (filebrowser
keeps 8083), morphos-server 8086→8089 (netbird-server keeps 8086),
lightning-stack 8087→8091/9737→9738 (netbird and fedimint-gateway
keep theirs) — and fix their healthchecks, which probed the HOST port
inside the container (same class as strfry 6fe62355; now 127.0.0.1 +
container port). load_manifests now flags collisions across the
merged disk+catalog set (bitcoin variants and the mempool umbrella
pair are exempt as mutually exclusive), and a repo-level test parses
apps/*/manifest.yml so a future latent pair fails CI instead of
shipping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Channel open fee control, channels tab in wallet settings, mempool tx
link, and the channel-open async peer-connect race fix. Conflict in
LightningChannels.vue resolved to the branch's extracted
LightningChannelsPanel component, with main's Teleport-to-body modal
fixes and glass-button-warning class (8256fde1) ported into the panel
so they aren't lost. lnd tests 16/16, LightningChannels vitest green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
24fd97ed cascades on package.start/restart, but a reconcile pass that
recreates or starts a bitcoin backend (desired-state recovery, repair
recreate, boot InstallMissing) also moves the RPC address behind a
running lnd's back — §C 'restart lnd after ANY bitcoin recreate'.
After the pass, dependents from the shared address_caching_dependents
table (moved to app_ops as the single source of truth) that sat
untouched (NoOp) and aren't user-stopped are restarted under their op
lock; a dependent the pass itself (re)created already resolved the
fresh address and is left alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wait_for_manifest_host_ports polls up to 420s while the caller holds
the per-app lock, so a user's stop request queued behind it and the
scanner painted the app Restarting for the whole wait (the dd3afbba
transitional-state stick; gate take-2 iter 5 vaultwarden). The wait
now selects against a user-stop-marker watcher (2s poll on
user-stopped.json, by app id or container name) and aborts with a
recognizable error, and the host-port repair path skips its restart
when a stop was requested while it settled — restarting there would
fight the queued stop worker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six manifests still pulled through git.tx1138.com/lfg2025, which is
dead — every boot reconcile of archy-btcpay-db/archy-nbxplorer failed
at the pull step (live-testing report 2026-07-10; the orchestrator
now also falls back to local storage, 3a5c5db1). All six tags
verified present on 146.59.87.168:3000/lfg2025 via the registry API.
Ships fleet-wide at the next catalog ceremony (catalog manifests
override disk ones).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generated HealthCmd reported healthy (exit 0) whenever the image
shipped neither wget nor curl — btcpay-server sat 'healthy' in podman
ps while completely non-functional (live-testing report 2026-07-10).
Try a bash /dev/tcp connect to the health URL's host:port before
giving up; only images with none of wget/curl/bash still skip Podman
health (dropping exit 0 entirely would restart-loop those). Podman's
health timeout bounds the connect attempt.
Note: the rendered unit text changes, so every http-healthcheck app
gets a one-time unit rewrite + restart on first reconcile after this
binary lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconcile recreates route through install_fresh, which pulled
unconditionally: with git.tx1138.com dead, every boot reconcile of
archy-btcpay-db/archy-nbxplorer hard-failed on 'pulling ...' even
though both image:tags sat in local storage the whole time
(live-testing report 2026-07-10). Check image_exists first and only
pull when the ref is truly absent — the same semantics as
ensure_resolved_source_available and the generated quadlets'
Pull=never.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The health monitor restarted containers that were legitimately down
between a package.start/stop/restart worker's stop and start halves —
the same interleave class as the reconciler's mempool repair-recreate
(e275494a). Before restarting, probe the app_ops op-lock registry via
the container name, the derived app id, and a '_'->'-' normalization
(legacy stack containers are underscore-named while lock keys use
hyphenated app ids) and skip the cycle when a worker holds the lock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The healthcheck runs INSIDE the container, so it must target the
container port (7777), not the host mapping (8090) — and 127.0.0.1
explicitly, because `localhost` resolves to ::1 in the image while
strfry binds IPv4 0.0.0.0 only. Both bugs verified empirically on .228
(container was Up but "unhealthy"; 127.0.0.1:7777/health returns 200).
Ships at the next catalog regen/re-sign (catalog overlay supremacy);
with the runtime manifest-reload now in place it will apply fleet-wide
on the next catalog refresh, no service restarts needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live verification on .228 showed catalog_changed:true on every
check-updates call: AppCatalog.apps is a HashMap, so the re-serialized
cache bytes flap with key order and the changed-detection never reports
"unchanged" — which would have reloaded the manifest overlay every
hourly poll. Cache the raw fetched body instead and compare against
that; as a bonus the cache now holds the exact signed preimage, so a
present release-root signature stays verifiable from disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run B failed the same probe with LND_ERROR (gRPC 'waiting to start' maps
there) — the 7b77462d window only matched WALLET_LOCKED, one of four
phases a cascade-restarted lnd passes through. Retry every code except
LND_WALLET_UNINITIALIZED (no wallet — never self-heals) within a bounded
180s window; persistent failures still fail after it. lnd verified
healthy minutes after both failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate test 24 probed lnd.newaddress moments after the bitcoin bounce
cascade-restarted lnd; the auto-unlocker was still retrying against a
starting gRPC, so the probe saw LND_WALLET_LOCKED on a node that unlocks
itself seconds later (verified on .228 — same request succeeds post-run).
Treat WALLET_LOCKED as settling with a bounded 120s retry, mirroring the
e21f3baf settle window for tests 123/124; every other error still fails
fast, and a genuinely stuck-locked wallet still fails after the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.install routed to the manifest-driven orchestrator only for a
hardcoded per-app allowlist; every other app fell through to the legacy
flow, which ignores manifests entirely and creates a bare container with
no ports or volumes (strfry crash-looped this way on .228, 2026-07-09).
New ContainerOrchestrator::knows_app(app_id) (default false; prod checks
its loaded-manifest map, disk + signed-catalog overlay). The install gate
is now allowlist OR knows_app — no per-app Rust for manifest-driven apps,
matching the packaging invariant. Apps without a manifest keep the legacy
flow including the container-exists adopt block; the unknown-app_id →
legacy fallback inside the orchestrator branch is unchanged.
Tests: knows_app_reflects_loaded_manifests + install suite 37/37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dockurr/strfry image listens on 7777 (not 8080) and its default config
demands a 1M NOFILES rlimit — above the rootless user-manager hard cap
(524288), so the relay exited at startup even with the mount fixed. Ship
the config declaratively (manifest files: + ro bind at /etc/strfry.conf)
with nofiles = 0; this also skips the entrypoint's copy into /etc, which a
readonly_root container cannot perform. Verified end-to-end on .228 in a
quadlet-equivalent systemd-run environment: active, GET / and /health 200.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catalog regenerated from fixed manifests (cfb8d953 strfry, loopback-only
bitcoin-core/knots binds) and re-signed by the release root. Semantic diff
vs the previous signed catalog is exactly those three manifests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Formal gate take-2 went 4/5: iteration 5's vaultwarden stop-wait saw
'restarting' for the full 120s window. A reconcile pass had queued every
app into pending_boot_starts moments before the stop marker landed, and
the scanner's overlay painted the deliberately-stopped app Restarting
until the slow sequential pass reached it (legacy container, so no unit
state to contradict it).
Two filters: the scanner overlay skips user-stopped ids (the marker is
written before the stop runs, so it's reliably present), and the
reconciler no longer queues apps whose lifecycle op is in flight into the
pending overlay (it skips them via the same probe anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two defects kept strfry crash-looping since install (62,868 systemd
restarts on .228): its host port 8082 collides with vaultwarden's, and the
volume mounted at /strfry which the dockurr image never reads — its LMDB
lives at /app/strfry-db (entrypoint default), so mdb_env_open got ENOENT
even once the port was freed. 8090 is unclaimed across all manifests.
Uninstalled from .228 (db was empty — it never ran); reinstall after the
next catalog regen ships this manifest.
NOTE: repo-wide host-port collision audit also flags did-wallet/filebrowser
(8083), morphos-server/netbird-server (8086), lightning-stack/netbird
(8087), fedimint-gateway/lightning-stack (9737) — latent, tracked in the
hardening plan; a cross-app collision check belongs in the port allocator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reconciler doesn't take the RPC layer's per-app FIFO op lock (known
limit of 891cbba4): between a restart worker's stop and start halves it saw
the mempool frontend "missing", repair-recreated it behind systemd's back,
killed the worker's fresh container 11s after start, and left the unit down
for ~3.5 min until the next heal — gate test 123 measured exactly that
window (.228 iteration 3, 2026-07-09).
New crate::app_ops module owns the op-lock registry + stack member table
(runtime.rs and dependencies.rs now delegate) so the reconciler can probe
lifecycle_op_in_flight(app_id) — covering both the app's own key and its
owning stack package — and skip that app for the cycle. The ownership-sweep
podman restart gets the same guard. Health monitor is a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mempool-api exits at startup while electrumx is catching up to the daemon
(7-day initial-sync ETA on .228), flapping through 'activating' for ~2 min
after each lifecycle cycle before systemd heals it. The instant asserts
probed inside that window both iterations. Poll up to 180s for
converges-to-active — a genuine crash-loop still fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lnd resolves the bitcoin RPC address once at startup and never re-resolves;
when bitcoin-core/knots restarts (new container IP) lnd spins on "dial tcp
<old-ip>:8332: no route to host" and its own RPC wedges until lnd restarts
(gate lnd getinfo test 75 failed every iteration on .228, 2026-07-09;
hardening plan §C cascade item). electrumx is already restarted by the
version-switch path — lnd was the only dependent left stranded.
After a successful package.start/restart of a bitcoin backend, restart
running, not-user-stopped address-caching dependents under their own app-op
lock (lock order backend→dependent only, so no deadlock).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wyoming-piper/wyoming-whisper are being integrated and are not yet part of
the platform contract; their hand-staged units (companion-style
Restart=always) failed gate test 121 on .228. Exclude wyoming-* from
backend_quadlet_units() until their packaging lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.stop/restart on orchestrator-managed stacks (immich, indeedhub,
btcpay, netbird, mempool) addressed live CONTAINER names; unknown app ids
fell through to raw podman stop/start, racing systemd's --rm cleanup of the
quadlet unit — the container vanished on stop and the start half found
nothing (immich RESTART FAIL + indeedhub stuck restarting, gate 2026-07-09).
Now stack ops resolve member APP ids (reverse start order for stop) so the
orchestrator drives the quadlet .service; user-stop markers are written for
member app ids too (covers absent containers), and the orchestrator stop
fan-out treats an already-missing container as stopped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three landed 2026-07-09 (dd3afbba, c3f0a306, 2683ad4f0). Files the
repair-wait user-stop preemption as the open follow-up of the stick fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/log/archipelago/container-installs.log has been 0 bytes since April:
the service sandbox (ProtectSystem=strict class) leaves /var/log
read-only for the unit, the open() fails, and install_log() drops every
line by design (fire-and-forget). These START/STOP/RESTART/FAIL
breadcrumbs are the primary forensic trail for gate failures, so mirror
every line to tracing (journald) unconditionally and keep the file
append as best-effort for hosts where the path is writable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backend_quadlet_units() enumerates every *.container file on disk and the
active-state tests asserted each unit active + container running. A
user-stopped app keeps its unit file (e.g. the inactive half of the
bitcoin-core/bitcoin-knots multi-version pair after tonight's crash-loop
wave left core's unit on disk), so its inactive service / absent container
is the CORRECT state — gate run E false-failed on it (tests 123/124,
.228 2026-07-09). Honour the orchestrator's user-stopped.json intent
marker and skip those units in both tests.
Verified on .228: bitcoin-core's lingering unit now exempted; suite parses
(bats --count = 6) and the two fixed tests pass user-stopped filtering.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One legacy app per gate run stuck at 'stopping'/'starting' past the test
window while its container was already settled (vaultwarden:stop run C,
jellyfin:stop run D, uptime-kuma:start run E — .228, 2026-07-09). The
package scanner already sees the truth every 60s but
merge_preserving_transitional refused to report it until the RPC worker
wrote the final state, and the workers can legitimately trail the
container by minutes:
- stop workers queue behind the orchestrator per-app lock, which the
reconcile host-port repair path holds through multi-minute stability
waits (repair_manifest_host_ports_after_stability: 5s + 5s probe +
restart + 60-420s port wait + 15-90s stable-running);
- start workers hold Starting through the full readiness wait —
host_port_wait_timeout_secs is 420s for uptime-kuma (HTTP probe),
longer than any UI/test patience — and the Starting stuck-timeout is
the 20-minute INSTALLING one.
New merge rules, both truth-driven by podman's live view:
- (Stopping, Stopped) + user-stop marker → Stopped: the scanner only
normalizes exited→Stopped for user-stopped apps, so this is exactly
"the user asked for a stop and the container has exited".
- (Starting, Running) → Running: the start visibly succeeded; live
health readings are merged separately and keep reporting readiness.
Restarting is deliberately NOT resolved by a Running scan — mid-restart
readings are the pre-stop container.
Tests: merge_tests 13/13 (4 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.start/stop/restart reply immediately and run their multi-container
sequences in spawned tasks with no coordination. Back-to-back RPCs on the
same app interleave those sequences: gate run D (.228, 2026-07-09 04:28)
fired stop→start→restart on mempool; the start's member bring-up raced the
still-running stop's member shutdown — archy-mempool-db was stopped 2s
after it started and the stack finished with ZERO containers (tests 95/123/
124). A user double-clicking restart in the UI can do the same.
Workers now take a per-app tokio Mutex (fair/FIFO, keyed by the normalized
orchestrator app id) as their first await, so queued ops run in RPC arrival
order and the final state matches the last request. The RPC reply stays
immediate.
Known limit: container lists are still computed at RPC time (pre-lock), so
a stop landing mid-start can still return 'No containers found' — it can no
longer interleave/corrupt, only error. Reconciler/health-monitor restarts
don't take the lock yet (separate paths, same candidate follow-up).
Also from run D, filed separately: jellyfin/vaultwarden (legacy containers
on a quadlet node) stop fine but the package state sticks at 'stopping' —
'Failed to stop jellyfin.service: Unit not loaded' error path suspected.
Tests: runtime 9/9; archipelago builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second instance of the stop→start destruction class from tonight's gate
runs (.228): package.stop of a multi-container stack marks every member
user-stopped and quadlet-removes their containers; package.start's fallback
then resurrected only the bare package id, leaving the other members absent
AND user-stopped — indeedhub lost all 7 containers this way (mempool needed
its umbrella alias for the same reason).
- New stack_member_app_ids map (real manifest app ids, dependency order) —
used by the start fallback when a stack has no live containers, and by
package.restart instead of hard-failing "No containers found".
- orchestrator_uninstall_app_ids grows the missing indeedhub arm (same
reconciler-ghost rationale as the immich arm).
- do_orchestrator_package_start no longer aborts the whole sequence when one
member's unknown-app-id fallback fails (the #73/#74 wish) — collects and
continues.
NOTE: extends the known §G per-app-map debt (tracker updated) — the proper
home for stack membership is a manifest field; parked post-tag by design.
Tests: package:: 48/48 incl. new fallback + indeedhub-uninstall coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Regenerated from disk manifests and signed by the release root (verified
against the pinned anchor). Carries: bitcoin-knots/-core RPC/ZMQ bind
hardening (dd61a204), immich ownership-fix capabilities, lnd templated
BITCOIN_HOST, bitcoin:archival dependency tags for electrumx/mempool stack.
Nodes pick this up on their next catalog refresh; running bitcoin containers
converge on their next restart (restart-sensitive guard) — restart lnd after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lnd.bats retry loop had no per-exec timeout — a wedged lnd RPC (chain-blind
after bitcoin-knots was recreated under it) hung one podman exec, and the
whole gate, indefinitely (.228, 30+ min at test 75). timeout 10 per attempt
keeps the intended ~4min bound. Tracker: new §C item for the root cause —
apps cache their backend container's IP; a backend recreate must cascade a
dependent restart/alert or lnd goes chain-blind silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the .228 lifecycle gate (2026-07-08, first quadlet-mode run):
stop→start on the legacy `mempool` umbrella app id destroyed the mempool
deployment. Under quadlet, package.stop removes the containers; package.start
then failed with `unknown app_id: mempool` because load_manifests drops the
umbrella manifest whenever the three split-stack members are loaded — leaving
nothing to recreate from and the stack down.
Two fixes:
- prod_orchestrator start/stop/restart now resolve the historical
`mempool`/`mempool-web` id to the split members (archy-mempool-db,
mempool-api, archy-mempool-web) whenever the umbrella manifest was dropped —
the same alias install already implements ("installing mempool assembles the
split stack"). Restart falls back to start for members whose container/unit
is gone. Legacy umbrella-only nodes are unaffected (alias inactive while the
umbrella manifest is live).
- is_missing_container_error (3 sites) now recognizes podman 5.x's
`no such object` inspect phrasing, so a genuinely absent container is
classified as missing instead of surfacing as a hard inspect error.
Tests: 2 new alias tests + 2 classifier tests; prod_orchestrator suite 61/61
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- multinode suites no longer carry real node passwords as defaults: *_PW
vars are required, auto-loaded from git-ignored tests/multinode/.env
(lib sources it; .env.example documents the shape). Suites fail fast
with a clear message when unset.
- node_login/node_rpc get --connect-timeout 10 --max-time 120 (override:
MULTINODE_RPC_TIMEOUT) so one hung server-side RPC can't stall the whole
suite. Verified live: login+rpc against .116 OK; unroutable node fails
in 10s instead of hanging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- create-release.sh: assert the freshly built frontend dist actually embeds
the bumped version before packaging — the ui-dist-version guard in
tests/release/run.sh is behind --with-build, which the release path never
passes, so a silently no-opped npm build could ship a stale dist with a
valid sha256. Verified against the real dist (passes on current version,
trips on a missing one).
- publish-release-assets.sh: verify published assets by downloading and
comparing sha256 against the manifest, not just Content-Length — a
size-correct/content-wrong mirror asset now fails the publish gate.
Verified live against the v1.7.99-alpha assets on the vps2 mirror.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- operations-runbook: redact the plaintext sudo password (rotate it — it
stays in git history); replace the nonexistent test-cross-node.sh /
test-reboot-survival.sh with the real entry points (lifecycle gate,
multinode smoke, e2e/post-install scripts)
- hotfix-process: de-hardcode the v1.0.x framing, point releases at
create-release.sh + primary Gitea, correct the rollback backup paths
(/opt/archipelago/rollback + updater backup dir, not /usr/local/bin)
- troubleshooting: connectivity probe start9.com -> debian.org
- INSTALL.sh: drop removed endurain app, point at developer-guide.md
(development-setup.md never existed)
- tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session
resume block with a resolved note; release-gate item 8 reflects the
2026-07-08 version decision (1.8.0-alpha)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second false-drift source found live on .228 right after the entrypoint
split fix: a YAML `>-` script with more-indented continuation lines
keeps literal newlines in custom_args, but shell_join flattens them to
spaces when writing the quadlet Exec= line, so Config.Cmd stores spaces
where the manifest has newlines. bitcoin-knots and fedimint-gateway
read as permanently command-drifted over line breaks alone (only the
restart-sensitive guard kept them from recreate-looping like electrumx).
Normalize both sides the way shell_join does before comparing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- app-manifest-spec.md: full rewrite from the real schema in
core/container/src/manifest.rs — it was 5 months stale and missing the
entire modern feature set (build, network/network_aliases, derived_env,
secret_env, generated_secrets/certs, data_uid, files, interfaces, hooks,
extensions flatten) and documented wrong network_policy values.
- app-developer-guide.md: add generated_secrets/generated_certs/
network_aliases/hooks to the field table.
- APP-PACKAGING-MIGRATION-PLAN.md: phase status stamped (1-3 done, 5 mostly,
4+6 open); deleted meshtastic app removed from regression-proof lists.
- registry-manifest-design.md: status design → implemented (phases 1-3),
stale manifest_dir:Option line fixed.
- marketplace-protocol.md: reframed proposal → as-built (marketplace.rs +
RPCs + UI shipped; create-invoice noted; schema disambiguated).
- manifest-hooks-design.md: phase 3 (indeedhub) done, phase 4 resolved via
orchestrator+generated_secrets instead of hooks.
- README/architecture: restore the NIP-07 signer-bridge claim — it is real
(neode-ui/public/nostr-provider.js), the earlier audit only grepped Rust.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
developer-guide: RPC dispatch moved from mod.rs handle_rpc_call to
dispatcher.rs dispatch() long ago — the registration instructions now point
at the real file; module tree updated (package/, federation/, mesh/ are
directories; mesh is tri-protocol); roadmap pointer loop/plan.md → docs/ROADMAP.md.
CONTRIBUTING: docs/development-setup.md never existed → developer-guide.md;
GitHub-specific fork URL generalized to the Gitea instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes: phantom parmanode crate removed and crate table now matches the real
workspace (5 members + 4 standalone), Debian 12 box → 13, LOC/file counts
were understated ~2x (49k→117k Rust, 47k→69k TS/Vue), 30→51 apps, dispatcher
is dispatcher.rs with ~380 methods (not mod.rs '100+'), NIP-07 claim replaced
with the NIPs actually implemented (33/44/04), Meshcore-only mesh → tri-protocol
with Reticulum daemon, dead MASTER_PLAN.md/BETA-PROGRESS.md links replaced.
Adds as-built sections for the app platform (manifest → Quadlet → signed
catalog overlay → marketplace) and drops the environment-specific node
IP inventory from a public doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
container_command_drifted compared the manifest's entrypoint and
custom_args against Config.Entrypoint and Config.Cmd separately, but
quadlet's Entrypoint= takes a single value, so a manifest entrypoint of
[sh, -lc] is written as Entrypoint=sh + Exec=-lc ... and podman reports
entry=[sh], cmd=[-lc, script]. Same argv, different split — every
quadlet-created app with a multi-element entrypoint read as drifted
forever. On .228 this recreated electrumx on every reconcile pass (114
times in 6h); bitcoin-knots and fedimint-gateway showed the same false
drift and were only spared by the restart-sensitive guard.
Compare the concatenated argv (entrypoint ++ args vs Entrypoint ++ Cmd)
instead — that is what actually runs. When the manifest declares no
custom_args, only the entrypoint prefix must match, since the trailing
Cmd may be the image's baked-in CMD.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README fixes: version badge 1.3.1→1.8.0-alpha, real LOC (117k Rust/69k TS-Vue
vs the stale 49k/47k), 51 apps not 29, tri-protocol mesh
(Meshtastic/MeshCore/Reticulum) not Meshcore-only, Nostr NIP-33/44/04 (NIP-07
was never implemented), Ed25519 signed catalog + pinned release-root anchor,
OTA host de-hardcoded (tx1138 retired), dead docs/MASTER_PLAN.md link removed,
crate map corrected (openwrt added, phantom crates dropped), new Philosophy and
Roadmap sections reflecting the manifest-driven/signed-registry north star.
docs/ROADMAP.md is the public-facing shipped/in-progress/planned summary,
synthesized from the master plan, unified tracker, and 1.8.0 hardening plan —
including the completed signing ceremony and the 1.8.0-alpha version decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session logs, handovers, point-in-time snapshots, the v0.1.0 security audit,
stale generated HTML guides, and shipped design docs (three-mode UI, install
screens) move out of the living docs/ tree. docs/archive/README.md explains
what each was and why it's archived; all in-repo links updated.
(docs/UNIFIED-TASK-TRACKER.md links intentionally untouched — it has another
agent's uncommitted work in flight.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The packaged reticulum-daemon is a PyInstaller one-file binary: the
bootloader extracts a ~48M _MEI* dir into TMPDIR and only deletes it on
clean exit. ReticulumLink's shutdown SIGKILLed the process group
immediately (kill_on_drop + instant SIGTERM+SIGKILL in Drop), so the
cleanup never ran and every reconnect/restart stranded another 48M in
/tmp — filling .116's 12G tmpfs in ~4h.
Two-pronged fix:
- terminate_group(): group-wide SIGTERM now, SIGKILL from a detached
backstop thread 5s later, used on every shutdown path (Drop and all
spawn-failure paths, which previously used bare start_kill and could
orphan the forked Python child too).
- Per-interface private TMPDIR under the runtime dir, wiped on every
(re)spawn — even a hard-killed or power-lost daemon can never
accumulate stale extraction dirs, and they land on the data disk
instead of the small tmpfs /tmp.
Verified: unit tests + the ignored mesh_service_connects_over_
reticulum_tcp_client integration test (spawns two real daemons over
loopback TCP through the changed spawn/terminate path), clean teardown
with no stray processes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fedimint sat 'failed' for 7+ hours (exit 255 at 21:21, pre-gate) — the
reconciler repairs missing/drifted containers but never reset-fails +
starts a failed quadlet .service, and the health monitor can't see an
app whose container is gone. Concrete lever added to the Tier 2
container-flapping item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
capabilities:[] was latent — the long-lived legacy container predated
strict manifest enforcement, so nothing noticed that a recreate against
this manifest produces a root process without DAC_OVERRIDE that
EACCESes on upload/encoded-video and crash-loops (49 systemd restarts
on .228 when the 2026-07-05 secret-env migration finally recreated
it). Any reinstall or reboot-repair would have tripped the same wire.
Cap set mirrors immich-postgres minus SETUID/SETGID.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Secret env used to merge into manifest.app.environment, landing in
'podman inspect' Config.Env on the API backend and — worse — as
plaintext Environment= lines in Quadlet unit files on disk. Now:
- expand_and_partition_env (container crate, pure + tested) expands
${KEY} placeholders and splits env into plain entries and
secret-bearing pairs. Plain entries that interpolate a secret
(btcpay's Password=${BTCPAY_DB_PASS} connection strings) are
tainted and travel as secrets too. Secret values themselves are
never expanded (a generated value containing '${' passes verbatim).
- values register as podman secrets: stdin (never argv/tempfile),
--replace, content-hash label to skip no-op rewrites; a per-app hash
cache in the orchestrator makes steady-state reconciles free of
podman secret calls. Registration goes through the runtime trait
(default no-op keeps mocks/docker inert).
- containers reference secrets by name: secret_env map in the libpod
create spec, Secret=<name>,type=env,target=<KEY> in Quadlet units.
Verified empirically on fleet podman 5.4.2: value absent from
inspect Config.Env, runtime injection works rootless.
- rotation detection: io.archipelago.secret-env-hash container label
(API) / the changed unit bytes (Quadlet). Pre-upgrade containers
lack the label, so every secret-bearing app recreates ONCE on the
first reconcile after deploy — deliberate, it scrubs the plaintext
secrets out of existing container configs. Data dirs untouched.
- docker dev fallback keeps plain -e injection (no secret store);
podman secrets persist across uninstall, matching the
preserve-credentials invariant (reinstall re-registers by hash).
In-container /proc/<pid>/environ is unchanged — env remains the
app-compat contract; the closed leaks are inspect output and unit
files on disk.
Tests: archipelago-container 61/61 (3 new: taint partition, verbatim
secrets, hash order-independence), archipelago container:: 160/160
(fedimint install test now asserts the secret arrives as a ref, not
env; quadlet render test asserts Secret=/Label= lines). NEEDS the
on-node gate re-run before the item counts as verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New container::image_verify gates PodmanClient::pull_image and the
dev-only DockerRuntime::pull_image. Signature claims classify three
ways: absent/empty (pull unverified, logged), the literal
'cosign://...' placeholder every fleet manifest carries today (same —
enforcement stays dormant until the signing ceremony ships real
values), or a declared signature, which must verify via
'cosign verify --key /etc/archipelago/cosign.pub
--insecure-ignore-tlog=true' (plus --allow-insecure-registry
--allow-http-registry for the HTTP mirror; flags checked against
cosign's own docs) before anything is fetched. Missing key, missing
cosign binary, timeout, or verification failure all hard-fail the
pull — a declared signature cannot be skipped on either runtime. Key
path overridable via ARCHIPELAGO_COSIGN_PUBKEY for tests/staging.
Deletes security::ImageVerifier: zero callers, blocking
std::process::Command on would-be async paths, and a fantasy
'cosign verify --signature' invocation (that flag belongs to
verify-blob).
Activation ships with the Workstream B ceremony, in order: pin
cosign.pub on nodes + install cosign, then publish real
image_signature values in the catalog.
Tests: archipelago-container 58/58 (5 new), archipelago container::
159/159, security check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh: verify_signature accepts a v2 preimage (t,v,ts,seq) alongside
legacy v1 (t,v,ts); signed_with_seq() is the v2 sender path, not yet
wired — senders stay v1 until the fleet verifies v2 (receivers
hard-drop bad sigs, so flipping send-side first would break
mixed-fleet alerts). Tests: v2 verify, v2 seq-tamper rejection,
v1 sign-then-set-seq compat.
- mesh listener: malformed radio-supplied DID shorter than the
'did🔑' prefix can no longer panic advert_name (slice -> .get()).
- auth: the pre-setup password123 dev login and the constant itself are
now #[cfg(debug_assertions)] — no release binary carries the bypass,
whatever its runtime config says.
- orchestrator: canned host-facts under #[cfg(test)] — awaiting real
subprocesses under tokio's paused test clock deadlocks against
auto-advanced timers (the old blocking detection only worked by never
yielding).
- drop two now-unused std::process::Command imports left by 4c75bb3d.
Tests: mesh 110/110 (incl. 2 new), api 68/68, container 159/159,
archipelago-container check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Was implicit across the Phase-3 Quadlet flip and Workstream F; now one
consolidated pre-tag item with the lever list, an observability proposal
(per-app restart counter + flap log line), and an already-landed list so
nothing gets re-done.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
verify_pending_update previously cleared the rollback marker on any
2xx/3xx from GET / — a release with a dead RPC API or broken podman
access passed and never rolled back. Verification now requires, in the
same attempt: the frontend via nginx, backend RPC liveness (an
unauthenticated POST /rpc/v1 — 401 proves the stack is up, 5xx/404/
refused fails it), and rootless podman reachability. A pre-loop check
also asserts the running binary's version matches what the marker says
was applied, catching a silent or half swap deterministically.
Per-app container assertions are deliberately excluded: the
pre-Quadlet service restart legitimately takes containers down and the
boot reconciler can need minutes for heavy apps — that would
false-rollback healthy updates. Revisit after the Phase-3 flip.
§B of the 1.8.0 hardening plan; update suite 38/38 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catalog- and manifest-supplied image refs reached pull_image without
ever passing the RPC boundary's validator — a malicious catalog entry
or manifest could pull from an arbitrary registry. The allowlist now
lives in container::image_policy (the RPC check delegates to it) and
both orchestrator pull sites (install_fresh and
ensure_resolved_source_available) refuse refs that fail it.
The shared policy accepts trusted-registry refs and registry-less
Docker Hub shorthand (grafana/grafana etc., used by 8 shipped
manifests — a registry-less ref cannot name an attacker host), and
rejects explicit non-allowlisted hosts, shell metacharacters, and
malformed refs. §A of the 1.8.0 hardening plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Server.vue rendered the backend-generated WireGuard peer QR with raw
v-html while the analogous TOTP QR was DOMPurify-sanitized — both now
use the same svg-profile sanitizer. Mesh.vue's 5s poll interval gets
the same start-guard as the arch poll (no leak on double-mount) and is
nulled on unmount. curatedApps.ts catalog fetches no longer fail
silently: each failed source logs a console.warn, including the final
all-sources-failed fallback to the hardcoded list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every production process spawn reachable from a tokio worker now uses
tokio::process: the install path's podman-port probe, the dependencies
disk check, factory-reset restart, config host-IP detection, the
orchestrator's host-facts helpers (resolve_dynamic_env and its call
sites made async to carry it through), and AutoRuntime's podman/docker
probes.
The FIPS transport probe is the special case: is_available() is a sync
trait method called from async route(), so instead of blocking ~50ms
on systemctl per stale-cache hit it now serves the cached value and
refreshes on a background thread (stale-while-revalidate) — bounded
staleness, zero stalled workers.
§C of the 1.8.0 hardening plan; container/transport/config/package
suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nostr bridge derived the caller from the launcher's own URL and
never checked event.origin, so any co-resident iframe could pull the
node's nostr pubkey or use nip04/nip44 decrypt as an oracle while an
app was open. The bridge now rejects senders whose real origin doesn't
match the open app's origin, and every identity-sensitive method
(getPublicKey, signEvent, encrypt/decrypt) requires user consent or a
remembered per-origin approval — previously only signEvent did.
share-to-mesh in App.vue likewise accepted messages from any sender
and force-navigated to /mesh with an attacker-staged CID; it now
requires same-origin, matching Chat.vue's existing handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
check_for_updates now fetches the manifest as raw JSON and runs
trust::verify_detached before parsing: a tampered or wrong-signer
signature rejects the mirror outright, and unsigned manifests are
offered for MANUAL apply only — the 3 AM auto-apply scheduler refuses
them, closing the unattended remote-root hole (§A of the 1.8.0
hardening plan). UpdateState gains manifest_signed so the UI can
surface authenticity.
Publisher side: create-release.sh signs the manifest during the
release (ceremony, mnemonic via TTY/env only), publish-release-assets
hard-refuses to ship an unsigned manifest (grep + new 'ceremony
verify' cryptographic gate), and scripts/sign-manifest.sh covers
re-signing outside a release run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin RELEASE_ROOT_PUBKEY_HEX from the 2026-07-02 release-root signing ceremony
(signer did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur) so nodes verify
the publisher identity of the app-catalog. Sign releases/app-catalog.json in place.
Fix two floats that made the catalog unsignable: archy-btcpay-db manifest version
-> string, fedimint-clientd cpu_limit 0.25 -> 1 (u32). Add scripts/sign-catalog.sh
helper, the 1.8.0 release-hardening plan/tracker, and the commit-and-push project
rule in CLAUDE.md.
Backward-compatible: old binaries still accept the signed catalog; the pinned-anchor
binary ships in the next build/OTA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
