The gateway crash-looped dialing bitcoind at host.archipelago:8332 (the
host gateway IP, where bitcoind does not listen). Root-cause fix, three
parts:
- fedimint-gateway manifest: --bitcoind-url now comes from
$FM_BITCOIND_URL, filled by a {{BITCOIN_HOST}} derived-env — works on
Knots, Core, or any future Bitcoin distro.
- fedimint manifest: same derived-env replaces the hardcoded
FM_BITCOIND_URL=http://bitcoin-knots:8332 environment entry.
- orchestrator: removed the hardcoded FM_BITCOIND_URL=bitcoin-knots
override that clobbered the derived-env, and bitcoin_host() now
accepts any running bitcoin*/bitcoin container (excluding -ui and
archy-* sidecars), preferring the known names in order.
Catalog regenerated + signed (nodes install from the signed catalog, so
the manifest fixes only reach them via this regen).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image default is beam 5 on x86 (1 on ARM). Benchmarked on framework-pt
(i5-1135G7, base-int8, real speech): beam 1 transcribes identical text
~45% faster; extra CPU threads made it slower, so only the beam changes.
App revision 3.4.2 > image 3.4.1 so catalog-driven nodes roll the args
change; a 3.4.1-1 suffix would semver-compare LOWER and never ship.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The creds-off-argv script used printf "rpcuser=%s" in the manifest's
custom_args; quadlet copies Exec= into the generated service's ExecStart,
where systemd expands %s (user's shell) at load time — bitcoind's rpc.conf
came out as rpcuser=/bin/bash and every RPC consumer got 401s on quadlet
nodes (framework-pt: bitcoin-status, HA block-height sensor, LND chain RPC).
Two-layer fix: quadlet.rs now escapes % -> %% in Exec/Entrypoint/HealthCmd/
Environment emission (with regression test), and the bitcoin manifests write
rpc.conf with plain echo so the catalog also heals nodes still running older
binaries. Release catalog regenerated with the fixed scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pine-openwakeword manifest (wyoming-openwakeword 2.1.0, :10400, /custom model
dir for the future Yo Archy model). Pine 1.3.0: launcher page gains a live
node-status card fed by /api/pine/status via a same-origin nginx proxy, and
copy for the new intents / Claude fallback / mesh announcements. Catalogs,
app-session config, drift ids regenerated; release catalog embeds 56
manifests (unsigned until the ceremony). Framework PT test plan in docs/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Wallet scan modal: fallback buttons no longer flash on open (spinner while
the camera auto-starts, video hidden until live — also kills the Android
WebView play-glyph), decode rate 10/s -> 4/s (preview visibly lagged on
phones), clear messages for BOLT12 offers and LNURL/lightning addresses
(unsupported by the LND backend; full LNURL-pay is a queued feature),
zero-amount invoices in BIP21 URIs prefill from amount=.
- Pine 1.2.0: launch-page copy reflects the auto-seeded HA wiring (only
speaker pairing stays manual), adds the ask-Archy hint and the
silent-answers power-cycle troubleshooting tip (firmware FIFO bug).
- bitcoin-core/knots: rpcuser/rpcpassword move from bitcoind argv (world-
readable in host ps) to a 0600 config file inside the container tmpfs;
the salted txrelay rpcauth hash stays on argv. Catalog regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HA 2024.1's Wyoming integration predates ping/pong keepalives. The PineVoice
satellite firmware (wyoming 1.7.2) drops connections idle for ~8s, so HA
looped on 'Satellite disconnected' forever and the speaker never worked.
HA 2026.7.3 pings satellites and holds the connection (verified on
framework-pt: link stays ESTABLISHED, wake word reaches Assist).
Catalog regenerated with the bump — needs the signing ceremony before the
origin publish.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The launcher must be a secure context for Web Bluetooth, but the UI opens local
apps as http://host:port (resolveAppUrl), so an https-only launcher (v1.1.0)
would break the Open button with a TLS mismatch. Serve http on 10380 (the Open
target) that 301-redirects to the https listener on 10381, so the new tab lands
on https where navigator.bluetooth is available. Bump pine 1.1.0 -> 1.1.1;
re-sign the catalog (only the pine entry changes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the launcher's text instructions with the real Improv-over-BLE
Web-Bluetooth provisioner (a "Connect Pine to WiFi" button, SSID/password
fields, live log) ported from the pine repo. Web Bluetooth needs a secure
context, so the launcher now serves HTTPS (self-signed): :80 (published 10380,
the Open target) 301-redirects to :10381, and open_in_new_tab keeps BLE out of
an iframe. Bump pine 1.0.0 -> 1.1.0.
Regenerate + re-sign releases/app-catalog.json (only the pine entry changes vs
v1.7.109) so nodes reconcile the button page live. Signed by the release-root
key did🔑z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Package the PineVoice/Wyoming local voice assistant as ONE store app "Pine"
that runs three rootless containers:
- pine-whisper — Wyoming faster-whisper STT (docker.io/rhasspy/wyoming-whisper
3.4.1, model base-int8/en), publishes :10300
- pine-piper — Wyoming Piper TTS (2.2.2, voice en_GB-alba-medium),
publishes :10200
- pine — nginx launcher serving a self-contained setup/status page
(WiFi provisioning + Home Assistant wiring instructions), the Open target
The two engines publish their Wyoming ports so Home Assistant reaches them via
host.containers.internal. All rootless (cap-drop=ALL, no_new_privileges),
manifest-driven — no hardcoded podman/installer. The engines are internal stack
members (added to the drift-check INTERNAL_MANIFEST_IDS allowlist); only "pine"
carries a catalog entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manifest baked bitcoin-knots:8332 into DAEMON_URL. Nodes whose
backend runs under the bitcoin-core name (and any future variant
without a knots DNS alias) left electrumx permanently disconnected —
'connection problem' forever and a block index stuck at 0. The
startup script now picks the first backend name that resolves on
archy-net and falls back to bitcoin-knots.
Catalog regenerated (catalog manifests override disk ones fleet-wide);
this regen also embeds the recently-merged barkd/Ark manifest for the
first time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two btcpay-server manifest gaps found on shorty-s (2026-07-10):
- BTCPAY_PLUGINDIR=/datadir/Plugins: the image default is container-local,
so every container recreate silently wiped installed plugins (the
reinstall-the-nostr-plugin-after-every-restart symptom, plus a soft-restart
loop while an install was pending).
- BTCPAY_BTCLIGHTNING via optional secret_env btcpay-lnd-connection: the
manifest never wired the internal LND node. The secret is generated by the
daemon (see previous commit); nodes without LND skip it.
releases/app-catalog.json regenerated (embeds the manifest; origin-wins
overlay means nodes only pick this up from the published catalog). Signature
stripped by regeneration — run scripts/sign-catalog.sh before publishing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strfry/vaultwarden 8082 collision shipped in a signed catalog
because nothing checks cross-app host ports. Fix the remaining latent
pairs from the cfb8d953 audit — did-wallet 8083→8088 (filebrowser
keeps 8083), morphos-server 8086→8089 (netbird-server keeps 8086),
lightning-stack 8087→8091/9737→9738 (netbird and fedimint-gateway
keep theirs) — and fix their healthchecks, which probed the HOST port
inside the container (same class as strfry 6fe62355; now 127.0.0.1 +
container port). load_manifests now flags collisions across the
merged disk+catalog set (bitcoin variants and the mempool umbrella
pair are exempt as mutually exclusive), and a repo-level test parses
apps/*/manifest.yml so a future latent pair fails CI instead of
shipping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six manifests still pulled through git.tx1138.com/lfg2025, which is
dead — every boot reconcile of archy-btcpay-db/archy-nbxplorer failed
at the pull step (live-testing report 2026-07-10; the orchestrator
now also falls back to local storage, 3a5c5db1). All six tags
verified present on 146.59.87.168:3000/lfg2025 via the registry API.
Ships fleet-wide at the next catalog ceremony (catalog manifests
override disk ones).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The healthcheck runs INSIDE the container, so it must target the
container port (7777), not the host mapping (8090) — and 127.0.0.1
explicitly, because `localhost` resolves to ::1 in the image while
strfry binds IPv4 0.0.0.0 only. Both bugs verified empirically on .228
(container was Up but "unhealthy"; 127.0.0.1:7777/health returns 200).
Ships at the next catalog regen/re-sign (catalog overlay supremacy);
with the runtime manifest-reload now in place it will apply fleet-wide
on the next catalog refresh, no service restarts needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dockurr/strfry image listens on 7777 (not 8080) and its default config
demands a 1M NOFILES rlimit — above the rootless user-manager hard cap
(524288), so the relay exited at startup even with the mount fixed. Ship
the config declaratively (manifest files: + ro bind at /etc/strfry.conf)
with nofiles = 0; this also skips the entrypoint's copy into /etc, which a
readonly_root container cannot perform. Verified end-to-end on .228 in a
quadlet-equivalent systemd-run environment: active, GET / and /health 200.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two defects kept strfry crash-looping since install (62,868 systemd
restarts on .228): its host port 8082 collides with vaultwarden's, and the
volume mounted at /strfry which the dockurr image never reads — its LMDB
lives at /app/strfry-db (entrypoint default), so mdb_env_open got ENOENT
even once the port was freed. 8090 is unclaimed across all manifests.
Uninstalled from .228 (db was empty — it never ran); reinstall after the
next catalog regen ships this manifest.
NOTE: repo-wide host-port collision audit also flags did-wallet/filebrowser
(8083), morphos-server/netbird-server (8086), lightning-stack/netbird
(8087), fedimint-gateway/lightning-stack (9737) — latent, tracked in the
hardening plan; a cross-app collision check belongs in the port allocator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
capabilities:[] was latent — the long-lived legacy container predated
strict manifest enforcement, so nothing noticed that a recreate against
this manifest produces a root process without DAC_OVERRIDE that
EACCESes on upload/encoded-video and crash-loops (49 systemd restarts
on .228 when the 2026-07-05 secret-env migration finally recreated
it). Any reinstall or reboot-repair would have tripped the same wire.
Cap set mirrors immich-postgres minus SETUID/SETGID.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
gap where installing it standalone (a legitimate direct orchestrator-install
target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
etc.) resurrecting itself after an explicit uninstall survives a restart or
reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
("latest") as the reported running version -- a stale local :latest cache
reported "latest" forever regardless of what latest had moved on to. Now
falls back to asking the Bitcoin backend directly via `bitcoind --version`
when the tag is floating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Master-plan backlog §10b/§10c: replace two per-app-hardcoded lookups with
generic, manifest-driven behavior so future apps are covered automatically
instead of needing a code edit.
- extract_lan_address (docker_packages.rs) now skips container-side ports
that are known non-HTTP (SSH, FTP, common DB ports) instead of blindly
taking podman's first-listed port. Fixes the whole class of bug the gitea
SSH-before-web static override was a one-off patch for.
- requires_unpruned_bitcoin (dependencies.rs) now checks the app's own
manifest for a `bitcoin:archival` dependency declaration first, falling
back to the old hardcoded id list. electrumx and mempool manifests now
declare it explicitly as the proof case.
869/869 Rust tests green, catalog drift clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three stacked bugs made "switch version" silently fail / crash-loop, and
the data-access mismatch corrupted a node's index during recovery attempts.
Backend renderer:
- sync_quadlet_unit ignored the per-app pinned version and re-rendered the
quadlet with the manifest's :latest every reconcile tick, reverting any
switch. Factor the install-time catalog/pin resolution into a shared
resolve_catalog_image() and call it in BOTH install_fresh and
sync_quadlet_unit.
- The renderer folded manifest `entrypoint: ["sh","-lc"]` into Exec=, which
only worked when the image entrypoint was a passthrough shell wrapper. The
versioned images use ENTRYPOINT ["bitcoind"], so Exec=sh -lc ... became
`bitcoind sh -lc ...` and crash-looped. Emit a real Entrypoint= override;
exec_changed now also compares Entrypoint=.
Images:
- Build all bitcoin images (Core + Knots, every version) as container-root
(USER removed) like the legacy :latest image. Chain data is owned by the
data_uid (container uid 102); root reads it via CAP_DAC_OVERRIDE (granted in
the manifest). A non-root USER (the previous uid 1000) can't read existing
chain data → "Error initializing block database". Still fully rootless:
container-root maps to the unprivileged host service user.
Catalog:
- bitcoin-knots versions[]: 29.3.knots20260508/20260507/20260210 +
29.2.knots20251110, "latest" tracking newest.
- bitcoin-core versions[]: add 29.2 + a "latest" entry. All images rebuilt
root and published to the mirror.
Frontend:
- AppSidebar version dropdown: rename the latest option to "Always use the
latest version" (no v prefix), fix right padding, and guarantee the current
selection matches a real option (was rendering blank).
- New InstallVersionModal: full-screen version chooser shown from the App
Store / Discover install button for multi-version apps (Bitcoin Knots/Core),
app icon + "Install <name>", latest pre-selected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The archy-mempool-web health_check endpoint used http://localhost:8080.
Inside the frontend image, wget resolves `localhost` to ::1 (IPv6) first,
but nginx binds 0.0.0.0:8080 (IPv4) only -> the baked HealthCmd gets
"connection refused" every probe -> container is perpetually unhealthy ->
the reconciler recreates it forever (observed on .228: mempool container
re-Started every ~3 min, Health=unhealthy). Proven live: in-container
`wget http://localhost:8080/` = refused, `wget http://127.0.0.1:8080/` = OK.
Pin the probe to 127.0.0.1 so it matches nginx's IPv4 bind. Updated both
the source manifest and the embedded copy in releases/app-catalog.json
(the catalog overlay wins over the disk manifest on fleet nodes, so the
catalog copy is the one that actually reaches .228).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mempool manifest + embedded catalog declared the frontend container
port as 4080, but mempool-frontend nginx listens on 8080 (the stack
creates it as -p 4080:8080 with FRONTEND_HTTP_PORT=8080, see
api/rpc/package/stacks.rs). So every reconcile rendered the quadlet as
PublishPort=4080:4080, disagreed with the working 4080:8080 container,
and restarted it ("port binding drift" -> "host port 4080 did not become
reachable within 5s" -> "host listener disappeared; restarting") in a
perpetual loop on .228. Correcting the manifest container port to 8080
makes the rendered quadlet match reality so the drift/restart loop stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On NAT'd nodes that can reach the iroh federation neither directly nor
via iroh's public relays, fmcd's embedded iroh networking enters a
relay/hole-punch reconnect hot-loop that pegs its entire CPU allotment
indefinitely (observed ~1 core sustained for 4 days on a Tailscale node,
while LAN nodes that reach the guardian directly stay <3%). fmcd 0.8.0
exposes no iroh/relay knobs, so:
- fmcd-run now samples fmcd's own CPU and restarts it when it stays near
its allotment for ~15 min (a restart demonstrably clears the stuck iroh
state; real work is bursty and never flat-pegs a core for minutes).
- Lower cpu_limit 1 -> 0.25 core so a stuck instance can't starve the
node (steady-state is <3% of a core; joins are brief).
Ships as fmcd:0.8.1 (launcher-only rebuild, same fmcd binary). Bumped the
image pin + cpu_limit in the manifest, image-versions.sh, the embedded
catalog manifest (releases/app-catalog.json), and the UI catalogs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workstream F now in-progress: the immich/grafana uninstall hang →
ghost/stuck-bar/reinstall-block is root-caused (unbounded systemctl/
podman in quadlet::disable_remove) and fixed (71cc9ac4); cascade-
uninstall.bats 7/7 on .228. Records the remaining F items + the pending
gate-wiring decision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The frontend nginx used a literal proxy_pass host with no resolver, so it
pinned mempool-api's IP at worker startup. When the backend restarts (gate,
OTA, crash, reboot re-IPAM) podman reassigns its IP and nginx keeps proxying
to the dead one -> /api hangs, websocket 502s, UI shows 'offline' until a
manual nginx reload. Same stale-upstream-IP class as the netbird 502.
Fix: mempool-frontend:v3.0.1 rewrites the generated nginx-mempool.conf to
re-resolve the backend per-request via 'resolver' + a variable proxy_pass.
Resolver address is read from /etc/resolv.conf (podman aardvark-dns answers
on the network gateway, not Docker's 127.0.0.11). Per-location path mapping
preserved (ws -> '/', /api/v1 identity via no-URI, /api/ -> /api/v1/ rewrite).
Proven on .228: backend IP change now auto-recovers with no reload; the
literal-host control still 502s. Migrated the manifest off the retired
tx1138 registry to vps2.
Also: mempool.bats #74 waited only 180s post-restart (the slow path) and
called an undefined 'fail' helper (status 127). Bumped to 300s to match the
passing parity probes and emit a real failure instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the loaded .198 the frontend churned (created → "unhealthy" → reconciler
recreates → loop). The http health check fetched / through nginx (SPA +
sub_filter) and false-failed under node load; the reconciler then treated the
frontend as wedged and recreated it. nginx binds 7777 at startup, so a tcp
liveness check passes immediately and stays green under load while still
catching a real "nginx not listening" failure. Generous retries/start_period.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live fresh-create on .228 (post special-case removal) had nginx workers die
with "setgid(101) failed (Operation not permitted)" → workers exited code 2,
port published but nothing served (HTTP 000). The orchestrator does
--cap-drop=ALL, so unlike the legacy `podman run` (default caps) nginx's master
couldn't drop workers to the nginx user. Declare CHOWN/DAC_OVERRIDE/SETGID/SETUID
(SET* to drop the worker user, CHOWN+DAC_OVERRIDE for the tmpfs proxy cache).
Verified on .228: frontend fresh-creates, caps applied, nginx serves, UI 200
incl. /api/ and /nostr-provider.js.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author the IndeedHub stack as 7 manifests (postgres/redis/minio/relay/api/
ffmpeg + frontend) and route install_indeedhub_stack through the
orchestrator first (immich pattern), falling back to the legacy installer
only when the manifests aren't deployed.
Data-preserving by construction — the manifests reproduce the live install
exactly so an existing node ADOPTS rather than recreates:
- container_name = the live hyphenated names the runtime already references
(health_monitor tiers/deps, crash_recovery).
- named volumes indeedhub-{postgres,redis,minio,relay}-data (not bind mounts).
- dedicated indeedhub-net + network_aliases [postgres|redis|minio|relay|api]
so the api/ffmpeg env hostnames and the frontend nginx upstreams resolve
unchanged.
- generated_secrets (indeedhub-db-password/-minio-password owned by their
backends, indeedhub-jwt by the api) reuse the live /var/lib/archipelago/
secrets values (ensure_one no-ops on existing files; postgres pw is fixed
at PGDATA init). minio user "indeeadmin" + AES_MASTER_SECRET literal kept.
The frontend carries the post_install hook (#20) that replaces the hardcoded
patch_indeedhub_nostr_provider: strip X-Frame-Options, refresh
nostr-provider.js from /opt/archipelago/web-ui, inject the <script> if
absent, reload nginx — defensive/idempotent since indeedhub:1.0.0 already
bakes these. Frontend manifest also corrected off its dead Next.js shape
(health check now nginx :7777, tmpfs /run + /var/cache/nginx).
Builds + unit-tested; live adoption/lifecycle verification on .228 next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
package.stop/start/restart broke ("no containers found" / "no such object
immich_postgres") because the runtime hardcodes the immich stack's container names
as immich_server/immich_postgres/immich_redis (underscore) across 8 files
(lifecycle, health, crash-recovery, ports, config). The migration had named the
containers by app_id (hyphen), mismatching all of it.
Root cause of the earlier failed attempt: container_name was nested under an
`extensions:` block, but `app.extensions` is serde(flatten) — container_name must
be a TOP-LEVEL app key to be read by compute_container_name. Fixed: set
container_name: immich_server / immich_postgres / immich_redis at top level, and
point DB_HOSTNAME/REDIS_HOSTNAME at the underscore aliases. App ids stay hyphen
(immich/immich-postgres/immich-redis) so the catalog identity (title+icon) holds.
Manifest-only change — container names now match existing runtime references, no
code edits to the 8 files. (Deriving stack containers from manifests instead of
hardcoded lists remains a north-star follow-up.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After the manifest migration the launcher installed as "immich-server" (app_id),
which has no catalog entry → showed the raw id and no icon. Rename the server
manifest app_id immich-server→immich so it matches the catalog/curated "immich"
entry (title "Immich", icon immich.png) and is recognised as a known launcher app
(APP_CATEGORY_MAP) → stays in My Apps. immich_stack_app_ids now installs
[immich-postgres, immich-redis, immich]; orchestrator.install bypasses package
routing so there's no recursion with the "immich"→stack-installer mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Classify databases/APIs/backends into Services (#10): add immich-postgres/redis
to SERVICE_NAMES; isServiceContainer matches -postgres/-redis/-valkey/-cache/-db
suffixes; isWebsitePackage final fallback now routes any no-UI, non-known package
to Services ("anything that isn't the frontend UI launcher").
- Services show their parent app's icon (#14): backends reuse the app logo
(immich-* → immich, archy-btcpay-db → btcpay, indeedhub-* → indeedhub, etc.)
via explicit APP_ICON_FALLBACKS + prefix map, instead of 404 → 📦.
- Categories sub-nav for Services (#12): getServiceCategory + buildServiceCategories
+ useServiceCategories; Services tab gets the same desktop/mobile category strips
(Databases/Caches/APIs/Backends), shown only for categories with items. Shared
selectedCategory resets to 'all' on tab switch.
- Mobile swipe (#11): the tab-swipe gesture is suppressed over .mobile-category-strip
so swiping the category chips scrolls them instead of changing tabs (covers both
My Apps and the new Services strip).
vue-tsc build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the immich migration off the legacy hardcoded install_immich_stack
(podman run + sudo chown) to the registry-manifest + orchestrator path. Validated
live on .228 (clean single set, healthy v2.7.4, data dir ownership correct).
- install_immich_stack now tries install_stack_via_orchestrator(immich_stack_app_ids)
first; legacy remains only as the no-manifests fallback.
- immich-{postgres,redis,server} manifests corrected from live findings:
* named by app_id (dropped container_name override) — using container_name
spawned DUPLICATE containers (app_id-named install vs name-override reconcile)
on the same PGDATA, which corrupted a postgres cluster. Server reaches its
siblings via app_id aliases (DB_HOSTNAME=immich-postgres, REDIS=immich-redis).
* immich-postgres data_uid 100998:100998 (postgres drops to container 999 →
host 100998 under rootless; verified the fresh dir is chowned correctly).
* immich-server version "release"→"2.7.4" (manifest validation requires a digit;
the bad version made the manifest silently skip → partial orchestrator install
→ legacy fallback → the duplicate corruption above).
- HARDEN install_stack_via_orchestrator: only fall back to the legacy installer
when NOTHING was installed yet. An "unknown app_id" AFTER a member is up now
errors instead of double-creating containers on shared data (the corruption
root cause).
- Strict the all-manifests round-trip test: fail (not skip) on any invalid shipped
manifest — this gap let the bad immich-server version through.
Known follow-up (pre-existing, platform-wide): orchestrator-installed backends
(immich, btcpay-db) run as podman --restart, not Quadlet, and podman-restart.service
is disabled on .228 → reboot-survival gap independent of this migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
immich becomes a manifest-driven stack (the legacy install_immich_stack — hardcoded
podman run + sudo chown — is the anti-pattern being retired). Three image-only
manifests modelled on the btcpay stack + the live .228 container config:
- immich-postgres / immich-redis / immich-server on archy-net; container_name set
to the underscore form (immich_postgres/_redis/_server) so the server's
DB_HOSTNAME/REDIS_HOSTNAME aliases resolve.
- generated_secrets: [immich-db-password] (idempotent — reuses the live secret on
existing nodes; postgres is already initialised with it).
- server depends on postgres+redis (install ordering); upload bind preserved.
Inert for now: not added to the UI catalog and install_immich_stack still the
default, so nothing installs these until the orchestrator wiring + on-node
ownership (data_uid) validation lands. Schema validated by the all-manifests
round-trip test. See docs/PRODUCTION-MASTER-PLAN.md §6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Buyer-side paid downloads now persist: purchases are cached on disk
(content_owned.rs) keyed by (seller onion, content_id), the gallery shows
an "Owned" badge unblurred, and items view/play in-app from the local
cache with no re-payment or reliance on a browser download (which
silently failed on the mobile companion). New RPCs content.owned-list /
content.owned-get. Validated e2e .116<-.198 (paid 100 sats via Fedimint,
166KB jpeg returns, survives restart).
fedimint-clientd manifest: restore the standard container capability set
(CHOWN/DAC_OVERRIDE/FOWNER/SETUID/SETGID) so fmcd's startup chown of an
existing-federation /data succeeds instead of dying EPERM (#7). Confirmed
the orchestrator applies these to the running container.
FIPS perf: tighten the supervisor warm-path keepalive 45s -> 25s so peer
paths stay inside the ~30-60s NAT cold window. Dials now reliably land on
FIPS instead of re-punching and falling back to Tor. Measured to the same
peer: cloud browse 18-22s -> 0.4s; full Fedimint paid download 29s -> 11s
(residual is the seller-side guardian reissue round-trip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fmcd crash-looped "Operation not permitted (os error 1)" on .116 (kernel
6.12.74): the default rootless seccomp profile blocks a syscall its Mainline-DHT
/ iroh transport needs, so the REST API never came up (:8178 → HTTP 000) and
federations couldn't be joined. Verified: with seccomp=unconfined fmcd boots and
answers /v2/* (HTTP 401 instead of dead). fmcd works on other nodes, so this is
kernel/seccomp-specific — but the relaxation is safe for an outbound-networking
daemon and harmless where not needed.
- new `security.seccomp_unconfined` manifest flag (SecurityPolicy);
- libpod backend sets `seccomp_profile_path: "unconfined"` (== --security-opt
seccomp=unconfined); quadlet backend emits `SeccompProfile=unconfined`;
- enabled in apps/fedimint-clientd/manifest.yml.
NOTE: manifests live on-disk at /opt/archipelago/apps/<id>/manifest.yml, so the
node needs the updated manifest deployed + the fmcd container recreated to apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- reissue_into_any now tries the UNION of the local registry AND fmcd's live
joined set (/v2/admin/info) before failing, so a valid Fedimint token isn't
wrongly rejected when the registry has drifted. On all-fail it returns a
friendly message: notes already redeemed into this wallet (funds safe) vs
didn't match any connected federation.
- Unified transaction history: a local Fedimint tx log (recorded on each
successful redeem) is merged with the Cashu history in wallet.ecash-history,
newest-first, each tagged kind=cashu|fedimint. Previously a Fedimint receive
appeared nowhere.
- fedimint-clientd healthcheck -> type:tcp. It was probing /health, which fmcd
doesn't serve (only /v2/*), pinning the container in (starting) forever; the
TCP probe is skipped by the Quadlet renderer (host-side lifecycle verifies),
so it reports running. Cosmetic for ecash, which worked throughout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mesh: stop broadcasting ARCHY:2 identity on the public channel (startup + every advert tick); receive path still parses inbound. No more public-channel spam.
- mesh assistant: trigger on !ai/!ask typed in 1:1 chat (was only the dead AssistQuery path + bare channel text); route the reply transport-aware via MeshService::send_message (Tor for federation peers, LoRa for radio) through a new AssistChatReply event consumed at the server layer — fixes replies never reaching federation askers.
- mesh assistant: per-contact !ai allowlist (allowed_contacts) bypassing trusted_only; config + RPC + is_sender_allowed.
- fedimint-clientd manifest: network_policy open -> bridge (invalid value made the loader skip the whole manifest, so fmcd never ran and federations never joined/listed).
- ui: AI panel — Claude model dropdown (Haiku/Sonnet/Opus presets) + allowlist contact picker.
- ui: Settings — App Updates + App Registry moved under Account.
- ui: mesh chat — overscroll-behavior: contain so chat scroll no longer bleeds to the contacts panel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
