The port map deferred to DISK manifests for any app with a build source
— which is exactly the four companion UIs (lnd-ui, bitcoin-ui,
electrs-ui, fips-ui). Their disk manifests reach nodes only via the
frontend runtime payload or a per-node repo checkout, and in the
v1.7.125 rollout both proved stale or entirely absent: one node had no
checkout at all, others restored an older payload over apps/ at every
boot. Result: session_passthrough never reached the gate, so the node's
own screens 401'd on every data call, and on nodes whose UI rebuilt
from a stale context the app held its port UNGATED.
Classification now uses a ports-only overlay that accepts build-source
manifests (install/orchestration still defers to disk — unchanged). The
signed catalog is the freshest, operator-signed source, and the gate's
address binds fail safely against a container publishing differently
(logged CANNOT PROTECT), so this can only tighten policy, never expose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chown_for_rootless_container prefers `podman unshare chown` (which maps
container uid N through the userns), but when that failed once it fell
back to `sudo chown -R <literal>` — writing e.g. host uid 999 for
container uid 999 and reporting success. Host-999 maps to nobody inside
the userns, so the app could not open its own data while everything
claimed the chown worked: botfights on framework-pt crash-looped every
10s on SQLITE_CANTOPEN over a data dir the daemon itself had just
"fixed".
The sudo fallback now translates container ids (1..99999) to
subuid_base + id - 1 (fleet base 100000; container root maps to the
service user, 1000). Already-mapped ids and uid 0 pass through.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The periodic reconcile runs ExistingOnly — merely listing a catalog
manifest must never install an app — and its only absent-container
recovery keyed on the last running-names snapshot, which ages out after
a few daemon restarts. An absent member of an installed stack then stays
absent forever: .38 ran indeedhub with no minio/postgres for days, nginx
down on 'host not found in upstream "minio"', and nothing ever put the
members back.
A live sibling container is proof the stack is installed on this node,
so an absent member is now treated as a hole to repair, not a choice to
respect: the recovery guard also fires when another member of the same
stack (app_ops::stack_member_app_ids) has a container in any state.
A stack with no containers at all is left untouched, and sibling app ids
resolve through the loaded-manifest container names (immich-postgres
runs as immich_postgres).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two daemon bugs, one debugging arc (2026-08-05, operator-reported):
1. The app gate removed the ENTIRE Cookie header before proxying. That
broke the data plane of every first-party companion UI behind the gate
(lnd-ui/bitcoin-ui/electrs-ui/fips-ui render their shell, then every
/proxy/* and /lnd-connect-info call 401s — observed as "LND UI
unreachable"), and silently logged users out of every gated app with
its own cookie login (vaultwarden, nextcloud, gitea) on each request.
The gate now strips only its own cookie pairs (session, csrf_token);
a new per-port manifest opt-in `session_passthrough: true` forwards
the node session to first-party UIs whose nginx proxies the daemon's
authenticated endpoints. Undeclared ports never get passthrough.
2. podman_client::create_container sent named volumes to the libpod API
as bind mounts with the bare volume name as source, so creating any
manifest app with a `type: volume` mount failed. On .38 the reconciler
removed indeedhub-postgres/-minio for env drift and then could never
create their replacements, leaving the stack half-missing forever.
Named volumes now ride the spec's `volumes` field ({Name, Dest,
Options}). Also: the reconcile-failure log now prints the full anyhow
chain — `%e` showed only "create_container X" and hid the real error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rebuilt image never reached a running companion. ensure_image_present
rebuilds in place under the same tag, so the quadlet body is identical,
write_if_changed reports no change, and enable_now is a no-op on a running
service — the container keeps the old layers indefinitely.
That is precisely how archi-dev-box kept serving the LND, FIPS, Electrs and
Guardian screens on 0.0.0.0 after v1.7.123 rebuilt every one of those images
to bind loopback: correct images on disk, three-day-old containers still
running. Closing those ports needed a manual 'podman rm -f' per container,
which no other node would ever get. Compare the running container's image ID
against the built one and restart when they diverge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scanning archi-dev-box from OUTSIDE found five ports serving their screens
with no login — lnd-ui 18083, bitcoin-ui 8334, fips-ui 8336, electrs-ui
50002 and the Fedimint Guardian 8175 — none of which appeared in the gate's
unprotected list. They are host-networked, so Podman publishes nothing to
pin and their manifests declared 'ports: []'; the gate builds its map from
declared ports, so it neither protected them nor reported them. An audit
that reports success while five screens are open is worse than no audit.
Their nginx now listens on 127.0.0.1 instead of 0.0.0.0, and each port is
declared 'auth: gated' so the daemon owns the outside. 'bind:' on a
host-networked app is a statement of where the container listens, not a
publish instruction — quadlet already skips PublishPort in host mode.
Guardian 8175 is declared on the fedimint app because its companion has no
manifest, and the gate keys on port, not container.
Credential paths were NOT exposed and are verified so: /lnd-connect-info,
the /proxy/lnd/ passthrough, container logs and every RPC method through
these screens all return 401 unauthenticated. What leaked was the page
shell.
Also fixes the delivery gap that would have made this unshippable: only
bitcoin-ui, lnd-ui and electrs-ui were ever rsynced to
/opt/archipelago/docker, so edits to fips-ui and fedimint-ui reached nodes
through no path at all. All five now sync; the two whose rebuilds the
daemon owns are synced without being handed to container-specs.
Every remaining undeclared port is now declared with a stated reason —
gated: botfights 9100, router 8084, pine 10380; exempt with rationale:
fedimint consensus 8173/8174, gateway 8176/9737, netbird 8086/8087 (TLS +
own auth, and enrolled devices cannot hold a session), pine TLS 10381,
lightning-stack REST 8091 (macaroon, mirrors lnd). Zero undeclared ports
remain across all 56 manifests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dev-box verification of the Tor/FIPS fixes caught a pre-existing split
brain: the orchestrator publishes containers from the signed catalog's
embedded manifests (origin-wins), but the gate classified ports from the
stale disk manifests — so it externally bound nbxplorer 32838, a port
the catalog declares auth: local and pins to loopback. Reachable behind
a login, but reachable where it deliberately was not.
- build_port_map now consults the catalog overlay first, via the same
parse/validate/image-only filter the orchestrator uses (moved to
app_catalog::catalog_manifest_overlay so the two cannot diverge again).
- GatedPort carries . The gated set still includes undeclared
Session-default ports for challenge/audit, but every action that
REDIRECTS traffic — the torrc 127.0.0.2 repoint, the FIPS relay
stand-down, the Tor-upstream bind — now keys on the declaration.
- The sweep releases held claims whose port left the gated set, so a
catalog refresh that withdraws a port (gated → local/none) takes
effect without a daemon restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Legacy pre-quadlet containers kept publishing 0.0.0.0 after the catalog
pinned their app to loopback, because host_port_bindings_drifted only
compared host PORT numbers — closing them needed a manual package.update
per app per node. The drift check now also compares the bind ADDRESS,
but only when the manifest declares one: an empty bind never fires,
since recreating a loopback-published container to wildcard on silence
is exactly the v1.7.121 Bitcoin-RPC incident. With this, every node
recreates its legacy containers to the declared state on its own after
the OTA.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two live incidents on archi-dev-box today, one bug. Both times a safety
decision read an ABSENT manifest field as if it were a value, and a
node's installed manifests always lag the binary — so "absent" is the
state of essentially every port on every node.
1. Gating any `session` port regardless of `bind` published Bitcoin's
loopback-only RPC 8332 on the LAN, Tailscale and IPv6 within seconds
of deploy.
2. The `bind`-keyed replacement looked safe because it protected
`bind: 127.0.0.1` ports — but LND's gRPC 10009 and REST 18080 carry
an EMPTY bind, so they fell through. One container recreate from
pinning them to loopback and breaking Zeus and every remote wallet.
`auth` is now `Option<PortAuth>`, separating two questions that were
conflated:
* `auth_policy()` — what to CLASSIFY the port as. Undeclared reports as
Session, i.e. shows in the audit as something that should be behind
the gate. Reporting is always safe.
* `auth_is_declared()` — whether the daemon may ACT. Only an explicit
declaration authorises changing how a port is published.
Also reverts the daemon-side publish rewriting entirely. The node proved
it wrong twice over: the recreate path that actually ran was in
package::install, not podman_client, so the pin never fired; and even
`bind: 127.0.0.1` written directly into the node's manifest was
overridden by the signed catalog. Publishes are built in several places
and all of them already honour `bind`, so the migration belongs in the
catalog as data — not in daemon-side inference that can only ever cover
one path and guess wrong on the rest.
Tests: 75/75 container, incl. the LND wallet-port shape (`host: 10009`,
empty bind, no auth) asserted to be non-actionable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Observed live on archi-dev-box: archy-fedimint-ui rebuilt 45 times in 10
minutes, every ~35s, indefinitely. bitcoin-ui, lnd-ui and electrs-ui were
all one reconcile away from the same loop.
`context_is_newer_than_image` decides to rebuild when the build context's
newest mtime is later than `podman image inspect .Created`. The rebuild
that follows is a full layer-cache hit, so podman reuses the identical
image and leaves .Created untouched — the condition that triggered the
rebuild is still true afterwards. The check cannot converge: it rebuilds
on every reconcile tick forever, burning CPU and churning the container.
It bites after any deploy that refreshes /opt/archipelago/docker/*, which
makes the contexts newer than the shipped images — so this is fleet-wide
on every OTA, not local to one node.
Fix: stamp the context mtime that was built into an image label and
compare against that instead. A label is part of the image config, so a
cache-hit build with a new value still produces a new image — the thing
being tested does change, and the comparison settles after exactly one
rebuild. Verified against real podman before writing it: two cache-hit
builds with different label values produced distinct image IDs
(6cfdbc9bcd3e vs a9b10eb9a558), each carrying its stamp; the indexed
inspect format was checked against an image with real labels, and a
missing label prints empty (handled, along with "<no value>").
Images built before this carry no label and fall back to .Created, so
behaviour is unchanged for them and each self-heals on its first
reconcile after upgrade — nodes fix themselves rather than needing the
manual `podman build --no-cache` pass this needed by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced again on this node today: with no session cookie, six app
ports answered HTTP 200 with their real UIs (18083 LND, 8334, 8175
Fedimint Guardian, 8336 FIPS Mesh, 8090, 7777), all bound 0.0.0.0 and so
served on every host address. Same bug class as the /lnd-connect-info
and /bitcoin-rpc/ leaks closed in v1.7.120, but across every app.
LAN, Tailscale, Tor and the FIPS mesh all converge on 127.0.0.1:<port>,
so this is one gate rather than four. It lives in the daemon rather than
a per-app sidecar (umbrel's app_proxy model): rootless, no extra
container per app, and it can reuse machinery that already exists.
It invents no authentication policy. verify_password, TOTP secret
decryption, verify_code with used-step replay protection, the session
store, and — importantly — the SAME LoginRateLimiter instance as the
JSON-RPC path, so an attacker cannot get a fresh budget of password
guesses by moving to an app port. Only the transport differs, an HTML
form instead of JSON-RPC, because a browser being sent to an app cannot
speak JSON-RPC.
2FA comes for free: a session still pending its TOTP step fails
validate(), so the gate rejects it without knowing what a second factor
is.
Details worth keeping:
- 401, not a redirect. A redirect to a login page is indistinguishable
from the app itself redirecting, and machine clients would follow it
and parse HTML as their API response.
- Cookie and Authorization are stripped before proxying. The app has no
use for the node session and must never be able to log or forward it.
- The challenge page names and pictures the app being opened, so the
visitor can confirm what they are authenticating to.
- device_tokens grew `apps: Option<Vec<String>>` and verify_for_app for
machine clients. None = node-wide, which every existing companion
token is; migrating them by guessing a scope would silently revoke
access nobody asked to revoke. An empty list is rejected rather than
minted, since it reads as unrestricted while authorising nothing.
The rollout is necessarily per-app and the gate is built to say so. A
container publishing 0.0.0.0:<port> claims every host address, so the
gate cannot bind that port until the app is pinned to bind: 127.0.0.1
and recreated — gate-first is impossible, and all-at-once would recreate
every container on a node simultaneously. Every port it cannot claim is
logged at warn each sweep and recorded in GateStatus::unprotected,
surfaced by security.app-gate-status. The failure mode being designed
against is a gate that binds nothing, logs at debug, and reports success
while every app stays exactly as open as before — worse than no gate,
because it stops anyone looking. Same reasoning that ruled out an
nft drop-in, whose absence is a silent no-op.
Not yet done: pinning the 39 gated ports to loopback, repointing
HiddenServicePort at the gate, and on-node verification.
Tests: 21/21 appgate, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate's cargo-fmt stage failed on my own additions — the
tests in message_types.rs and lnd/info.rs and the reconcile branch in
prod_orchestrator.rs were written programmatically and never passed
through rustfmt. Formatting only; rustfmt is semantics-preserving and
the gate re-runs the suites before building.
Caught by the gate rather than in review, which is the gate working. Also
a reminder that a piped command's exit code is the pipe's, not the
script's: the task notification reported success while the log said
CREATE_RELEASE_EXIT=1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things needed for the new UI to actually reach users.
The rendered nginx.conf served index.html with only ETag/Last-Modified and
no Cache-Control, so browsers applied heuristic caching to it. Confirmed on
archi-dev-box: after rebuilding and recreating the container, :8334 and
/app/bitcoin-ui/ both served the new markup immediately, but the app iframe
in the main UI kept showing the previous UI until a hard refresh.
docker/lnd-ui/nginx.conf has always carried this header, which is why only
bitcoin-ui showed the stale copy. Using "no-cache" (revalidate) rather than
"no-store" keeps the ETag doing its job when nothing has changed.
Validated by mounting the rendered config into a throwaway container from
the built image and running nginx -t. (An earlier attempt to test it inside
the running container was meaningless — conf.d/default.conf is a read-only
bind mount, so the copy failed and nginx -t just re-checked the original.)
The 8 container::bitcoin_ui tests still pass; their assertions cover the
placeholder, the 8332 proxy_pass and the listen directive, none of which
this touches.
BITCOIN_UI_IMAGE was still pinned to 1.7.84-alpha, so a fresh install would
pull a bitcoin-ui from many releases ago regardless of what the OTA ships —
first-boot-containers.sh tries the registry image before building from
source. Bumped to 1.7.119-alpha, matching the current release, and the
image is pushed under that tag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while VERIFYING a05956c4 on archi-dev-box rather than assuming it.
GET /lnd-connect-info is correctly 401 with no cookies over the LAN
address. But POST /bitcoin-rpc/ on :8334 still answered an
unauthenticated caller with a real block height, and still carried
`Access-Control-Allow-Origin: *`. The node looked patched. Half of it
was not.
The rendered /var/lib/archipelago/bitcoin-ui/nginx.conf was dated
2026-06-30 — the pre-fix version — even though the running binary
carries the new template. a05956c4's commit message claimed the
template "is re-rendered on every reconcile pass, so this ships
atomically with the binary". That is false in one specific state, and
this node was in it:
1. bitcoin-ui sits in the durable user-uninstalled marker.
2. reconcile returns on that marker BEFORE run_pre_start_hooks, which
is what renders the config.
3. The container keeps running regardless, because it is owned by
systemd via a Quadlet unit (archy-bitcoin-ui.service, active,
restarted 17:25 after the daemon restart) — not by this reconciler.
So a container systemd keeps alive, that the orchestrator has stopped
reconciling, never receives a config fix shipped inside the binary. An
OTA carrying a05956c4 would have silently failed to close this on every
node in that state, while the LND half closed correctly — the most
misleading possible outcome. archy-electrs-ui is in the same state on
this node, so it is not a one-app accident.
A container that is actually running is a live attack surface whatever a
marker says about it. Its security-relevant config is now reconciled
even behind the marker, and it is restarted so nginx actually loads it.
Deliberately narrow:
- Nothing is created, pulled, built, started or resurrected. The "must
stay removed" contract only ever gets weaker if a container is ALREADY
running, which by definition means it was never removed.
- A hook error is swallowed, not propagated: an app the user uninstalled
must not be able to fail the reconcile pass for everything after it.
- The pre-existing marker test still passes unchanged, which is what
proves the removal contract survived.
Verified: 11/11 reconcile tests and 9/9 bitcoin_ui tests pass, including
a new regression test that pins the whole chain — stale conf in, gate
present out, container restarted, nothing created.
No node has been touched. The live exposure on archi-dev-box stands
until this is deployed and the operator restarts archy-bitcoin-ui.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes F-10a. Nothing here fixes a present defect: on the pinned rand
0.8.5, rand::random() and thread_rng() both resolve to a ChaCha12 CSPRNG
seeded from getrandom(2). What they lack is a STATED backend — it is
fixed by dependency and build configuration rather than by the calling
code, with no compile error if that changes. That is the structural
shape behind the 2026-07-30 COLDCARD entropy defect, and here the blast
radius includes Cashu blinded-key-exchange values, X3DH prekey material,
session bearer tokens and a ChaCha20-Poly1305 nonce.
Layer (a) — every production key, nonce and token draw now names
rand::rngs::OsRng at its own call site. The mnemonic seam is bound to
entropy::KeyGenRng, a SEALED allowlist whose supertrait lives in a
private module, so the set of RNGs that can drive the master key
hierarchy is exactly what one file says it is. This retires the false
promise at seed.rs:656: rand::CryptoRng is a marker with no
compiler-checked content, and the crate now contains zero impls of it.
Layer (d) — key material and AEAD nonces of >=12 bytes run a
degenerate-entropy predicate that refuses all-zero, all-identical and
wrapping +/-1 counter draws. Nothing heuristic: no entropy estimator, no
chi-squared. Each of the three shapes has a false-positive probability
computable in closed form (3 * 2^-88 at 12 bytes, 3 * 2^-248 at 32), and
a predicate whose false-positive rate cannot be computed cannot be
argued safe on a key-generation path. There is deliberately no retry — a
retry would paper over the broken RNG this exists to surface.
Layer (e) — the kernel-CSPRNG readiness verdict at master-seed
generation is now durable (backlog R-09). It was previously computed,
logged and thrown away, so a node could never answer after the fact
whether its keys were born from a seeded pool. The record holds a schema
version, timestamp, verdict and event name — no entropy, no key bytes.
Formats and wire shapes are proven unchanged rather than asserted:
storage_crypto and the credential store each open a HARDCODED
pre-migration ciphertext vector (a same-process round trip would pass
even if the envelope had changed), the vector was produced by an
independent RFC 8439 implementation so it pins the documented
nonce||ciphertext format rather than this implementation's output, and
the x3dh prekey bundle and bdhke values keep their field set and order.
totp.rs migrates its SOURCE only: the % charset.len() reduction and the
32-char charset are untouched. The bias there is presently zero (32
divides 256) and fixing the latent bias is R-12, which stays deferred.
Verified: cargo build clean; cargo test -p archipelago 1068 passed,
2 failed. Both failures are container::boot_reconciler timing tests
(second_pass_fires_after_interval, shutdown_terminates_loop) in a file
this change does not touch — pre-existing, not caused here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CRITICAL. Two app-UI ports handed unauthenticated callers full control of
the node's money. Both verified live on archi-dev-box 2026-08-02 over the
fips0 mesh ULA with no cookies.
GET /lnd-connect-info returned 200 with the LND ADMIN MACAROON, the TLS
cert, the gRPC/REST ports and the node's onion address — a complete remote
wallet-drain package, and the onion means an attacker keeps that ability
after losing network access. POST /bitcoin-rpc/ reached Bitcoin Core RPC
with credentials the proxy injected on the caller's behalf, with a wallet
loaded, so wallet methods were reachable too.
Both were reachable because ports 18083 (lnd-ui) and 8334 (bitcoin-ui)
bind 0.0.0.0 AND sit on the fips0 mesh allowlist in fips/app_ports.rs. Any
mesh peer, LAN host or Tailscale peer could take either path.
The root cause is one mistaken idea in two places: that a check performed
by a reverse proxy is an auth check. It is not — it only holds for traffic
that arrived through that proxy. /lnd-connect-info's comment said "nginx
validates session cookie (presence check), backend is bound to 127.0.0.1
so only nginx can reach it". Both clauses were false in production: the
lnd-ui container runs its OWN nginx on :18083 that proxies straight to the
backend forwarding whatever cookies arrived, including none, and that
second front door never performed the check the premise named.
So authorisation moves to the resource:
- /lnd-connect-info now requires a session, like /proxy/lnd/ beside it.
The 401 carries CORS headers so the wallet UI shows a readable error
rather than an opaque CORS failure.
- New GET /auth/session-check returns 204/401 and nothing else, giving
container nginx an auth_request gate it can actually use.
- bitcoin-ui's /bitcoin-rpc/ is gated by that auth_request. Its
`Access-Control-Allow-Origin *` is also gone: on a proxy that injects
credentials, it let any page a user visited drive the node's RPC.
Preflight is answered before the gate, since OPTIONS carries no cookies.
The nginx template is include_str!'d and re-rendered on every reconcile
pass, so this ships atomically with the binary.
Operators must treat the LND admin macaroon and the Bitcoin RPC password
on every affected node as compromised and rotate them AFTER this is
deployed — rotating first just re-leaks through the same hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The checkpoint on archi-dev-box proved rotation alone doesn't close FED-07:
the credential file went unique while the RUNNING container kept serving the
compromised one, because the Quadlet path rewrites a unit without restarting
it and fedimint-gateway is classified restart-sensitive, so drift was detected
and deliberately ignored on every tick.
Rotation now records the app id, and the drift check consumes that flag to
recreate even a restart-sensitive app, with a WARN naming the reason. This
mirrors the published-port carve-out a few lines above, which already makes
the same trade for the same reason: a container that is already broken (there)
or already compromised (here) is not protected by leaving it running.
Restart-sensitivity protects working services. A gateway answering to a
credential published in this repository is not working, it is compromised, and
gateway admin can drain Lightning liquidity — indefinite exposure loses to a
few seconds of restart. Rotating-but-only-alerting was rejected: the
monitoring system fires on metric thresholds only, so it would have needed new
event-alert plumbing to deliver something strictly weaker.
Re-verified on the same node, same scenario: rotation at 06:39:23, recreate at
06:39:27, PID 3923125 -> 148426, running credential now matches the file,
container healthy with the same name and ports, gatewayd.db intact at 18 files
with IDENTITY present, 32 containers untouched, no repeat rotation.
3 new tests. Also lands the missing 01-19 and 01-20 SUMMARYs: both had code
committed 2026-07-31 but no summary and no roadmap tick, so they read as
unstarted. Phase 1 is 11/20. FED-09 carries 15h of Tor uptime and 0
permission-fixes across 542 doctor runs on archi-dev-box.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01-11 stopped new installs from ever taking a shipped credential, but did
nothing for the nodes that already did — those gateways still answer to a
password published in this repository.
rotate_compromised_gateway_credential() detects an EXACT match against the
denylist and replaces the pair; absent, unique, or merely unrecognised values
are left alone and return false. That distinction is the point: an operator
who deliberately set their own credential also has an "unrecognised" one, and
rotating it would be the same class of harm as leaving the default in place.
It hangs off resolve_dynamic_env beside ensure_generated_secrets, gated on the
gateway's app id, so an affected node heals on its next reconcile tick. There
is deliberately no teardown here: the new hash changes the resolved secret env,
which changes secret_env_hash, which the drift check reads as a container-label
mismatch — so the platform's own recreate path rebuilds the gateway around its
unchanged data directory, ports, volumes and name.
Rotation is self-terminating (the value written is not on the denylist, so the
next tick is a no-op) and errors propagate rather than being swallowed, because
the atomic write leaves the previous credential intact on failure.
Bcrypt generation was factored out of ensure_one into write_bcrypt_pair, which
both generation and rotation call — 01-11's SUMMARY claimed such a helper
existed but the arm was still inline, and rotation cannot reuse
ensure_gateway_credential because its idempotent fast path returns early
exactly when the file is present, which is the case rotation acts on.
Also fixes cargo fmt drift left by 42652547 in install.rs.
Verified: 6 new tests, secrets suite 16/16; full suite 1008 passed with one
known wall-clock flake (green 4/4 in isolation). NOT verified on a node —
Task 2's blocking checkpoint has not been run, so FED-07 stays open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six code paths configured the Lightning gateway with a bcrypt hash committed
to this repository — and one deploy path with a plaintext password literal —
whenever the per-install secret was missing. Anyone holding a copy of the repo
held the admin credential for every gateway that ever took a fallback.
container::secrets now owns the credential end to end: ensure_gateway_credential
(idempotent, delegates to ensure_one's bcrypt arm) and gateway_bcrypt_hash,
which returns Err when the secret is missing/empty and when the stored value is
on the KNOWN_DEFAULT_GATEWAY_HASHES denylist — so this codebase cannot hand
back the compromised value even to a node already carrying it.
get_app_config was widened to Result so a credential-less install cannot reach
podman run at all; configure_fedimint_lnd takes the resolved hash instead of
re-reading with its own fallback. The four shell paths stop generating
credentials entirely (dropping the htpasswd host dependency) and skip container
creation with a printed reason rather than substituting anything.
Naming converges on the manifest's fedimint-gateway-hash/.pw, with legacy
fedimint-gateway-password values copied forward rather than regenerated so no
node loses a working unique credential. Plan 01-16 owns rotation of installs
already carrying the default.
Verified: cargo build clean; cargo test -p archipelago 999 passed (2
boot_reconciler timing tests failed under concurrent load, green in isolation,
untouched by this diff); bash -n clean on all five scripts; the compromised
literal now appears exactly once in the tree, as the denylist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A published port no longer implies a web UI. The package scanner used to
synthesize interfaces.main.ui="true" for any container with a port or
onion address, so headless backends — including self-deployed compose
stacks like podsteadr — showed up as launchable apps. New ui_detection
module decides instead: a manifest interfaces declaration (catalog
overlay first, disk second) is definitive; undeclared apps get a short
HTTP probe of the launch port (HTML page, redirect, or browser auth
wall = UI; JSON APIs, raw TCP, dead ports = service), with cached
verdicts and probes gated on running containers. Frontend canLaunch
now refuses curated services outright and only treats a bare runtime
address as launchable for curated known apps.
Works identically for manifest apps and containers deployed by hand
outside the orchestrator. ui_detection tests 6/6, frontend suite
696/696.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the .198 load spiral (2026-07-28): needs_repair() called
ensure_image_present() every 30s tick to render the expected unit, so
under IO pressure the image-existence check timed out, read as "image
missing", and a 900s podman build ran inside the PROBE while the
companion was up — each build pegging the disk that made probes fail.
- needs_repair() is now build/pull-free: unit file present → service
is-active (10s cap; a hung systemctl reads as "assume active", never
as dead) → unit matches one of the three image refs install_one could
have written → context-newer-than-image staleness only when the unit
uses the auto-built :latest.
- Per-companion 10-min repair cooldown after a failed install_one, so a
failing build retries at most every REPAIR_COOLDOWN instead of every
reconcile tick.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rootless podman's wildcard publish claims [::] but BLACK-HOLES inbound
v6 (accepts, forwards nothing — vaultwarden 'empty response' from the
phone, 2026-07-26), while most apps got no v6 listener at all. Two
halves: quadlets now publish unbound ports on 0.0.0.0 explicitly
(frees the v6 side; the one-time drift/recreate at upgrade is the
deploy vehicle), and the daemon runs a self-selecting V6ONLY relay on
every catalog launch port forwarding raw TCP to the v4 loopback
listener. Rescans every 60s so new installs bridge without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway crash-looped dialing bitcoind at host.archipelago:8332 (the
host gateway IP, where bitcoind does not listen). Root-cause fix, three
parts:
- fedimint-gateway manifest: --bitcoind-url now comes from
$FM_BITCOIND_URL, filled by a {{BITCOIN_HOST}} derived-env — works on
Knots, Core, or any future Bitcoin distro.
- fedimint manifest: same derived-env replaces the hardcoded
FM_BITCOIND_URL=http://bitcoin-knots:8332 environment entry.
- orchestrator: removed the hardcoded FM_BITCOIND_URL=bitcoin-knots
override that clobbered the derived-env, and bitcoin_host() now
accepts any running bitcoin*/bitcoin container (excluding -ui and
archy-* sidecars), preferring the known names in order.
Catalog regenerated + signed (nodes install from the signed catalog, so
the manifest fixes only reach them via this regen).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fleet log sweep (2026-07-22): (1) recovery paths podman-start'ed
containers that were already running, producing hundreds of benign but
scary conmon 'Failed to create container' + cgroup Permission-denied
pairs per node per day — both paths now check state first; (2) the 30s
companion-reconcile loop retried registry pulls/builds forever against
unreachable registries (~174×/image/day on an offline node) — failing
rounds now back off exponentially to a 1h cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The creds-off-argv script used printf "rpcuser=%s" in the manifest's
custom_args; quadlet copies Exec= into the generated service's ExecStart,
where systemd expands %s (user's shell) at load time — bitcoind's rpc.conf
came out as rpcuser=/bin/bash and every RPC consumer got 401s on quadlet
nodes (framework-pt: bitcoin-status, HA block-height sensor, LND chain RPC).
Two-layer fix: quadlet.rs now escapes % -> %% in Exec/Entrypoint/HealthCmd/
Environment emission (with regression test), and the bitcoin manifests write
rpc.conf with plain echo so the catalog also heals nodes still running older
binaries. Release catalog regenerated with the fixed scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Register "pine" as a manifest-driven stack (like netbird) across every
lifecycle site so install/stop/start/restart/uninstall/crash-recovery treat
the launcher + two Wyoming engines as one app:
- stacks.rs: pine_stack_app_ids() + install_pine_stack() (orchestrator-first,
adopt existing, else bail — no hardcoded fallback)
- install.rs: dispatch package_id == "pine" to the stack installer
- app_ops.rs: canonical stack-member table + owning_package STACKS
- dependencies.rs: startup_order + needs_archy_net
- config.rs: all_container_names
- crash_recovery.rs: auto-start members + a StackRecoverySpec (anchor
pine-whisper) so a genuinely-installed stack self-heals but orphan debris
does not crash-loop
- docker_packages.rs: exclude the engines from the apps list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sweep podman-execs into every running container; doing that on each 30s
reconcile tick was a permanent conmon 'Failed to create container' storm on
hosts where exec from the backend's cgroup context fails (Debian 13 first
boot). Ownership drift is an install/OTA-time event — sweep on first pass
after a container appears, then at most hourly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a fresh ISO node, NetBird's own WireGuard tunnel (10.44.0.1) sorted
ahead of the real NIC in hostname -I, so the NetBird launch URL (and the
{{HOST_IP}} baked into its cert/config) pointed at the tunnel instead of
the LAN address. The main routing table's default route names the
physical uplink even while a VPN is active (NetBird/Tailscale steer
traffic via policy-routing rules, not the main table), so read src/dev
from 'ip -4 route show default' first, then fall back to a connected UDP
socket's source address, then to the old hostname -I scan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
936b4cca made ANY loaded manifest with an absent container self-heal via
install_fresh — but the signed catalog and the ISO ship manifests for
EVERY app, installed or not, so 'manifest loaded' is not installation
evidence. On a fresh node this mass-installed the catalog: the framework
node grew portainer, vaultwarden, searxng, strfry, mempool, immich and
fedimint uninvited within an hour (fleet nodes showed the same loop as
reconcile-failed noise for unpullable images).
ExistingOnly (boot) reconcile now creates containers from nothing only
for required-baseline apps (filebrowser, fedimint-clientd); every other
absent app is Left('absent') and recreated ONLY via desired-state
recovery when it was running at the last snapshot — which still covers
the vanished-container regressions (indeedhub/immich) that motivated
936b4cca. Reconcile suite 16/16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aezeed words are now captured at BOTH wallet-init paths and stored
encrypted (Argon2 + ChaCha20-Poly1305, per-node wallet secret) at
identity/lnd_aezeed.enc; ack marker cleared when a wallet is recreated
- lnd.init-wallet-from-seed was posting seed_entropy to /v1/initwallet,
which is a GenSeed field — the wallet was never actually derived from
the master seed; now GenSeed(entropy) → InitWallet(words)
- new RPCs: lnd.seed-backup-status / lnd.seed-reveal (password + 2FA
gated, same as seed.reveal via shared verify_reveal_auth) /
lnd.seed-backup-ack
- LND app detail page: Lightning-seed card with first-launch backup
prompt, tap-to-reveal modal, 'I've backed it up' acknowledgment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Materialise /var/lib/archipelago/secrets/btcpay-lnd-connection
(type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<sha256>)
in ensure_app_secrets before btcpay-server env resolution.
The macaroon travels inline as hex because LND's datadir is owned by its
container subuid (100999) — bind-mounting it into btcpay's userns EACCESes.
Reads the macaroon via the existing read_file_as_root sudo fallback; pins the
TLS cert by DER SHA256 thumbprint and rewrites on cert rotation. No-op while
LND is absent (btcpay's secret_env entry is optional), and generation errors
log-and-continue so btcpay never fails to start over this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
btcpay-server started and reported healthy while its declared
dependencies (archy-btcpay-db, archy-nbxplorer) failed every boot
reconcile (live-testing report 2026-07-10, Fix 3). After each
reconcile pass, flag every app the pass left up whose manifest
dependency landed in the failure list — a loud DEGRADED journal error
instead of silence. True start-gating on dependency health is a
larger design change (boot-ordering/deadlock risk) and stays open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strfry/vaultwarden 8082 collision shipped in a signed catalog
because nothing checks cross-app host ports. Fix the remaining latent
pairs from the cfb8d953 audit — did-wallet 8083→8088 (filebrowser
keeps 8083), morphos-server 8086→8089 (netbird-server keeps 8086),
lightning-stack 8087→8091/9737→9738 (netbird and fedimint-gateway
keep theirs) — and fix their healthchecks, which probed the HOST port
inside the container (same class as strfry 6fe62355; now 127.0.0.1 +
container port). load_manifests now flags collisions across the
merged disk+catalog set (bitcoin variants and the mempool umbrella
pair are exempt as mutually exclusive), and a repo-level test parses
apps/*/manifest.yml so a future latent pair fails CI instead of
shipping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
24fd97ed cascades on package.start/restart, but a reconcile pass that
recreates or starts a bitcoin backend (desired-state recovery, repair
recreate, boot InstallMissing) also moves the RPC address behind a
running lnd's back — §C 'restart lnd after ANY bitcoin recreate'.
After the pass, dependents from the shared address_caching_dependents
table (moved to app_ops as the single source of truth) that sat
untouched (NoOp) and aren't user-stopped are restarted under their op
lock; a dependent the pass itself (re)created already resolved the
fresh address and is left alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wait_for_manifest_host_ports polls up to 420s while the caller holds
the per-app lock, so a user's stop request queued behind it and the
scanner painted the app Restarting for the whole wait (the dd3afbba
transitional-state stick; gate take-2 iter 5 vaultwarden). The wait
now selects against a user-stop-marker watcher (2s poll on
user-stopped.json, by app id or container name) and aborts with a
recognizable error, and the host-port repair path skips its restart
when a stop was requested while it settled — restarting there would
fight the queued stop worker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generated HealthCmd reported healthy (exit 0) whenever the image
shipped neither wget nor curl — btcpay-server sat 'healthy' in podman
ps while completely non-functional (live-testing report 2026-07-10).
Try a bash /dev/tcp connect to the health URL's host:port before
giving up; only images with none of wget/curl/bash still skip Podman
health (dropping exit 0 entirely would restart-loop those). Podman's
health timeout bounds the connect attempt.
Note: the rendered unit text changes, so every http-healthcheck app
gets a one-time unit rewrite + restart on first reconcile after this
binary lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconcile recreates route through install_fresh, which pulled
unconditionally: with git.tx1138.com dead, every boot reconcile of
archy-btcpay-db/archy-nbxplorer hard-failed on 'pulling ...' even
though both image:tags sat in local storage the whole time
(live-testing report 2026-07-10). Check image_exists first and only
pull when the ref is truly absent — the same semantics as
ensure_resolved_source_available and the generated quadlets'
Pull=never.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live verification on .228 showed catalog_changed:true on every
check-updates call: AppCatalog.apps is a HashMap, so the re-serialized
cache bytes flap with key order and the changed-detection never reports
"unchanged" — which would have reloaded the manifest overlay every
hourly poll. Cache the raw fetched body instead and compare against
that; as a bonus the cache now holds the exact signed preimage, so a
present release-root signature stays verifiable from disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.install routed to the manifest-driven orchestrator only for a
hardcoded per-app allowlist; every other app fell through to the legacy
flow, which ignores manifests entirely and creates a bare container with
no ports or volumes (strfry crash-looped this way on .228, 2026-07-09).
New ContainerOrchestrator::knows_app(app_id) (default false; prod checks
its loaded-manifest map, disk + signed-catalog overlay). The install gate
is now allowlist OR knows_app — no per-app Rust for manifest-driven apps,
matching the packaging invariant. Apps without a manifest keep the legacy
flow including the container-exists adopt block; the unknown-app_id →
legacy fallback inside the orchestrator branch is unchanged.
Tests: knows_app_reflects_loaded_manifests + install suite 37/37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Formal gate take-2 went 4/5: iteration 5's vaultwarden stop-wait saw
'restarting' for the full 120s window. A reconcile pass had queued every
app into pending_boot_starts moments before the stop marker landed, and
the scanner's overlay painted the deliberately-stopped app Restarting
until the slow sequential pass reached it (legacy container, so no unit
state to contradict it).
Two filters: the scanner overlay skips user-stopped ids (the marker is
written before the stop runs, so it's reliably present), and the
reconciler no longer queues apps whose lifecycle op is in flight into the
pending overlay (it skips them via the same probe anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
