Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate failed cargo-fmt. The domain that replaced the IP-based
registry is longer, pushing several test assertions past the width limit, so
rustfmt wanted to re-wrap them. Pure line re-wrapping — no semantic change.
Caught by the pre-flight gate rather than after tagging, which is what it is
for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app store offered "update to 2.3.9" on a node already running 2.4.2 — the
release that fixes an actively exploited 2FA bypass. Taking it would have
rolled the node back onto the vulnerable version.
Root cause: available_update_for_images compared tags for inequality only.
Same repo + different tag meant "update available", with no ordering. Every
version claim upstream of it can go stale — the signed catalog, a legacy
catalog entry, the image-versions.sh baseline pin — and any one of them
lagging turned into a backwards Update button.
Guard added: when both tags parse as dotted-numeric versions, a lower pinned
version is never offered. Tags that cannot be ordered (RELEASE.2024-11-07…,
14-vectorchord0.4.3) keep the previous behaviour rather than silently losing
updates. This makes stale data fail safe, which matters more than any single
pin being correct.
Four sources still named 2.3.9, three of them able to act on it:
- releases/app-catalog.json — a LEGACY `btcpay` entry, distinct from
`btcpay-server`, carrying a concrete 2.3.9 image. catalog_primary_image
treats that as authoritative, so this is what drove the button. Fixed, but
held back from this commit: it needs re-signing.
- scripts/image-versions.sh — the baseline pin used when the catalog does not
cover an app.
- stacks.rs — the legacy BTCPay installer, twice. The fallback install path
would have deployed 2.3.9 outright.
- neode-ui curatedApps/marketplaceData and public/catalog.json — the store's
displayed version, hardcoded rather than read from the catalog, which is why
it still showed 2.3.9 after the update landed.
Audited every other installer for the same shape. The remaining literals are
the immich stack, which currently agrees with its manifests; hits in
set_config.rs and app_catalog.rs are test fixtures. To keep it that way,
scripts/check-installer-image-pins.py asserts that any installer literal
naming the same repository as an app manifest carries the same tag, and runs
blocking in CI. Verified it catches a simulated revert to 2.3.9.
Tests: 13/13 in image_versions including the exact BTCPay case, a genuine
upgrade still offered, equal versions silent, prerelease suffixes ordered on
their numbers, and opaque tags unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes 4 test failures introduced by 8e814ca0, which I pushed after running
only the container-crate tests while the full suite was still compiling. Both
failures were real defects, not stale assertions.
1. Catalog-driven installs would have failed fleet-wide.
8e814ca0 dropped the old registry address from TRUSTED_REGISTRIES, but the
signed catalog still advertises image refs on it — deliberately, since
rewriting a signed artifact invalidates its signature. Nodes resolve apps
through the catalog, so every install would have been refused with "not
from a trusted registry". Reinstated as LEGACY_REGISTRY_HOST, documented
as transitional and removable only once the catalog is re-signed.
2. The update fallback lost the property it exists for.
update.rs keeps two mirrors on purpose: the domain as primary, and the
old IP over plain HTTP as a fallback, because a node whose DNS or clock is
wrong (both break TLS) must still be able to update itself — the signature,
not the transport, is what makes either source safe. The bulk rewrite
pointed both constants at the domain, leaving the escape hatch dependent on
exactly what it exists to survive. Restored to its original value.
Separately, validate-app-manifest.sh is ported from ruby to python3+PyYAML.
It shelled out to ruby with stderr discarded, so on any machine without ruby
a missing interpreter was reported as "Valid YAML with top-level app block:
FAIL" and every manifest came back REJECTED. This is the first tool an app
developer runs, and it sent them to fix YAML that was never broken. Ruby was
also the odd dependency out — the repo already ships three python scripts.
It now checks for python3 and PyYAML up front and names what is missing, then
parses with PyYAML. Missing keys resolve to an absent-value object that
indexes to itself and prints empty, so call sites lost their per-hop guards:
(((app["container"] || {})["build"] || {})["context"])
becomes app["container"]["build"]["context"]. Booleans still print as
true/false rather than Python's True/False — call sites compare == "true",
so Python's capitalisation would have silently inverted the readonly_root
and no_new_privileges security checks.
Verified: full rust suite 1148/1148, 0 failed. All 56 app manifests validate
(0 rejected, 0 errored) where previously every one was rejected. No signed
artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the registry host across 86 files: 309 references, covering all 40
app manifests, the orchestrator and container crates, the release and catalog
scripts, both demo-images workflows, the ISO builder, demo-deploy, and the
frontend marketplace data.
Verified the domain actually serves the registry before rewriting anything,
rather than assuming the web host implies the registry:
- TLS verifies clean, HTTP/2 on the web root
- an anonymous token grants a manifest fetch (HTTP 200) with no credentials
- skopeo inspect --no-creds resolves an image and lists its tags
That last check is the one that matters: an outside developer with no account
can now pull, which was the functional blocker for publishing at all.
Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop
crossing the network in the clear.
Deliberately NOT rewritten:
- The public FIPS anchor on port 8444. It is a functional network endpoint
every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded
seeds than to leaked infrastructure. The domain does resolve to the same
host, so it could become a hostname, but that adds a DNS dependency to the
path used precisely when things are broken. Worth a deliberate decision,
not a side effect of this change.
- The companion APK on port 2100. The domain returns 404 for that path, so
rewriting it would swap a working URL for a broken one. The Releases page
does serve (200), which is where the plan already wants those binaries.
- releases/app-catalog.json, releases/manifest.json and release-manifest.json.
These carry `signature` and `signed_by`; editing their contents invalidates
the signature and the fleet refuses artifacts that fail verification. They
were rewritten in a first pass and reverted — they must be regenerated and
re-signed through the signing ceremony instead, which needs the mnemonic.
So the catalog still advertises the old host until that ceremony runs. Nodes
resolve images through the signed catalog, not the on-disk manifests, so this
commit alone does not change what a node pulls.
Verified: archipelago-container 75/75; every manifest still parses with a
top-level app block; no signed artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The port map deferred to DISK manifests for any app with a build source
— which is exactly the four companion UIs (lnd-ui, bitcoin-ui,
electrs-ui, fips-ui). Their disk manifests reach nodes only via the
frontend runtime payload or a per-node repo checkout, and in the
v1.7.125 rollout both proved stale or entirely absent: one node had no
checkout at all, others restored an older payload over apps/ at every
boot. Result: session_passthrough never reached the gate, so the node's
own screens 401'd on every data call, and on nodes whose UI rebuilt
from a stale context the app held its port UNGATED.
Classification now uses a ports-only overlay that accepts build-source
manifests (install/orchestration still defers to disk — unchanged). The
signed catalog is the freshest, operator-signed source, and the gate's
address binds fail safely against a container publishing differently
(logged CANNOT PROTECT), so this can only tighten policy, never expose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chown_for_rootless_container prefers `podman unshare chown` (which maps
container uid N through the userns), but when that failed once it fell
back to `sudo chown -R <literal>` — writing e.g. host uid 999 for
container uid 999 and reporting success. Host-999 maps to nobody inside
the userns, so the app could not open its own data while everything
claimed the chown worked: botfights on framework-pt crash-looped every
10s on SQLITE_CANTOPEN over a data dir the daemon itself had just
"fixed".
The sudo fallback now translates container ids (1..99999) to
subuid_base + id - 1 (fleet base 100000; container root maps to the
service user, 1000). Already-mapped ids and uid 0 pass through.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The periodic reconcile runs ExistingOnly — merely listing a catalog
manifest must never install an app — and its only absent-container
recovery keyed on the last running-names snapshot, which ages out after
a few daemon restarts. An absent member of an installed stack then stays
absent forever: .38 ran indeedhub with no minio/postgres for days, nginx
down on 'host not found in upstream "minio"', and nothing ever put the
members back.
A live sibling container is proof the stack is installed on this node,
so an absent member is now treated as a hole to repair, not a choice to
respect: the recovery guard also fires when another member of the same
stack (app_ops::stack_member_app_ids) has a container in any state.
A stack with no containers at all is left untouched, and sibling app ids
resolve through the loaded-manifest container names (immich-postgres
runs as immich_postgres).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two daemon bugs, one debugging arc (2026-08-05, operator-reported):
1. The app gate removed the ENTIRE Cookie header before proxying. That
broke the data plane of every first-party companion UI behind the gate
(lnd-ui/bitcoin-ui/electrs-ui/fips-ui render their shell, then every
/proxy/* and /lnd-connect-info call 401s — observed as "LND UI
unreachable"), and silently logged users out of every gated app with
its own cookie login (vaultwarden, nextcloud, gitea) on each request.
The gate now strips only its own cookie pairs (session, csrf_token);
a new per-port manifest opt-in `session_passthrough: true` forwards
the node session to first-party UIs whose nginx proxies the daemon's
authenticated endpoints. Undeclared ports never get passthrough.
2. podman_client::create_container sent named volumes to the libpod API
as bind mounts with the bare volume name as source, so creating any
manifest app with a `type: volume` mount failed. On .38 the reconciler
removed indeedhub-postgres/-minio for env drift and then could never
create their replacements, leaving the stack half-missing forever.
Named volumes now ride the spec's `volumes` field ({Name, Dest,
Options}). Also: the reconcile-failure log now prints the full anyhow
chain — `%e` showed only "create_container X" and hid the real error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rebuilt image never reached a running companion. ensure_image_present
rebuilds in place under the same tag, so the quadlet body is identical,
write_if_changed reports no change, and enable_now is a no-op on a running
service — the container keeps the old layers indefinitely.
That is precisely how archi-dev-box kept serving the LND, FIPS, Electrs and
Guardian screens on 0.0.0.0 after v1.7.123 rebuilt every one of those images
to bind loopback: correct images on disk, three-day-old containers still
running. Closing those ports needed a manual 'podman rm -f' per container,
which no other node would ever get. Compare the running container's image ID
against the built one and restart when they diverge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scanning archi-dev-box from OUTSIDE found five ports serving their screens
with no login — lnd-ui 18083, bitcoin-ui 8334, fips-ui 8336, electrs-ui
50002 and the Fedimint Guardian 8175 — none of which appeared in the gate's
unprotected list. They are host-networked, so Podman publishes nothing to
pin and their manifests declared 'ports: []'; the gate builds its map from
declared ports, so it neither protected them nor reported them. An audit
that reports success while five screens are open is worse than no audit.
Their nginx now listens on 127.0.0.1 instead of 0.0.0.0, and each port is
declared 'auth: gated' so the daemon owns the outside. 'bind:' on a
host-networked app is a statement of where the container listens, not a
publish instruction — quadlet already skips PublishPort in host mode.
Guardian 8175 is declared on the fedimint app because its companion has no
manifest, and the gate keys on port, not container.
Credential paths were NOT exposed and are verified so: /lnd-connect-info,
the /proxy/lnd/ passthrough, container logs and every RPC method through
these screens all return 401 unauthenticated. What leaked was the page
shell.
Also fixes the delivery gap that would have made this unshippable: only
bitcoin-ui, lnd-ui and electrs-ui were ever rsynced to
/opt/archipelago/docker, so edits to fips-ui and fedimint-ui reached nodes
through no path at all. All five now sync; the two whose rebuilds the
daemon owns are synced without being handed to container-specs.
Every remaining undeclared port is now declared with a stated reason —
gated: botfights 9100, router 8084, pine 10380; exempt with rationale:
fedimint consensus 8173/8174, gateway 8176/9737, netbird 8086/8087 (TLS +
own auth, and enrolled devices cannot hold a session), pine TLS 10381,
lightning-stack REST 8091 (macaroon, mirrors lnd). Zero undeclared ports
remain across all 56 manifests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dev-box verification of the Tor/FIPS fixes caught a pre-existing split
brain: the orchestrator publishes containers from the signed catalog's
embedded manifests (origin-wins), but the gate classified ports from the
stale disk manifests — so it externally bound nbxplorer 32838, a port
the catalog declares auth: local and pins to loopback. Reachable behind
a login, but reachable where it deliberately was not.
- build_port_map now consults the catalog overlay first, via the same
parse/validate/image-only filter the orchestrator uses (moved to
app_catalog::catalog_manifest_overlay so the two cannot diverge again).
- GatedPort carries . The gated set still includes undeclared
Session-default ports for challenge/audit, but every action that
REDIRECTS traffic — the torrc 127.0.0.2 repoint, the FIPS relay
stand-down, the Tor-upstream bind — now keys on the declaration.
- The sweep releases held claims whose port left the gated set, so a
catalog refresh that withdraws a port (gated → local/none) takes
effect without a daemon restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Legacy pre-quadlet containers kept publishing 0.0.0.0 after the catalog
pinned their app to loopback, because host_port_bindings_drifted only
compared host PORT numbers — closing them needed a manual package.update
per app per node. The drift check now also compares the bind ADDRESS,
but only when the manifest declares one: an empty bind never fires,
since recreating a loopback-published container to wildcard on silence
is exactly the v1.7.121 Bitcoin-RPC incident. With this, every node
recreates its legacy containers to the declared state on its own after
the OTA.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two live incidents on archi-dev-box today, one bug. Both times a safety
decision read an ABSENT manifest field as if it were a value, and a
node's installed manifests always lag the binary — so "absent" is the
state of essentially every port on every node.
1. Gating any `session` port regardless of `bind` published Bitcoin's
loopback-only RPC 8332 on the LAN, Tailscale and IPv6 within seconds
of deploy.
2. The `bind`-keyed replacement looked safe because it protected
`bind: 127.0.0.1` ports — but LND's gRPC 10009 and REST 18080 carry
an EMPTY bind, so they fell through. One container recreate from
pinning them to loopback and breaking Zeus and every remote wallet.
`auth` is now `Option<PortAuth>`, separating two questions that were
conflated:
* `auth_policy()` — what to CLASSIFY the port as. Undeclared reports as
Session, i.e. shows in the audit as something that should be behind
the gate. Reporting is always safe.
* `auth_is_declared()` — whether the daemon may ACT. Only an explicit
declaration authorises changing how a port is published.
Also reverts the daemon-side publish rewriting entirely. The node proved
it wrong twice over: the recreate path that actually ran was in
package::install, not podman_client, so the pin never fired; and even
`bind: 127.0.0.1` written directly into the node's manifest was
overridden by the signed catalog. Publishes are built in several places
and all of them already honour `bind`, so the migration belongs in the
catalog as data — not in daemon-side inference that can only ever cover
one path and guess wrong on the rest.
Tests: 75/75 container, incl. the LND wallet-port shape (`host: 10009`,
empty bind, no auth) asserted to be non-actionable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Observed live on archi-dev-box: archy-fedimint-ui rebuilt 45 times in 10
minutes, every ~35s, indefinitely. bitcoin-ui, lnd-ui and electrs-ui were
all one reconcile away from the same loop.
`context_is_newer_than_image` decides to rebuild when the build context's
newest mtime is later than `podman image inspect .Created`. The rebuild
that follows is a full layer-cache hit, so podman reuses the identical
image and leaves .Created untouched — the condition that triggered the
rebuild is still true afterwards. The check cannot converge: it rebuilds
on every reconcile tick forever, burning CPU and churning the container.
It bites after any deploy that refreshes /opt/archipelago/docker/*, which
makes the contexts newer than the shipped images — so this is fleet-wide
on every OTA, not local to one node.
Fix: stamp the context mtime that was built into an image label and
compare against that instead. A label is part of the image config, so a
cache-hit build with a new value still produces a new image — the thing
being tested does change, and the comparison settles after exactly one
rebuild. Verified against real podman before writing it: two cache-hit
builds with different label values produced distinct image IDs
(6cfdbc9bcd3e vs a9b10eb9a558), each carrying its stamp; the indexed
inspect format was checked against an image with real labels, and a
missing label prints empty (handled, along with "<no value>").
Images built before this carry no label and fall back to .Created, so
behaviour is unchanged for them and each self-heals on its first
reconcile after upgrade — nodes fix themselves rather than needing the
manual `podman build --no-cache` pass this needed by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced again on this node today: with no session cookie, six app
ports answered HTTP 200 with their real UIs (18083 LND, 8334, 8175
Fedimint Guardian, 8336 FIPS Mesh, 8090, 7777), all bound 0.0.0.0 and so
served on every host address. Same bug class as the /lnd-connect-info
and /bitcoin-rpc/ leaks closed in v1.7.120, but across every app.
LAN, Tailscale, Tor and the FIPS mesh all converge on 127.0.0.1:<port>,
so this is one gate rather than four. It lives in the daemon rather than
a per-app sidecar (umbrel's app_proxy model): rootless, no extra
container per app, and it can reuse machinery that already exists.
It invents no authentication policy. verify_password, TOTP secret
decryption, verify_code with used-step replay protection, the session
store, and — importantly — the SAME LoginRateLimiter instance as the
JSON-RPC path, so an attacker cannot get a fresh budget of password
guesses by moving to an app port. Only the transport differs, an HTML
form instead of JSON-RPC, because a browser being sent to an app cannot
speak JSON-RPC.
2FA comes for free: a session still pending its TOTP step fails
validate(), so the gate rejects it without knowing what a second factor
is.
Details worth keeping:
- 401, not a redirect. A redirect to a login page is indistinguishable
from the app itself redirecting, and machine clients would follow it
and parse HTML as their API response.
- Cookie and Authorization are stripped before proxying. The app has no
use for the node session and must never be able to log or forward it.
- The challenge page names and pictures the app being opened, so the
visitor can confirm what they are authenticating to.
- device_tokens grew `apps: Option<Vec<String>>` and verify_for_app for
machine clients. None = node-wide, which every existing companion
token is; migrating them by guessing a scope would silently revoke
access nobody asked to revoke. An empty list is rejected rather than
minted, since it reads as unrestricted while authorising nothing.
The rollout is necessarily per-app and the gate is built to say so. A
container publishing 0.0.0.0:<port> claims every host address, so the
gate cannot bind that port until the app is pinned to bind: 127.0.0.1
and recreated — gate-first is impossible, and all-at-once would recreate
every container on a node simultaneously. Every port it cannot claim is
logged at warn each sweep and recorded in GateStatus::unprotected,
surfaced by security.app-gate-status. The failure mode being designed
against is a gate that binds nothing, logs at debug, and reports success
while every app stays exactly as open as before — worse than no gate,
because it stops anyone looking. Same reasoning that ruled out an
nft drop-in, whose absence is a silent no-op.
Not yet done: pinning the 39 gated ports to loopback, repointing
HiddenServicePort at the gate, and on-node verification.
Tests: 21/21 appgate, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate's cargo-fmt stage failed on my own additions — the
tests in message_types.rs and lnd/info.rs and the reconcile branch in
prod_orchestrator.rs were written programmatically and never passed
through rustfmt. Formatting only; rustfmt is semantics-preserving and
the gate re-runs the suites before building.
Caught by the gate rather than in review, which is the gate working. Also
a reminder that a piped command's exit code is the pipe's, not the
script's: the task notification reported success while the log said
CREATE_RELEASE_EXIT=1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things needed for the new UI to actually reach users.
The rendered nginx.conf served index.html with only ETag/Last-Modified and
no Cache-Control, so browsers applied heuristic caching to it. Confirmed on
archi-dev-box: after rebuilding and recreating the container, :8334 and
/app/bitcoin-ui/ both served the new markup immediately, but the app iframe
in the main UI kept showing the previous UI until a hard refresh.
docker/lnd-ui/nginx.conf has always carried this header, which is why only
bitcoin-ui showed the stale copy. Using "no-cache" (revalidate) rather than
"no-store" keeps the ETag doing its job when nothing has changed.
Validated by mounting the rendered config into a throwaway container from
the built image and running nginx -t. (An earlier attempt to test it inside
the running container was meaningless — conf.d/default.conf is a read-only
bind mount, so the copy failed and nginx -t just re-checked the original.)
The 8 container::bitcoin_ui tests still pass; their assertions cover the
placeholder, the 8332 proxy_pass and the listen directive, none of which
this touches.
BITCOIN_UI_IMAGE was still pinned to 1.7.84-alpha, so a fresh install would
pull a bitcoin-ui from many releases ago regardless of what the OTA ships —
first-boot-containers.sh tries the registry image before building from
source. Bumped to 1.7.119-alpha, matching the current release, and the
image is pushed under that tag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while VERIFYING a05956c4 on archi-dev-box rather than assuming it.
GET /lnd-connect-info is correctly 401 with no cookies over the LAN
address. But POST /bitcoin-rpc/ on :8334 still answered an
unauthenticated caller with a real block height, and still carried
`Access-Control-Allow-Origin: *`. The node looked patched. Half of it
was not.
The rendered /var/lib/archipelago/bitcoin-ui/nginx.conf was dated
2026-06-30 — the pre-fix version — even though the running binary
carries the new template. a05956c4's commit message claimed the
template "is re-rendered on every reconcile pass, so this ships
atomically with the binary". That is false in one specific state, and
this node was in it:
1. bitcoin-ui sits in the durable user-uninstalled marker.
2. reconcile returns on that marker BEFORE run_pre_start_hooks, which
is what renders the config.
3. The container keeps running regardless, because it is owned by
systemd via a Quadlet unit (archy-bitcoin-ui.service, active,
restarted 17:25 after the daemon restart) — not by this reconciler.
So a container systemd keeps alive, that the orchestrator has stopped
reconciling, never receives a config fix shipped inside the binary. An
OTA carrying a05956c4 would have silently failed to close this on every
node in that state, while the LND half closed correctly — the most
misleading possible outcome. archy-electrs-ui is in the same state on
this node, so it is not a one-app accident.
A container that is actually running is a live attack surface whatever a
marker says about it. Its security-relevant config is now reconciled
even behind the marker, and it is restarted so nginx actually loads it.
Deliberately narrow:
- Nothing is created, pulled, built, started or resurrected. The "must
stay removed" contract only ever gets weaker if a container is ALREADY
running, which by definition means it was never removed.
- A hook error is swallowed, not propagated: an app the user uninstalled
must not be able to fail the reconcile pass for everything after it.
- The pre-existing marker test still passes unchanged, which is what
proves the removal contract survived.
Verified: 11/11 reconcile tests and 9/9 bitcoin_ui tests pass, including
a new regression test that pins the whole chain — stale conf in, gate
present out, container restarted, nothing created.
No node has been touched. The live exposure on archi-dev-box stands
until this is deployed and the operator restarts archy-bitcoin-ui.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes F-10a. Nothing here fixes a present defect: on the pinned rand
0.8.5, rand::random() and thread_rng() both resolve to a ChaCha12 CSPRNG
seeded from getrandom(2). What they lack is a STATED backend — it is
fixed by dependency and build configuration rather than by the calling
code, with no compile error if that changes. That is the structural
shape behind the 2026-07-30 COLDCARD entropy defect, and here the blast
radius includes Cashu blinded-key-exchange values, X3DH prekey material,
session bearer tokens and a ChaCha20-Poly1305 nonce.
Layer (a) — every production key, nonce and token draw now names
rand::rngs::OsRng at its own call site. The mnemonic seam is bound to
entropy::KeyGenRng, a SEALED allowlist whose supertrait lives in a
private module, so the set of RNGs that can drive the master key
hierarchy is exactly what one file says it is. This retires the false
promise at seed.rs:656: rand::CryptoRng is a marker with no
compiler-checked content, and the crate now contains zero impls of it.
Layer (d) — key material and AEAD nonces of >=12 bytes run a
degenerate-entropy predicate that refuses all-zero, all-identical and
wrapping +/-1 counter draws. Nothing heuristic: no entropy estimator, no
chi-squared. Each of the three shapes has a false-positive probability
computable in closed form (3 * 2^-88 at 12 bytes, 3 * 2^-248 at 32), and
a predicate whose false-positive rate cannot be computed cannot be
argued safe on a key-generation path. There is deliberately no retry — a
retry would paper over the broken RNG this exists to surface.
Layer (e) — the kernel-CSPRNG readiness verdict at master-seed
generation is now durable (backlog R-09). It was previously computed,
logged and thrown away, so a node could never answer after the fact
whether its keys were born from a seeded pool. The record holds a schema
version, timestamp, verdict and event name — no entropy, no key bytes.
Formats and wire shapes are proven unchanged rather than asserted:
storage_crypto and the credential store each open a HARDCODED
pre-migration ciphertext vector (a same-process round trip would pass
even if the envelope had changed), the vector was produced by an
independent RFC 8439 implementation so it pins the documented
nonce||ciphertext format rather than this implementation's output, and
the x3dh prekey bundle and bdhke values keep their field set and order.
totp.rs migrates its SOURCE only: the % charset.len() reduction and the
32-char charset are untouched. The bias there is presently zero (32
divides 256) and fixing the latent bias is R-12, which stays deferred.
Verified: cargo build clean; cargo test -p archipelago 1068 passed,
2 failed. Both failures are container::boot_reconciler timing tests
(second_pass_fires_after_interval, shutdown_terminates_loop) in a file
this change does not touch — pre-existing, not caused here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CRITICAL. Two app-UI ports handed unauthenticated callers full control of
the node's money. Both verified live on archi-dev-box 2026-08-02 over the
fips0 mesh ULA with no cookies.
GET /lnd-connect-info returned 200 with the LND ADMIN MACAROON, the TLS
cert, the gRPC/REST ports and the node's onion address — a complete remote
wallet-drain package, and the onion means an attacker keeps that ability
after losing network access. POST /bitcoin-rpc/ reached Bitcoin Core RPC
with credentials the proxy injected on the caller's behalf, with a wallet
loaded, so wallet methods were reachable too.
Both were reachable because ports 18083 (lnd-ui) and 8334 (bitcoin-ui)
bind 0.0.0.0 AND sit on the fips0 mesh allowlist in fips/app_ports.rs. Any
mesh peer, LAN host or Tailscale peer could take either path.
The root cause is one mistaken idea in two places: that a check performed
by a reverse proxy is an auth check. It is not — it only holds for traffic
that arrived through that proxy. /lnd-connect-info's comment said "nginx
validates session cookie (presence check), backend is bound to 127.0.0.1
so only nginx can reach it". Both clauses were false in production: the
lnd-ui container runs its OWN nginx on :18083 that proxies straight to the
backend forwarding whatever cookies arrived, including none, and that
second front door never performed the check the premise named.
So authorisation moves to the resource:
- /lnd-connect-info now requires a session, like /proxy/lnd/ beside it.
The 401 carries CORS headers so the wallet UI shows a readable error
rather than an opaque CORS failure.
- New GET /auth/session-check returns 204/401 and nothing else, giving
container nginx an auth_request gate it can actually use.
- bitcoin-ui's /bitcoin-rpc/ is gated by that auth_request. Its
`Access-Control-Allow-Origin *` is also gone: on a proxy that injects
credentials, it let any page a user visited drive the node's RPC.
Preflight is answered before the gate, since OPTIONS carries no cookies.
The nginx template is include_str!'d and re-rendered on every reconcile
pass, so this ships atomically with the binary.
Operators must treat the LND admin macaroon and the Bitcoin RPC password
on every affected node as compromised and rotate them AFTER this is
deployed — rotating first just re-leaks through the same hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The checkpoint on archi-dev-box proved rotation alone doesn't close FED-07:
the credential file went unique while the RUNNING container kept serving the
compromised one, because the Quadlet path rewrites a unit without restarting
it and fedimint-gateway is classified restart-sensitive, so drift was detected
and deliberately ignored on every tick.
Rotation now records the app id, and the drift check consumes that flag to
recreate even a restart-sensitive app, with a WARN naming the reason. This
mirrors the published-port carve-out a few lines above, which already makes
the same trade for the same reason: a container that is already broken (there)
or already compromised (here) is not protected by leaving it running.
Restart-sensitivity protects working services. A gateway answering to a
credential published in this repository is not working, it is compromised, and
gateway admin can drain Lightning liquidity — indefinite exposure loses to a
few seconds of restart. Rotating-but-only-alerting was rejected: the
monitoring system fires on metric thresholds only, so it would have needed new
event-alert plumbing to deliver something strictly weaker.
Re-verified on the same node, same scenario: rotation at 06:39:23, recreate at
06:39:27, PID 3923125 -> 148426, running credential now matches the file,
container healthy with the same name and ports, gatewayd.db intact at 18 files
with IDENTITY present, 32 containers untouched, no repeat rotation.
3 new tests. Also lands the missing 01-19 and 01-20 SUMMARYs: both had code
committed 2026-07-31 but no summary and no roadmap tick, so they read as
unstarted. Phase 1 is 11/20. FED-09 carries 15h of Tor uptime and 0
permission-fixes across 542 doctor runs on archi-dev-box.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01-11 stopped new installs from ever taking a shipped credential, but did
nothing for the nodes that already did — those gateways still answer to a
password published in this repository.
rotate_compromised_gateway_credential() detects an EXACT match against the
denylist and replaces the pair; absent, unique, or merely unrecognised values
are left alone and return false. That distinction is the point: an operator
who deliberately set their own credential also has an "unrecognised" one, and
rotating it would be the same class of harm as leaving the default in place.
It hangs off resolve_dynamic_env beside ensure_generated_secrets, gated on the
gateway's app id, so an affected node heals on its next reconcile tick. There
is deliberately no teardown here: the new hash changes the resolved secret env,
which changes secret_env_hash, which the drift check reads as a container-label
mismatch — so the platform's own recreate path rebuilds the gateway around its
unchanged data directory, ports, volumes and name.
Rotation is self-terminating (the value written is not on the denylist, so the
next tick is a no-op) and errors propagate rather than being swallowed, because
the atomic write leaves the previous credential intact on failure.
Bcrypt generation was factored out of ensure_one into write_bcrypt_pair, which
both generation and rotation call — 01-11's SUMMARY claimed such a helper
existed but the arm was still inline, and rotation cannot reuse
ensure_gateway_credential because its idempotent fast path returns early
exactly when the file is present, which is the case rotation acts on.
Also fixes cargo fmt drift left by 42652547 in install.rs.
Verified: 6 new tests, secrets suite 16/16; full suite 1008 passed with one
known wall-clock flake (green 4/4 in isolation). NOT verified on a node —
Task 2's blocking checkpoint has not been run, so FED-07 stays open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six code paths configured the Lightning gateway with a bcrypt hash committed
to this repository — and one deploy path with a plaintext password literal —
whenever the per-install secret was missing. Anyone holding a copy of the repo
held the admin credential for every gateway that ever took a fallback.
container::secrets now owns the credential end to end: ensure_gateway_credential
(idempotent, delegates to ensure_one's bcrypt arm) and gateway_bcrypt_hash,
which returns Err when the secret is missing/empty and when the stored value is
on the KNOWN_DEFAULT_GATEWAY_HASHES denylist — so this codebase cannot hand
back the compromised value even to a node already carrying it.
get_app_config was widened to Result so a credential-less install cannot reach
podman run at all; configure_fedimint_lnd takes the resolved hash instead of
re-reading with its own fallback. The four shell paths stop generating
credentials entirely (dropping the htpasswd host dependency) and skip container
creation with a printed reason rather than substituting anything.
Naming converges on the manifest's fedimint-gateway-hash/.pw, with legacy
fedimint-gateway-password values copied forward rather than regenerated so no
node loses a working unique credential. Plan 01-16 owns rotation of installs
already carrying the default.
Verified: cargo build clean; cargo test -p archipelago 999 passed (2
boot_reconciler timing tests failed under concurrent load, green in isolation,
untouched by this diff); bash -n clean on all five scripts; the compromised
literal now appears exactly once in the tree, as the denylist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A published port no longer implies a web UI. The package scanner used to
synthesize interfaces.main.ui="true" for any container with a port or
onion address, so headless backends — including self-deployed compose
stacks like podsteadr — showed up as launchable apps. New ui_detection
module decides instead: a manifest interfaces declaration (catalog
overlay first, disk second) is definitive; undeclared apps get a short
HTTP probe of the launch port (HTML page, redirect, or browser auth
wall = UI; JSON APIs, raw TCP, dead ports = service), with cached
verdicts and probes gated on running containers. Frontend canLaunch
now refuses curated services outright and only treats a bare runtime
address as launchable for curated known apps.
Works identically for manifest apps and containers deployed by hand
outside the orchestrator. ui_detection tests 6/6, frontend suite
696/696.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the .198 load spiral (2026-07-28): needs_repair() called
ensure_image_present() every 30s tick to render the expected unit, so
under IO pressure the image-existence check timed out, read as "image
missing", and a 900s podman build ran inside the PROBE while the
companion was up — each build pegging the disk that made probes fail.
- needs_repair() is now build/pull-free: unit file present → service
is-active (10s cap; a hung systemctl reads as "assume active", never
as dead) → unit matches one of the three image refs install_one could
have written → context-newer-than-image staleness only when the unit
uses the auto-built :latest.
- Per-companion 10-min repair cooldown after a failed install_one, so a
failing build retries at most every REPAIR_COOLDOWN instead of every
reconcile tick.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rootless podman's wildcard publish claims [::] but BLACK-HOLES inbound
v6 (accepts, forwards nothing — vaultwarden 'empty response' from the
phone, 2026-07-26), while most apps got no v6 listener at all. Two
halves: quadlets now publish unbound ports on 0.0.0.0 explicitly
(frees the v6 side; the one-time drift/recreate at upgrade is the
deploy vehicle), and the daemon runs a self-selecting V6ONLY relay on
every catalog launch port forwarding raw TCP to the v4 loopback
listener. Rescans every 60s so new installs bridge without a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway crash-looped dialing bitcoind at host.archipelago:8332 (the
host gateway IP, where bitcoind does not listen). Root-cause fix, three
parts:
- fedimint-gateway manifest: --bitcoind-url now comes from
$FM_BITCOIND_URL, filled by a {{BITCOIN_HOST}} derived-env — works on
Knots, Core, or any future Bitcoin distro.
- fedimint manifest: same derived-env replaces the hardcoded
FM_BITCOIND_URL=http://bitcoin-knots:8332 environment entry.
- orchestrator: removed the hardcoded FM_BITCOIND_URL=bitcoin-knots
override that clobbered the derived-env, and bitcoin_host() now
accepts any running bitcoin*/bitcoin container (excluding -ui and
archy-* sidecars), preferring the known names in order.
Catalog regenerated + signed (nodes install from the signed catalog, so
the manifest fixes only reach them via this regen).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fleet log sweep (2026-07-22): (1) recovery paths podman-start'ed
containers that were already running, producing hundreds of benign but
scary conmon 'Failed to create container' + cgroup Permission-denied
pairs per node per day — both paths now check state first; (2) the 30s
companion-reconcile loop retried registry pulls/builds forever against
unreachable registries (~174×/image/day on an offline node) — failing
rounds now back off exponentially to a 1h cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The creds-off-argv script used printf "rpcuser=%s" in the manifest's
custom_args; quadlet copies Exec= into the generated service's ExecStart,
where systemd expands %s (user's shell) at load time — bitcoind's rpc.conf
came out as rpcuser=/bin/bash and every RPC consumer got 401s on quadlet
nodes (framework-pt: bitcoin-status, HA block-height sensor, LND chain RPC).
Two-layer fix: quadlet.rs now escapes % -> %% in Exec/Entrypoint/HealthCmd/
Environment emission (with regression test), and the bitcoin manifests write
rpc.conf with plain echo so the catalog also heals nodes still running older
binaries. Release catalog regenerated with the fixed scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GET /api/pine/status: public tier (version/uptime/bitcoin/mesh facts) for
the Pine page; bearer-token tier (Lightning balances, latest mesh message)
for the seeded Home Assistant sensors. Token minted 0600 at
secrets/pine-status-token; nginx location ships via the bootstrap
self-heal + canonical ISO conf. Replaces the per-node socat forwarder and
bitcoind RPC credentials living in HA's configuration.yaml.
- pine_ha seeder: REST sensors + four ask-Archy intents (block height,
peers, sync %, Lightning balance) with LLM tool descriptions; Claude
conversation agent (anthropic entry from the node's claude-api-key) set as
pipeline default with prefer_local_intents so exact phrases stay local and
fuzzy ones tool-route through Claude; mesh-message announce automation on
every Assist satellite; bounded config markers with legacy-block migration.
- pine-openwakeword joins the pine stack (all lifecycle sites) — on-node
wake-word engine groundwork for the custom "Yo Archy" model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Register "pine" as a manifest-driven stack (like netbird) across every
lifecycle site so install/stop/start/restart/uninstall/crash-recovery treat
the launcher + two Wyoming engines as one app:
- stacks.rs: pine_stack_app_ids() + install_pine_stack() (orchestrator-first,
adopt existing, else bail — no hardcoded fallback)
- install.rs: dispatch package_id == "pine" to the stack installer
- app_ops.rs: canonical stack-member table + owning_package STACKS
- dependencies.rs: startup_order + needs_archy_net
- config.rs: all_container_names
- crash_recovery.rs: auto-start members + a StackRecoverySpec (anchor
pine-whisper) so a genuinely-installed stack self-heals but orphan debris
does not crash-loop
- docker_packages.rs: exclude the engines from the apps list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sweep podman-execs into every running container; doing that on each 30s
reconcile tick was a permanent conmon 'Failed to create container' storm on
hosts where exec from the backend's cgroup context fails (Debian 13 first
boot). Ownership drift is an install/OTA-time event — sweep on first pass
after a container appears, then at most hourly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a fresh ISO node, NetBird's own WireGuard tunnel (10.44.0.1) sorted
ahead of the real NIC in hostname -I, so the NetBird launch URL (and the
{{HOST_IP}} baked into its cert/config) pointed at the tunnel instead of
the LAN address. The main routing table's default route names the
physical uplink even while a VPN is active (NetBird/Tailscale steer
traffic via policy-routing rules, not the main table), so read src/dev
from 'ip -4 route show default' first, then fall back to a connected UDP
socket's source address, then to the old hostname -I scan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
936b4cca made ANY loaded manifest with an absent container self-heal via
install_fresh — but the signed catalog and the ISO ship manifests for
EVERY app, installed or not, so 'manifest loaded' is not installation
evidence. On a fresh node this mass-installed the catalog: the framework
node grew portainer, vaultwarden, searxng, strfry, mempool, immich and
fedimint uninvited within an hour (fleet nodes showed the same loop as
reconcile-failed noise for unpullable images).
ExistingOnly (boot) reconcile now creates containers from nothing only
for required-baseline apps (filebrowser, fedimint-clientd); every other
absent app is Left('absent') and recreated ONLY via desired-state
recovery when it was running at the last snapshot — which still covers
the vanished-container regressions (indeedhub/immich) that motivated
936b4cca. Reconcile suite 16/16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
484.kitchen, Arch Presentation, Call the Operator, Next Web News
Network, Syntropy Institute, and T-0 were hardcoded web-only bookmark
apps (no manifests/containers). Removed across the frontend app grid /
curated / marketplace / session / launcher data, the backend
AppMetadata table, their icons, and the nginx /ext/ proxy blocks
(external-app-proxies.conf deleted — nothing installed it anymore).
botfights/nostrudel/gitea keep the shared external-app code paths.
Frontend: vue-tsc clean, 672 tests pass, production build verified
clean of the removed slugs. Backend: cargo check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aezeed words are now captured at BOTH wallet-init paths and stored
encrypted (Argon2 + ChaCha20-Poly1305, per-node wallet secret) at
identity/lnd_aezeed.enc; ack marker cleared when a wallet is recreated
- lnd.init-wallet-from-seed was posting seed_entropy to /v1/initwallet,
which is a GenSeed field — the wallet was never actually derived from
the master seed; now GenSeed(entropy) → InitWallet(words)
- new RPCs: lnd.seed-backup-status / lnd.seed-reveal (password + 2FA
gated, same as seed.reveal via shared verify_reveal_auth) /
lnd.seed-backup-ack
- LND app detail page: Lightning-seed card with first-launch backup
prompt, tap-to-reveal modal, 'I've backed it up' acknowledgment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Materialise /var/lib/archipelago/secrets/btcpay-lnd-connection
(type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<sha256>)
in ensure_app_secrets before btcpay-server env resolution.
The macaroon travels inline as hex because LND's datadir is owned by its
container subuid (100999) — bind-mounting it into btcpay's userns EACCESes.
Reads the macaroon via the existing read_file_as_root sudo fallback; pins the
TLS cert by DER SHA256 thumbprint and rewrites on cert rotation. No-op while
LND is absent (btcpay's secret_env entry is optional), and generation errors
log-and-continue so btcpay never fails to start over this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
btcpay-server started and reported healthy while its declared
dependencies (archy-btcpay-db, archy-nbxplorer) failed every boot
reconcile (live-testing report 2026-07-10, Fix 3). After each
reconcile pass, flag every app the pass left up whose manifest
dependency landed in the failure list — a loud DEGRADED journal error
instead of silence. True start-gating on dependency health is a
larger design change (boot-ordering/deadlock risk) and stays open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strfry/vaultwarden 8082 collision shipped in a signed catalog
because nothing checks cross-app host ports. Fix the remaining latent
pairs from the cfb8d953 audit — did-wallet 8083→8088 (filebrowser
keeps 8083), morphos-server 8086→8089 (netbird-server keeps 8086),
lightning-stack 8087→8091/9737→9738 (netbird and fedimint-gateway
keep theirs) — and fix their healthchecks, which probed the HOST port
inside the container (same class as strfry 6fe62355; now 127.0.0.1 +
container port). load_manifests now flags collisions across the
merged disk+catalog set (bitcoin variants and the mempool umbrella
pair are exempt as mutually exclusive), and a repo-level test parses
apps/*/manifest.yml so a future latent pair fails CI instead of
shipping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
24fd97ed cascades on package.start/restart, but a reconcile pass that
recreates or starts a bitcoin backend (desired-state recovery, repair
recreate, boot InstallMissing) also moves the RPC address behind a
running lnd's back — §C 'restart lnd after ANY bitcoin recreate'.
After the pass, dependents from the shared address_caching_dependents
table (moved to app_ops as the single source of truth) that sat
untouched (NoOp) and aren't user-stopped are restarted under their op
lock; a dependent the pass itself (re)created already resolved the
fresh address and is left alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wait_for_manifest_host_ports polls up to 420s while the caller holds
the per-app lock, so a user's stop request queued behind it and the
scanner painted the app Restarting for the whole wait (the dd3afbba
transitional-state stick; gate take-2 iter 5 vaultwarden). The wait
now selects against a user-stop-marker watcher (2s poll on
user-stopped.json, by app id or container name) and aborts with a
recognizable error, and the host-port repair path skips its restart
when a stop was requested while it settled — restarting there would
fight the queued stop worker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
