chown_for_rootless_container prefers `podman unshare chown` (which maps
container uid N through the userns), but when that failed once it fell
back to `sudo chown -R <literal>` — writing e.g. host uid 999 for
container uid 999 and reporting success. Host-999 maps to nobody inside
the userns, so the app could not open its own data while everything
claimed the chown worked: botfights on framework-pt crash-looped every
10s on SQLITE_CANTOPEN over a data dir the daemon itself had just
"fixed".
The sudo fallback now translates container ids (1..99999) to
subuid_base + id - 1 (fleet base 100000; container root maps to the
service user, 1000). Already-mapped ids and uid 0 pass through.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A sweep of 92 unconfirmed/dust sats failed with LND's debug-flavored
"insufficient input to create sweep tx: input_sum=0 BTC, output_sum=
0.00000092 BTC" — and the RPC sanitizer then masked even that behind
"Operation failed. Check server logs." (framework-pt, 2026-08-06).
The sweep mechanics are untouched (balance minus fee, as always) —
this only makes the refusal say WHY in plain language.
- lnd.sendcoins translates the sweep refusal: balance below Bitcoin's
dust minimum or not yet confirmed, so no transaction can be built
(LND's original message kept in parens).
- "Failed to send" joins the sanitizer's user-facing allowlist — the
same lesson as "Insufficient balance"/"Payment failed" before it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The periodic reconcile runs ExistingOnly — merely listing a catalog
manifest must never install an app — and its only absent-container
recovery keyed on the last running-names snapshot, which ages out after
a few daemon restarts. An absent member of an installed stack then stays
absent forever: .38 ran indeedhub with no minio/postgres for days, nginx
down on 'host not found in upstream "minio"', and nothing ever put the
members back.
A live sibling container is proof the stack is installed on this node,
so an absent member is now treated as a hole to repair, not a choice to
respect: the recovery guard also fires when another member of the same
stack (app_ops::stack_member_app_ids) has a container in any state.
A stack with no containers at all is left untouched, and sibling app ids
resolve through the loaded-manifest container names (immich-postgres
runs as immich_postgres).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both predate this session's changes and were masked by the release
gate's cargo-test-weekly compile timeout:
- login_page_sources_its_art_from_the_gate still asserted the retired
wordmark (logo-archipelago.svg); the login page ships the sidebar A
mark (favico-black-v2.svg) since the 2026-08-05 rework.
- unauthenticated_ports_are_all_accounted_for lagged at 17; the
v1.7.123 port-policy round grew the rationale-carrying exempt set
to 25 (reviewed and enumerated in the test comment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two daemon bugs, one debugging arc (2026-08-05, operator-reported):
1. The app gate removed the ENTIRE Cookie header before proxying. That
broke the data plane of every first-party companion UI behind the gate
(lnd-ui/bitcoin-ui/electrs-ui/fips-ui render their shell, then every
/proxy/* and /lnd-connect-info call 401s — observed as "LND UI
unreachable"), and silently logged users out of every gated app with
its own cookie login (vaultwarden, nextcloud, gitea) on each request.
The gate now strips only its own cookie pairs (session, csrf_token);
a new per-port manifest opt-in `session_passthrough: true` forwards
the node session to first-party UIs whose nginx proxies the daemon's
authenticated endpoints. Undeclared ports never get passthrough.
2. podman_client::create_container sent named volumes to the libpod API
as bind mounts with the bare volume name as source, so creating any
manifest app with a `type: volume` mount failed. On .38 the reconciler
removed indeedhub-postgres/-minio for env drift and then could never
create their replacements, leaving the stack half-missing forever.
Named volumes now ride the spec's `volumes` field ({Name, Dest,
Options}). Also: the reconcile-failure log now prints the full anyhow
chain — `%e` showed only "create_container X" and hid the real error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unbreaks Bitcoin on every node running the 1.7.124 catalog: the embedded
start script had a shell syntax error, so bitcoind never launched and the
app vanished. Delivered by catalog rather than a release because manifests
reach nodes through the signed catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate-app-catalog.py writes APP_LAUNCH_PORTS one entry per line; rustfmt
packs it. The release gate checks formatting, so the generated file has to
be formatted after regeneration or every catalog sync fails the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portainer's image reaches the public catalog (the release gate caught the
manifest and catalog disagreeing), and fips-ui 8336 joins the mesh relay's
port list now that it declares a port — it is auth: gated, so the relay
withholds it rather than bridging it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-bumped so the release gate compiles the test profile at the final
version — create-release bumps after the gate, so the gate would otherwise
run on the old version and the bump would invalidate the cache, timing out
cargo-test-weekly on the compile rather than the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
auth.session-policy.get/set plus a card under Account. Presented as two
plain questions rather than the token mechanism underneath, because the
distinction that matters to an operator is which control actually ends a
session: the dashboard polls constantly, so an idle timeout alone never
fires on an open tab — the absolute cap is what guarantees it.
Values are clamped server-side and the stored result is echoed back, so
the bounds are discoverable instead of an error. Presets rather than a free
number field: a box accepting '5' invites locking yourself out. A short
idle choice warns that it is the payments-industry posture.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
austin-sapien (100.70.96.88) sat dead for over two hours after taking
v1.7.122 — 'server starting' in the UI, service inactive, exit status
0/SUCCESS. It did not crash: the in-process updater replaces the binary and
exits cleanly for systemd to restart it, and that node's unit still carried
Restart=on-failure from an older install. systemd read the clean exit as
success and left it stopped. Every node with the old unit has this waiting
for it on the next update.
self-update.sh does refresh units, but the in-process update path never
runs it, so nothing was repairing them. The daemon now checks its own unit
at boot and rewrites only the Restart= line, so a node that starts even
once ends up with a policy that survives the next update.
Also carries the session-policy wiring: validate() now honours the
configured idle and absolute limits and the per-device class, instead of
the single hard-coded 24h constant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mesh right panel: a >=2560px screen hid the tab bar and stacked all five
tool panels in fixed grid rows. On a real display that clipped the Bitcoin,
Dead Man and AI headings to a few pixels each, letterboxed the map, and
pushed Radio Settings into a scroll — more screen producing a worse view.
Very wide now uses the same tabbed column as every other desktop width,
with the selected panel filling the column and the map running edge to edge
(it is the one panel with nothing to scroll).
Session policy: idle timeout, absolute cap and a re-prompt-for-funds flag,
persisted and clamped. Two tokens already existed — a session token and a
30-day login token — so the knob changes how long a quiet tab stays usable
without putting a long-lived credential on every request. Kiosk screens are
exempt from the idle timeout (nobody is there to log a TV back in) but keep
the absolute cap so a stolen box does not stay authenticated forever. The
cap is not optional theatre: idle alone never fires on a polling dashboard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portainer: nodes have been running :latest — which is 2.39.1 — while the
manifest pinned 2.19.4 from two years ago. The port migration recreated the
container onto that old pin and Portainer refused to start: it migrates a
database forward, never backward, so an existing install died with 'schema
version does not align' and My Apps showed 'app is not responding'
(100.82.34.38). 2.39.1 published as an immutable tag and pinned forward, so
existing databases keep working and older ones migrate up.
Bitcoin: complements PR #131. That removes the code which kept writing a
datadir bitcoin.conf; -allowignoredconf=1 additionally makes an existing
one non-fatal, so a node already carrying the file recovers on restart
instead of crash-looping until something reinstalls it.
App gate login: rebuilt against the dashboard's own design — rotating
intro backgrounds served from the gate, the glass panel, the Archipelago
mark in its gradient ring, the app's icon as a My Apps tile, and the glass
button. Crucially it no longer sends X-Frame-Options: DENY, which made
every gated app render as unreachable inside My Apps' embedded frame;
frame-ancestors expresses 'only this node may frame me', which
X-Frame-Options cannot.
OTA origin: primary mirror is now source.archipelago-foundation.org over
TLS instead of a bare IP on plaintext. The IP stays as an automatic
fallback for nodes whose DNS or clock is broken — both break TLS, and the
signature, not the transport, is what establishes trust.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the Bitcoin crash-loop on 100.82.34.38: since a597c1d9
bitcoind launches with -conf=/tmp/rpc.conf and never reads the datadir
bitcoin.conf, but write_bitcoin_conf / ensure_bitcoin_rpc_config /
run_bitcoin_rpc_repair kept writing one on every install and restart.
Bitcoin Core's own datadir-conflict check then refuses to start at all.
Conflict resolved in favour of the PR: HEAD still carried
write_bitcoin_conf, whose deletion is the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mint failures surfaced raw JSON ({"detail":"proofs already spent"}) to
the user; now the top-level message is actionable while the raw body stays
in logs via {:#}.
A rebuilt image never reached a running companion. ensure_image_present
rebuilds in place under the same tag, so the quadlet body is identical,
write_if_changed reports no change, and enable_now is a no-op on a running
service — the container keeps the old layers indefinitely.
That is precisely how archi-dev-box kept serving the LND, FIPS, Electrs and
Guardian screens on 0.0.0.0 after v1.7.123 rebuilt every one of those images
to bind loopback: correct images on disk, three-day-old containers still
running. Closing those ports needed a manual 'podman rm -f' per container,
which no other node would ever get. Compare the running container's image ID
against the built one and restart when they diverge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scanning archi-dev-box from OUTSIDE found five ports serving their screens
with no login — lnd-ui 18083, bitcoin-ui 8334, fips-ui 8336, electrs-ui
50002 and the Fedimint Guardian 8175 — none of which appeared in the gate's
unprotected list. They are host-networked, so Podman publishes nothing to
pin and their manifests declared 'ports: []'; the gate builds its map from
declared ports, so it neither protected them nor reported them. An audit
that reports success while five screens are open is worse than no audit.
Their nginx now listens on 127.0.0.1 instead of 0.0.0.0, and each port is
declared 'auth: gated' so the daemon owns the outside. 'bind:' on a
host-networked app is a statement of where the container listens, not a
publish instruction — quadlet already skips PublishPort in host mode.
Guardian 8175 is declared on the fedimint app because its companion has no
manifest, and the gate keys on port, not container.
Credential paths were NOT exposed and are verified so: /lnd-connect-info,
the /proxy/lnd/ passthrough, container logs and every RPC method through
these screens all return 401 unauthenticated. What leaked was the page
shell.
Also fixes the delivery gap that would have made this unshippable: only
bitcoin-ui, lnd-ui and electrs-ui were ever rsynced to
/opt/archipelago/docker, so edits to fips-ui and fedimint-ui reached nodes
through no path at all. All five now sync; the two whose rebuilds the
daemon owns are synced without being handed to container-specs.
Every remaining undeclared port is now declared with a stated reason —
gated: botfights 9100, router 8084, pine 10380; exempt with rationale:
fedimint consensus 8173/8174, gateway 8176/9737, netbird 8086/8087 (TLS +
own auth, and enrolled devices cannot hold a session), pine TLS 10381,
lightning-stack REST 8091 (macaroon, mirrors lnd). Zero undeclared ports
remain across all 56 manifests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mint HTTP failures (swap/melt/mint-quote) were surfacing raw JSON bodies
like {"detail":"proofs already spent","code":11001} straight to the
user. Add a translator for the NUT-02/03/04/05 transaction-validation
error codes (10001-11017, 12001-12003; see
https://github.com/cashubtc/nuts/blob/main/error_codes.md) and layer it
onto the mint_client bail sites via anyhow context, so the top-level
message is actionable while the raw status/body stays available via
{:#} for logs. receive_token now surfaces the real reason (e.g. "This
ecash has already been redeemed") instead of a generic "Failed to
receive any proofs from token" when every mint in a token fails.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Leads with what changes for the operator: app screens now require the node
password across LAN, Tailscale, mesh and Tor; the wallet/protocol ports that
must stay open stayed open; the mesh leak found during on-node verification;
nodes repairing their own legacy containers; and the signing-key rotation.
Known gaps disclosed, including the eleven still-undeclared ports and that
non-browser clients will now meet the login page.
The new block uses <strong> rather than the literal ** markers in earlier
entries, which render as asterisks in the modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DO NOT MERGE INTO A RELEASE SIGNED WITH THE NEW KEY. See below.
The previous release root (z6Mkkid…q7ur, pinned 2026-07-02) was exposed
in a chat transcript and is treated as compromised. It signs both OTA
manifests and the app catalog, so anyone holding it could sign updates
the fleet would install.
Pins the new key in trust::anchor and moves EXPECTED_DID in all three
signing/publishing scripts.
ORDERING IS CRITICAL — nodes pin the OLD key:
* The release CARRYING this commit must be signed with the OLD key.
That is the only signature a node running the previous binary will
accept, and it is what installs the binary pinning the new key.
* Only the release AFTER that may be signed with the new key.
* Signing this release with the new key makes every node reject it,
ending OTA fleet-wide and requiring hands-on recovery per node.
sign-catalog.sh moves in the same commit, so the app catalog must also be
re-signed with the new key once this ships, or nodes accept the binary
and reject the catalog.
Key verified before pinning: the hex and the did:key are the same
keypair, checked with a base58 decoder round-tripped against the previous
known-good pair. An earlier candidate hex (cb830e13…) was rejected
because it decoded to a different DID than the one supplied — pinning it
would have made every node reject every future update.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught verifying the gate fixes on archi-dev-box: [fips0-ULA]:32838
answered HTTP 200 straight from nbxplorer with no credential. The
catalog declares that port auth: local — host-local by intent, pinned to
loopback, the gate deliberately keeps its hands off — but the mesh relay
bridges a STATIC port list to 127.0.0.1, so it republished it to the
whole mesh. Same bug class as the Tor onion gap: a transport that
converges on the app loopback without consulting the declaration.
PortMap now records declared-local ports and the relay withholds them
(tearing down an existing bridge if a catalog refresh newly declares
one), alongside the declared-gated withhold. Undeclared ports keep
todays behaviour — silence is not an instruction in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dev-box verification of the Tor/FIPS fixes caught a pre-existing split
brain: the orchestrator publishes containers from the signed catalog's
embedded manifests (origin-wins), but the gate classified ports from the
stale disk manifests — so it externally bound nbxplorer 32838, a port
the catalog declares auth: local and pins to loopback. Reachable behind
a login, but reachable where it deliberately was not.
- build_port_map now consults the catalog overlay first, via the same
parse/validate/image-only filter the orchestrator uses (moved to
app_catalog::catalog_manifest_overlay so the two cannot diverge again).
- GatedPort carries . The gated set still includes undeclared
Session-default ports for challenge/audit, but every action that
REDIRECTS traffic — the torrc 127.0.0.2 repoint, the FIPS relay
stand-down, the Tor-upstream bind — now keys on the declaration.
- The sweep releases held claims whose port left the gated set, so a
catalog refresh that withdraws a port (gated → local/none) takes
effect without a daemon restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Legacy pre-quadlet containers kept publishing 0.0.0.0 after the catalog
pinned their app to loopback, because host_port_bindings_drifted only
compared host PORT numbers — closing them needed a manual package.update
per app per node. The drift check now also compares the bind ADDRESS,
but only when the manifest declares one: an empty bind never fires,
since recreating a loopback-published container to wildcard on silence
is exactly the v1.7.121 Bitcoin-RPC incident. With this, every node
recreates its legacy containers to the declared state on its own after
the OTA.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mesh relay is a raw unauthenticated forward to the app's loopback,
and whether it or the gate owned a fips0 ULA port was decided by a bind
race — the dev box happened to be safe because the gate bound first.
The relay now skips ports declared auth: gated and tears down any
existing bridge for a port that became gated since it was bridged
(catalog refresh), releasing the bind for the gate's next sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tor carries no session cookie, so HiddenServicePort → 127.0.0.1:<port>
reached the app around the gate — the last transport the gate did not
cover. The gate now binds 127.0.0.2 (its own loopback, distinct from the
app's 127.0.0.1, so no app needs a second port), and regenerate_torrc
forwards declared-gated ports there. Undeclared ports keep today's
target: absence of the field is not an instruction.
The 127.0.0.2 claim deliberately does not count toward the unprotected
audit — a port whose only claim is the Tor loopback is still wide open
on the LAN and must keep warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two live incidents on archi-dev-box today, one bug. Both times a safety
decision read an ABSENT manifest field as if it were a value, and a
node's installed manifests always lag the binary — so "absent" is the
state of essentially every port on every node.
1. Gating any `session` port regardless of `bind` published Bitcoin's
loopback-only RPC 8332 on the LAN, Tailscale and IPv6 within seconds
of deploy.
2. The `bind`-keyed replacement looked safe because it protected
`bind: 127.0.0.1` ports — but LND's gRPC 10009 and REST 18080 carry
an EMPTY bind, so they fell through. One container recreate from
pinning them to loopback and breaking Zeus and every remote wallet.
`auth` is now `Option<PortAuth>`, separating two questions that were
conflated:
* `auth_policy()` — what to CLASSIFY the port as. Undeclared reports as
Session, i.e. shows in the audit as something that should be behind
the gate. Reporting is always safe.
* `auth_is_declared()` — whether the daemon may ACT. Only an explicit
declaration authorises changing how a port is published.
Also reverts the daemon-side publish rewriting entirely. The node proved
it wrong twice over: the recreate path that actually ran was in
package::install, not podman_client, so the pin never fired; and even
`bind: 127.0.0.1` written directly into the node's manifest was
overridden by the signed catalog. Publishes are built in several places
and all of them already honour `bind`, so the migration belongs in the
catalog as data — not in daemon-side inference that can only ever cover
one path and guess wrong on the rest.
Tests: 75/75 container, incl. the LND wallet-port shape (`host: 10009`,
empty bind, no auth) asserted to be non-actionable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Peering a node was not enough to message it — you had to be in radio
range once before chat worked, which defeats the point of federating.
`send_message` chose its transport from the attached radio:
let use_typed_envelope =
archy && matches!(device_type, Meshcore | Reticulum);
Only the typed path knows about FIPS/Tor. Everything else fell through to
`peer_dest_prefix`, which resolves an over-the-air ROUTING key — so on a
node running Meshtastic, or with no radio at all, sending to a federated
peer failed. It only worked once a LoRa advert had created a radio twin
for the same archipelago identity, which is precisely the "connect on
LoRa first" the operator hit.
Federation contacts are reachable off-radio by definition — that is what
`upsert_federation_peer` records with `reachable: true` — so the
transport choice must not depend on which radio is plugged in. A
federation-synthetic contact id now always takes the typed path.
This loses no radio-first behaviour: `send_typed_wire` already prefers a
REACHABLE radio twin when the payload fits the frame, and only then falls
back to FIPS and Tor. The fix routes federation contacts INTO that logic
rather than around it.
Test pins the predicate across every device type, including the two that
failed (Meshtastic, Unknown), and asserts ordinary radio contacts and
stock clients still route exactly as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Observed live on archi-dev-box: archy-fedimint-ui rebuilt 45 times in 10
minutes, every ~35s, indefinitely. bitcoin-ui, lnd-ui and electrs-ui were
all one reconcile away from the same loop.
`context_is_newer_than_image` decides to rebuild when the build context's
newest mtime is later than `podman image inspect .Created`. The rebuild
that follows is a full layer-cache hit, so podman reuses the identical
image and leaves .Created untouched — the condition that triggered the
rebuild is still true afterwards. The check cannot converge: it rebuilds
on every reconcile tick forever, burning CPU and churning the container.
It bites after any deploy that refreshes /opt/archipelago/docker/*, which
makes the contexts newer than the shipped images — so this is fleet-wide
on every OTA, not local to one node.
Fix: stamp the context mtime that was built into an image label and
compare against that instead. A label is part of the image config, so a
cache-hit build with a new value still produces a new image — the thing
being tested does change, and the comparison settles after exactly one
rebuild. Verified against real podman before writing it: two cache-hit
builds with different label values produced distinct image IDs
(6cfdbc9bcd3e vs a9b10eb9a558), each carrying its stamp; the indexed
inspect format was checked against an image with real labels, and a
missing label prints empty (handled, along with "<no value>").
Images built before this carry no label and fall back to .Created, so
behaviour is unchanged for them and each self-heals on its first
reconcile after upgrade — nodes fix themselves rather than needing the
manual `podman build --no-cache` pass this needed by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both found while setting up the on-node test, and both fail silently in
the same direction — the gate reports success while protecting nothing,
which is the exact failure the module was written to prevent.
1. Loopback-pinned ports were skipped entirely.
`identity.rs` dropped any port whose manifest sets `bind: 127.0.0.1`,
reasoning that a loopback publish is not externally reachable. But
`listener.rs` requires loopback-pinning as the PRECONDITION for gating —
while an app holds 0.0.0.0:<port> the kernel will not let the gate bind
that port at all. So the two contradicted each other: pinning an app, the
one action that lets the gate take over, was also what removed it from
the gated set. Completing the entire migration would have gated nothing,
and GateStatus would have reported zero unprotected ports while doing it.
`bind` cannot carry this decision, because two unrelated intentions
produce an identical loopback publish: Bitcoin's RPC 8332 is pinned so
the LAN CANNOT reach it (fronting it would newly expose it on every host
address, behind a login but exposed where it deliberately was not),
whereas a migrated app is pinned precisely so the gate CAN. Inferring
from `bind` breaks one or the other, so the intent is now declared:
`PortAuth::Local` means the first case. The three ports that are
host-local by intent (bitcoin-core/knots 8332, aiui 5180 — all already
`bind: 127.0.0.1`) say so, and a loopback publish with `auth: session`
stays gated. A test pins that property.
2. The port map was never refreshed.
`AppGate::refresh()` existed, was documented as making catalog changes
apply without a restart, and was called by nothing. The map was built
once in `new()`, so an app installed while the daemon runs would never be
gated — and would never appear in `unprotected` either, so the node would
report itself fully enforced while serving a brand-new app to anyone who
asked. The sweep now refreshes before classifying.
Tests: 22/22 appgate, 73/73 archipelago-container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced again on this node today: with no session cookie, six app
ports answered HTTP 200 with their real UIs (18083 LND, 8334, 8175
Fedimint Guardian, 8336 FIPS Mesh, 8090, 7777), all bound 0.0.0.0 and so
served on every host address. Same bug class as the /lnd-connect-info
and /bitcoin-rpc/ leaks closed in v1.7.120, but across every app.
LAN, Tailscale, Tor and the FIPS mesh all converge on 127.0.0.1:<port>,
so this is one gate rather than four. It lives in the daemon rather than
a per-app sidecar (umbrel's app_proxy model): rootless, no extra
container per app, and it can reuse machinery that already exists.
It invents no authentication policy. verify_password, TOTP secret
decryption, verify_code with used-step replay protection, the session
store, and — importantly — the SAME LoginRateLimiter instance as the
JSON-RPC path, so an attacker cannot get a fresh budget of password
guesses by moving to an app port. Only the transport differs, an HTML
form instead of JSON-RPC, because a browser being sent to an app cannot
speak JSON-RPC.
2FA comes for free: a session still pending its TOTP step fails
validate(), so the gate rejects it without knowing what a second factor
is.
Details worth keeping:
- 401, not a redirect. A redirect to a login page is indistinguishable
from the app itself redirecting, and machine clients would follow it
and parse HTML as their API response.
- Cookie and Authorization are stripped before proxying. The app has no
use for the node session and must never be able to log or forward it.
- The challenge page names and pictures the app being opened, so the
visitor can confirm what they are authenticating to.
- device_tokens grew `apps: Option<Vec<String>>` and verify_for_app for
machine clients. None = node-wide, which every existing companion
token is; migrating them by guessing a scope would silently revoke
access nobody asked to revoke. An empty list is rejected rather than
minted, since it reads as unrestricted while authorising nothing.
The rollout is necessarily per-app and the gate is built to say so. A
container publishing 0.0.0.0:<port> claims every host address, so the
gate cannot bind that port until the app is pinned to bind: 127.0.0.1
and recreated — gate-first is impossible, and all-at-once would recreate
every container on a node simultaneously. Every port it cannot claim is
logged at warn each sweep and recorded in GateStatus::unprotected,
surfaced by security.app-gate-status. The failure mode being designed
against is a gate that binds nothing, logs at debug, and reports success
while every app stays exactly as open as before — worse than no gate,
because it stops anyone looking. Same reasoning that ruled out an
nft drop-in, whose absence is a silent no-op.
Not yet done: pinning the 39 gated ports to loopback, repointing
HiddenServicePort at the gate, and on-node verification.
Tests: 21/21 appgate, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for the app gate (item 1): before anything can enforce
authentication on app ports, the node has to know which ports are
*supposed* to be reachable without it.
`PortMapping` grows `auth` (PortAuth::Session | None, defaulting to
Session) and `auth_rationale`. The default is deliberately the protected
one. Every app port on this node answered with no credential at all over
LAN, Tailscale, Tor and the FIPS mesh alike — reproduced 2026-08-03 —
precisely because exposure was what you got by saying nothing. Inverting
the default means a new app is protected unless its manifest argues for
an exemption.
Validation makes the argument mandatory: `auth: none` without a
rationale is rejected, and so is a rationale without `auth: none` (that
combination means the author wrote an exemption and did not get one —
shipping it silently would leave them believing otherwise).
17 ports across 12 apps are declared exempt, each with its reason. They
are the ports that cannot sit behind an HTTP login page at all: Lightning
p2p (BOLT-8 noise), LND gRPC/REST and CLN gRPC (macaroon / mutual TLS —
Zeus and remote wallets dial these directly), Bitcoin p2p gossip,
electrum wire protocol, Wyoming voice streams, git-over-SSH, and the UDP
discovery protocols (mDNS, SSDP, STUN). Everything else — 39 published
ports — now defaults to gated.
Bitcoin's RPC 8332 is deliberately NOT exempted: it is already
`bind: 127.0.0.1`, so the gate never sees it, and claiming an exemption
it does not need would put a line in the audit list that means nothing.
If the loopback bind is ever dropped, it fails closed.
Two corpus tests keep this honest: every shipped manifest must parse
under the new rules, and the exempt set is pinned at 17 so any change to
the node's unauthenticated surface has to be a deliberate edit.
Tests: 73/73 archipelago-container, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Promotion to Trusted is a privilege escalation — a Trusted peer can read
node state, be deployed to, and is exempt from the `!= Untrusted` gates
federation/DWN/messaging use. It must therefore cost a fresh proof that
the person at the keyboard is the operator, not merely that a session
cookie exists. Same reasoning as node.rotate-identity and TOTP setup,
both of which already re-verify.
Both entry points are covered:
- `federation.invite` gates on the RESOLVED level, not on an explicit
request for Trusted: "Link Your Nodes" sends no `trust_level` at all
and falls through to the Trusted default. The invite is a bearer grant
of Trusted to whoever redeems it, so minting it IS the escalation.
Observer invites are untouched.
- `federation.set-trust` gates only when the peer is not already
Trusted, so the dropdown re-emitting its own value doesn't demand a
password for a no-op.
Demotion is deliberately NOT gated: making something less privileged
must never be harder than leaving it alone, or the safe action becomes
the inconvenient one.
The backend is the sole authority on what counts as an escalation — it
returns a `PASSWORD_REQUIRED:`-prefixed error and the UI prompts and
retries only on that, so the rule lives in exactly one place and the
frontend never pre-judges. TrustPasswordModal.vue (modelled on
RotateDidModal.vue) serves both flows. NodeDetailModal's select snaps
back to the node's real level on change, since a cancelled or failed
promotion would otherwise leave the dropdown displaying a level the node
never accepted.
The operator path stamps TrustSource::Manual; set_trust_level grew an
`Option<TrustSource>` so automatic adjustments (the discovery-handshake
demotion safety net) pass None and leave the recorded provenance alone
rather than laundering an uninvited-join peer into looking approved.
Follow-up, deliberately out of scope: `federation.join` also reaches
Trusted when redeeming someone else's Trusted invite, with no re-auth.
Tests: 44/44 federation, 79/79 rpc-client, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported: "peers seem to be slipping into trusted status somehow which is
absolutely terrible for security". Two independent fail-open paths, both
granting Trusted with no operator decision anywhere in the loop.
1. federation.peer-joined is UNAUTHENTICATED (middleware's no-session
list — federated peers call it over Tor without cookies) and reachable
on /rpc/v1, which is peer-allowed. It does verify an ed25519 signature,
but against THE PUBKEY THE CALLER SUPPLIED, so it proves the caller
holds its own key and nothing about whether we ever invited it. A join
presenting no invite_token fell through to
None => TrustLevel::Trusted.min(claimed_trust)
and claimed_trust itself defaults to Trusted when the field is absent.
So anything able to reach the node could generate a keypair, omit the
token, and be recorded as Trusted. Now capped at Observer: an invite
WE minted is the only path to Trusted. `min` is kept so a peer's own
lower claim is still honoured — this can only ever reduce trust.
2. merge_transitive_peers added every peer advertised by a Trusted source
as Trusted. That makes trust viral rather than transitive-by-one-hop:
the merged node is itself synced with, its peers merged in turn, so a
single invite anywhere in the graph eventually marked the entire graph
Trusted on every node. Now Observer — which is what this feature's own
spec always said. NodeStateSnapshot.federated_peers is documented as
"adds them as Observers on her side… doesn't auto-promote Observer-via-
Bob to Trusted". The code contradicted the comment directly above it.
Observer is deliberate rather than Untrusted: the merge exists for
routing, and Observer still passes the `!= Untrusted` gates that
federation, DWN and messaging actually check, so a legacy peer degrades
instead of breaking. Per the operator's decision, existing peers are NOT
auto-demoted — silently rewriting live trust relationships across the
fleet would be worse than the bug.
Instead they are made auditable: FederatedNode.trust_source records WHY a
level was granted (invite | uninvited-join | transitive-merge | manual).
It deliberately has no default provenance — None means "recorded before
this existed", which is exactly the population worth reviewing.
The one failing test was asserting the vulnerable behaviour
(merge_transitive_peers_skips_source_and_local_node expected Trusted); it
now asserts the security property and says why, so the escalation cannot
be reintroduced by making a test go green.
Verified: 42/42 federation tests, cargo check --all-targets clean.
Still open, tracked in .planning/RELEASE-1.7.121-TASKS.md: surface
trust_source in the UI, and require the node password to grant Trusted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
create-release.sh builds and commits the manifest BEFORE the signing
step, so the release commit carried an UNSIGNED manifest. Nodes fetch
releases/manifest.json from branch main and refuse to auto-apply an
unsigned one, so publishing without this would have shipped an OTA the
fleet silently declines.
Signature verified against the pinned release root before committing:
signed_by did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur
Cargo.lock carries the 1.7.120-alpha version bump from the release build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate's cargo-fmt stage failed on my own additions — the
tests in message_types.rs and lnd/info.rs and the reconcile branch in
prod_orchestrator.rs were written programmatically and never passed
through rustfmt. Formatting only; rustfmt is semantics-preserving and
the gate re-runs the suites before building.
Caught by the gate rather than in review, which is the gate working. Also
a reminder that a piped command's exit code is the pipe's, not the
script's: the task notification reported success while the log said
CREATE_RELEASE_EXIT=1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enabling half of "Bitcoin Core has no Tor proxy at all", handed over from
the app-UI work. Core reported `onion reachable=False, proxy=''` with all
11 peers on clearnet, and the reason was not a missing bitcoind flag: the
container sits on the archy-net bridge (10.89.0.0/24 here), so
127.0.0.1:9050 inside it is its OWN loopback. The host's Tor was
genuinely unreachable, and no flag on bitcoind could have fixed that
alone.
torrc now binds a second SOCKS listener on the archy-net gateway.
The gateway is DERIVED at runtime via `podman network inspect`, never
hardcoded: archy-net is created without an explicit subnet, so podman
allocates one. It is 10.89.0.0/24 on this node with no guarantee of that
elsewhere, and a hardcoded guess would fail silently — binding SOCKS to
an address no container can reach, which looks identical to working.
Two deliberate safety properties:
- FAIL CLOSED. If archy-net is absent or its inspect output does not
parse, no second listener is emitted and SOCKS stays loopback-only. An
exposure boundary is not something to widen on a guess.
- 127.0.0.1 is accepted FIRST in the SocksPolicy. SocksPolicy applies to
every SocksPort, so an accept-list naming only the bridge subnet would
have locked the daemon out of its own loopback SOCKS — breaking the
node's Tor usage in a way that looks nothing like "we added a
listener". The list is accept-loopback, accept-subnet, reject *.
This widens Tor SOCKS from loopback-only to the archy-net subnet, which
is a real change to the node's exposure surface and was explicitly
approved by the operator rather than assumed. Inbound onion for Core
remains impossible without reversing the deliberate "ControlPort disabled
for security" decision — this is outbound only, and the node stays
unlisted on Tor.
Not yet wired: bitcoind still has no -onion flag, because the operator
wants network mode to be a UI setting with Tor rather than clearnet as
the default. Hardcoding the flag in the three places that currently
define bitcoind's arguments would be the wrong shape for that, so it is
deferred to the settings work rather than done twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 3, completing 01-04.
mesh.lightning-peers returns the peers that have advertised a Lightning
URI: filtered, deduplicated, deterministically ordered, and an empty
array rather than an error when nobody has — "nobody yet" is a normal
state on a fresh node, not a fault.
mesh.send-lightning-info advertises this node's own URI to ONE chosen
peer. There is deliberately no broadcast form: this discloses the node's
payment endpoint, and who learns it is the operator's choice rather than
a side effect of being in radio range (T-01-13). It refuses to send when
LND advertises no URI, instead of sending an empty one a peer would
store as an undialable target.
The list-building and target-parsing logic is extracted into pure
functions because this file has no handler test harness and the
handlers need a live mesh service. That keeps the three contracts that
actually matter provable rather than merely readable:
- dedup is keyed on identity_pubkey_hex() — the AUTHENTICATING key,
lowercased — never the firmware routing key, so a radio contact and
its federation twin collapse to one entry (T-01-11)
- "newest advertisement wins" compares PARSED RFC3339 timestamps, not
strings: 09:30-01:00 is later than 10:00Z while sorting earlier as
text, and there is a test that fails if that is ever string-compared
- ordering is name-then-contact_id and asserted byte-identical across
eight rotations of the input, because a HashMap's iteration order is
not stable and a picker that reshuffles between reads means an
operator can click a different node than the one they aimed at
The peer allow-list is untouched: server.rs has an empty diff and
is_peer_allowed_path still occurs 13 times (T-01-15).
Verified: cargo test -p archipelago 1087 passed / 0 failed; clippy
--all-targets clean in every touched module (two useless_format lints in
the new test code fixed, not waived).
The SUMMARY records one deviation honestly: Task 1's tests were written
alongside its implementation rather than before, so no pre-implementation
failing output exists. A mutation test was run in its place — disabling
the pubkey validation fails 3 of the 5 tests — which proves the
assertions bind, and the mutation was reverted and verified gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tasks 1 and 2 of 01-04. This node's own shareable URI, and a mesh
message a peer uses to advertise theirs.
lnd.getinfo now deserializes identity_pubkey and uris, which its
response struct simply did not declare before (RESEARCH.md Pitfall 5).
The identity mapping is split into a pure map_identity() so it is
testable without a live LND. A pubkey that is not 66 hex characters maps
to None rather than being forwarded: the same rule lnd.openchannel
enforces, applied where the operator is reading their own node's
identity instead of at the moment they try to open a channel. An absent
field yields an honest absence — never a fabricated or placeholder
identity.
MeshMessageType::LightningInfo = 26 is additive on a wire format shared
with every fleet node: 26 was unused, so a peer that predates this fails
to decode it rather than mis-decoding it as something else. Its payload
is deliberately two fields — this rides LoRa, where every byte is paid
for on air, and the optional alias is skip_serializing_if so an absent
one costs nothing (asserted, not assumed).
is_valid_lightning_uri() validates before anything is stored, because
this is unauthenticated RF input: 66-hex pubkey, non-empty host, optional
numeric :port, exactly one '@'. It deliberately does NOT resolve or dial
the host — that would turn a received advertisement into an outbound
connection an attacker chose.
Two preservation hazards found while wiring MeshPeer.lightning_uri, both
of which would have silently emptied the picker:
- decode.rs's identity-advert path does a WHOLESALE insert, preserving
only advert_name and lat/lon by hand. Reticulum re-emits identity
adverts every announce tick, so a stored URI would have been wiped
about once a minute. Now preserved, alongside the same guard the name
and position already had.
- session.rs's refresh_contacts and mod.rs's federation seeding rebuild
the peer record wholesale too. Neither carries a Lightning datum, so
both now carry the previous value forward rather than nulling it.
A malformed inbound URI is rejected before the write, leaving any
previously stored good URI intact — otherwise anyone in range could
blank out a real peer's picker entry (T-01-12).
Verified: 5/5 new lnd::info tests, 18/18 mesh::message_types (5 new),
cargo check --all-targets clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things needed for the new UI to actually reach users.
The rendered nginx.conf served index.html with only ETag/Last-Modified and
no Cache-Control, so browsers applied heuristic caching to it. Confirmed on
archi-dev-box: after rebuilding and recreating the container, :8334 and
/app/bitcoin-ui/ both served the new markup immediately, but the app iframe
in the main UI kept showing the previous UI until a hard refresh.
docker/lnd-ui/nginx.conf has always carried this header, which is why only
bitcoin-ui showed the stale copy. Using "no-cache" (revalidate) rather than
"no-store" keeps the ETag doing its job when nothing has changed.
Validated by mounting the rendered config into a throwaway container from
the built image and running nginx -t. (An earlier attempt to test it inside
the running container was meaningless — conf.d/default.conf is a read-only
bind mount, so the copy failed and nginx -t just re-checked the original.)
The 8 container::bitcoin_ui tests still pass; their assertions cover the
placeholder, the 8332 proxy_pass and the listen directive, none of which
this touches.
BITCOIN_UI_IMAGE was still pinned to 1.7.84-alpha, so a fresh install would
pull a bitcoin-ui from many releases ago regardless of what the OTA ships —
first-boot-containers.sh tries the registry image before building from
source. Bumped to 1.7.119-alpha, matching the current release, and the
image is pushed under that tag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while VERIFYING a05956c4 on archi-dev-box rather than assuming it.
GET /lnd-connect-info is correctly 401 with no cookies over the LAN
address. But POST /bitcoin-rpc/ on :8334 still answered an
unauthenticated caller with a real block height, and still carried
`Access-Control-Allow-Origin: *`. The node looked patched. Half of it
was not.
The rendered /var/lib/archipelago/bitcoin-ui/nginx.conf was dated
2026-06-30 — the pre-fix version — even though the running binary
carries the new template. a05956c4's commit message claimed the
template "is re-rendered on every reconcile pass, so this ships
atomically with the binary". That is false in one specific state, and
this node was in it:
1. bitcoin-ui sits in the durable user-uninstalled marker.
2. reconcile returns on that marker BEFORE run_pre_start_hooks, which
is what renders the config.
3. The container keeps running regardless, because it is owned by
systemd via a Quadlet unit (archy-bitcoin-ui.service, active,
restarted 17:25 after the daemon restart) — not by this reconciler.
So a container systemd keeps alive, that the orchestrator has stopped
reconciling, never receives a config fix shipped inside the binary. An
OTA carrying a05956c4 would have silently failed to close this on every
node in that state, while the LND half closed correctly — the most
misleading possible outcome. archy-electrs-ui is in the same state on
this node, so it is not a one-app accident.
A container that is actually running is a live attack surface whatever a
marker says about it. Its security-relevant config is now reconciled
even behind the marker, and it is restarted so nginx actually loads it.
Deliberately narrow:
- Nothing is created, pulled, built, started or resurrected. The "must
stay removed" contract only ever gets weaker if a container is ALREADY
running, which by definition means it was never removed.
- A hook error is swallowed, not propagated: an app the user uninstalled
must not be able to fail the reconcile pass for everything after it.
- The pre-existing marker test still passes unchanged, which is what
proves the removal contract survived.
Verified: 11/11 reconcile tests and 9/9 bitcoin_ui tests pass, including
a new regression test that pins the whole chain — stale conf in, gate
present out, container restarted, nothing created.
No node has been touched. The live exposure on archi-dev-box stands
until this is deployed and the operator restarts archy-bitcoin-ui.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Layer (b) — core/clippy.toml bans rand::random and rand::thread_rng
crate-wide, each with a reason naming KEY-05 and pointing at the
evidence doc. No CI change was needed: the Rust job already runs
`cargo clippy --all-targets --all-features -- -D warnings` from core/,
so a disallowed_methods hit is already a build failure. --all-targets
covers tests deliberately — a fixture keeping the default is a template
for the next production call site.
Ordering was asserted before the file was written, not after: the
residual count of unmigrated call sites is 0, so this cannot turn CI red
for other agents on this shared tree.
Layer (c) — core/deny.toml makes the rand major split change-detecting:
global multiple-versions = "allow", a per-crate deny-multiple-versions
for rand, and a dated grandfather skip pinning =0.9.2 exactly. The tree
as it stands passes; a third version or a change to either member fails.
Both gates were OBSERVED working, not assumed:
- Reintroducing one banned call produced the disallowed_methods error
with the reason text reaching the developer at the failure point;
reverting returned the residual count to 0.
- `cargo deny check bans` exits 0 as-is. Removing the grandfather entry
made it exit 2 and print both dependency trees, independently
confirming F-07's account of where each rand version comes from.
Restored, it exits 0 again.
Policy (checkpoint Task 5, human-approved): bans-only. The advisories
gate is NOT enabled — it fails builds when a new CVE is published
against an existing dep with no local change, which on a tree where
several agents push continuously would block everyone at an arbitrary
hour, with remediation often meaning a bump to an exactly-pinned crypto
dependency. No break-glass procedure exists. F-07's advisory half stays
OPEN and is recorded as such.
cargo-deny is pinned to 0.20.2 and installed from crates.io rather than
via EmbarkStudios/cargo-deny-action, because that action exposes no
input to pin the tool version — an unpinned supply-chain checker would
reintroduce, at the CI layer, the exact "backend fixed by configuration
rather than stated" shape this plan exists to remove. crates.io is also
the source vetted at the Task 5 legitimacy gate (EmbarkStudios, repo
resolves, ~4.79M downloads).
RECORDED HONESTLY: layer (b)'s gate is live but not yet EFFECTIVE. The
tree carries 42 pre-existing clippy warnings — unused imports, dead
code, ~39 style lints — that are already errors under -D warnings, so
that CI step cannot pass today for reasons unrelated to KEY-05. Until a
dedicated lint-clearing pass lands, a new banned RNG call would be one
error among many rather than a distinctive build-stopper. Pre-existing
and out of scope; clearing it right before an OTA would be poor
sequencing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
