The release tarball bakes demo/aiui in for fresh installs; the previous dist
predated the phase-13 security waves (S1-S7, W1.4) and would have shipped a
pre-hardening bundle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Quadlet renders --rm, so every stop DELETES the container: a scan during
the stop->start window legitimately sees nothing where an installed app
lives. The state layer treated that as the app not existing — after three
absent scans it removed the entry entirely, so dashboard tiles vanished
mid-restart, gate waits read 'absent' (grafana, at load ~2, ruling out
contention), and under daemon-restart churn the FIRST scan could publish
a completely empty My Apps map (observed twice on 2026-08-09).
Two guards, both keyed to the durable installed-apps.json registry:
- Eviction: an id the registry says is installed, and the user has not
uninstalled, is held as Stopped (health/exit cleared) instead of being
removed. The next scan that sees the container restores live state;
desired-state recovery still recreates genuinely lost containers; a
deliberate uninstall clears the registry first and still disappears.
- Empty scans: the first-scan exemption no longer applies when the
registry is non-empty — better to keep saying "scanning" than "empty".
Verified: scan-merge/absence/registry test filters 34/34.
Also carries the v1.7.127-alpha changelog (user-benefit entries for the
Tor self-heal, trusted invites, the three-layer mempool fix, vanishing
apps, the Bitcoin version pin, and the smaller UI fixes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An OTA updates the binary and web assets — not /etc/nginx, and not
/opt/archipelago/scripts. Auditing the 2026-08-09 fixes' delivery paths
found two that would silently reach nobody already installed:
- scripts/tor-helper.sh (reset-failed + truthful restart result) shipped
only via ISO builds and manual deploys. Now embedded via include_str!
like the doctor script, staged and installed at boot when the on-disk
copy differs.
- The /app/mempool/ nginx Upgrade/Connection headers existed only in repo
snippet sources consumed at image build time. A boot repair now
idempotently patches any mempool location block missing them — in the
live vhost (archipelago-http, the one sites-enabled actually links to),
the legacy file, and the installed snippet — and reloads nginx once.
Without this, every fleet node's mempool UI keeps loading-but-never-
connecting after the OTA that supposedly fixed it.
Both are non-fatal boot repairs in the existing bootstrap chain, no-ops
when everything is already current.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sticky tab header painted bg-black/70 across the modal — reported as
a black container where the design is transparent glass. The backdrop
blur alone keeps pinned tabs legible over rows scrolling beneath; the
tint drops to the standard bg-white/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate accepted a client's websocket upgrade (.with_upgrades() on the
serving side) but proxy_to_app forwarded every request with a plain
hyper::Client, which completes the app's 101 handshake and then drops the
upgraded connection. Result: any ws-driven app behind a gated port loads
its page fine and then dies with close code 1006 on every connect —
mempool's entire UI is such an app, and the operator's browser console
was the only place the failure was visible. This was the THIRD layer of
the same outage (electrumx sync, then the nginx proxy path, now the app
tile's gated-port path — each fix exposed the next).
On an Upgrade request the gate now takes the client's OnUpgrade handle
from the request extensions, hands the handshake to the app with the same
sanitized headers (cookie/authorization stripping applies unchanged), and
on a 101 bridges the two upgraded connections with copy_bidirectional.
Non-101 upstream answers pass through so auth failures stay visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's /app/mempool/ proxy forwarded plain HTTP only: no
Upgrade/Connection headers, so the browser's /api/v1/ws handshake reached
mempool-api as a bare GET and Express 404'd it. The mempool page loaded
fine and every REST probe was green while the user saw a UI that never
connects — the backend was fully healthy and completely unreachable in
the only way that matters to the page.
This hid behind the electrumx initial-sync outage: once sync finished and
the API answered, "mempool works" was declared from REST checks while the
websocket path stayed dead. Reported by the operator from the browser,
which is the only place it was visible.
Adds Upgrade/$http_upgrade + Connection "upgrade" to the /app/mempool/
location in both shipped sources (image-recipe snippet + scripts copy).
Live-verified on the dev node: ws through the proxy now answers 101; the
control probe without upgrade headers reproduces the 404 signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full project build (vue-tsc -b, which also checks test files) caught
two default-ServerInfo constructors the scoped typecheck missed. Rather
than adding the field to both, the type now tells the truth for a
mixed-version fleet: daemons older than 2026-08-09 omit the field, so
consumers must treat absent as "unknown" and only === true as connected —
which is exactly how Home.vue already reads it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Home's Network card computed "Connected" from server-info's tor-address —
a string read from the hidden-service hostname file, which persists on
disk however dead the daemon is. Three fleet nodes ran with Tor down for
days while their dashboards said Connected; that indicator is why nobody
noticed. Server.vue's Tor label had the same flaw one step removed,
inferring "running" from services having onion_address values while the
correct signal (torDaemonRunning, backed by a real socket probe) sat
computed and unused twelve lines above.
Both now read liveness signals only: Home reads the new tor-running field
(server-side connect to 127.0.0.1:9050), Server.vue uses
torDaemonRunning. The address remains displayed as what it is — an
address — never as proof of life.
Verified: vue-tsc clean; rpc-client suite 79/79.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three of five reachable fleet nodes had Tor completely dead — Home said
"Connected" on all of them — and shipping the generator fix alone would
have repaired none of them. Two additions close that loop.
heal_on_boot (wired into the bootstrap repair chain): regenerate torrc
from current config with the fixed generator, and apply-and-restart ONLY
if the live file drifted or Tor is not answering on 9050. A healthy node
is left untouched. Without this, regenerate_torrc runs only from the Tor
RPC handlers and package install, so a node carrying a poisoned torrc
keeps it until someone happens to toggle a Tor setting — and worse, the
still-running OLD binary re-poisons on any such toggle: observed live on
the dev node at 07:20, when the running daemon rewrote torrc with the
unbindable gateway line hours after it had been hand-fixed. The heal
makes the fix self-applying on every restart, i.e. the OTA itself.
ServerInfo gains tor-running, populated by a real connect to
127.0.0.1:9050 each state refresh. tor-address is read from the
hidden-service hostname file, which OUTLIVES a dead daemon — it is a
configuration artifact, and the dashboard treating it as liveness is
precisely why three dead nodes showed "Connected" for days. Liveness now
comes only from the probe; the address stays a separate fact.
The heal reports the truth: it re-probes after restarting and warns if
Tor still is not answering, rather than assuming success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No node in the fleet could mint a Trusted federation invite or promote a
peer to Trusted from the dashboard. The cause is one case comparison.
Granting Trusted requires the node password; when none is supplied the
handler bails with "PASSWORD_REQUIRED: node password required to grant
Trusted" — a sentinel the frontend machine-reads (isPasswordRequired
checks includes('PASSWORD_REQUIRED')) to know it should open the password
prompt. sanitize_error_message allowlists "Password", but starts_with is
case-sensitive, so the ALL-CAPS sentinel failed the allowlist and
collapsed into "Operation failed. Check server logs for details." The
frontend never saw the sentinel, the prompt never opened, and the flow
died with a generic error on every node.
Reproduced via direct RPC: federation.invite {trust_level:"trusted"}
without a password returned the generic message; the same call WITH the
password minted a code fine — the machinery was healthy, only the signal
was being eaten.
"Tor address not available. Tor may not be running." — the invite
handler's other user-actionable precondition — was masked the same way
and is allowlisted too.
Regression tests pin both passing through verbatim, and
internal_errors_stay_generic still passes, so the generic masking of real
internals is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All ten AI data-access categories offer a context toggle, but only five
(Apps, System, Network, Media, Bitcoin) have any tool behind them — Wallet,
Files, Search, Local AI and Notes grant the assistant visibility with
nothing to act on. The Settings copy implied actions that do not exist.
Operator decision (2026-08-09): ship honest copy now, build the tools as
ordinary backlog. Each of the five carries a contextOnly flag, a "context
only" badge, and an appended description line saying the assistant can see
this but can't take actions with it yet. Drop the flag per category as
real tools land.
Verified: vue-tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two problems with running the gate against a mid-initial-sync electrumx:
1. The (now honest) protocol probe can only fail — ElectrumX serves no
sessions until it has caught up to its daemon, so the failure names a
state nobody can act on.
2. Worse, the destructive stop/start/restart tests actively destroy sync
progress: electrumx flushes its DB cache at 1GB, i.e. rarely, and every
restart discards all unflushed work back to the last flush. This node
spent 8d14h in initial sync largely because gate runs and reboots kept
taking hours of progress away — it restarted at 06:16 and resumed from
959,774, the same height it had reported hours earlier.
The suite now detects initial sync POSITIVELY — a fresh (<30 min)
"our height: N daemon: M" line from electrumx's own log, gap > 10 — and
skips the probe and the four destructive tests with the gap named:
# skip electrumx initial sync in progress (1672 blocks behind) — ...
This is not the container-absent skip trap fixed earlier: absence of the
log line means "unknown" and the tests run and fail honestly. Validated
against the live mid-sync node: all four guards fired with the real gap;
on a synced node the line shows gap 0-1 and everything runs.
Unblocks the release gate from waiting hours on a sync it was itself
prolonging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
changeTrust() wrote failures into `error`, which renders inside <NodeList>
— page content that sits BEHIND the open NodeDetailModal. So when
promoting a peer to Trusted failed, the operator saw the dropdown snap
back and nothing else: the error banner was painted on a surface the
modal was covering. Reported as "the error doesn't show in the modal",
2026-08-09.
NodeDetailModal gains an actionError prop rendered directly under the
Trust Level control, and Federation.vue routes failures there whenever
that node's modal is open (clearing on close and on the next attempt).
The page banner still serves failures with no modal in the way, and the
password-retry path keeps its own in-modal error as before.
Verified: vue-tsc clean; federation component tests 5/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's "Restart Tor" button dispatched to this helper and always
got {"ok":true} back. Two real defects, in order of importance — and one
disproved theory, recorded so nobody re-chases it:
1. reset-failed was missing. Once tor@default fails enough times systemd
latches "Start request repeated too quickly" and refuses to start it at
all; a plain restart is then a no-op no matter which unit you name.
All three fleet nodes found dead on 2026-08-09 were in exactly that
state, which is why the button appeared to do nothing.
2. The result was unconditional. The write-torrc branch waited up to 30s
for SOCKS and then ignored the outcome; the restart branch slept 3s and
claimed success. The UI reported "restarted" over a dead daemon.
Disproved: this was NOT wrong-unit targeting. `systemctl restart tor`
does propagate to tor@default — measured on austin-sapien, MainPID
changed. tor@default is still addressed explicitly because it is the
unit carrying the failed state worth resetting.
restart_tor_daemon() now: reset-failed, restart tor@default (fall back to
tor on single-instance installs), wait up to 30s for SOCKS on 9050, and
return {"ok":false,"error":...} pointing at journalctl when it never
comes up. Callers may no longer report success without a live SOCKS port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
regenerate_torrc widened SOCKS to the archy-net gateway whenever `podman
network inspect archy-net` reported one. Podman reporting a gateway is not
proof the host can bind it: under rootless podman the bridge and its gateway
live inside a network namespace and never appear on a host interface.
Tor binds listeners at STARTUP, not at config-check time — `--verify-config`
passes happily — so the failure lands as a refusal to start. austin-sapien,
2026-08-09:
[warn] Could not bind to 10.89.0.1:9050: Cannot assign requested address
[warn] Failed to parse/validate config: Failed to bind one of the listener ports.
[err] Reading config failed--see warnings above.
Tor then died completely — loopback SOCKS and every hidden service with it —
having run 3d 18h before something restarted it. The node's Home tab still
read "connected" because tor.service is Debian's multi-instance MASTER unit,
which is active(exited) by design; the real instance, tor@default, was failed.
Whether a node is affected is pure timing: archi-dev-box's torrc has only
`SocksPort 9050` because archy-net was not discoverable when its torrc was
last regenerated, so it fell into the loopback branch and Tor starts fine.
Nodes whose torrc was generated while archy-net was visible are one Tor
restart away from the same outage.
The gateway is now probed the way Tor will use it — bind an ephemeral port on
that exact address — and a failure falls through to the existing loopback-only
branch, which was already written and commented "fail closed". Widening SOCKS
must never be able to take the whole daemon down.
Verified on this host: 127.0.0.1 binds, 10.89.0.1 returns EADDRNOTAVAIL.
Tor restored on austin-sapien and confirmed end to end — bootstrapped 100%,
and a request through the proxy returns {"IsTor":true}.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both electrumx tests did socket.create_connection(("127.0.0.1", 50001)) and
called it proof the service was up. It is not: that is the HOST-published
port, and podman's port forwarder completes the TCP handshake even when
nothing inside the container is listening.
The cost was real. Both tests were green while mempool-api sat in a hard
ECONNREFUSED loop against electrumx:50001 from inside archy-net — the
operator's "mempool doesn't connect to bitcoin". Measured side by side:
127.0.0.1:50001 from the host OPEN, electrumx:50001 from a sibling container
REFUSED.
The cause is not a misconfiguration — mempool-api's CORE_RPC_HOST,
ELECTRUM_HOST and ports are all correct. electrumx has not finished its
initial sync (DB height 959,774 vs chain tip 961,706, ~1,932 blocks, 8d 14h
elapsed) and ElectrumX does not serve sessions until it has caught up.
Now both tests do a server.version round-trip and require a JSON result, so a
forwarder that accepts-and-drops fails. Verified against the live node: the
new probe FAILS today where the old one passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moves the default off 29.3.knots20260508, which applies the BIP110/RDTS
network upgrade and HALTS until an operator sets consensusrules=rdts. A node
running it sits frozen — 100.64.204.114 is stuck at block 961,692 with blocks
AND headers static, 11 peers and no pruning, while reporting
initialblockdownload=false and verificationprogress 0.99998, i.e. "synced".
The nodes on 20260210 are at the tip.
The old catalog's default was {"version":"latest", image: ...20260508} while
the manifest's :latest tag resolves to 20260210 in the registry, so a fresh
install and a plain pull produced different daemons. That is almost certainly
how one node ended up on the RDTS build and its siblings did not.
20260508 stays selectable — adopting RDTS remains possible, just deliberate.
Diff against the previously-signed catalog is exactly one app, all four
fields: version, image, default and the embedded manifest. Registry hosts
unchanged and inside the trust floor; verified by
check-catalog-registry-trust.py and `ceremony verify`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sign-catalog.sh refused to sign a freshly regenerated catalog:
REFUSING: the catalog names registry hosts the deployed fleet does not
trust. source.archipelago-foundation.org — 78 image refs
It was right, and the check earned its place. apps/*/manifest.yml moved to
the public domain in 8e814ca0, but releases/registry-trust-floor.json still
lists only docker.io, ghcr.io, localhost and the OVH host — the migration is
ship-binary -> confirm-fleet -> promote-floor -> regenerate, and only the
first move happened. A full regen therefore produced a catalog naming a host
no deployed binary trusts, which would have failed every install in the field
with "not from a trusted registry". Verifying that an image PULLS from the
domain (it does, including from a fleet node) does not verify that shipped
binaries ACCEPT it — different checks.
Two fixes, no manifest churn:
- image-versions.sh honours a caller-supplied ARCHY_REGISTRY instead of
overwriting it, so generation can target a host inside the floor.
- the generator retargets OUR registry host inside embedded manifests to
whatever it is generating against, leaving docker.io/ghcr.io refs alone.
REGISTRY moves above the embed block, which now depends on it.
The repo keeps naming the public domain; only the generated artifact is
pinned to what the fleet can actually use. When the floor is promoted,
generating against the domain becomes a no-op here.
Verified: check-catalog-registry-trust.py passes, 11 docker.io/ghcr.io refs
untouched, and the diff against the signed catalog is exactly one app —
bitcoin-knots, :latest/20260508 -> :29.3.knots20260210 across all four fields.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalog's default version was {"version": "latest", image:
bitcoin-knots:29.3.knots20260508, default: true}, so a fresh install — or
anyone selecting "latest" — silently got the BIP110/RDTS build. Meanwhile
the manifest's own :latest tag resolves to 29.3.knots20260210 in the
registry, so the two paths installed different daemons.
That matters because 20260508 does not merely enable RDTS, it HALTS until
an operator sets consensusrules=rdts. Node 100.64.204.114 runs it and is
frozen at block 961,692 — blocks AND headers static across 30s, 11 peers,
unpruned — while reporting initialblockdownload=false and
verificationprogress 0.99998, i.e. "synced". The two nodes on 20260210 sit
at the tip, 961,706. Re-pointing a tag could have done that to every node
at once.
Pinned in all three places that would otherwise re-introduce a moving tag:
the manifest, image-versions.sh, and the catalog VERSIONS table. The
"latest" pseudo-version is removed outright — a moving default is not
appropriate for a consensus-critical app — and the default is now the
explicit non-RDTS 29.3.knots20260210. 20260508 remains selectable, so
adopting RDTS stays possible but becomes deliberate.
Verified: both tags' daemons read via `bitcoind --version` — :latest and
:29.3.knots20260210 are both v29.3.knots20260210, so this pin is
behaviour-neutral on nodes already running latest.
NOT DONE HERE: releases/app-catalog.json still carries the old signed
entry. It must be regenerated and re-signed by the operator before it
takes effect — catalog entries override on-disk manifests (origin-wins).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two fixes from the first full 125-test run (121 ok / 4 not ok).
1. The Quadlet unit test still asserted Restart=on-failure for backends,
which 8908fb4f deliberately changed. All 32 units on the node are
Restart=always; only archy-btcpay-db showed up because `fail` aborts at
the first offender. The old rationale — "backends need on-failure so an
operator `systemctl stop` stays stopped" — was wrong twice over:
systemd never applies Restart= to a unit stopped via `systemctl stop`,
and because quadlet renders --rm, a cleanly-exited container is deleted
and on-failure never brings it back (bitcoind exits 0 on SIGTERM, so
backends vanished). Verified on-device before that change: `podman stop
bitcoin-knots` came back in 12s, a dashboard stop stayed stopped 90s.
Now asserts Restart=always, with the corrected reasoning in-place.
2. all-apps-lifecycle cycled ten apps back-to-back and manufactured the
contention that then failed later tests: load went 10.8 at preflight to
17.6 mid-loop, and btcpay's recovery stretched from 52s on a quiet box
to 216s and then 512s, defeating waits of 180s and 300s. Widening waits
cannot win when the run sets the load they must survive, so the loop now
drains between apps — waits for load1 under 2x nproc, capped at 90s.
ARCHY_APP_SETTLE_SECS=0 disables; ARCHY_APP_SETTLE_LOAD overrides.
Verified: run.sh use-quadlet-backends-install → 6/6 (was 1 failure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both failed a gate run on a healthy node.
bitcoin-receive already tolerated WALLET_LOCKED, but its 180s window
starts when the TEST starts and the lnd restart that locks the wallet can
land partway in. On 2026-08-08 the restart hit 65s in and the wallet
unlocked at 2m25s (journal: lnd.service started 20:11:05, "wallet has
been unlocked without a time limit" 20:13:48) — 48s after the deadline
expired. The daemon's own unlock budget is ~10 min because opening the
channel and graph dbs takes minutes on a loaded box, so 180s was stricter
than the thing under test. Now 420s, ARCHY_LND_UNLOCK_SECS.
btcpay's start wait was 180s, but stopping btcpay DELETES the container
(quadlet renders --rm), so package.start is a full dotnet recreate rather
than a container start. Measured 52s on a quiet box; it exceeded 180s
during a gate run on the same node at load ~11. Now 300s,
ARCHY_BTCPAY_START_SECS.
Neither change masks a lifecycle fault: both paths were verified by hand
to complete correctly, just slower than the assertion allowed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop/start/restart loop carefully accumulates per-app failures into
$fails and prints them, but the three rpc_result calls were bare. Under
bats' errexit a bare call ends the test immediately, so the summary that
names the app never ran.
On 2026-08-08 that turned a single transient error into an unattributable
failure: the test died at package.stop with no indication which of the
ten targets was involved. It was mempool, and the identical call returned
{"status":"stopping"} by hand a few minutes later.
Each call now records <id>:<phase>-rpc and moves to the next app, so the
run reports what actually broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ui-coverage.bats and ui-probes.bash hardcoded https:// while the rest of
the harness builds URLs from ARCHY_SCHEME via lib/rpc.bash. That made the
suite unrunnable on a node serving the dashboard over http.
On archi-dev-box :443 is bound to the Tailscale / WireGuard / LAN
interface addresses but NOT to loopback, while :80 is bound on 0.0.0.0.
So five probes failed with "curl failed (network/timeout)" against
endpoints that were serving 200 the whole time — http://127.0.0.1/,
/catalog.json, /app/lnd/, /app/electrumx/ and /app/mempool/ all verified
200 by hand.
Default stays https, so nodes that already exercise the TLS path keep
doing so. Test titles drop the hardcoded scheme, since they no longer
describe which one ran.
Verified: ARCHY_SCHEME=http ./run.sh ui-coverage → 9/9 (1 skip),
previously 5 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gate run on a box at load ~14 failed five times over, and every failure
read "could not create a container" — searxng:start, package.start
btcpay-server, 3x electrumx — never a lifecycle fault. That sent two
separate sessions hunting a phantom host-wide cgroup failure. It was
contention.
Measured on the 4-core node: at load ~14 podman runs 9-16 processes deep
and healthchecks time out 3-8/min; at load ~3.7, podman ~1 and zero
timeouts. Restoring four containers whose HealthTimeout equals their
HealthInterval, unchanged, made the box *better* once load fell — so the
config is a latent hazard, not the cause here.
Preflight now checks, once, before iteration 1:
- aardvark-dns is singular (duplicates desync name resolution)
- load1 is under nproc+1, waiting up to 15 min for a spike to pass
- podman can actually create a container, 3/3
Deliberately NOT checked: the count of "Failed to create container" in
the journal. Those lines come from healthcheck exec churn and post-boot
settling, never reach 0 on a busy node, and gating on them would block
the gate forever. The probe proves creation positively instead.
Escape hatches: ARCHY_PREFLIGHT=0, ARCHY_MAX_LOAD, ARCHY_PREFLIGHT_SECS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An app whose container disappeared could be lost permanently. Desired-state
recovery decided whether to recreate it from running-containers.json — "what
was running at the last snapshot" — which is a different question and a
perishable answer: it records only what is running NOW, so an app that stays
down long enough simply ages out. Once out, boot's ExistingOnly mode will not
recreate it, because it cannot tell "installed and lost" from "merely
available in the catalog". Manifest still on disk, nothing to bring it back.
This is the second occurrence of one root cause. indeedhub-minio/-postgres
went permanently absent on one node (2026-08-06); the fix then was
`absent_stack_member_with_live_sibling`, which only rescues a stack member
that still has a living sibling. bitcoin-knots is standalone, so on
archi-dev-box (2026-08-08) it vanished, aged out, and stayed gone — LND
crash-looping on `lookup bitcoin-knots: no such host` for hours, electrumx
unable to reach its daemon, and an orphaned fedimint container waiting 30
hours for a host that no longer resolved. Recovering it took a manual
reinstall. This is the general fix the narrow one implied.
Installation is a DECISION, not a runtime observation, so it gets a record
that no amount of downtime erodes: installed-apps.json, written when an
install succeeds and cleared on uninstall, in the same breath as
mark_user_uninstalled — leaving a stale claim would let recovery recreate the
app that was just removed. It is the durable counterpart to the
user-uninstalled marker that already existed.
Safety, in order of how badly each could go wrong:
- Cannot resurrect a deliberate uninstall: user_uninstalled is checked
earlier in ensure_running_with_mode and returns before anything is created,
and uninstall clears this record too.
- Cannot install an app nobody asked for: only names in the record qualify,
and ExistingOnly's other guards are untouched.
- Cannot mislead a node upgrading into the feature: backfill seeds from
ADOPTED containers only — evidence that something is really there — skips
anything user-uninstalled, is additive so a momentarily-down app is never
dropped, and no-ops on an empty adoption list (podman unreachable must not
read as "nothing is installed").
Four tests, including the one that states the point: the record must outlive
a running-snapshot that has gone empty.
Container suite 221/221, crash_recovery 15/15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two operator reports on the same screen.
The section had no card. Every other Settings section wraps itself in
`glass-card px-6 py-6 mb-6` — AccountSection, AIDataAccessSection,
NodeCertificateSection, BackupSection, the lot — and this one rendered as
bare text on the page. Reported twice, because the wrapper lives in the new
component and nothing about adding `<LightningCredentialsSection />` to
SystemSection.vue's list tells you it is missing. Heading moved to h2/text-xl
to match its siblings. A test now asserts the card, so a third report is not
needed.
And rotating told the operator Lightning did not exist. Rotation restarts
LND, so `status.installed` reads false for a moment — and the template read
that literally: "Lightning is not set up on this node yet, so there are no
credentials to rotate. Install the Lightning app first." Seconds after
rotating. On a node with a working wallet. It also replaced the progress they
had every reason to be watching, on the one action that invalidates every
credential their wallet holds.
A container briefly absent is what rotating LOOKS like, not evidence
Lightning was never there. The not-installed message is now gated on
`!rotationInFlight`, which covers both `running: true` and the awaitUntil
window between asking for a rotation and the node reporting one — `installed`
can already be false in that gap, so gating on `running` alone would have
left the same hole. Mid-rotation with no status yet says "Rotating
credentials — Lightning is restarting" instead of falling through to a
details block with empty fields.
awaitUntil became a ref so the computed re-evaluates rather than holding a
stale value until some other reactive dependency happens to change.
Three tests: the card exists; a running rotation does not claim Lightning is
missing; and — the half that matters just as much — a node with genuinely no
Lightning still gets told there is nothing to rotate, so the fix has not
simply hidden a true statement. 16/16, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced live: the operator pressed install on ElectrumX shortly after
Bitcoin Knots started and got
Bitcoin RPC returned 500 Internal Server Error while checking pruning
status: {"error":{"code":-28,"message":"Verifying blocks…"}}
-28 is RPC_IN_WARMUP — bitcoind's normal path on every start, lasting
minutes on a large chainstate. check_bitcoin_pruning_compatibility retried
`for _ in 0..3` with a 2s sleep, so it gave the node about six seconds and
then reported warm-up as a hard failure. Any app requiring unpruned Bitcoin
was therefore uninstallable in the ordinary window after Bitcoin starts.
This is the likely mechanism behind the operator's "fedimint gateway
disappeared at 88% install": both fedimint apps declare a bitcoin-core
dependency and sit in exactly that window.
The same install path already knew better. wait_for_bitcoin_rpc_gate waits
180s precisely because getblockchaininfo answers during sync. This check
runs earlier and disagreed — one concern, two contradictory answers, in one
install. It now shares the budget, and a test asserts the two stay equal.
Only NOT-READY is waited out: -28 by code, plus the "loading block index" /
"verifying blocks" / "rewinding blocks" message shapes for a proxy that
rewrites the envelope. A genuine fault — bad auth, method not found,
unparseable body — still ends the loop on the first response, so a broken
RPC fails fast instead of burning 180s. Both directions are pinned by tests,
because being too loose here is as bad as being too strict.
Warm-up is announced ONCE to the install log, so a slow install reads as
"waiting for Bitcoin" rather than a stall. And the failure message now says
what to do — "Bitcoin is still starting up… wait until it reports it's
synced, then try again" — instead of pasting the raw JSON-RPC envelope,
which was accurate and useless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught on archi-dev-box within minutes of deploying 3ac59a73: the reaper
removed archy-bitcoin-ui and archy-lnd-ui, whose backends ARE installed.
archy-bitcoin-ui was gone for 36 minutes, until the operator reinstalled
bitcoin-knots and `reconcile` put the companion back.
Not a logic error — the arithmetic did what it was told. The inputs were
false. Both backends' containers were missing because of the clean-exit
vanishing bug (8908fb4f), and both had already aged out of
running-containers.json, which only ever records what is CURRENTLY RUNNING.
So the two signals `installed_app_ids` combines are not independent: one root
cause falsifies both simultaneously. ORPHAN_GRACE could not help either — the
condition was persistent, not transient, which is exactly the case the grace
period cannot distinguish.
The asymmetry decides it. An un-reaped orphan costs a stale UI tile. A
wrongly-reaped companion costs a working screen and turns one lost app into
two — the reaper amplifies the very failure it was meant to tidy up after.
`reap_orphans` and its tests stay, documented as NOT TO BE WIRED until a
durable record of "this app is installed" exists to drive it. Inferring
installation from runtime state cannot answer that question, however many
runtime signals are combined.
The provisioning half is untouched and is the actual fix for "fedimint
installs but does not work": driving `reconcile` from installed_app_ids means
a companion is never stood up for an app nobody installed, so no NEW orphans
appear. The one genuine orphan on this node (archy-fedimint-ui, for an app
never installed) was correctly removed before this change landed.
Unit tests passed the reaper because they verify the set arithmetic, not
whether the "installed" signal is truthful. Only the device could show that.
Container suite 221/221.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`sign_nip98` read the node's pubkey through `get_nostr_pubkey`, which goes
through `load_or_create_nostr_keys`. On a node with no Nostr identity that
does not fail — it GENERATES a keypair, writes the secret to disk, and signs
with it. So an IndeeHub auth header could quietly create a new node identity
as a side effect, then authenticate as a stranger holding a key nobody has
ever seen. The `.context("node has no Nostr identity")` guarding the call
could never fire, because the call could never fail that way.
`nostr_identity_exists` is the missing distinction: bootstrap may create,
but anything AUTHENTICATING as this node must prove the identity it already
has. sign_nip98 now gates on it and bails loudly.
Caught by `a_nip98_event_names_the_exact_url_and_method`, which asserts
exactly this ("must fail loudly rather than sign something empty") and had
been failing since the file landed in 58c759c1 — invisible because the
earlier runs on this branch filtered to `container::`.
Full bin suite 1381/1381.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AIUI has carried 353/356 as a known-open item (W1.7) for long enough that
the three reds were treated as background noise. All three were the tests
being wrong. 357/357 now.
`injects web results into system prompt when enabled` asserted
`body.webSearch === true`. The code deliberately sends false there:
`proxyWebSearch = webSearchEnabled && !clientSearchSucceeded`, so when the
client-side search has already run and injected its results into the system
prompt, asking the proxy to search again would be a second redundant search
on every turn. The assertion predates that change. Fixed, and the other half
of the contract added as its own case — zero client results must still ask
the proxy to search.
The two seed-songs failures ("extracts 10 songs", and the conversation
regression built on the same fixture) were one wrong number:
`expected: { songs: 10 }` against an assistantResponse containing exactly
six `song_ext` entries and ending coherently on Treefingers. Not truncated —
just miscounted. The extractor was returning the right answer the whole
time. Counted from the fixture rather than from intent, so the number now
describes the input instead of contradicting it.
Full branch state after the main merge: AIUI 357/357, neode-ui 972/972
across 114 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e36f36ee re-encoded the chat backdrop to webp (1052K -> 478K) to fix "the
background takes ages to load", but left the JPG in place. ChatPage.vue
references only `bg-intro-3.webp`, so the 1053K JPG has been shipping in
every AIUI bundle, every deploy and every ISO since, referenced by nothing.
Checked before removing, because the same filename IS live elsewhere:
`appgate/mod.rs` serves `bg-intro-3.jpg` as one of four LOGIN_BACKGROUNDS,
but from `/opt/archipelago/web-ui/assets/img` — the neode-ui copy, which is
untouched here. Only AIUI's duplicate goes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported: "`files` context request times out". sanitizeFiles makes three
sequential calls into the File Browser app — login, getUsage, listDirectory
— wrapped in a try/catch. A catch only sees a REJECTION. A socket that
connects and then says nothing leaves the promise pending forever, so
handleContextRequest never posts a `context:response` and the AIUI side sits
until its own bridge timeout instead. The File Browser is a plausible source
of exactly that: on this node `/app/filebrowser/api/resources/` does not even
route (404), and its session-cookie path is the subject of a separate open
bug.
The guard goes at handleContextRequest rather than inside sanitizeFiles, so
no category — present or future — can strand the bridge. `files` is merely
the one with three network hops today; sanitizeSystem is also async.
withTimeout resolves rather than rejects, because the caller's one job is to
always answer, and a rejection would just relocate the problem into a catch.
A late null is safe by the protocol's existing shape: the AIUI reader
already treats a response with no usable data as "nothing to show", the same
as an empty category.
Two tests: a never-settling File Browser still produces a
`context:response`, and a healthy category still returns real data rather
than being flattened to null. 27/27 contextBroker, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13-15's gate requires the DEPLOYED surface to be checked, not only the
source. That half is currently void: archi-dev-box runs a binary dated
2026-08-08 03:20 built from main, not this branch — no `app_uninstall` in
`strings`, and the ownership hooks chown unconditionally with no drift-gate
`stat` calls, so b9e64eb6/db8937f9/ca106c5a/b8869307 are all absent. Every
row of the acceptance table was verified against a binary the node no longer
runs, 417 commits back.
Also corrects row 2. The record captured scope `own` only, which cannot
discharge check 2's "real peer/owned files"; and the 2026-08-06 note saying
peers/owned "exist only in type signatures" is obsolete — 05b459a6 and
9abc1623 made requestArchyAllContent fetch all three scopes from init(),
deduped through one sink, which the operator confirmed on 2026-08-08 is the
intended auto-load-at-init behaviour. Check 2 is code-complete and
verification-pending, not unbuilt.
Records the ordered steps to actually close the phase, and the pre-deploy
orphan-companion evidence snapshot the reaper will consume.
The phase stays OPEN. Nothing here closes 13-15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box logged `reconcile failed app_id=btcpay-server error=chown
/var/lib/archipelago/postgres-btcpay failed with status exit status: 1`
while BTCPay was running and healthy and there was nothing to repair:
`find /var/lib/archipelago/postgres-btcpay ! -uid 100998` returns zero
files, and the identical command run by hand exits 0. The chown through
`sudo systemd-run` had simply failed once, and that transient failure
propagated out of the pre-start hook and took the app's entire reconcile
with it.
These hooks exist to repair OLD installs. On a healthy node the repair is
already a no-op, so its failure is not evidence of anything being wrong.
repair_dir_ownership folds the gate, the chown and the verdict into one
place: skip when ownership is already right, chown when it is not, and on a
failed chown RE-PROBE before deciding it matters. If the ownership is
correct anyway — a concurrent repair, or a transient sudo/systemd-run
failure on an already-correct tree — warn and continue. Only a chown that
fails AND leaves the ownership wrong is an error, which is the case the
loud failure was written for: a mis-owned volume the app genuinely cannot
open.
Replaces the three hand-rolled gate+chown+bail blocks in
ensure_btcpay_stack_dirs and the one in ensure_fedimint_dirs.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writing the first tests for this section found the bug they were written to look
for. `rotate()` started the poll, then the `load()` immediately behind it took a
status snapshot that did not yet carry `running: true` and cancelled the interval
— so the screen froze on the one action that most needs to show progress. The
operator has just invalidated every credential their wallet holds, the rotation
is genuinely running on the node, and the page tells them nothing is happening
until they reload it by hand.
It survived manual review because the backend flips `running` inside the same
critical section that accepts the request, so the happy path usually wins the
race. "Usually wins a race" is not a property to ship on a credential rotation.
Polling now continues for a bounded window after a request the node accepted,
and stops early as soon as `running` is observed. Bounded, so a request that was
accepted but never acted on stops polling rather than hammering the node.
12 component tests cover the states that carry consequences: the channel census
shown before the button is offered, the stale-BTCPay warning, the difference
between "BTCPay has no internal node" (silence — an absence, not a fault) and
"BTCPay's credential is dead" (a warning), the block on rotating while LND is
unreachable, both poll races above, and that an idle tab does not wake the node.
Verified: 12/12 new, 880/880 frontend tests, vue-tsc clean, and the rebuilt
bundle contains the new strings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint wait and the post-rotation verify shared one 15-minute budget. A
rotation that legitimately spent 14 of those minutes waiting for LND to mint a
fresh macaroon — normal on a loaded node, where opening channel.db/graph.db/
wallet.db alone has been measured at 2m38s — then had 60 seconds to confirm the
node identity and channel census came back, and would report FAILURE on a wallet
that was completely healthy.
That is the most alarming possible way to be wrong about someone's Lightning
node: it names a backup directory and tells them to investigate before retrying,
at the exact moment nothing is actually broken. Each wait now gets its own
budget. Waiting longer costs nothing here — the failure this step exists to catch
(changed identity, missing channels) is not time-sensitive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 2.
The `indeedhub` submodule points at a Gitea repo that is not public and carries
no known licence (the licence audit defers it: "partnership in place; license
the submodule before/at public release"). An outside developer running
`git clone --recursive` today either fails on auth or pulls unlicensed code —
a bad first five minutes with the project either way. It was never checked out
in this tree.
Removing it costs nothing, because nothing builds from it:
- `indeedhub-demo/Dockerfile` states in its own header "No submodule or local
source needed" and clones the public GitHub mirror instead.
- Every other `indeedhub/` reference in the tree is `apps/indeedhub/` — the app
package — which is a different path and untouched. The app itself ships as a
container image from the registry and is unaffected.
Kept `indeedhub-demo/` rather than dropping it as the plan suggested: it is a
working, self-contained demo build with no submodule dependency, which is
exactly the shape the rest of Phase 2 is moving toward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 2. Removed (~17 MB, 12.8k lines):
- `.githooks/pre-push` — the hook that re-committed the 27 MB companion APK on
every push, which the plan names as the root cause of the 5.5 GB history.
Verified inert first: `core.hooksPath` is unset, so it only ever ran for a dev
who opted in by hand.
- `neode-ui/dev-dist/` — generated vite-plugin-pwa output (a Workbox bundle),
tracked and not ignored. Added to .gitignore so it cannot come back.
- `Android/archipelago-0.3.0-debug.apk.zip` — 16 MB, stale, zero references.
- `RELEASE-NOTES-v1.0.0.md` — superseded by CHANGELOG.md.
- `docs/container-architecture.html` (311 KB) and the two generated archive
HTML artefacts, whose rows are removed from the archive index in the same
commit so the table doesn't point at deleted files.
THREE items the plan lists were verified and deliberately NOT deleted — the
plan is wrong about each, and following it literally would have lost content or
broken a build:
- `neode-ui/docs/GAMEPAD-NAV-MAP.md` is called "a duplicate of
docs/GAMEPAD-NAV.md". It is 660 lines against that file's 159 — four times the
content, not a copy. Needs a human read to decide what to keep.
- `Android/app/debug.keystore` is called "standard practice" to remove. This
repo deliberately commits it: `build.gradle.kts` sets
`storeFile = file("debug.keystore")` and `Android/.gitignore` carries an
explicit `!/app/debug.keystore`, with a comment explaining it exists so every
machine produces the same debug signing identity. Deleting it breaks Android
debug builds.
- The three "move to release assets" binaries are not a pure git operation —
two have live consumers. Detailed in the next message rather than guessed at.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Bitcoin Knots disappeared again, plus other apps." Root cause is a pairing,
not a single bug: quadlet renders `podman run ... --replace --rm`, so the
container is deleted the moment it stops, and from_manifest set
Restart=on-failure, which declines to restart after a CLEAN exit. bitcoind
exits 0 on SIGTERM. So any clean stop deleted the container AND left it
deleted — the app vanished from podman and from My Apps until a later
archipelago reconcile tick noticed and recreated it. That is the
"previously-running app has no container after boot — recreating
(desired-state recovery)" line, which fired for bitcoin-knots at 18:53,
19:57 and 20:39 and for electrumx at 19:57 and 20:42 on 2026-08-07.
A crash always self-healed: on-failure restarted the unit and podman run
recreated the container. Only a clean exit stranded it, which is why this
survived so long.
The justification for on-failure was wrong on systemd's own semantics. It
read "clean exits — e.g. operator-issued systemctl stop — stay stopped", but
Restart= is never consulted for a unit stopped via systemctl stop
(systemd.service(5)), and that is exactly how archipelago stops these apps
(prod_orchestrator -> stop_service_with_timeout). Always keeps the
stopped-stays-stopped behaviour and drops the failure mode.
Always also restores the premise of the Quadlet migration — systemd owns
supervision, so an app returns without archipelago alive to notice it left.
Checked before flipping: no manifest declares a one-shot container and there
is no manifest-level restart field, so nothing gets restart-looped.
Propagation to existing nodes is via sync_quadlet_unit's drift re-render,
which rewrites the unit and daemon-reloads WITHOUT restarting the service —
running containers are undisturbed and the new policy governs the next start.
OnFailure is kept as a deliberate opt-in with a note not to wire it back to
backends. Two tests now pin the new default and assert on-failure is absent
from a rendered backend unit.
Container suite 215/215.
NOTE FOR THE OPERATOR: this changes supervision semantics for every app on
the Quadlet canary path. Wants sign-off and a lifecycle-gate run before OTA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
installed_app_ids judged installation on live containers alone. Watched on
archi-dev-box within the hour: lnd read as ABSENT, then as EXISTS again.
Containers on this node come and go — the boot reconciler logs
"previously-running app has no container after boot — recreating" for
bitcoin-knots and electrumx repeatedly — so a momentary gap looked exactly
like a removal, and the reaper would have taken a healthy companion's unit
with it. ORPHAN_GRACE narrows that window but cannot close it: nothing
bounds how long a gap lasts.
An app now counts as installed if its container exists in any state OR its
container name is in the durable last-running snapshot. That snapshot is
what crash_recovery itself calls "installation evidence" and what
reconcile_all_with_mode already trusts to recreate a previously-running app
whose container vanished — the same signal, for the same reason, now shared
rather than reinvented.
Only fedimint is a true orphan on this box: it appears in no adoption list
and has no quadlet unit of its own. lnd is installed and merely flapping,
which is a separate bug.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both default update mirrors resolve to the SAME host — the primary by name over
HTTPS, the fallback by IP over plain HTTP — while SystemUpdate.vue told the
operator "Servers this node checks for updates. The primary is tried first; if
it's slow or unreachable, the next one in the list is tried automatically."
That promises availability redundancy the pair cannot provide: if the origin is
down, both entries are down. Reported by the operator, who read the list and
correctly concluded the fallback made no sense.
The mechanism is fine and deliberate — it recovers a node whose DNS is broken
or whose clock is wrong, both of which fail TLS while plain HTTP still works,
and it is safe because the manifest carries an Ed25519 signature verified
against the pinned release-root anchor, so transport integrity is not what
protects the update. (That last part only became true once Workstream B pinned
the anchor; before then this fallback would have been a real hole.)
So the bug was the labelling, not the design:
- Backend label "Direct (fallback)" -> "Same server, no DNS/TLS", and the
comment now states plainly that it is the same host, what it recovers, and
that real redundancy needs a different one.
- UI copy now scopes the redundancy sentence to genuine mirrors and adds a
paragraph saying the two built-in entries are one server, what the second
actually recovers, that it does not help if the server is down, why an
unencrypted fetch is acceptable, and how to get real redundancy.
The relabel reaches existing nodes: force_ovh_update_primary rewrites labels for
the two default URLs on every load, while the merge matches on URL and never on
label — without that rewrite path a renamed default would have sat in the code
and never propagated to a single deployed node. Noted inline so it is not
re-broken.
Verified: 40/40 update tests pass (including the mirror load/merge/strip ones),
vue-tsc clean, build green, and the new copy is present in the freshly built
SystemUpdate chunk. Nothing in the tree pinned the old label string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Failed to scroll to index N after 10 attempts" appeared in the console on
every send. It was not a real failure — it was two scroll controllers
arguing.
scrollToBottom() called virtualizer.scrollToIndex(last) AND then assigned
el.scrollTop on the next tick. scrollToIndex runs a retry loop that nudges
scrollTop toward the target row's measured offset and re-checks, up to ten
times, because dynamically-measured rows move the target as they settle.
The manual assignment overwrote each nudge, so the loop never observed
itself converge and always exhausted its attempts.
For "go to the end" the index-settling machinery buys nothing: scrollHeight
already is the bottom, the virtualizer renders whatever window that offset
implies, and it keeps working while a response streams and the last row
grows — the case the manual fallback was added for in the first place.
scrollToMessageIndex still uses scrollToIndex, which is the right tool for
jumping to an arbitrary row.
Console-only change; needs a device check that the chat still pins to the
bottom while streaming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 1 items 3 and 5.
Item 3 turned out to be far narrower than the plan's "93 files" once each hit
was classified rather than bulk-replaced. Sanitized only genuine operator
identifiers:
- FIPS test fixtures and a pine_ha comment carried real node LAN addresses ->
RFC 5737 TEST-NET-1, the convention already used elsewhere in this repo.
- Real tailnet addresses in fips/endpoints.rs, mock-backend.js and the mesh
test runner -> the base of the CGNAT range, obviously synthetic.
- Incident comments in appgate/mod.rs and apps/fedimint/manifest.yml named a
specific node; the role is what carries the meaning, so the address is gone.
- CHANGELOG.md held five real addresses in published release notes — the most
exposed of the lot.
Deliberately NOT touched, because the plan's item-3 list is over-broad and
following it literally would break working code:
- 192.168.1.1 / .254, 192.168.0.0/16 and 100.64.0.0/10 are generic router
defaults, RFC1918 classification in backup_rpc, and CGNAT range logic in
pine_ha / CompanionIntroOverlay. Not leaked infra.
- `tx1138` is listed as a hostname to scrub but is two live things: the
user-facing default block explorer (`DEFAULT_TX_EXPLORER`) and
`RETIRED_TX1138_HOST`, the migration constant whose entire job is stripping
that retired registry from existing nodes' saved mirror lists. Scrubbing
either breaks a feature. The plan needs this correction.
- Android's `192.168.1.100` strings are UI placeholder text.
Item 5: added *.key, *.pem, id_rsa*, *.sqlite, *.db to .gitignore, with a
negation for core/archipelago/src/appgate/testdata/*.key. Checked those first —
they are documented throwaway TLS fixtures compiled in via include_bytes!, not
node identity — and the negation stops the new rule silently dropping them if
they are ever regenerated. Verified both directions: fixtures not ignored, a
stray key elsewhere caught.
Verified: residual grep for real infra addresses is clean; audit-secrets.sh
still 5/5; app-catalog drift 0 (the fedimint edit is a YAML comment, which does
not survive parsing into the signed catalog); 44/44 fips tests pass with the
rewritten assertion fixtures.
Note: these test runs shared the working tree with another agent's in-flight
LND work, which was present but unstaged and is not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
