- The fit now scales screen X and Y independently (capped at 1.75x
anisotropy), so both 2D and 3D stretch to the container's aspect ratio —
a portrait phone uses its full height instead of shrinking the orbit to
the narrow width, and wide desktop panels spread horizontally.
- Legend/key centres at the top on mobile, mirroring the bottom-centre
2D/3D toggle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Compact containers (<480px) shrink the fit margins so the scene fills the
phone screen instead of floating in padding.
- 3D mode steepens its tilt on portrait (-0.95 vs -0.5) so the orbit reads
as a tall ellipse with depth, not a squashed horizontal band; projection
params are resolved per-aspect via modeParams() and re-resolve on resize.
- The 2D/3D toggle moves to bottom-centre on mobile (thumb reach, frees the
top edge); the hint line tucks above it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root causes of the flaky intro / janky revisits, all fixed structurally:
- Graph updates now RECONCILE against the live scene by identity (peer DID /
request id) instead of wiping the SVG: in-place updates for label/online/
trust changes, animated arrivals for new nodes, animated departures for
removed ones, and short-way-around slot glides when the ring layout shifts.
The old wipe-and-rebuild orphaned every in-flight intro tween whenever the
5s poll or the async self-DID fetch landed mid-intro — that was the
'sometimes parts don't show'.
- The intro is gated on readiness: first real container measurement (post-
layout, via ResizeObserver) AND the self node being present, with a 1.2s
fallback. Graph changes during the intro queue and apply on completion.
- Layout is deterministic per node: peers sort by DID for slot assignment and
jitter/phase hash from the DID, so backend response order can't shuffle
the map between polls.
- The long cinematic intro plays once per browser session; revisits get a
shorter, lighter entrance that doesn't fight the route transition.
- selfDid seeds from the cached localStorage value (re-cached on fetch) so
the centre node and links exist from the first frame.
- Requests' waiting-room orbit pushed clearly outside the peer rings (+1.0)
with loose per-request hover jitter — outside until accepted, then the
join animation glides them in.
- Full unmount hygiene: every tween killed by target (incl. the repeating
sonar pulse), fallback timer cleared, window listeners removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Inbound pending requests render as blinking yellow point-globes on an
orbit outside the peers, dotted-linked to self, with their own dashed
guide ring and a Request legend chip.
- Tapping one opens a black-glass popover (name, request message,
Accept/Reject) centred over the scene; tap-away or ✕ dismisses.
- Reject: the node swells and pops out of existence, taking its link.
- Accept: green burst ring, the point cloud/link morph to the trusted
colour and the globe glides inward onto the peer orbit, then the data
refresh replaces it with the real peer node.
- Wired to the same approve/reject RPCs as the pending panel; new
motionTokens.color.pending (#facc15) token.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Small glass 2D/3D toggle top-right of the map: tweens tilt/perspective/fit
(0.9s inOut) so the scene folds flat into the original radial 2D layout and
back. Default: portrait/mobile opens 2D, desktop 3D; last choice saved to
localStorage.
- Fix: setPointerCapture retargeted pointerup to the container, suppressing
click synthesis on children — node taps and the toggle never fired. Drag
now tracks via window listeners, no capture.
- Fix: stale post-drag distance made the click-suppressor swallow toggle taps
indefinitely ('stuck' toggle). Distance resets on every pointerdown and the
suppressor is one-shot.
- Map no longer slides under the floating mobile back pill: the
mobile-scroll-pad-back panel keeps its full 64px clearance when filled.
- Mobile DID copy/rotate card moved out of the header to below the view tabs
(new DidCardMobile.vue) and hidden on the Network Map tab.
- Node labels: dark stroke halo removed; 10px on mobile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portrait containers now render near top-down and near-orthographic (tilt
-1.35, persp 16) so the map reads like the previous 2D radial layout —
rings become circles with no perspective squash. Desktop keeps the 3D
depth camera. Globe point-spheres unchanged on both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The map view turns the root into a flex column; default stretch alignment
was pulling the mode-switcher full-width on desktop. md:self-start restores
the content-fit pill (mobile stays full-width by design).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Nodes are now spheres made of points: fibonacci point-cloud globes with
depth-shaded dots, limb outline, and a slow local spin (drag adds parallax).
Self node is black — dark dots over a soft light backing disc with the
brand-orange sonar pulse marking 'you'. Spheres and dots sized down.
- Scene auto-fits and centres to the container on every device: the outermost
orbit is sampled through the real camera projection to get true bounds,
then scaled/centred between the overlays. Portrait screens tilt the camera
towards top-down so the orbit uses the full height (mobile readability).
- No idle orbiting: nodes hold position with a gentle side-to-side sway;
drag inertia now settles to a stop. Intro dolly pushes in (0.82→1) instead
of pulling back, so nothing clips during the intro.
- Labels get a dark paint-order halo and bump to 12.5px on mobile.
- Point clouds update at half frame rate to keep mobile/companion smooth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add gsap 3.15 + design-system-aware motion module (src/utils/motion.ts):
shared colour/duration/ease tokens mirrored from style.css, gsap.defaults,
live prefers-reduced-motion check.
- Replace the d3 force NetworkMap with NetworkMap3D.vue: peers on projected
3D orbital rings around the self node, cinematic intro (camera dolly +
staggered fly-in + ring draw), idle rotation with drag-to-orbit inertia,
depth-sorted painter's order, trust-colour palette, online/offline states,
sonar pulse on self, tap-a-node opens the detail modal.
- Map view now fills the dashboard panel to the bottom edge on desktop,
mobile and companion: .dashboard-scroll-panel:has(.node-map-stage) turns
the panel into a column (tab-bar/safe-area/audio-player aware padding)
instead of leaving the old dead bottom margin.
- Reduced motion: intro/idle skipped, scene renders static.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signed release catalog was pinned in 6c41852c; the curation source still
said :latest, which ec8d88a6 established can stall the fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The release tarball bakes demo/aiui in for fresh installs; the previous dist
predated the phase-13 security waves (S1-S7, W1.4) and would have shipped a
pre-hardening bundle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Quadlet renders --rm, so every stop DELETES the container: a scan during
the stop->start window legitimately sees nothing where an installed app
lives. The state layer treated that as the app not existing — after three
absent scans it removed the entry entirely, so dashboard tiles vanished
mid-restart, gate waits read 'absent' (grafana, at load ~2, ruling out
contention), and under daemon-restart churn the FIRST scan could publish
a completely empty My Apps map (observed twice on 2026-08-09).
Two guards, both keyed to the durable installed-apps.json registry:
- Eviction: an id the registry says is installed, and the user has not
uninstalled, is held as Stopped (health/exit cleared) instead of being
removed. The next scan that sees the container restores live state;
desired-state recovery still recreates genuinely lost containers; a
deliberate uninstall clears the registry first and still disappears.
- Empty scans: the first-scan exemption no longer applies when the
registry is non-empty — better to keep saying "scanning" than "empty".
Verified: scan-merge/absence/registry test filters 34/34.
Also carries the v1.7.127-alpha changelog (user-benefit entries for the
Tor self-heal, trusted invites, the three-layer mempool fix, vanishing
apps, the Bitcoin version pin, and the smaller UI fixes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An OTA updates the binary and web assets — not /etc/nginx, and not
/opt/archipelago/scripts. Auditing the 2026-08-09 fixes' delivery paths
found two that would silently reach nobody already installed:
- scripts/tor-helper.sh (reset-failed + truthful restart result) shipped
only via ISO builds and manual deploys. Now embedded via include_str!
like the doctor script, staged and installed at boot when the on-disk
copy differs.
- The /app/mempool/ nginx Upgrade/Connection headers existed only in repo
snippet sources consumed at image build time. A boot repair now
idempotently patches any mempool location block missing them — in the
live vhost (archipelago-http, the one sites-enabled actually links to),
the legacy file, and the installed snippet — and reloads nginx once.
Without this, every fleet node's mempool UI keeps loading-but-never-
connecting after the OTA that supposedly fixed it.
Both are non-fatal boot repairs in the existing bootstrap chain, no-ops
when everything is already current.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sticky tab header painted bg-black/70 across the modal — reported as
a black container where the design is transparent glass. The backdrop
blur alone keeps pinned tabs legible over rows scrolling beneath; the
tint drops to the standard bg-white/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate accepted a client's websocket upgrade (.with_upgrades() on the
serving side) but proxy_to_app forwarded every request with a plain
hyper::Client, which completes the app's 101 handshake and then drops the
upgraded connection. Result: any ws-driven app behind a gated port loads
its page fine and then dies with close code 1006 on every connect —
mempool's entire UI is such an app, and the operator's browser console
was the only place the failure was visible. This was the THIRD layer of
the same outage (electrumx sync, then the nginx proxy path, now the app
tile's gated-port path — each fix exposed the next).
On an Upgrade request the gate now takes the client's OnUpgrade handle
from the request extensions, hands the handshake to the app with the same
sanitized headers (cookie/authorization stripping applies unchanged), and
on a 101 bridges the two upgraded connections with copy_bidirectional.
Non-101 upstream answers pass through so auth failures stay visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's /app/mempool/ proxy forwarded plain HTTP only: no
Upgrade/Connection headers, so the browser's /api/v1/ws handshake reached
mempool-api as a bare GET and Express 404'd it. The mempool page loaded
fine and every REST probe was green while the user saw a UI that never
connects — the backend was fully healthy and completely unreachable in
the only way that matters to the page.
This hid behind the electrumx initial-sync outage: once sync finished and
the API answered, "mempool works" was declared from REST checks while the
websocket path stayed dead. Reported by the operator from the browser,
which is the only place it was visible.
Adds Upgrade/$http_upgrade + Connection "upgrade" to the /app/mempool/
location in both shipped sources (image-recipe snippet + scripts copy).
Live-verified on the dev node: ws through the proxy now answers 101; the
control probe without upgrade headers reproduces the 404 signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full project build (vue-tsc -b, which also checks test files) caught
two default-ServerInfo constructors the scoped typecheck missed. Rather
than adding the field to both, the type now tells the truth for a
mixed-version fleet: daemons older than 2026-08-09 omit the field, so
consumers must treat absent as "unknown" and only === true as connected —
which is exactly how Home.vue already reads it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Home's Network card computed "Connected" from server-info's tor-address —
a string read from the hidden-service hostname file, which persists on
disk however dead the daemon is. Three fleet nodes ran with Tor down for
days while their dashboards said Connected; that indicator is why nobody
noticed. Server.vue's Tor label had the same flaw one step removed,
inferring "running" from services having onion_address values while the
correct signal (torDaemonRunning, backed by a real socket probe) sat
computed and unused twelve lines above.
Both now read liveness signals only: Home reads the new tor-running field
(server-side connect to 127.0.0.1:9050), Server.vue uses
torDaemonRunning. The address remains displayed as what it is — an
address — never as proof of life.
Verified: vue-tsc clean; rpc-client suite 79/79.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three of five reachable fleet nodes had Tor completely dead — Home said
"Connected" on all of them — and shipping the generator fix alone would
have repaired none of them. Two additions close that loop.
heal_on_boot (wired into the bootstrap repair chain): regenerate torrc
from current config with the fixed generator, and apply-and-restart ONLY
if the live file drifted or Tor is not answering on 9050. A healthy node
is left untouched. Without this, regenerate_torrc runs only from the Tor
RPC handlers and package install, so a node carrying a poisoned torrc
keeps it until someone happens to toggle a Tor setting — and worse, the
still-running OLD binary re-poisons on any such toggle: observed live on
the dev node at 07:20, when the running daemon rewrote torrc with the
unbindable gateway line hours after it had been hand-fixed. The heal
makes the fix self-applying on every restart, i.e. the OTA itself.
ServerInfo gains tor-running, populated by a real connect to
127.0.0.1:9050 each state refresh. tor-address is read from the
hidden-service hostname file, which OUTLIVES a dead daemon — it is a
configuration artifact, and the dashboard treating it as liveness is
precisely why three dead nodes showed "Connected" for days. Liveness now
comes only from the probe; the address stays a separate fact.
The heal reports the truth: it re-probes after restarting and warns if
Tor still is not answering, rather than assuming success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No node in the fleet could mint a Trusted federation invite or promote a
peer to Trusted from the dashboard. The cause is one case comparison.
Granting Trusted requires the node password; when none is supplied the
handler bails with "PASSWORD_REQUIRED: node password required to grant
Trusted" — a sentinel the frontend machine-reads (isPasswordRequired
checks includes('PASSWORD_REQUIRED')) to know it should open the password
prompt. sanitize_error_message allowlists "Password", but starts_with is
case-sensitive, so the ALL-CAPS sentinel failed the allowlist and
collapsed into "Operation failed. Check server logs for details." The
frontend never saw the sentinel, the prompt never opened, and the flow
died with a generic error on every node.
Reproduced via direct RPC: federation.invite {trust_level:"trusted"}
without a password returned the generic message; the same call WITH the
password minted a code fine — the machinery was healthy, only the signal
was being eaten.
"Tor address not available. Tor may not be running." — the invite
handler's other user-actionable precondition — was masked the same way
and is allowlisted too.
Regression tests pin both passing through verbatim, and
internal_errors_stay_generic still passes, so the generic masking of real
internals is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All ten AI data-access categories offer a context toggle, but only five
(Apps, System, Network, Media, Bitcoin) have any tool behind them — Wallet,
Files, Search, Local AI and Notes grant the assistant visibility with
nothing to act on. The Settings copy implied actions that do not exist.
Operator decision (2026-08-09): ship honest copy now, build the tools as
ordinary backlog. Each of the five carries a contextOnly flag, a "context
only" badge, and an appended description line saying the assistant can see
this but can't take actions with it yet. Drop the flag per category as
real tools land.
Verified: vue-tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two problems with running the gate against a mid-initial-sync electrumx:
1. The (now honest) protocol probe can only fail — ElectrumX serves no
sessions until it has caught up to its daemon, so the failure names a
state nobody can act on.
2. Worse, the destructive stop/start/restart tests actively destroy sync
progress: electrumx flushes its DB cache at 1GB, i.e. rarely, and every
restart discards all unflushed work back to the last flush. This node
spent 8d14h in initial sync largely because gate runs and reboots kept
taking hours of progress away — it restarted at 06:16 and resumed from
959,774, the same height it had reported hours earlier.
The suite now detects initial sync POSITIVELY — a fresh (<30 min)
"our height: N daemon: M" line from electrumx's own log, gap > 10 — and
skips the probe and the four destructive tests with the gap named:
# skip electrumx initial sync in progress (1672 blocks behind) — ...
This is not the container-absent skip trap fixed earlier: absence of the
log line means "unknown" and the tests run and fail honestly. Validated
against the live mid-sync node: all four guards fired with the real gap;
on a synced node the line shows gap 0-1 and everything runs.
Unblocks the release gate from waiting hours on a sync it was itself
prolonging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
changeTrust() wrote failures into `error`, which renders inside <NodeList>
— page content that sits BEHIND the open NodeDetailModal. So when
promoting a peer to Trusted failed, the operator saw the dropdown snap
back and nothing else: the error banner was painted on a surface the
modal was covering. Reported as "the error doesn't show in the modal",
2026-08-09.
NodeDetailModal gains an actionError prop rendered directly under the
Trust Level control, and Federation.vue routes failures there whenever
that node's modal is open (clearing on close and on the next attempt).
The page banner still serves failures with no modal in the way, and the
password-retry path keeps its own in-modal error as before.
Verified: vue-tsc clean; federation component tests 5/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's "Restart Tor" button dispatched to this helper and always
got {"ok":true} back. Two real defects, in order of importance — and one
disproved theory, recorded so nobody re-chases it:
1. reset-failed was missing. Once tor@default fails enough times systemd
latches "Start request repeated too quickly" and refuses to start it at
all; a plain restart is then a no-op no matter which unit you name.
All three fleet nodes found dead on 2026-08-09 were in exactly that
state, which is why the button appeared to do nothing.
2. The result was unconditional. The write-torrc branch waited up to 30s
for SOCKS and then ignored the outcome; the restart branch slept 3s and
claimed success. The UI reported "restarted" over a dead daemon.
Disproved: this was NOT wrong-unit targeting. `systemctl restart tor`
does propagate to tor@default — measured on austin-sapien, MainPID
changed. tor@default is still addressed explicitly because it is the
unit carrying the failed state worth resetting.
restart_tor_daemon() now: reset-failed, restart tor@default (fall back to
tor on single-instance installs), wait up to 30s for SOCKS on 9050, and
return {"ok":false,"error":...} pointing at journalctl when it never
comes up. Callers may no longer report success without a live SOCKS port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
regenerate_torrc widened SOCKS to the archy-net gateway whenever `podman
network inspect archy-net` reported one. Podman reporting a gateway is not
proof the host can bind it: under rootless podman the bridge and its gateway
live inside a network namespace and never appear on a host interface.
Tor binds listeners at STARTUP, not at config-check time — `--verify-config`
passes happily — so the failure lands as a refusal to start. austin-sapien,
2026-08-09:
[warn] Could not bind to 10.89.0.1:9050: Cannot assign requested address
[warn] Failed to parse/validate config: Failed to bind one of the listener ports.
[err] Reading config failed--see warnings above.
Tor then died completely — loopback SOCKS and every hidden service with it —
having run 3d 18h before something restarted it. The node's Home tab still
read "connected" because tor.service is Debian's multi-instance MASTER unit,
which is active(exited) by design; the real instance, tor@default, was failed.
Whether a node is affected is pure timing: archi-dev-box's torrc has only
`SocksPort 9050` because archy-net was not discoverable when its torrc was
last regenerated, so it fell into the loopback branch and Tor starts fine.
Nodes whose torrc was generated while archy-net was visible are one Tor
restart away from the same outage.
The gateway is now probed the way Tor will use it — bind an ephemeral port on
that exact address — and a failure falls through to the existing loopback-only
branch, which was already written and commented "fail closed". Widening SOCKS
must never be able to take the whole daemon down.
Verified on this host: 127.0.0.1 binds, 10.89.0.1 returns EADDRNOTAVAIL.
Tor restored on austin-sapien and confirmed end to end — bootstrapped 100%,
and a request through the proxy returns {"IsTor":true}.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both electrumx tests did socket.create_connection(("127.0.0.1", 50001)) and
called it proof the service was up. It is not: that is the HOST-published
port, and podman's port forwarder completes the TCP handshake even when
nothing inside the container is listening.
The cost was real. Both tests were green while mempool-api sat in a hard
ECONNREFUSED loop against electrumx:50001 from inside archy-net — the
operator's "mempool doesn't connect to bitcoin". Measured side by side:
127.0.0.1:50001 from the host OPEN, electrumx:50001 from a sibling container
REFUSED.
The cause is not a misconfiguration — mempool-api's CORE_RPC_HOST,
ELECTRUM_HOST and ports are all correct. electrumx has not finished its
initial sync (DB height 959,774 vs chain tip 961,706, ~1,932 blocks, 8d 14h
elapsed) and ElectrumX does not serve sessions until it has caught up.
Now both tests do a server.version round-trip and require a JSON result, so a
forwarder that accepts-and-drops fails. Verified against the live node: the
new probe FAILS today where the old one passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moves the default off 29.3.knots20260508, which applies the BIP110/RDTS
network upgrade and HALTS until an operator sets consensusrules=rdts. A node
running it sits frozen — 100.64.204.114 is stuck at block 961,692 with blocks
AND headers static, 11 peers and no pruning, while reporting
initialblockdownload=false and verificationprogress 0.99998, i.e. "synced".
The nodes on 20260210 are at the tip.
The old catalog's default was {"version":"latest", image: ...20260508} while
the manifest's :latest tag resolves to 20260210 in the registry, so a fresh
install and a plain pull produced different daemons. That is almost certainly
how one node ended up on the RDTS build and its siblings did not.
20260508 stays selectable — adopting RDTS remains possible, just deliberate.
Diff against the previously-signed catalog is exactly one app, all four
fields: version, image, default and the embedded manifest. Registry hosts
unchanged and inside the trust floor; verified by
check-catalog-registry-trust.py and `ceremony verify`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sign-catalog.sh refused to sign a freshly regenerated catalog:
REFUSING: the catalog names registry hosts the deployed fleet does not
trust. source.archipelago-foundation.org — 78 image refs
It was right, and the check earned its place. apps/*/manifest.yml moved to
the public domain in 8e814ca0, but releases/registry-trust-floor.json still
lists only docker.io, ghcr.io, localhost and the OVH host — the migration is
ship-binary -> confirm-fleet -> promote-floor -> regenerate, and only the
first move happened. A full regen therefore produced a catalog naming a host
no deployed binary trusts, which would have failed every install in the field
with "not from a trusted registry". Verifying that an image PULLS from the
domain (it does, including from a fleet node) does not verify that shipped
binaries ACCEPT it — different checks.
Two fixes, no manifest churn:
- image-versions.sh honours a caller-supplied ARCHY_REGISTRY instead of
overwriting it, so generation can target a host inside the floor.
- the generator retargets OUR registry host inside embedded manifests to
whatever it is generating against, leaving docker.io/ghcr.io refs alone.
REGISTRY moves above the embed block, which now depends on it.
The repo keeps naming the public domain; only the generated artifact is
pinned to what the fleet can actually use. When the floor is promoted,
generating against the domain becomes a no-op here.
Verified: check-catalog-registry-trust.py passes, 11 docker.io/ghcr.io refs
untouched, and the diff against the signed catalog is exactly one app —
bitcoin-knots, :latest/20260508 -> :29.3.knots20260210 across all four fields.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalog's default version was {"version": "latest", image:
bitcoin-knots:29.3.knots20260508, default: true}, so a fresh install — or
anyone selecting "latest" — silently got the BIP110/RDTS build. Meanwhile
the manifest's own :latest tag resolves to 29.3.knots20260210 in the
registry, so the two paths installed different daemons.
That matters because 20260508 does not merely enable RDTS, it HALTS until
an operator sets consensusrules=rdts. Node 100.64.204.114 runs it and is
frozen at block 961,692 — blocks AND headers static across 30s, 11 peers,
unpruned — while reporting initialblockdownload=false and
verificationprogress 0.99998, i.e. "synced". The two nodes on 20260210 sit
at the tip, 961,706. Re-pointing a tag could have done that to every node
at once.
Pinned in all three places that would otherwise re-introduce a moving tag:
the manifest, image-versions.sh, and the catalog VERSIONS table. The
"latest" pseudo-version is removed outright — a moving default is not
appropriate for a consensus-critical app — and the default is now the
explicit non-RDTS 29.3.knots20260210. 20260508 remains selectable, so
adopting RDTS stays possible but becomes deliberate.
Verified: both tags' daemons read via `bitcoind --version` — :latest and
:29.3.knots20260210 are both v29.3.knots20260210, so this pin is
behaviour-neutral on nodes already running latest.
NOT DONE HERE: releases/app-catalog.json still carries the old signed
entry. It must be regenerated and re-signed by the operator before it
takes effect — catalog entries override on-disk manifests (origin-wins).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two fixes from the first full 125-test run (121 ok / 4 not ok).
1. The Quadlet unit test still asserted Restart=on-failure for backends,
which 8908fb4f deliberately changed. All 32 units on the node are
Restart=always; only archy-btcpay-db showed up because `fail` aborts at
the first offender. The old rationale — "backends need on-failure so an
operator `systemctl stop` stays stopped" — was wrong twice over:
systemd never applies Restart= to a unit stopped via `systemctl stop`,
and because quadlet renders --rm, a cleanly-exited container is deleted
and on-failure never brings it back (bitcoind exits 0 on SIGTERM, so
backends vanished). Verified on-device before that change: `podman stop
bitcoin-knots` came back in 12s, a dashboard stop stayed stopped 90s.
Now asserts Restart=always, with the corrected reasoning in-place.
2. all-apps-lifecycle cycled ten apps back-to-back and manufactured the
contention that then failed later tests: load went 10.8 at preflight to
17.6 mid-loop, and btcpay's recovery stretched from 52s on a quiet box
to 216s and then 512s, defeating waits of 180s and 300s. Widening waits
cannot win when the run sets the load they must survive, so the loop now
drains between apps — waits for load1 under 2x nproc, capped at 90s.
ARCHY_APP_SETTLE_SECS=0 disables; ARCHY_APP_SETTLE_LOAD overrides.
Verified: run.sh use-quadlet-backends-install → 6/6 (was 1 failure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both failed a gate run on a healthy node.
bitcoin-receive already tolerated WALLET_LOCKED, but its 180s window
starts when the TEST starts and the lnd restart that locks the wallet can
land partway in. On 2026-08-08 the restart hit 65s in and the wallet
unlocked at 2m25s (journal: lnd.service started 20:11:05, "wallet has
been unlocked without a time limit" 20:13:48) — 48s after the deadline
expired. The daemon's own unlock budget is ~10 min because opening the
channel and graph dbs takes minutes on a loaded box, so 180s was stricter
than the thing under test. Now 420s, ARCHY_LND_UNLOCK_SECS.
btcpay's start wait was 180s, but stopping btcpay DELETES the container
(quadlet renders --rm), so package.start is a full dotnet recreate rather
than a container start. Measured 52s on a quiet box; it exceeded 180s
during a gate run on the same node at load ~11. Now 300s,
ARCHY_BTCPAY_START_SECS.
Neither change masks a lifecycle fault: both paths were verified by hand
to complete correctly, just slower than the assertion allowed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop/start/restart loop carefully accumulates per-app failures into
$fails and prints them, but the three rpc_result calls were bare. Under
bats' errexit a bare call ends the test immediately, so the summary that
names the app never ran.
On 2026-08-08 that turned a single transient error into an unattributable
failure: the test died at package.stop with no indication which of the
ten targets was involved. It was mempool, and the identical call returned
{"status":"stopping"} by hand a few minutes later.
Each call now records <id>:<phase>-rpc and moves to the next app, so the
run reports what actually broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ui-coverage.bats and ui-probes.bash hardcoded https:// while the rest of
the harness builds URLs from ARCHY_SCHEME via lib/rpc.bash. That made the
suite unrunnable on a node serving the dashboard over http.
On archi-dev-box :443 is bound to the Tailscale / WireGuard / LAN
interface addresses but NOT to loopback, while :80 is bound on 0.0.0.0.
So five probes failed with "curl failed (network/timeout)" against
endpoints that were serving 200 the whole time — http://127.0.0.1/,
/catalog.json, /app/lnd/, /app/electrumx/ and /app/mempool/ all verified
200 by hand.
Default stays https, so nodes that already exercise the TLS path keep
doing so. Test titles drop the hardcoded scheme, since they no longer
describe which one ran.
Verified: ARCHY_SCHEME=http ./run.sh ui-coverage → 9/9 (1 skip),
previously 5 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gate run on a box at load ~14 failed five times over, and every failure
read "could not create a container" — searxng:start, package.start
btcpay-server, 3x electrumx — never a lifecycle fault. That sent two
separate sessions hunting a phantom host-wide cgroup failure. It was
contention.
Measured on the 4-core node: at load ~14 podman runs 9-16 processes deep
and healthchecks time out 3-8/min; at load ~3.7, podman ~1 and zero
timeouts. Restoring four containers whose HealthTimeout equals their
HealthInterval, unchanged, made the box *better* once load fell — so the
config is a latent hazard, not the cause here.
Preflight now checks, once, before iteration 1:
- aardvark-dns is singular (duplicates desync name resolution)
- load1 is under nproc+1, waiting up to 15 min for a spike to pass
- podman can actually create a container, 3/3
Deliberately NOT checked: the count of "Failed to create container" in
the journal. Those lines come from healthcheck exec churn and post-boot
settling, never reach 0 on a busy node, and gating on them would block
the gate forever. The probe proves creation positively instead.
Escape hatches: ARCHY_PREFLIGHT=0, ARCHY_MAX_LOAD, ARCHY_PREFLIGHT_SECS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An app whose container disappeared could be lost permanently. Desired-state
recovery decided whether to recreate it from running-containers.json — "what
was running at the last snapshot" — which is a different question and a
perishable answer: it records only what is running NOW, so an app that stays
down long enough simply ages out. Once out, boot's ExistingOnly mode will not
recreate it, because it cannot tell "installed and lost" from "merely
available in the catalog". Manifest still on disk, nothing to bring it back.
This is the second occurrence of one root cause. indeedhub-minio/-postgres
went permanently absent on one node (2026-08-06); the fix then was
`absent_stack_member_with_live_sibling`, which only rescues a stack member
that still has a living sibling. bitcoin-knots is standalone, so on
archi-dev-box (2026-08-08) it vanished, aged out, and stayed gone — LND
crash-looping on `lookup bitcoin-knots: no such host` for hours, electrumx
unable to reach its daemon, and an orphaned fedimint container waiting 30
hours for a host that no longer resolved. Recovering it took a manual
reinstall. This is the general fix the narrow one implied.
Installation is a DECISION, not a runtime observation, so it gets a record
that no amount of downtime erodes: installed-apps.json, written when an
install succeeds and cleared on uninstall, in the same breath as
mark_user_uninstalled — leaving a stale claim would let recovery recreate the
app that was just removed. It is the durable counterpart to the
user-uninstalled marker that already existed.
Safety, in order of how badly each could go wrong:
- Cannot resurrect a deliberate uninstall: user_uninstalled is checked
earlier in ensure_running_with_mode and returns before anything is created,
and uninstall clears this record too.
- Cannot install an app nobody asked for: only names in the record qualify,
and ExistingOnly's other guards are untouched.
- Cannot mislead a node upgrading into the feature: backfill seeds from
ADOPTED containers only — evidence that something is really there — skips
anything user-uninstalled, is additive so a momentarily-down app is never
dropped, and no-ops on an empty adoption list (podman unreachable must not
read as "nothing is installed").
Four tests, including the one that states the point: the record must outlive
a running-snapshot that has gone empty.
Container suite 221/221, crash_recovery 15/15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two operator reports on the same screen.
The section had no card. Every other Settings section wraps itself in
`glass-card px-6 py-6 mb-6` — AccountSection, AIDataAccessSection,
NodeCertificateSection, BackupSection, the lot — and this one rendered as
bare text on the page. Reported twice, because the wrapper lives in the new
component and nothing about adding `<LightningCredentialsSection />` to
SystemSection.vue's list tells you it is missing. Heading moved to h2/text-xl
to match its siblings. A test now asserts the card, so a third report is not
needed.
And rotating told the operator Lightning did not exist. Rotation restarts
LND, so `status.installed` reads false for a moment — and the template read
that literally: "Lightning is not set up on this node yet, so there are no
credentials to rotate. Install the Lightning app first." Seconds after
rotating. On a node with a working wallet. It also replaced the progress they
had every reason to be watching, on the one action that invalidates every
credential their wallet holds.
A container briefly absent is what rotating LOOKS like, not evidence
Lightning was never there. The not-installed message is now gated on
`!rotationInFlight`, which covers both `running: true` and the awaitUntil
window between asking for a rotation and the node reporting one — `installed`
can already be false in that gap, so gating on `running` alone would have
left the same hole. Mid-rotation with no status yet says "Rotating
credentials — Lightning is restarting" instead of falling through to a
details block with empty fields.
awaitUntil became a ref so the computed re-evaluates rather than holding a
stale value until some other reactive dependency happens to change.
Three tests: the card exists; a running rotation does not claim Lightning is
missing; and — the half that matters just as much — a node with genuinely no
Lightning still gets told there is nothing to rotate, so the fix has not
simply hidden a true statement. 16/16, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced live: the operator pressed install on ElectrumX shortly after
Bitcoin Knots started and got
Bitcoin RPC returned 500 Internal Server Error while checking pruning
status: {"error":{"code":-28,"message":"Verifying blocks…"}}
-28 is RPC_IN_WARMUP — bitcoind's normal path on every start, lasting
minutes on a large chainstate. check_bitcoin_pruning_compatibility retried
`for _ in 0..3` with a 2s sleep, so it gave the node about six seconds and
then reported warm-up as a hard failure. Any app requiring unpruned Bitcoin
was therefore uninstallable in the ordinary window after Bitcoin starts.
This is the likely mechanism behind the operator's "fedimint gateway
disappeared at 88% install": both fedimint apps declare a bitcoin-core
dependency and sit in exactly that window.
The same install path already knew better. wait_for_bitcoin_rpc_gate waits
180s precisely because getblockchaininfo answers during sync. This check
runs earlier and disagreed — one concern, two contradictory answers, in one
install. It now shares the budget, and a test asserts the two stay equal.
Only NOT-READY is waited out: -28 by code, plus the "loading block index" /
"verifying blocks" / "rewinding blocks" message shapes for a proxy that
rewrites the envelope. A genuine fault — bad auth, method not found,
unparseable body — still ends the loop on the first response, so a broken
RPC fails fast instead of burning 180s. Both directions are pinned by tests,
because being too loose here is as bad as being too strict.
Warm-up is announced ONCE to the install log, so a slow install reads as
"waiting for Bitcoin" rather than a stall. And the failure message now says
what to do — "Bitcoin is still starting up… wait until it reports it's
synced, then try again" — instead of pasting the raw JSON-RPC envelope,
which was accurate and useless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught on archi-dev-box within minutes of deploying 3ac59a73: the reaper
removed archy-bitcoin-ui and archy-lnd-ui, whose backends ARE installed.
archy-bitcoin-ui was gone for 36 minutes, until the operator reinstalled
bitcoin-knots and `reconcile` put the companion back.
Not a logic error — the arithmetic did what it was told. The inputs were
false. Both backends' containers were missing because of the clean-exit
vanishing bug (8908fb4f), and both had already aged out of
running-containers.json, which only ever records what is CURRENTLY RUNNING.
So the two signals `installed_app_ids` combines are not independent: one root
cause falsifies both simultaneously. ORPHAN_GRACE could not help either — the
condition was persistent, not transient, which is exactly the case the grace
period cannot distinguish.
The asymmetry decides it. An un-reaped orphan costs a stale UI tile. A
wrongly-reaped companion costs a working screen and turns one lost app into
two — the reaper amplifies the very failure it was meant to tidy up after.
`reap_orphans` and its tests stay, documented as NOT TO BE WIRED until a
durable record of "this app is installed" exists to drive it. Inferring
installation from runtime state cannot answer that question, however many
runtime signals are combined.
The provisioning half is untouched and is the actual fix for "fedimint
installs but does not work": driving `reconcile` from installed_app_ids means
a companion is never stood up for an app nobody installed, so no NEW orphans
appear. The one genuine orphan on this node (archy-fedimint-ui, for an app
never installed) was correctly removed before this change landed.
Unit tests passed the reaper because they verify the set arithmetic, not
whether the "installed" signal is truthful. Only the device could show that.
Container suite 221/221.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`sign_nip98` read the node's pubkey through `get_nostr_pubkey`, which goes
through `load_or_create_nostr_keys`. On a node with no Nostr identity that
does not fail — it GENERATES a keypair, writes the secret to disk, and signs
with it. So an IndeeHub auth header could quietly create a new node identity
as a side effect, then authenticate as a stranger holding a key nobody has
ever seen. The `.context("node has no Nostr identity")` guarding the call
could never fire, because the call could never fail that way.
`nostr_identity_exists` is the missing distinction: bootstrap may create,
but anything AUTHENTICATING as this node must prove the identity it already
has. sign_nip98 now gates on it and bails loudly.
Caught by `a_nip98_event_names_the_exact_url_and_method`, which asserts
exactly this ("must fail loudly rather than sign something empty") and had
been failing since the file landed in 58c759c1 — invisible because the
earlier runs on this branch filtered to `container::`.
Full bin suite 1381/1381.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AIUI has carried 353/356 as a known-open item (W1.7) for long enough that
the three reds were treated as background noise. All three were the tests
being wrong. 357/357 now.
`injects web results into system prompt when enabled` asserted
`body.webSearch === true`. The code deliberately sends false there:
`proxyWebSearch = webSearchEnabled && !clientSearchSucceeded`, so when the
client-side search has already run and injected its results into the system
prompt, asking the proxy to search again would be a second redundant search
on every turn. The assertion predates that change. Fixed, and the other half
of the contract added as its own case — zero client results must still ask
the proxy to search.
The two seed-songs failures ("extracts 10 songs", and the conversation
regression built on the same fixture) were one wrong number:
`expected: { songs: 10 }` against an assistantResponse containing exactly
six `song_ext` entries and ending coherently on Treefingers. Not truncated —
just miscounted. The extractor was returning the right answer the whole
time. Counted from the fixture rather than from intent, so the number now
describes the input instead of contradicting it.
Full branch state after the main merge: AIUI 357/357, neode-ui 972/972
across 114 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e36f36ee re-encoded the chat backdrop to webp (1052K -> 478K) to fix "the
background takes ages to load", but left the JPG in place. ChatPage.vue
references only `bg-intro-3.webp`, so the 1053K JPG has been shipping in
every AIUI bundle, every deploy and every ISO since, referenced by nothing.
Checked before removing, because the same filename IS live elsewhere:
`appgate/mod.rs` serves `bg-intro-3.jpg` as one of four LOGIN_BACKGROUNDS,
but from `/opt/archipelago/web-ui/assets/img` — the neode-ui copy, which is
untouched here. Only AIUI's duplicate goes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported: "`files` context request times out". sanitizeFiles makes three
sequential calls into the File Browser app — login, getUsage, listDirectory
— wrapped in a try/catch. A catch only sees a REJECTION. A socket that
connects and then says nothing leaves the promise pending forever, so
handleContextRequest never posts a `context:response` and the AIUI side sits
until its own bridge timeout instead. The File Browser is a plausible source
of exactly that: on this node `/app/filebrowser/api/resources/` does not even
route (404), and its session-cookie path is the subject of a separate open
bug.
The guard goes at handleContextRequest rather than inside sanitizeFiles, so
no category — present or future — can strand the bridge. `files` is merely
the one with three network hops today; sanitizeSystem is also async.
withTimeout resolves rather than rejects, because the caller's one job is to
always answer, and a rejection would just relocate the problem into a catch.
A late null is safe by the protocol's existing shape: the AIUI reader
already treats a response with no usable data as "nothing to show", the same
as an empty category.
Two tests: a never-settling File Browser still produces a
`context:response`, and a healthy category still returns real data rather
than being flattened to null. 27/27 contextBroker, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
