An app whose container disappeared could be lost permanently. Desired-state
recovery decided whether to recreate it from running-containers.json — "what
was running at the last snapshot" — which is a different question and a
perishable answer: it records only what is running NOW, so an app that stays
down long enough simply ages out. Once out, boot's ExistingOnly mode will not
recreate it, because it cannot tell "installed and lost" from "merely
available in the catalog". Manifest still on disk, nothing to bring it back.
This is the second occurrence of one root cause. indeedhub-minio/-postgres
went permanently absent on one node (2026-08-06); the fix then was
`absent_stack_member_with_live_sibling`, which only rescues a stack member
that still has a living sibling. bitcoin-knots is standalone, so on
archi-dev-box (2026-08-08) it vanished, aged out, and stayed gone — LND
crash-looping on `lookup bitcoin-knots: no such host` for hours, electrumx
unable to reach its daemon, and an orphaned fedimint container waiting 30
hours for a host that no longer resolved. Recovering it took a manual
reinstall. This is the general fix the narrow one implied.
Installation is a DECISION, not a runtime observation, so it gets a record
that no amount of downtime erodes: installed-apps.json, written when an
install succeeds and cleared on uninstall, in the same breath as
mark_user_uninstalled — leaving a stale claim would let recovery recreate the
app that was just removed. It is the durable counterpart to the
user-uninstalled marker that already existed.
Safety, in order of how badly each could go wrong:
- Cannot resurrect a deliberate uninstall: user_uninstalled is checked
earlier in ensure_running_with_mode and returns before anything is created,
and uninstall clears this record too.
- Cannot install an app nobody asked for: only names in the record qualify,
and ExistingOnly's other guards are untouched.
- Cannot mislead a node upgrading into the feature: backfill seeds from
ADOPTED containers only — evidence that something is really there — skips
anything user-uninstalled, is additive so a momentarily-down app is never
dropped, and no-ops on an empty adoption list (podman unreachable must not
read as "nothing is installed").
Four tests, including the one that states the point: the record must outlive
a running-snapshot that has gone empty.
Container suite 221/221, crash_recovery 15/15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced live: the operator pressed install on ElectrumX shortly after
Bitcoin Knots started and got
Bitcoin RPC returned 500 Internal Server Error while checking pruning
status: {"error":{"code":-28,"message":"Verifying blocks…"}}
-28 is RPC_IN_WARMUP — bitcoind's normal path on every start, lasting
minutes on a large chainstate. check_bitcoin_pruning_compatibility retried
`for _ in 0..3` with a 2s sleep, so it gave the node about six seconds and
then reported warm-up as a hard failure. Any app requiring unpruned Bitcoin
was therefore uninstallable in the ordinary window after Bitcoin starts.
This is the likely mechanism behind the operator's "fedimint gateway
disappeared at 88% install": both fedimint apps declare a bitcoin-core
dependency and sit in exactly that window.
The same install path already knew better. wait_for_bitcoin_rpc_gate waits
180s precisely because getblockchaininfo answers during sync. This check
runs earlier and disagreed — one concern, two contradictory answers, in one
install. It now shares the budget, and a test asserts the two stay equal.
Only NOT-READY is waited out: -28 by code, plus the "loading block index" /
"verifying blocks" / "rewinding blocks" message shapes for a proxy that
rewrites the envelope. A genuine fault — bad auth, method not found,
unparseable body — still ends the loop on the first response, so a broken
RPC fails fast instead of burning 180s. Both directions are pinned by tests,
because being too loose here is as bad as being too strict.
Warm-up is announced ONCE to the install log, so a slow install reads as
"waiting for Bitcoin" rather than a stall. And the failure message now says
what to do — "Bitcoin is still starting up… wait until it reports it's
synced, then try again" — instead of pasting the raw JSON-RPC envelope,
which was accurate and useless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught on archi-dev-box within minutes of deploying 3ac59a73: the reaper
removed archy-bitcoin-ui and archy-lnd-ui, whose backends ARE installed.
archy-bitcoin-ui was gone for 36 minutes, until the operator reinstalled
bitcoin-knots and `reconcile` put the companion back.
Not a logic error — the arithmetic did what it was told. The inputs were
false. Both backends' containers were missing because of the clean-exit
vanishing bug (8908fb4f), and both had already aged out of
running-containers.json, which only ever records what is CURRENTLY RUNNING.
So the two signals `installed_app_ids` combines are not independent: one root
cause falsifies both simultaneously. ORPHAN_GRACE could not help either — the
condition was persistent, not transient, which is exactly the case the grace
period cannot distinguish.
The asymmetry decides it. An un-reaped orphan costs a stale UI tile. A
wrongly-reaped companion costs a working screen and turns one lost app into
two — the reaper amplifies the very failure it was meant to tidy up after.
`reap_orphans` and its tests stay, documented as NOT TO BE WIRED until a
durable record of "this app is installed" exists to drive it. Inferring
installation from runtime state cannot answer that question, however many
runtime signals are combined.
The provisioning half is untouched and is the actual fix for "fedimint
installs but does not work": driving `reconcile` from installed_app_ids means
a companion is never stood up for an app nobody installed, so no NEW orphans
appear. The one genuine orphan on this node (archy-fedimint-ui, for an app
never installed) was correctly removed before this change landed.
Unit tests passed the reaper because they verify the set arithmetic, not
whether the "installed" signal is truthful. Only the device could show that.
Container suite 221/221.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`sign_nip98` read the node's pubkey through `get_nostr_pubkey`, which goes
through `load_or_create_nostr_keys`. On a node with no Nostr identity that
does not fail — it GENERATES a keypair, writes the secret to disk, and signs
with it. So an IndeeHub auth header could quietly create a new node identity
as a side effect, then authenticate as a stranger holding a key nobody has
ever seen. The `.context("node has no Nostr identity")` guarding the call
could never fire, because the call could never fail that way.
`nostr_identity_exists` is the missing distinction: bootstrap may create,
but anything AUTHENTICATING as this node must prove the identity it already
has. sign_nip98 now gates on it and bails loudly.
Caught by `a_nip98_event_names_the_exact_url_and_method`, which asserts
exactly this ("must fail loudly rather than sign something empty") and had
been failing since the file landed in 58c759c1 — invisible because the
earlier runs on this branch filtered to `container::`.
Full bin suite 1381/1381.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box logged `reconcile failed app_id=btcpay-server error=chown
/var/lib/archipelago/postgres-btcpay failed with status exit status: 1`
while BTCPay was running and healthy and there was nothing to repair:
`find /var/lib/archipelago/postgres-btcpay ! -uid 100998` returns zero
files, and the identical command run by hand exits 0. The chown through
`sudo systemd-run` had simply failed once, and that transient failure
propagated out of the pre-start hook and took the app's entire reconcile
with it.
These hooks exist to repair OLD installs. On a healthy node the repair is
already a no-op, so its failure is not evidence of anything being wrong.
repair_dir_ownership folds the gate, the chown and the verdict into one
place: skip when ownership is already right, chown when it is not, and on a
failed chown RE-PROBE before deciding it matters. If the ownership is
correct anyway — a concurrent repair, or a transient sudo/systemd-run
failure on an already-correct tree — warn and continue. Only a chown that
fails AND leaves the ownership wrong is an error, which is the case the
loud failure was written for: a mis-owned volume the app genuinely cannot
open.
Replaces the three hand-rolled gate+chown+bail blocks in
ensure_btcpay_stack_dirs and the one in ensure_fedimint_dirs.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint wait and the post-rotation verify shared one 15-minute budget. A
rotation that legitimately spent 14 of those minutes waiting for LND to mint a
fresh macaroon — normal on a loaded node, where opening channel.db/graph.db/
wallet.db alone has been measured at 2m38s — then had 60 seconds to confirm the
node identity and channel census came back, and would report FAILURE on a wallet
that was completely healthy.
That is the most alarming possible way to be wrong about someone's Lightning
node: it names a backup directory and tells them to investigate before retrying,
at the exact moment nothing is actually broken. Each wait now gets its own
budget. Waiting longer costs nothing here — the failure this step exists to catch
(changed identity, missing channels) is not time-sensitive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Bitcoin Knots disappeared again, plus other apps." Root cause is a pairing,
not a single bug: quadlet renders `podman run ... --replace --rm`, so the
container is deleted the moment it stops, and from_manifest set
Restart=on-failure, which declines to restart after a CLEAN exit. bitcoind
exits 0 on SIGTERM. So any clean stop deleted the container AND left it
deleted — the app vanished from podman and from My Apps until a later
archipelago reconcile tick noticed and recreated it. That is the
"previously-running app has no container after boot — recreating
(desired-state recovery)" line, which fired for bitcoin-knots at 18:53,
19:57 and 20:39 and for electrumx at 19:57 and 20:42 on 2026-08-07.
A crash always self-healed: on-failure restarted the unit and podman run
recreated the container. Only a clean exit stranded it, which is why this
survived so long.
The justification for on-failure was wrong on systemd's own semantics. It
read "clean exits — e.g. operator-issued systemctl stop — stay stopped", but
Restart= is never consulted for a unit stopped via systemctl stop
(systemd.service(5)), and that is exactly how archipelago stops these apps
(prod_orchestrator -> stop_service_with_timeout). Always keeps the
stopped-stays-stopped behaviour and drops the failure mode.
Always also restores the premise of the Quadlet migration — systemd owns
supervision, so an app returns without archipelago alive to notice it left.
Checked before flipping: no manifest declares a one-shot container and there
is no manifest-level restart field, so nothing gets restart-looped.
Propagation to existing nodes is via sync_quadlet_unit's drift re-render,
which rewrites the unit and daemon-reloads WITHOUT restarting the service —
running containers are undisturbed and the new policy governs the next start.
OnFailure is kept as a deliberate opt-in with a note not to wire it back to
backends. Two tests now pin the new default and assert on-failure is absent
from a rendered backend unit.
Container suite 215/215.
NOTE FOR THE OPERATOR: this changes supervision semantics for every app on
the Quadlet canary path. Wants sign-off and a lifecycle-gate run before OTA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
installed_app_ids judged installation on live containers alone. Watched on
archi-dev-box within the hour: lnd read as ABSENT, then as EXISTS again.
Containers on this node come and go — the boot reconciler logs
"previously-running app has no container after boot — recreating" for
bitcoin-knots and electrumx repeatedly — so a momentary gap looked exactly
like a removal, and the reaper would have taken a healthy companion's unit
with it. ORPHAN_GRACE narrows that window but cannot close it: nothing
bounds how long a gap lasts.
An app now counts as installed if its container exists in any state OR its
container name is in the durable last-running snapshot. That snapshot is
what crash_recovery itself calls "installation evidence" and what
reconcile_all_with_mode already trusts to recreate a previously-running app
whose container vanished — the same signal, for the same reason, now shared
rather than reinvented.
Only fedimint is a true orphan on this box: it appears in no adoption list
and has no quadlet unit of its own. lnd is installed and merely flapping,
which is a separate bug.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both default update mirrors resolve to the SAME host — the primary by name over
HTTPS, the fallback by IP over plain HTTP — while SystemUpdate.vue told the
operator "Servers this node checks for updates. The primary is tried first; if
it's slow or unreachable, the next one in the list is tried automatically."
That promises availability redundancy the pair cannot provide: if the origin is
down, both entries are down. Reported by the operator, who read the list and
correctly concluded the fallback made no sense.
The mechanism is fine and deliberate — it recovers a node whose DNS is broken
or whose clock is wrong, both of which fail TLS while plain HTTP still works,
and it is safe because the manifest carries an Ed25519 signature verified
against the pinned release-root anchor, so transport integrity is not what
protects the update. (That last part only became true once Workstream B pinned
the anchor; before then this fallback would have been a real hole.)
So the bug was the labelling, not the design:
- Backend label "Direct (fallback)" -> "Same server, no DNS/TLS", and the
comment now states plainly that it is the same host, what it recovers, and
that real redundancy needs a different one.
- UI copy now scopes the redundancy sentence to genuine mirrors and adds a
paragraph saying the two built-in entries are one server, what the second
actually recovers, that it does not help if the server is down, why an
unencrypted fetch is acceptable, and how to get real redundancy.
The relabel reaches existing nodes: force_ovh_update_primary rewrites labels for
the two default URLs on every load, while the merge matches on URL and never on
label — without that rewrite path a renamed default would have sat in the code
and never propagated to a single deployed node. Noted inline so it is not
re-broken.
Verified: 40/40 update tests pass (including the mirror load/merge/strip ones),
vue-tsc clean, build green, and the new copy is present in the freshly built
SystemUpdate chunk. Nothing in the tree pinned the old label string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 1 items 3 and 5.
Item 3 turned out to be far narrower than the plan's "93 files" once each hit
was classified rather than bulk-replaced. Sanitized only genuine operator
identifiers:
- FIPS test fixtures and a pine_ha comment carried real node LAN addresses ->
RFC 5737 TEST-NET-1, the convention already used elsewhere in this repo.
- Real tailnet addresses in fips/endpoints.rs, mock-backend.js and the mesh
test runner -> the base of the CGNAT range, obviously synthetic.
- Incident comments in appgate/mod.rs and apps/fedimint/manifest.yml named a
specific node; the role is what carries the meaning, so the address is gone.
- CHANGELOG.md held five real addresses in published release notes — the most
exposed of the lot.
Deliberately NOT touched, because the plan's item-3 list is over-broad and
following it literally would break working code:
- 192.168.1.1 / .254, 192.168.0.0/16 and 100.64.0.0/10 are generic router
defaults, RFC1918 classification in backup_rpc, and CGNAT range logic in
pine_ha / CompanionIntroOverlay. Not leaked infra.
- `tx1138` is listed as a hostname to scrub but is two live things: the
user-facing default block explorer (`DEFAULT_TX_EXPLORER`) and
`RETIRED_TX1138_HOST`, the migration constant whose entire job is stripping
that retired registry from existing nodes' saved mirror lists. Scrubbing
either breaks a feature. The plan needs this correction.
- Android's `192.168.1.100` strings are UI placeholder text.
Item 5: added *.key, *.pem, id_rsa*, *.sqlite, *.db to .gitignore, with a
negation for core/archipelago/src/appgate/testdata/*.key. Checked those first —
they are documented throwaway TLS fixtures compiled in via include_bytes!, not
node identity — and the negation stops the new rule silently dropping them if
they are ever regenerated. Verified both directions: fixtures not ignored, a
stray key elsewhere caught.
Verified: residual grep for real infra addresses is clean; audit-secrets.sh
still 5/5; app-catalog drift 0 (the fedimint edit is a YAML comment, which does
not survive parsing into the signed catalog); 44/44 fips tests pass with the
rewritten assertion fixtures.
Note: these test runs shared the working tree with another agent's in-flight
LND work, which was present but unstaged and is not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box was running archy-fedimint-ui and archy-lnd-ui with no
fedimint and no lnd container anywhere on the box. The Fedimint Guardian
UI sat on :8175 serving its "waiting for Bitcoin" page forever with
nothing behind it, which is what the operator reported as "fedimint
guardian installs but does not work" — there was nothing to install, the
UI was already up.
The boot reconciler drove companion provisioning from manifest_ids(),
which is every manifest the node can SEE: the whole apps/ directory plus
the signed-catalog overlay, 56 of them. The app reconciler has drawn this
line since phase 3 (ReconcileMode::ExistingOnly, "merely listing a
catalog manifest never installs an unqualified app"); the companion stage
never got the equivalent guard, so it stood up a UI for every app that
merely had a manifest and then self-healed it forever.
The other half is that reconcile() could only ever ADD. remove_for fires
only on the explicit uninstall RPC, so nothing ever subtracted: an
install that failed after its companion landed, or a container removed
by any other route, left a Restart=always unit alive permanently.
- installed_app_ids() replaces manifest_ids(): app ids whose container
actually exists. Returns Option, because a caller that removes things
on absence must not read "I could not look" as "nothing is installed".
Container presence in ANY state is the whole test — it deliberately
does not inherit the user_stopped/disabled filters, since a stopped app
is still an installed app and treating it otherwise would tear its
companion down and rebuild it on the next start.
- manifest_ids() is deleted rather than left unused. Its contract reads
as "installed" to anyone skimming, which is the whole bug.
- reap_orphans() removes companions whose backend is not installed, after
ORPHAN_GRACE (300s). The grace period is required, not defensive: this
node runs ARCHIPELAGO_USE_QUADLET_BACKENDS=true and a Quadlet app is
briefly containerless while restarting, so reaping on the first absent
tick would cost a healthy companion a teardown plus a possible 900s
image rebuild. A backend that reappears clears its clock.
- Reap failures are logged but kept out of the backoff input. Repair
keeps a companion available; reaping only tidies one away, and a wedged
reap must not back the repair path off to its 1h ceiling.
Every uncertain signal resolves toward not removing: no unit file and a
hung is-active reads as leave-it-alone.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/marketplace-protocol.md described a full authorship-verification chain and
was marked "shipped end-to-end". It wasn't: `signatures.manifest_hash` and
`signatures.did_signature` existed only as two struct fields that nothing read.
The authenticity actually delivered was the Nostr event's NIP-01 Schnorr
signature — which proves who *relayed* an event, not who *authored* the manifest
inside it. Anyone could republish someone else's manifest under their own DID.
Implemented:
- `canonical_signing_bytes` / `manifest_digest` — the signed preimage is the
manifest as canonical JSON (recursively sorted keys, no whitespace) with
`signatures` omitted, SHA-256'd. Canonicalisation is load-bearing, not
cosmetic: `container.env` is a HashMap with per-process random iteration
order, and `serde_json::Map` is only sorted while the `preserve_order` feature
stays off — a feature any crate in the graph can enable for everyone via
feature unification. Either would make the digest vary between runs, so
signatures would fail *intermittently*, which is far worse to diagnose than
failing cleanly.
- `sign_manifest` / `verify_manifest_signature` — Ed25519 over the 32 raw digest
bytes, verified against the key `author.did` encodes (reusing the existing
`identity::pubkey_bytes_from_did_key`).
- `publish` signs before broadcasting, fills `author.did` when empty, and
**refuses** to publish under a DID this node cannot sign as — otherwise we'd
spray manifests across every relay that every verifier then rejects.
- `discover` verifies before caching. A `missing` signature is a normal
unsigned publisher: listed, but earning no identity trust. An `invalid` one is
tampered or forged, so it is **dropped entirely** and logged — it fails closed
rather than appearing behind a warning badge a user can click through.
Trust scoring now requires proof for both identity-derived factors:
- The 30-point identity factor was `did.starts_with("did:")`. An unsigned
manifest with a plausible DID string and a pinned image scored 65 —
"Community" — on no cryptography at all. It now scores 35, "Unverified".
- **The 20-point federation factor is gated too**, which the original spec did
not say. An unverified `author.did` is just a string the publisher chose, so
without this an attacker could copy the DID of a peer the user federates with
and be rewarded for impersonating the party they trust most.
`marketplace.verify` now returns the signature verdict separately from the
advisory policy issues — `valid` has always meant "passes the advisory security
checks", so conflating it with authenticity would have been its own trap.
Tests (22 pass), weighted to the adversarial cases: tampering; tampering that
also rewrites `manifest_hash` while reusing the stolen signature; signing with
key A while claiming B's DID; undecodable did:keys including the old
`z6MkTest123` fixture that used to score 30/30; malformed base64 and
wrong-length signatures; digest stability across map insertion order; the digest
ignoring the `signatures` block; the federation-impersonation case; and a legacy
cache without the new field loading as `missing` rather than defaulting trusted.
Protocol doc rewritten so the preimage rules are normative — a third-party
implementation that canonicalises differently produces signatures we reject, so
"sorted keys, no whitespace, signatures omitted, sign the raw digest" now has to
be stated exactly rather than sketched.
Not included: surfacing the verdict in Marketplace.vue, which reads only
trust_score/trust_tier today. The field reaches the frontend; where the badge
goes is a UI call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`zbase32 0.1.2` is LGPL-3.0+ — the only hard copyleft dependency in the whole
Rust graph and the last remaining blocker for the MIT release
(docs/LICENSE-COMPLIANCE-AUDIT.md §2). Statically linking LGPL code into a Rust
binary obliges us to ship relinkable objects, which is impractical for a node
image.
The audit offered two routes: the MIT `z32` crate, or an original
implementation. Took the latter — z-base-32 is an alphabet substitution over a
bit stream, so ~60 lines removes the blocker while adding *zero* new
dependencies rather than trading one supply-chain entry for another.
**Byte-compatibility was the requirement, not a nice-to-have.** A `did:dht`
identifier IS this encoding of an Ed25519 public key, so any drift would
silently rotate every node's DID and orphan its already-published DHT records.
So the semantics were not guessed: I read the vendored zbase32-0.1.2 source to
extract exactly what `encode_full_bytes` and `decode_full_bytes_str` do —
including that decode truncates to the next lower byte boundary, which is why a
52-character string round-trips to 32 bytes while discarding 4 padding bits.
A model implementation was then validated against three independent sources
before any Rust was written, all five vectors agreeing:
encode(b"testdata", 64) -> qt1zg7drcf4gn (crate doctest)
encode_full_bytes("Just an…") -> jj4zg7bycfzn… (crate doctest)
decode_full_bytes("qb1ze3m1") -> b"peter" (crate doctest)
encode([f0,bf,c7]) -> 6n9hq (Zimmermann spec)
encode([d4,7a,04]) -> 4t7ye (Zimmermann spec)
The module pins all of those plus four known 32-byte keys, a 0..40-byte
round-trip sweep, a 52-char/round-trip check over 64 keys, rejection of the
characters z-base-32 deliberately omits (`l`, `v`, `2`, `0`) and of non-ASCII,
and an alphabet/decode-table consistency check so the compile-time reverse table
can't drift from the alphabet.
`did_dht.rs` gains `did_for_a_known_key_is_stable`, which pins the full
identifier string for a known key — the regression that would actually hurt,
asserted at the call site that gives the string its meaning.
Dropped from Cargo.toml and Cargo.lock (7 lines); no other user in the tree.
Verified: 28/28 network tests pass, zero copyleft crates remain in the lockfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_node_key_known_answer_vs_python_verifier` pinned the node Ed25519 and node
Nostr keys, and `test_release_root_known_answer` covers the release root. The
remaining six — FIPS mesh transport, identity Ed25519, identity Nostr (NIP-06),
Bitcoin BIP-84 and LND aezeed entropy — were only asserted to be mutually
distinct by `test_full_derivation_from_known_mnemonic`.
Distinctness is satisfied by ANY change to an HKDF info string or BIP-32 path.
So redefining `archipelago/lnd/entropy/v1` — the seed behind a user's Lightning
wallet — broke no test, while invalidating every backup verification a user had
already performed against docs/SEED-VERIFICATION.md. Same for the FIPS key that
authenticates a node on the mesh.
Expected values were produced independently by the Python verifier published in
that doc, whose primitives were themselves cross-checked against bip_utils and
cryptography's own HKDF (BIP-39 seed, both BIP-32 paths, x-only pubkey, bech32
and HKDF-SHA256 salt=None all matched byte for byte). This commit closes the
loop in the other direction: the Rust implementation now agrees with those same
bytes, so the doc and the code are pinned to each other.
Verified: 26/26 seed tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reconciler's pre-start hooks for the btcpay stack, fedimint and fmcd
chowned unconditionally on EVERY prepare — and prepare re-runs far more
often than install (every reconcile that touches the app). archi-dev-box's
journal showed the same three dirs re-chowned every ~15s. The hooks exist
to repair old installs; they now skip when ownership is already correct
(root stat probe — the daemon's rootless metadata read can't see the
subuid-owned dirs).
Co-Authored-By: Claude <noreply@anthropic.com>
The first drift-gate attempt called plain sudo stat, which the daemon's
privilege path doesn't answer — the probe silently failed and the chown
loop continued. host_sudo_output mirrors host_sudo (systemd-run --pipe)
but returns the process output, so the ownership check gets a real answer.
Co-Authored-By: Claude <noreply@anthropic.com>
The direct metadata read can be denied in the service's rootless context even
when the directory is already correctly owned, which kept the reconciler
calling sudo chown on the same Postgres volume every minute. A root
fallback gives the guard a reliable answer on deployed nodes while remaining
much cheaper than a recursive chown.
Co-Authored-By: Claude <noreply@anthropic.com>
get_data_dirs_for_app had no btcpay arm — the default mapped to
/var/lib/archipelago/btcpay alone, leaving postgres-btcpay (where the
ACCOUNT lives) and nbxplorer on disk. Uninstall-with-wipe then reinstalled
to the old account still enabled. The btcpay arm now covers all three dirs,
for every alias and stack-member id. The map stays deliberately hardcoded:
deletion code must never derive its targets from a manifest at uninstall
time (a bad manifest could aim the wipe at another app's data).
Co-Authored-By: Claude <noreply@anthropic.com>
apply_data_uid ran a recursive sudo chown on every prepare_for_start, and the
reconciler re-prepares — archi-dev-box's journal showed postgres-btcpay rechowned
every ~45s despite already-correct ownership, and on framework-pt the same loop
surfaced as operator-visible 'chown failed' noise. chown_for_rootless_container
now stats the target first and returns early when the top-level owner already
matches the host-mapped uid:gid. Deep drift in a running container is still
caught by ensure_running_container_ownership's in-container write-probe, which
is the authority that actually matters (it probes writability, not stat bits).
Co-Authored-By: Claude <noreply@anthropic.com>
The repo ships an MIT LICENSE and the README carries an MIT badge, but the
crates themselves declared no license, so `cargo metadata`, packaging and any
downstream mirror saw "license: null". Adds [workspace.package] license = "MIT"
and inherits it in all five members via license.workspace = true. Verified with
cargo metadata: all five now report MIT.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live on archi-dev-box: the node still proxied web-search straight to
SearXNG :8888 unauthenticated — the repo conf was fixed in d0c9ea6e but
existing nodes' /etc/nginx/sites-enabled never gets rewritten by a source
edit. Added to the nginx self-heal battery: stale 8888/search proxy_pass →
session-gated 5678 with the Cookie forwarded (heal_stale_web_search_block,
pure + idempotent + tested). Fresh ISOs already ship the gated block.
Co-Authored-By: Claude <noreply@anthropic.com>
Mesh AssistQuery answered with a bare LLM call — no tools, no actions.
The CallerScope::Mesh variant was designed for this wiring ('the variant
exists so the shape is right when a future plan wires mesh callers into
the shared loop'); this is that plan. A trusted/allowlisted asker's prompt
now runs assistant::chat with CallerScope::Mesh { authorized } — the
operator's persisted grants cap what the model may touch (never wider),
and writes suspend on the node's own confirm gate. The reply is capped
for airtime as before, with a brevity instruction for mesh turns.
Wiring follows the blob_store pattern: RpcHandler::set_mesh_service (now
&Arc<Self>) forward-propagates an Arc<RpcHandler> into the mesh state's
new assistant_handler slot; absent (early boot) falls back to the legacy
bare-LLM answer.
Test: mesh_caller_authority_is_capped_at_operator_grants.
Co-Authored-By: Claude <noreply@anthropic.com>
Two changes, one binary batch:
1. app_install/app_uninstall (task 3): '!ai please install bitcoin knots'
correctly said it can't. Both tools are category-Apps, destructive, and
ride the 13-08 confirm gate (node-authored descriptions added). Install
validates catalog membership BEFORE the dialog (a typo never spends an
approval); uninstall resolves installed ids. Both reach the SAME
package.install/package.uninstall spawns every authenticated caller
uses, via a curated Arc-taking sibling of assistant_dispatch_tool.
2. S6: cloud legs no longer strip prior USER turns from replayed history.
Turn-minimality's allowlist is now the whole conversation's operator
turns (the node's own D-08 transcript, same trust class as this turn),
still mechanically matched, B1 secret scan and 64KB cap unchanged,
fabricated user messages still truncated. The model no longer sees its
own answers without the questions.
Co-Authored-By: Claude <noreply@anthropic.com>
create-release.sh builds the frontend at step 4 and validates the curated
changelog at step 5, then requires the freshly built bundle to contain the new
version. The version reaches the bundle only through the hand-written What's
New list, so on a fresh release that check can only pass if the changelog and
What's New entries are written BEFORE the script runs. Writing them after is
what aborted the first attempt.
Leads with the downgrade bug, since that is the one users saw: an Update button
offering the release withdrawn for an actively exploited 2FA bypass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate failed cargo-fmt. The domain that replaced the IP-based
registry is longer, pushing several test assertions past the width limit, so
rustfmt wanted to re-wrap them. Pure line re-wrapping — no semantic change.
Caught by the pre-flight gate rather than after tagging, which is what it is
for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
package.versions answered installedVersion "15.17" for btcpay-server while
offering "2.4.2" — 15.17 being its postgres dependency's tag. With BTCPay's
own container absent, installed_version fell back to `containers.first()`,
which for a multi-container stack is an arbitrary sibling.
That is the number the update decision is made from, and it is what the UI
shows next to the available version, so a nonsense pair like "installed 15.17,
available 2.4.2" is presented as a legitimate upgrade.
The fallback now only applies when there is exactly one container, which still
covers apps whose container is named differently from their id (immich_server
for immich). With several containers and no identifiable backend, the honest
answer is "unknown" rather than a guess at a sibling.
Extracted as select_backend_container so the rule is testable directly.
Tests: the BTCPay stack case, the lone differently-named container, and the
archy- prefixed preference. Full suite 1157/1157.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app store offered "update to 2.3.9" on a node already running 2.4.2 — the
release that fixes an actively exploited 2FA bypass. Taking it would have
rolled the node back onto the vulnerable version.
Root cause: available_update_for_images compared tags for inequality only.
Same repo + different tag meant "update available", with no ordering. Every
version claim upstream of it can go stale — the signed catalog, a legacy
catalog entry, the image-versions.sh baseline pin — and any one of them
lagging turned into a backwards Update button.
Guard added: when both tags parse as dotted-numeric versions, a lower pinned
version is never offered. Tags that cannot be ordered (RELEASE.2024-11-07…,
14-vectorchord0.4.3) keep the previous behaviour rather than silently losing
updates. This makes stale data fail safe, which matters more than any single
pin being correct.
Four sources still named 2.3.9, three of them able to act on it:
- releases/app-catalog.json — a LEGACY `btcpay` entry, distinct from
`btcpay-server`, carrying a concrete 2.3.9 image. catalog_primary_image
treats that as authoritative, so this is what drove the button. Fixed, but
held back from this commit: it needs re-signing.
- scripts/image-versions.sh — the baseline pin used when the catalog does not
cover an app.
- stacks.rs — the legacy BTCPay installer, twice. The fallback install path
would have deployed 2.3.9 outright.
- neode-ui curatedApps/marketplaceData and public/catalog.json — the store's
displayed version, hardcoded rather than read from the catalog, which is why
it still showed 2.3.9 after the update landed.
Audited every other installer for the same shape. The remaining literals are
the immich stack, which currently agrees with its manifests; hits in
set_config.rs and app_catalog.rs are test fixtures. To keep it that way,
scripts/check-installer-image-pins.py asserts that any installer literal
naming the same repository as an app manifest carries the same tag, and runs
blocking in CI. Verified it catches a simulated revert to 2.3.9.
Tests: 13/13 in image_versions including the exact BTCPay case, a genuine
upgrade still offered, equal versions silent, prerelease suffixes ordered on
their numbers, and opaque tags unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes 4 test failures introduced by 8e814ca0, which I pushed after running
only the container-crate tests while the full suite was still compiling. Both
failures were real defects, not stale assertions.
1. Catalog-driven installs would have failed fleet-wide.
8e814ca0 dropped the old registry address from TRUSTED_REGISTRIES, but the
signed catalog still advertises image refs on it — deliberately, since
rewriting a signed artifact invalidates its signature. Nodes resolve apps
through the catalog, so every install would have been refused with "not
from a trusted registry". Reinstated as LEGACY_REGISTRY_HOST, documented
as transitional and removable only once the catalog is re-signed.
2. The update fallback lost the property it exists for.
update.rs keeps two mirrors on purpose: the domain as primary, and the
old IP over plain HTTP as a fallback, because a node whose DNS or clock is
wrong (both break TLS) must still be able to update itself — the signature,
not the transport, is what makes either source safe. The bulk rewrite
pointed both constants at the domain, leaving the escape hatch dependent on
exactly what it exists to survive. Restored to its original value.
Separately, validate-app-manifest.sh is ported from ruby to python3+PyYAML.
It shelled out to ruby with stderr discarded, so on any machine without ruby
a missing interpreter was reported as "Valid YAML with top-level app block:
FAIL" and every manifest came back REJECTED. This is the first tool an app
developer runs, and it sent them to fix YAML that was never broken. Ruby was
also the odd dependency out — the repo already ships three python scripts.
It now checks for python3 and PyYAML up front and names what is missing, then
parses with PyYAML. Missing keys resolve to an absent-value object that
indexes to itself and prints empty, so call sites lost their per-hop guards:
(((app["container"] || {})["build"] || {})["context"])
becomes app["container"]["build"]["context"]. Booleans still print as
true/false rather than Python's True/False — call sites compare == "true",
so Python's capitalisation would have silently inverted the readonly_root
and no_new_privileges security checks.
Verified: full rust suite 1148/1148, 0 failed. All 56 app manifests validate
(0 rejected, 0 errored) where previously every one was rejected. No signed
artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live evidence, two ways: the 9abc1623 banner never fired because D-16 hides
ungranted tools (model never calls → refused_categories always empty), and
the [[needs:id>]] marker fix failed because a small local model answers with
a workaround narrative instead of emitting structured markers.
The model's reliable, trained behavior is tool CALLING — so disabled tools
are now listed in a DISABLED prompt section and remain in the schema. A call
hits the execution gate, which refuses and records the category → the
trusted chrome offers Settings → AI Data Access. Deterministic and
model-independent. The prompt split is UX/attack-surface shaping; the
security boundary remains the server-side grant re-check in execute_tool
(loop_.rs), unchanged and now the single enforcement layer by design.
Tests: ungranted_tool_only_ever_in_disabled_section (section-aware),
disabled_tools_are_listed_as_callable_but_refused, marker extraction kept
as a harmless safety net. 127/127 assistant suite green.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the registry host across 86 files: 309 references, covering all 40
app manifests, the orchestrator and container crates, the release and catalog
scripts, both demo-images workflows, the ISO builder, demo-deploy, and the
frontend marketplace data.
Verified the domain actually serves the registry before rewriting anything,
rather than assuming the web host implies the registry:
- TLS verifies clean, HTTP/2 on the web root
- an anonymous token grants a manifest fetch (HTTP 200) with no credentials
- skopeo inspect --no-creds resolves an image and lists its tags
That last check is the one that matters: an outside developer with no account
can now pull, which was the functional blocker for publishing at all.
Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop
crossing the network in the clear.
Deliberately NOT rewritten:
- The public FIPS anchor on port 8444. It is a functional network endpoint
every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded
seeds than to leaked infrastructure. The domain does resolve to the same
host, so it could become a hostname, but that adds a DNS dependency to the
path used precisely when things are broken. Worth a deliberate decision,
not a side effect of this change.
- The companion APK on port 2100. The domain returns 404 for that path, so
rewriting it would swap a working URL for a broken one. The Releases page
does serve (200), which is where the plan already wants those binaries.
- releases/app-catalog.json, releases/manifest.json and release-manifest.json.
These carry `signature` and `signed_by`; editing their contents invalidates
the signature and the fleet refuses artifacts that fail verification. They
were rewritten in a first pass and reverted — they must be regenerated and
re-signed through the signing ceremony instead, which needs the mnemonic.
So the catalog still advertises the old host until that ceremony runs. Nodes
resolve images through the signed catalog, not the on-disk manifests, so this
commit alone does not change what a node pulls.
Verified: archipelago-container 75/75; every manifest still parses with a
top-level app block; no signed artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 9abc1623 banner waited on refused_categories, but refused_categories
only fills when the model CALLS a gated tool — and D-16 hides ungranted
tools from the prompt, so the model never calls: it answered 'I can't do
that' in prose and the banner never fired. Live-verified: revoke media,
ask for content, no banner.
- build_system_prompt takes the disabled categories and teaches a marker:
'say it can be switched on in Settings → AI Data Access and end with
[[needs:<id>]]' — category names only, never tool names (D-16 holds)
- extract_needs_markers strips the markers from the reply and folds them
into refused_categories; unknown ids pass through as text (an offer is
the worst a bad marker can cause — never a grant)
- egress's seed-screen test now covers the new paragraph too
Co-Authored-By: Claude <noreply@anthropic.com>
Keeps the dev and test tooling an outside contributor would want, and takes
our node addresses out of it.
Scripts that silently defaulted to one of our nodes now require an explicit
host and exit 2 without one: smoke-test.sh, trust-archipelago-cert.sh,
dev-container-test.sh (which also derives its RPC and health URLs from the
SSH target instead of a second hardcoded copy), and image-recipe/dev-branding.sh.
A default that points at a machine the user does not own is worse than no
default: it fails confusingly, or reaches a stranger's device.
Usage examples, mock data and test fixtures move to the RFC 5737
documentation range (192.0.2.0/24). CGNAT test values stay inside
100.64.0.0/10 so the range-check semantics they exercise still hold, and
192.168.1.0/.1/.254 are left alone — those are gateway logic and UI
placeholders, not our addresses.
Playwright and the perf spec defaulted their baseURL to one of our nodes;
they now default to localhost:8100, the local dev server.
Removed neode-ui APP_URLS entirely. It is dead code — exported, never
imported — and it pinned fedimint's *prod* launch URL to 192.168.1.228:8175.
Had anything consumed it, every user's node would have tried to reach an
address that on their LAN is either nothing or someone else's machine.
Deleting beats sanitizing dead config.
Verified: frontend 868/868 vitest across 108 files; archipelago-container
75/75; mesh tests 9/9; audit-secrets 5/5. Zero node addresses and zero node
names remain in tracked files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two reasons the content surface 'often doesn't surface the content':
- content.indeehub-projects items carried no mime/filename, so the UI
adapter classified every film 'excluded' and the films grid could never
render. They are films: they now declare video/mp4.
- apps_list surfaced the container-list RPC's BARE ARRAY; the broker reads
{ items: [...] }, so the apps grid was silently dropped every turn.
Wrapped at the tool boundary — the shared RPC's own shape is untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
'recommend me 10 scifi films' answered prose-only and OFFERED to check the
catalogue — the preamble invited knowledge recommendations (paragraph 2) but
only ordered tool calls for existence questions (paragraph 3), so the model
never ran the tool and emitted none of the tags the iframe renders as rich
preview cards.
- discovery of a kind the node could hold (films/music/books…) now gets a
catalogue-and-peers check FIRST, knowledge picks on top
- the 'would you like me to look?' stall is banned outright — looking is
one tool call, do it then answer
- the preamble teaches the exact [[film_ext:Title|Year|Director]] /
tv_ext / song_ext / book_ext / podcast_ext formats, same-line with the
one-line reason, real titles only, never for items the tool returned
Co-Authored-By: Claude <noreply@anthropic.com>
- serve_content takes owner_session: a validated operator session skips the
availability/paid gates (Availability::Nobody stays delisted); the cookie
is re-validated in the content handler, same discipline as the model proxy
- the Tor proxy serves already-purchased items from the local content_owned
cache with Range slicing (206) instead of re-hitting the seller's 402 —
the buyer-side store exists so an owned item is never bought twice, and
its cards were rendering as permanent placeholders
- adapter: 'own'-scope items never render locked (a locked card suppresses
the playable URL — the placeholder-only grid the operator reported)
- broker: normalize 'purchased' OwnedRpcItems per item with the seller's
onion, and group 'peers' items per seller onion, so buildMediaUrl gets a
peerOnion and card URLs stop coming out empty
Co-Authored-By: Claude <noreply@anthropic.com>
S4: /aiui/api/web-search proxied straight to SearXNG with no auth — anyone
reaching the web port ran searches attributed to the node's IP. Now routed
through the daemon's session-gated model proxy like the claude/ollama legs
(both nginx server blocks), forcing format=json upstream (the client never
sent it — search could 200 with HTML that parsed as nothing).
S3: the forwarder also serves the STANDALONE frontend, whose bodies carry
full history/images with no assistant loop behind them — a pasted seed
phrase went to Anthropic unscreened. The forwarder now runs the egress
secret-shape scan (G-B1) with the node's own secrets dir as deny corpus on
Claude bodies and search queries; blocked requests get a plain-language 400.
Also fixes a REAL gap in the egress tokenizer found by these tests: a JSON
key glued to a string value's first word ('content":"abandon...') dropped
that word, so an exactly-12-word seed pasted as a bare message yielded an
11-member run — checksum misses, backstop misses. Non-member words now
rescan within the token. egress 15/15 + model_proxy 10/10 green.
Co-Authored-By: Claude <noreply@anthropic.com>
The browser broker redacted log lines (password=/token=/macaroon key=value,
64+ hex, 64+ base64) while the node-side tool only untrusted-wrapped — so a
log line carrying rpcpassword=<32-hex> crossed to cloud backends below the
egress screen's threshold. Port the broker's three patterns to the tool
boundary as a pure line redactor + JSON walker; unit-tested (124 assistant
tests green).
Co-Authored-By: Claude <noreply@anthropic.com>
ai_grants_unified UNIONED the assistant grants.json with the legacy
settings/ai_permissions.json on every read. On archi-dev-box legacy held
all-ten and grants.json held four, so the Settings UI and the AIUI frame
saw every category ON while the assistant refused six — and no UI toggle
could fix it, because both write paths existed but only ai.permissions.set
synced both files. Now: an existing grants.json answers alone; the legacy
file is consulted only when no grants file exists (pre-unification
upgrade), and that read migrates forward and persists the authority.
assistant.grants-set now also rewrites the legacy file in step. Regression
tests: authority is not widened by legacy; migration folds forward once.
Co-Authored-By: Claude <noreply@anthropic.com>
The Network permission's Settings label promises 'no IP addresses', and the
browser-side broker honours it — but the node-side tool forwarded
network.diagnostics verbatim, so a granted Network category sent the node's
WAN IP and SSID (both location-identifying) to cloud model backends. Strip
both at the tool boundary; NAT/UPnP/Tor/DNS connectivity shape stays.
Pure helper + unit test (123 assistant tests green).
Co-Authored-By: Claude <noreply@anthropic.com>
SearXNG defaults to `formats: [html]`. Its JSON API answers 403 —
and JSON is the only thing AIUI's web search speaks, since
`/aiui/api/web-search` proxies straight through to `/search`. Both
places that seed settings.yml (the first-boot script and the installer)
omitted `search.formats`, so web search has never worked on a node
whose SearXNG was installed, running and healthy. It reads as the
assistant being unable to search rather than as one missing config key.
Verified on archi-dev-box: `format=json` went 403 -> 200, returning 28
results for "bitcoin halving" from Brave and DuckDuckGo. Google and
Startpage self-suspend on a self-hosted instance (access denied /
CAPTCHA), which is expected and costs little given Brave's independent
index.
Existing nodes need the same two lines added to
/var/lib/archipelago/searxng/settings.yml and a restart; this commit
only fixes what new installs get.
Also fixes a build break: `fetchLibraryContent` built a bundle literal
that predates the images bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects, one visible symptom: a correct prose answer beside an
empty grid.
1. The assistant's curated RPC bridge had an arm only for
`content.list-mine`. `tools.rs` mapped the `peers`, `purchased` and
`films` scopes onto three real, dispatcher-registered handlers that
`assistant_dispatch_tool` had never heard of, so every non-"own"
scope died on its catch-all. Downstream that read as "the peers have
no content" — it was a missing match arm, and the tool never ran.
Regression test added: every scope the schema advertises must reach a
real handler.
2. `content.browse-all-peers` wrapped its whole fan-out in one
`timeout(..).unwrap_or_default()`, which DISCARDED every completed
batch the moment the budget expired. One slow peer turned a
partly-successful browse into "0 reached, 16 unreachable". Observed
live on archi-dev-box: back-to-back calls returned real peer items,
then nothing. Now accumulates per batch and checks a deadline between
them, so partial results always survive. Budget 20s -> 45s: two
batches of eight at a 10s per-peer timeout had no headroom at all.
3. `assistant.chat` returned only `{ text }`. The structured results of
any content tool the turn ran were dropped inside the loop, so the
surface had nothing to render. The turn now carries them through
(captured raw, before the untrusted wrap, since they go to a renderer
that treats every field as inert data, never back into the prompt).
4. The adapter classified images as 'excluded' and dropped them. A node
sharing mostly photos rendered as an empty grid while AIUI's image
grid sat unused. Images now have a bucket, with the paid-lock and
extension-fallback handling audio and video already had.
Also: the panel says "Loading…" while a turn is in flight and "Nothing
found" when it comes back empty, instead of leaving the previous
query's heading standing as though it answered this one; the system
prompt tells the model to call the content tool and summarise rather
than re-list what the cards already show; and a refused tool now names
its permission category so the trusted chrome can offer the settings
screen instead of leaving "I don't have a tool for that" as the only
clue.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
UNIFIED GRANTS. There were two stores for the same ten categories:
settings/ai_permissions.json (what Settings wrote) and
assistant/grants.json (what actually gates the tool list). Toggling Settings
did nothing for the assistant, so with grants stuck at {"apps","system"} the
model truthfully answered "I don't have a tool for that" no matter what the
operator enabled — the real cause behind "the settings I enable keep
disabling". Their serde forms already matched one-for-one, so this is a
duplicate rather than two concepts. ai.permissions.get/set now read and write
the assistant's grants; the legacy file is still written so a downgrade does
not lose grants, and anything recorded only there is folded in on read.
PEER BROWSE now mirrors Cloud.vue's peer-files fan-out, as the operator asked:
concurrent with a cap and a per-peer timeout, rather than sequential. Cloud
caps at 3 because CHROMIUM's connection pool was starved (02-08) — a browser
constraint the daemon does not share, and measurably wrong here: at 3 a 20s
budget got through 2 batches of 16 peers and reached none. At 8 every peer is
attempted inside the budget.
Measured after deploying: 20.0s, peers_total 16, peers_reached 0. FIPS itself
is healthy (anchor connected, 3 authenticated peers, 4 fips_ok dials) but 14
dials fall back and fail, so the peers are not serving /content. The empty
film list is therefore correct — the transport works and the peers are down.
Reported as partial with counts so the assistant can say so instead of
implying the peers have nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
content.browse-all-peers had a per-peer timeout but no OVERALL budget. On this
node that meant >45s with no answer, which the assistant reported to the
operator as "having trouble accessing the peer content list". Measured cause:
16 federated peers, 1 reachable. Now bounded to 20s total, returning partial
results with peers_reached / peers_total / peers_unreachable / partial, so the
assistant can say "1 of 16 peers answered" instead of implying the rest have
nothing. Verified on the node: 20.015s, was >45s.
NodeCertificateSection had no container — I copied a section that sits INSIDE a
card rather than one that provides its own. Now uses the same
`glass-card px-6 py-6 mb-6` shell and heading level as every other settings
section, so it matches on desktop and mobile.
setup-node-ca.sh now also ensures the nginx HTTPS listener, because a CA is
useless if nothing serves TLS. It binds LAN addresses ONLY: tailscaled already
owns :443 on the tailnet addresses with its own Let's Encrypt cert, so a plain
`listen 443 default_server` binds 0.0.0.0 and fails EADDRINUSE — and nginx then
keeps running the OLD config while the reload reports success. Hit exactly that
on archi-dev-box. Port 80 keeps serving: nodes are reached by IP on LANs where
forcing a redirect would strand anyone who has not installed the CA.
Live now: https://192.168.63.240/ and https://<host>.local/ both 200 with
verify=0 against the node CA, http still 200, tailscaled's 443 untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operator asked "what films are there to watch from my peers" and the model
answered, honestly, that it had no tool for it. It was right: content_list
mapped only to content.list-mine — this node's own shared files. Peer
catalogues and IndeeHub were unreachable from the assistant entirely.
content_list now takes scope: own | peers | purchased | films, dispatching to
content.list-mine / content.browse-all-peers / content.owned-list /
content.indeehub-projects. The model picks from a closed enum and never names
a method, so an invented scope falls back to "own" rather than reaching
anything it was not granted (T-13-34).
Two new RPCs behind it:
- content.browse-all-peers aggregates every federated peer in ONE call. The
dashboard fans this out client-side, but asking a model to enumerate peers
and loop is how it ends up claiming it has no tool. Rides FIPS —
PeerRequest::new(fips_npub, onion, "/content") with a 6s FIPS fast-fail then
Tor — so the onion is the peer's identity and FIPS is the transport.
Sequential with a per-peer timeout, not an unbounded fan-out: 02-08 traced a
real UI stall to browse-peer starving the connection pool. One peer being
down is the normal case and contributes nothing rather than failing the call.
- content.indeehub-projects fetches IndeeHub's catalogue, public plus (via a
node-signed NIP-98 login) the operator's private titles. Node-side because
signing that in the browser would put identity material next to the model,
which this phase rules out by name. Tolerant of IndeeHub's field spellings
across versions, and absent/stopped/empty all yield an empty list rather
than failing the caller.
action_key includes the scope, so listing peers cannot be replayed as listing
own files. 15/15 assistant::tools.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operator: "the AI Data Access settings are not persistent through sessions,
often turns them all off."
They were stored in localStorage, which is scoped to an ORIGIN — and a node
answers on several: LAN address, Tailscale address, <host>.local, hostname.
Granting Media over the LAN and returning over Tailscale showed every switch
off again. Not reset: never set *there*. It also made a working content path
look broken, because every scope silently returns nothing without a grant, so
an ungranted permission is indistinguishable from an empty library — that is
exactly what an empty films search turned out to be.
The grant answers "what may the assistant read about THIS NODE", which is a
property of the node, not of one browser at one address. New
settings/ai_permissions.rs (same shape as session_policy: atomic temp+rename,
sanitised on read and write, fails closed on a corrupt file — an unreadable
grant file must never read as "everything allowed"). New ai.permissions.get /
.set, absent from the unauthenticated allowlist so they require a session.
Migration, not replacement: if this browser holds grants and the node holds
none, the local set is pushed UP rather than wiped. Without that, upgrading
would silently revoke the grants of everyone who set them before this change.
The node still wins in every other direction, so a revocation made on one
device takes effect everywhere — otherwise revoking would be impossible from a
second device.
Unknown category ids are stored verbatim rather than validated against a
hardcoded list: a third copy of that list would silently drop a new category on
upgrade. Storing a category grants nothing by itself — the broker checks before
fetching and the node re-checks before answering (T-13-33).
Hydration happens ONCE at broker start, not inside each permission gate: the
gates are hot-path, and awaiting there adds an RPC to every content and context
request. The first attempt did it per-gate and the existing broker tests caught
it by failing on consumed mocks.
Rust 7/7, store 18/18, broker 23/23.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
