Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint wait and the post-rotation verify shared one 15-minute budget. A
rotation that legitimately spent 14 of those minutes waiting for LND to mint a
fresh macaroon — normal on a loaded node, where opening channel.db/graph.db/
wallet.db alone has been measured at 2m38s — then had 60 seconds to confirm the
node identity and channel census came back, and would report FAILURE on a wallet
that was completely healthy.
That is the most alarming possible way to be wrong about someone's Lightning
node: it names a backup directory and tells them to investigate before retrying,
at the exact moment nothing is actually broken. Each wait now gets its own
budget. Waiting longer costs nothing here — the failure this step exists to catch
(changed identity, missing channels) is not time-sensitive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both default update mirrors resolve to the SAME host — the primary by name over
HTTPS, the fallback by IP over plain HTTP — while SystemUpdate.vue told the
operator "Servers this node checks for updates. The primary is tried first; if
it's slow or unreachable, the next one in the list is tried automatically."
That promises availability redundancy the pair cannot provide: if the origin is
down, both entries are down. Reported by the operator, who read the list and
correctly concluded the fallback made no sense.
The mechanism is fine and deliberate — it recovers a node whose DNS is broken
or whose clock is wrong, both of which fail TLS while plain HTTP still works,
and it is safe because the manifest carries an Ed25519 signature verified
against the pinned release-root anchor, so transport integrity is not what
protects the update. (That last part only became true once Workstream B pinned
the anchor; before then this fallback would have been a real hole.)
So the bug was the labelling, not the design:
- Backend label "Direct (fallback)" -> "Same server, no DNS/TLS", and the
comment now states plainly that it is the same host, what it recovers, and
that real redundancy needs a different one.
- UI copy now scopes the redundancy sentence to genuine mirrors and adds a
paragraph saying the two built-in entries are one server, what the second
actually recovers, that it does not help if the server is down, why an
unencrypted fetch is acceptable, and how to get real redundancy.
The relabel reaches existing nodes: force_ovh_update_primary rewrites labels for
the two default URLs on every load, while the merge matches on URL and never on
label — without that rewrite path a renamed default would have sat in the code
and never propagated to a single deployed node. Noted inline so it is not
re-broken.
Verified: 40/40 update tests pass (including the mirror load/merge/strip ones),
vue-tsc clean, build green, and the new copy is present in the freshly built
SystemUpdate chunk. Nothing in the tree pinned the old label string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 1 items 3 and 5.
Item 3 turned out to be far narrower than the plan's "93 files" once each hit
was classified rather than bulk-replaced. Sanitized only genuine operator
identifiers:
- FIPS test fixtures and a pine_ha comment carried real node LAN addresses ->
RFC 5737 TEST-NET-1, the convention already used elsewhere in this repo.
- Real tailnet addresses in fips/endpoints.rs, mock-backend.js and the mesh
test runner -> the base of the CGNAT range, obviously synthetic.
- Incident comments in appgate/mod.rs and apps/fedimint/manifest.yml named a
specific node; the role is what carries the meaning, so the address is gone.
- CHANGELOG.md held five real addresses in published release notes — the most
exposed of the lot.
Deliberately NOT touched, because the plan's item-3 list is over-broad and
following it literally would break working code:
- 192.168.1.1 / .254, 192.168.0.0/16 and 100.64.0.0/10 are generic router
defaults, RFC1918 classification in backup_rpc, and CGNAT range logic in
pine_ha / CompanionIntroOverlay. Not leaked infra.
- `tx1138` is listed as a hostname to scrub but is two live things: the
user-facing default block explorer (`DEFAULT_TX_EXPLORER`) and
`RETIRED_TX1138_HOST`, the migration constant whose entire job is stripping
that retired registry from existing nodes' saved mirror lists. Scrubbing
either breaks a feature. The plan needs this correction.
- Android's `192.168.1.100` strings are UI placeholder text.
Item 5: added *.key, *.pem, id_rsa*, *.sqlite, *.db to .gitignore, with a
negation for core/archipelago/src/appgate/testdata/*.key. Checked those first —
they are documented throwaway TLS fixtures compiled in via include_bytes!, not
node identity — and the negation stops the new rule silently dropping them if
they are ever regenerated. Verified both directions: fixtures not ignored, a
stray key elsewhere caught.
Verified: residual grep for real infra addresses is clean; audit-secrets.sh
still 5/5; app-catalog drift 0 (the fedimint edit is a YAML comment, which does
not survive parsing into the signed catalog); 44/44 fips tests pass with the
rewritten assertion fixtures.
Note: these test runs shared the working tree with another agent's in-flight
LND work, which was present but unstaged and is not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/marketplace-protocol.md described a full authorship-verification chain and
was marked "shipped end-to-end". It wasn't: `signatures.manifest_hash` and
`signatures.did_signature` existed only as two struct fields that nothing read.
The authenticity actually delivered was the Nostr event's NIP-01 Schnorr
signature — which proves who *relayed* an event, not who *authored* the manifest
inside it. Anyone could republish someone else's manifest under their own DID.
Implemented:
- `canonical_signing_bytes` / `manifest_digest` — the signed preimage is the
manifest as canonical JSON (recursively sorted keys, no whitespace) with
`signatures` omitted, SHA-256'd. Canonicalisation is load-bearing, not
cosmetic: `container.env` is a HashMap with per-process random iteration
order, and `serde_json::Map` is only sorted while the `preserve_order` feature
stays off — a feature any crate in the graph can enable for everyone via
feature unification. Either would make the digest vary between runs, so
signatures would fail *intermittently*, which is far worse to diagnose than
failing cleanly.
- `sign_manifest` / `verify_manifest_signature` — Ed25519 over the 32 raw digest
bytes, verified against the key `author.did` encodes (reusing the existing
`identity::pubkey_bytes_from_did_key`).
- `publish` signs before broadcasting, fills `author.did` when empty, and
**refuses** to publish under a DID this node cannot sign as — otherwise we'd
spray manifests across every relay that every verifier then rejects.
- `discover` verifies before caching. A `missing` signature is a normal
unsigned publisher: listed, but earning no identity trust. An `invalid` one is
tampered or forged, so it is **dropped entirely** and logged — it fails closed
rather than appearing behind a warning badge a user can click through.
Trust scoring now requires proof for both identity-derived factors:
- The 30-point identity factor was `did.starts_with("did:")`. An unsigned
manifest with a plausible DID string and a pinned image scored 65 —
"Community" — on no cryptography at all. It now scores 35, "Unverified".
- **The 20-point federation factor is gated too**, which the original spec did
not say. An unverified `author.did` is just a string the publisher chose, so
without this an attacker could copy the DID of a peer the user federates with
and be rewarded for impersonating the party they trust most.
`marketplace.verify` now returns the signature verdict separately from the
advisory policy issues — `valid` has always meant "passes the advisory security
checks", so conflating it with authenticity would have been its own trap.
Tests (22 pass), weighted to the adversarial cases: tampering; tampering that
also rewrites `manifest_hash` while reusing the stolen signature; signing with
key A while claiming B's DID; undecodable did:keys including the old
`z6MkTest123` fixture that used to score 30/30; malformed base64 and
wrong-length signatures; digest stability across map insertion order; the digest
ignoring the `signatures` block; the federation-impersonation case; and a legacy
cache without the new field loading as `missing` rather than defaulting trusted.
Protocol doc rewritten so the preimage rules are normative — a third-party
implementation that canonicalises differently produces signatures we reject, so
"sorted keys, no whitespace, signatures omitted, sign the raw digest" now has to
be stated exactly rather than sketched.
Not included: surfacing the verdict in Marketplace.vue, which reads only
trust_score/trust_tier today. The field reaches the frontend; where the badge
goes is a UI call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`zbase32 0.1.2` is LGPL-3.0+ — the only hard copyleft dependency in the whole
Rust graph and the last remaining blocker for the MIT release
(docs/LICENSE-COMPLIANCE-AUDIT.md §2). Statically linking LGPL code into a Rust
binary obliges us to ship relinkable objects, which is impractical for a node
image.
The audit offered two routes: the MIT `z32` crate, or an original
implementation. Took the latter — z-base-32 is an alphabet substitution over a
bit stream, so ~60 lines removes the blocker while adding *zero* new
dependencies rather than trading one supply-chain entry for another.
**Byte-compatibility was the requirement, not a nice-to-have.** A `did:dht`
identifier IS this encoding of an Ed25519 public key, so any drift would
silently rotate every node's DID and orphan its already-published DHT records.
So the semantics were not guessed: I read the vendored zbase32-0.1.2 source to
extract exactly what `encode_full_bytes` and `decode_full_bytes_str` do —
including that decode truncates to the next lower byte boundary, which is why a
52-character string round-trips to 32 bytes while discarding 4 padding bits.
A model implementation was then validated against three independent sources
before any Rust was written, all five vectors agreeing:
encode(b"testdata", 64) -> qt1zg7drcf4gn (crate doctest)
encode_full_bytes("Just an…") -> jj4zg7bycfzn… (crate doctest)
decode_full_bytes("qb1ze3m1") -> b"peter" (crate doctest)
encode([f0,bf,c7]) -> 6n9hq (Zimmermann spec)
encode([d4,7a,04]) -> 4t7ye (Zimmermann spec)
The module pins all of those plus four known 32-byte keys, a 0..40-byte
round-trip sweep, a 52-char/round-trip check over 64 keys, rejection of the
characters z-base-32 deliberately omits (`l`, `v`, `2`, `0`) and of non-ASCII,
and an alphabet/decode-table consistency check so the compile-time reverse table
can't drift from the alphabet.
`did_dht.rs` gains `did_for_a_known_key_is_stable`, which pins the full
identifier string for a known key — the regression that would actually hurt,
asserted at the call site that gives the string its meaning.
Dropped from Cargo.toml and Cargo.lock (7 lines); no other user in the tree.
Verified: 28/28 network tests pass, zero copyleft crates remain in the lockfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_node_key_known_answer_vs_python_verifier` pinned the node Ed25519 and node
Nostr keys, and `test_release_root_known_answer` covers the release root. The
remaining six — FIPS mesh transport, identity Ed25519, identity Nostr (NIP-06),
Bitcoin BIP-84 and LND aezeed entropy — were only asserted to be mutually
distinct by `test_full_derivation_from_known_mnemonic`.
Distinctness is satisfied by ANY change to an HKDF info string or BIP-32 path.
So redefining `archipelago/lnd/entropy/v1` — the seed behind a user's Lightning
wallet — broke no test, while invalidating every backup verification a user had
already performed against docs/SEED-VERIFICATION.md. Same for the FIPS key that
authenticates a node on the mesh.
Expected values were produced independently by the Python verifier published in
that doc, whose primitives were themselves cross-checked against bip_utils and
cryptography's own HKDF (BIP-39 seed, both BIP-32 paths, x-only pubkey, bech32
and HKDF-SHA256 salt=None all matched byte for byte). This commit closes the
loop in the other direction: the Rust implementation now agrees with those same
bytes, so the doc and the code are pinned to each other.
Verified: 26/26 seed tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo ships an MIT LICENSE and the README carries an MIT badge, but the
crates themselves declared no license, so `cargo metadata`, packaging and any
downstream mirror saw "license: null". Adds [workspace.package] license = "MIT"
and inherits it in all five members via license.workspace = true. Verified with
cargo metadata: all five now report MIT.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
create-release.sh builds the frontend at step 4 and validates the curated
changelog at step 5, then requires the freshly built bundle to contain the new
version. The version reaches the bundle only through the hand-written What's
New list, so on a fresh release that check can only pass if the changelog and
What's New entries are written BEFORE the script runs. Writing them after is
what aborted the first attempt.
Leads with the downgrade bug, since that is the one users saw: an Update button
offering the release withdrawn for an actively exploited 2FA bypass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release gate failed cargo-fmt. The domain that replaced the IP-based
registry is longer, pushing several test assertions past the width limit, so
rustfmt wanted to re-wrap them. Pure line re-wrapping — no semantic change.
Caught by the pre-flight gate rather than after tagging, which is what it is
for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
package.versions answered installedVersion "15.17" for btcpay-server while
offering "2.4.2" — 15.17 being its postgres dependency's tag. With BTCPay's
own container absent, installed_version fell back to `containers.first()`,
which for a multi-container stack is an arbitrary sibling.
That is the number the update decision is made from, and it is what the UI
shows next to the available version, so a nonsense pair like "installed 15.17,
available 2.4.2" is presented as a legitimate upgrade.
The fallback now only applies when there is exactly one container, which still
covers apps whose container is named differently from their id (immich_server
for immich). With several containers and no identifiable backend, the honest
answer is "unknown" rather than a guess at a sibling.
Extracted as select_backend_container so the rule is testable directly.
Tests: the BTCPay stack case, the lone differently-named container, and the
archy- prefixed preference. Full suite 1157/1157.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app store offered "update to 2.3.9" on a node already running 2.4.2 — the
release that fixes an actively exploited 2FA bypass. Taking it would have
rolled the node back onto the vulnerable version.
Root cause: available_update_for_images compared tags for inequality only.
Same repo + different tag meant "update available", with no ordering. Every
version claim upstream of it can go stale — the signed catalog, a legacy
catalog entry, the image-versions.sh baseline pin — and any one of them
lagging turned into a backwards Update button.
Guard added: when both tags parse as dotted-numeric versions, a lower pinned
version is never offered. Tags that cannot be ordered (RELEASE.2024-11-07…,
14-vectorchord0.4.3) keep the previous behaviour rather than silently losing
updates. This makes stale data fail safe, which matters more than any single
pin being correct.
Four sources still named 2.3.9, three of them able to act on it:
- releases/app-catalog.json — a LEGACY `btcpay` entry, distinct from
`btcpay-server`, carrying a concrete 2.3.9 image. catalog_primary_image
treats that as authoritative, so this is what drove the button. Fixed, but
held back from this commit: it needs re-signing.
- scripts/image-versions.sh — the baseline pin used when the catalog does not
cover an app.
- stacks.rs — the legacy BTCPay installer, twice. The fallback install path
would have deployed 2.3.9 outright.
- neode-ui curatedApps/marketplaceData and public/catalog.json — the store's
displayed version, hardcoded rather than read from the catalog, which is why
it still showed 2.3.9 after the update landed.
Audited every other installer for the same shape. The remaining literals are
the immich stack, which currently agrees with its manifests; hits in
set_config.rs and app_catalog.rs are test fixtures. To keep it that way,
scripts/check-installer-image-pins.py asserts that any installer literal
naming the same repository as an app manifest carries the same tag, and runs
blocking in CI. Verified it catches a simulated revert to 2.3.9.
Tests: 13/13 in image_versions including the exact BTCPay case, a genuine
upgrade still offered, equal versions silent, prerelease suffixes ordered on
their numbers, and opaque tags unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes 4 test failures introduced by 8e814ca0, which I pushed after running
only the container-crate tests while the full suite was still compiling. Both
failures were real defects, not stale assertions.
1. Catalog-driven installs would have failed fleet-wide.
8e814ca0 dropped the old registry address from TRUSTED_REGISTRIES, but the
signed catalog still advertises image refs on it — deliberately, since
rewriting a signed artifact invalidates its signature. Nodes resolve apps
through the catalog, so every install would have been refused with "not
from a trusted registry". Reinstated as LEGACY_REGISTRY_HOST, documented
as transitional and removable only once the catalog is re-signed.
2. The update fallback lost the property it exists for.
update.rs keeps two mirrors on purpose: the domain as primary, and the
old IP over plain HTTP as a fallback, because a node whose DNS or clock is
wrong (both break TLS) must still be able to update itself — the signature,
not the transport, is what makes either source safe. The bulk rewrite
pointed both constants at the domain, leaving the escape hatch dependent on
exactly what it exists to survive. Restored to its original value.
Separately, validate-app-manifest.sh is ported from ruby to python3+PyYAML.
It shelled out to ruby with stderr discarded, so on any machine without ruby
a missing interpreter was reported as "Valid YAML with top-level app block:
FAIL" and every manifest came back REJECTED. This is the first tool an app
developer runs, and it sent them to fix YAML that was never broken. Ruby was
also the odd dependency out — the repo already ships three python scripts.
It now checks for python3 and PyYAML up front and names what is missing, then
parses with PyYAML. Missing keys resolve to an absent-value object that
indexes to itself and prints empty, so call sites lost their per-hop guards:
(((app["container"] || {})["build"] || {})["context"])
becomes app["container"]["build"]["context"]. Booleans still print as
true/false rather than Python's True/False — call sites compare == "true",
so Python's capitalisation would have silently inverted the readonly_root
and no_new_privileges security checks.
Verified: full rust suite 1148/1148, 0 failed. All 56 app manifests validate
(0 rejected, 0 errored) where previously every one was rejected. No signed
artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the registry host across 86 files: 309 references, covering all 40
app manifests, the orchestrator and container crates, the release and catalog
scripts, both demo-images workflows, the ISO builder, demo-deploy, and the
frontend marketplace data.
Verified the domain actually serves the registry before rewriting anything,
rather than assuming the web host implies the registry:
- TLS verifies clean, HTTP/2 on the web root
- an anonymous token grants a manifest fetch (HTTP 200) with no credentials
- skopeo inspect --no-creds resolves an image and lists its tags
That last check is the one that matters: an outside developer with no account
can now pull, which was the functional blocker for publishing at all.
Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop
crossing the network in the clear.
Deliberately NOT rewritten:
- The public FIPS anchor on port 8444. It is a functional network endpoint
every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded
seeds than to leaked infrastructure. The domain does resolve to the same
host, so it could become a hostname, but that adds a DNS dependency to the
path used precisely when things are broken. Worth a deliberate decision,
not a side effect of this change.
- The companion APK on port 2100. The domain returns 404 for that path, so
rewriting it would swap a working URL for a broken one. The Releases page
does serve (200), which is where the plan already wants those binaries.
- releases/app-catalog.json, releases/manifest.json and release-manifest.json.
These carry `signature` and `signed_by`; editing their contents invalidates
the signature and the fleet refuses artifacts that fail verification. They
were rewritten in a first pass and reverted — they must be regenerated and
re-signed through the signing ceremony instead, which needs the mnemonic.
So the catalog still advertises the old host until that ceremony runs. Nodes
resolve images through the signed catalog, not the on-disk manifests, so this
commit alone does not change what a node pulls.
Verified: archipelago-container 75/75; every manifest still parses with a
top-level app block; no signed artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keeps the dev and test tooling an outside contributor would want, and takes
our node addresses out of it.
Scripts that silently defaulted to one of our nodes now require an explicit
host and exit 2 without one: smoke-test.sh, trust-archipelago-cert.sh,
dev-container-test.sh (which also derives its RPC and health URLs from the
SSH target instead of a second hardcoded copy), and image-recipe/dev-branding.sh.
A default that points at a machine the user does not own is worse than no
default: it fails confusingly, or reaches a stranger's device.
Usage examples, mock data and test fixtures move to the RFC 5737
documentation range (192.0.2.0/24). CGNAT test values stay inside
100.64.0.0/10 so the range-check semantics they exercise still hold, and
192.168.1.0/.1/.254 are left alone — those are gateway logic and UI
placeholders, not our addresses.
Playwright and the perf spec defaulted their baseURL to one of our nodes;
they now default to localhost:8100, the local dev server.
Removed neode-ui APP_URLS entirely. It is dead code — exported, never
imported — and it pinned fedimint's *prod* launch URL to 192.168.1.228:8175.
Had anything consumed it, every user's node would have tried to reach an
address that on their LAN is either nothing or someone else's machine.
Deleting beats sanitizing dead config.
Verified: frontend 868/868 vitest across 108 files; archipelago-container
75/75; mesh tests 9/9; audit-secrets 5/5. Zero node addresses and zero node
names remain in tracked files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An app port must answer whatever the browser asks for: an HTTP dashboard
embeds http://host:PORT, an HTTPS one embeds https://host:PORT, and an HTTPS
page cannot embed an HTTP frame at all. So the choice is per-node, not
per-fleet, and a second port number would mean every manifest changes and
torrc doubles.
Instead the gate peeks the first byte. A TLS ClientHello is 0x16; no HTTP
method starts with it. peek() leaves the bytes in the socket buffer, so the
acceptor still sees a complete, untouched ClientHello. TLS and plain share one
generic serve_http(), so authentication, proxying and upgrade handling cannot
drift apart by scheme.
EXISTING NODES ARE UNAFFECTED BY CONSTRUCTION. Anything that is not a TLS
handshake takes the identical path as before, and a node with no certificate
serves plain HTTP exactly as today — TLS is strictly additive.
rustls does NOT verify that a private key matches its certificate. Established
by test, not assumed: with_single_cert accepted a pair from two different keys
and would only have failed mid-handshake in a user's browser — a security
control that reports success and does nothing, the exact shape this module's
own docs warn about. So the pairing is now proven explicitly (sign a fixed
message with the key, verify against the certificate's public key) and a
mismatch refuses to serve.
Also: cert and key mtimes are stamped as a PAIR, because reissuing writes them
separately and keying on one would serve a certificate that no longer matches
its key; a 15s first-byte timeout closes the slowloris window one step earlier
than the existing header-read timeout; PKCS#8 and PKCS#1 keys are both
accepted so a hand-made key does not silently downgrade a working node.
Deps pinned to the rustls 0.21 line reqwest already resolves — no new vendor,
no second rustls major. Test fixtures are throwaway (localhost SANs only), not
any node's identity.
38/38 appgate tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applying RF settings deliberately restarts the radio daemon (~15-20s).
Two things treated that healthy, expected gap as a fault (operator,
2026-08-06):
- radio_state was single-shot: a query landing inside the restart
window reported "The radio daemon did not answer the state query"
for a restart that was working correctly. It now retries for ~30s
and says the radio is restarting while it waits. A real device-level
refusal (not an RNode) still returns immediately.
- The device-setup modal auto-opens for any detected-but-unconnected
port, so the restart looked like a newly plugged stick and
interrupted the apply. Apply and Reboot now suppress auto-detect for
90s via mesh.suppressDeviceDetect().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying the .126 LoRa panel ahead of its daemon made every button
report "Operation failed. Check server logs for details." — the panel
was calling RPCs the older binary doesn't have, and the sanitizer
masked "Unknown method: mesh.rnode-config" into that generic string.
Read as "the feature is broken" rather than "this node needs its
update" (operator, 2026-08-06).
Allowlisted: "Unknown method" (a frontend newer than its daemon should
say so), every RNode RF validation message (each names the field and
its legal range — the entire point of validating before touching the
radio), and the actionable mesh preconditions (no device connected,
mesh service not running, MeshCore has no remote reboot, radio daemon
did not answer, RNode interface disabled).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The port map deferred to DISK manifests for any app with a build source
— which is exactly the four companion UIs (lnd-ui, bitcoin-ui,
electrs-ui, fips-ui). Their disk manifests reach nodes only via the
frontend runtime payload or a per-node repo checkout, and in the
v1.7.125 rollout both proved stale or entirely absent: one node had no
checkout at all, others restored an older payload over apps/ at every
boot. Result: session_passthrough never reached the gate, so the node's
own screens 401'd on every data call, and on nodes whose UI rebuilt
from a stale context the app held its port UNGATED.
Classification now uses a ports-only overlay that accepts build-source
manifests (install/orchestration still defers to disk — unchanged). The
signed catalog is the freshest, operator-signed source, and the gate's
address binds fail safely against a container publishing differently
(logged CANNOT PROTECT), so this can only tighten policy, never expose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh.rnode-config: persisted RF settings + best-effort live radio
state (radio-confirmed r_* values) for the LoRa panel.
- mesh.rnode-config-apply: validate → persist → restart the radio
daemon → poll the read-back until the radio reports online, returning
{applied, confirmed, live, message}. Failure modes report what
actually happened instead of pretending success.
- RebootRadio carries a reply channel: Meshtastic reboots firmware,
Reticulum restarts the sidecar (re-detect + reapply RF config),
MeshCore honestly reports it has no remote reboot — previously the
Reticulum/MeshCore arms returned Ok(()) doing NOTHING: the operator's
"button gives no feedback" bug.
- MeshCommand::QueryRadioState plumbs the sidecar's radio_state to the
service layer with a timeout instead of fire-and-forget.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The .126 LoRa panel's Rust half:
- mesh::rnode_settings: RNodeRfSettings persisted at
<data_dir>/rnode-rf-settings.json — every RNodeInterface parameter
(enabled, port override, frequency, bandwidth, sf, cr, txpower,
airtime_limit_short/long), validated against the bounds RNS itself
enforces. Defaults are byte-identical to the sidecar's historical
argparse defaults.
- FIRST-RUN ADOPTION (operator requirement: the update must change NO
device's applied settings): with no settings file yet, the node's
existing RNS config (~/.archy-reticulum, else ~/.reticulum) is parsed
and its RNodeInterface values adopted verbatim as the initial
settings — proven by a test carrying the operator's literal
"RNode LoRa Portugal" config.
- Serial spawns pass the settings as explicit sidecar args (frequency/
bandwidth/txpower/sf/cr + airtime locks); the operator port override
wins over auto-detect but still passes the KISS probe gate; a
disabled interface refuses to open with a readable error.
- ReticulumLink::query_radio_state(): asks the sidecar for the live
RNodeInterface state (radio-confirmed r_* values) — the panel's
apply-confirmation read-back source.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chown_for_rootless_container prefers `podman unshare chown` (which maps
container uid N through the userns), but when that failed once it fell
back to `sudo chown -R <literal>` — writing e.g. host uid 999 for
container uid 999 and reporting success. Host-999 maps to nobody inside
the userns, so the app could not open its own data while everything
claimed the chown worked: botfights on framework-pt crash-looped every
10s on SQLITE_CANTOPEN over a data dir the daemon itself had just
"fixed".
The sudo fallback now translates container ids (1..99999) to
subuid_base + id - 1 (fleet base 100000; container root maps to the
service user, 1000). Already-mapped ids and uid 0 pass through.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A sweep of 92 unconfirmed/dust sats failed with LND's debug-flavored
"insufficient input to create sweep tx: input_sum=0 BTC, output_sum=
0.00000092 BTC" — and the RPC sanitizer then masked even that behind
"Operation failed. Check server logs." (framework-pt, 2026-08-06).
The sweep mechanics are untouched (balance minus fee, as always) —
this only makes the refusal say WHY in plain language.
- lnd.sendcoins translates the sweep refusal: balance below Bitcoin's
dust minimum or not yet confirmed, so no transaction can be built
(LND's original message kept in parens).
- "Failed to send" joins the sanitizer's user-facing allowlist — the
same lesson as "Insufficient balance"/"Payment failed" before it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The periodic reconcile runs ExistingOnly — merely listing a catalog
manifest must never install an app — and its only absent-container
recovery keyed on the last running-names snapshot, which ages out after
a few daemon restarts. An absent member of an installed stack then stays
absent forever: .38 ran indeedhub with no minio/postgres for days, nginx
down on 'host not found in upstream "minio"', and nothing ever put the
members back.
A live sibling container is proof the stack is installed on this node,
so an absent member is now treated as a hole to repair, not a choice to
respect: the recovery guard also fires when another member of the same
stack (app_ops::stack_member_app_ids) has a container in any state.
A stack with no containers at all is left untouched, and sibling app ids
resolve through the loaded-manifest container names (immich-postgres
runs as immich_postgres).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both predate this session's changes and were masked by the release
gate's cargo-test-weekly compile timeout:
- login_page_sources_its_art_from_the_gate still asserted the retired
wordmark (logo-archipelago.svg); the login page ships the sidebar A
mark (favico-black-v2.svg) since the 2026-08-05 rework.
- unauthenticated_ports_are_all_accounted_for lagged at 17; the
v1.7.123 port-policy round grew the rationale-carrying exempt set
to 25 (reviewed and enumerated in the test comment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two daemon bugs, one debugging arc (2026-08-05, operator-reported):
1. The app gate removed the ENTIRE Cookie header before proxying. That
broke the data plane of every first-party companion UI behind the gate
(lnd-ui/bitcoin-ui/electrs-ui/fips-ui render their shell, then every
/proxy/* and /lnd-connect-info call 401s — observed as "LND UI
unreachable"), and silently logged users out of every gated app with
its own cookie login (vaultwarden, nextcloud, gitea) on each request.
The gate now strips only its own cookie pairs (session, csrf_token);
a new per-port manifest opt-in `session_passthrough: true` forwards
the node session to first-party UIs whose nginx proxies the daemon's
authenticated endpoints. Undeclared ports never get passthrough.
2. podman_client::create_container sent named volumes to the libpod API
as bind mounts with the bare volume name as source, so creating any
manifest app with a `type: volume` mount failed. On .38 the reconciler
removed indeedhub-postgres/-minio for env drift and then could never
create their replacements, leaving the stack half-missing forever.
Named volumes now ride the spec's `volumes` field ({Name, Dest,
Options}). Also: the reconcile-failure log now prints the full anyhow
chain — `%e` showed only "create_container X" and hid the real error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unbreaks Bitcoin on every node running the 1.7.124 catalog: the embedded
start script had a shell syntax error, so bitcoind never launched and the
app vanished. Delivered by catalog rather than a release because manifests
reach nodes through the signed catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate-app-catalog.py writes APP_LAUNCH_PORTS one entry per line; rustfmt
packs it. The release gate checks formatting, so the generated file has to
be formatted after regeneration or every catalog sync fails the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portainer's image reaches the public catalog (the release gate caught the
manifest and catalog disagreeing), and fips-ui 8336 joins the mesh relay's
port list now that it declares a port — it is auth: gated, so the relay
withholds it rather than bridging it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-bumped so the release gate compiles the test profile at the final
version — create-release bumps after the gate, so the gate would otherwise
run on the old version and the bump would invalidate the cache, timing out
cargo-test-weekly on the compile rather than the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
auth.session-policy.get/set plus a card under Account. Presented as two
plain questions rather than the token mechanism underneath, because the
distinction that matters to an operator is which control actually ends a
session: the dashboard polls constantly, so an idle timeout alone never
fires on an open tab — the absolute cap is what guarantees it.
Values are clamped server-side and the stored result is echoed back, so
the bounds are discoverable instead of an error. Presets rather than a free
number field: a box accepting '5' invites locking yourself out. A short
idle choice warns that it is the payments-industry posture.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
austin-sapien (100.70.96.88) sat dead for over two hours after taking
v1.7.122 — 'server starting' in the UI, service inactive, exit status
0/SUCCESS. It did not crash: the in-process updater replaces the binary and
exits cleanly for systemd to restart it, and that node's unit still carried
Restart=on-failure from an older install. systemd read the clean exit as
success and left it stopped. Every node with the old unit has this waiting
for it on the next update.
self-update.sh does refresh units, but the in-process update path never
runs it, so nothing was repairing them. The daemon now checks its own unit
at boot and rewrites only the Restart= line, so a node that starts even
once ends up with a policy that survives the next update.
Also carries the session-policy wiring: validate() now honours the
configured idle and absolute limits and the per-device class, instead of
the single hard-coded 24h constant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mesh right panel: a >=2560px screen hid the tab bar and stacked all five
tool panels in fixed grid rows. On a real display that clipped the Bitcoin,
Dead Man and AI headings to a few pixels each, letterboxed the map, and
pushed Radio Settings into a scroll — more screen producing a worse view.
Very wide now uses the same tabbed column as every other desktop width,
with the selected panel filling the column and the map running edge to edge
(it is the one panel with nothing to scroll).
Session policy: idle timeout, absolute cap and a re-prompt-for-funds flag,
persisted and clamped. Two tokens already existed — a session token and a
30-day login token — so the knob changes how long a quiet tab stays usable
without putting a long-lived credential on every request. Kiosk screens are
exempt from the idle timeout (nobody is there to log a TV back in) but keep
the absolute cap so a stolen box does not stay authenticated forever. The
cap is not optional theatre: idle alone never fires on a polling dashboard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portainer: nodes have been running :latest — which is 2.39.1 — while the
manifest pinned 2.19.4 from two years ago. The port migration recreated the
container onto that old pin and Portainer refused to start: it migrates a
database forward, never backward, so an existing install died with 'schema
version does not align' and My Apps showed 'app is not responding'
(100.82.34.38). 2.39.1 published as an immutable tag and pinned forward, so
existing databases keep working and older ones migrate up.
Bitcoin: complements PR #131. That removes the code which kept writing a
datadir bitcoin.conf; -allowignoredconf=1 additionally makes an existing
one non-fatal, so a node already carrying the file recovers on restart
instead of crash-looping until something reinstalls it.
App gate login: rebuilt against the dashboard's own design — rotating
intro backgrounds served from the gate, the glass panel, the Archipelago
mark in its gradient ring, the app's icon as a My Apps tile, and the glass
button. Crucially it no longer sends X-Frame-Options: DENY, which made
every gated app render as unreachable inside My Apps' embedded frame;
frame-ancestors expresses 'only this node may frame me', which
X-Frame-Options cannot.
OTA origin: primary mirror is now source.archipelago-foundation.org over
TLS instead of a bare IP on plaintext. The IP stays as an automatic
fallback for nodes whose DNS or clock is broken — both break TLS, and the
signature, not the transport, is what establishes trust.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the Bitcoin crash-loop on 100.82.34.38: since a597c1d9
bitcoind launches with -conf=/tmp/rpc.conf and never reads the datadir
bitcoin.conf, but write_bitcoin_conf / ensure_bitcoin_rpc_config /
run_bitcoin_rpc_repair kept writing one on every install and restart.
Bitcoin Core's own datadir-conflict check then refuses to start at all.
Conflict resolved in favour of the PR: HEAD still carried
write_bitcoin_conf, whose deletion is the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mint failures surfaced raw JSON ({"detail":"proofs already spent"}) to
the user; now the top-level message is actionable while the raw body stays
in logs via {:#}.
A rebuilt image never reached a running companion. ensure_image_present
rebuilds in place under the same tag, so the quadlet body is identical,
write_if_changed reports no change, and enable_now is a no-op on a running
service — the container keeps the old layers indefinitely.
That is precisely how archi-dev-box kept serving the LND, FIPS, Electrs and
Guardian screens on 0.0.0.0 after v1.7.123 rebuilt every one of those images
to bind loopback: correct images on disk, three-day-old containers still
running. Closing those ports needed a manual 'podman rm -f' per container,
which no other node would ever get. Compare the running container's image ID
against the built one and restart when they diverge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scanning archi-dev-box from OUTSIDE found five ports serving their screens
with no login — lnd-ui 18083, bitcoin-ui 8334, fips-ui 8336, electrs-ui
50002 and the Fedimint Guardian 8175 — none of which appeared in the gate's
unprotected list. They are host-networked, so Podman publishes nothing to
pin and their manifests declared 'ports: []'; the gate builds its map from
declared ports, so it neither protected them nor reported them. An audit
that reports success while five screens are open is worse than no audit.
Their nginx now listens on 127.0.0.1 instead of 0.0.0.0, and each port is
declared 'auth: gated' so the daemon owns the outside. 'bind:' on a
host-networked app is a statement of where the container listens, not a
publish instruction — quadlet already skips PublishPort in host mode.
Guardian 8175 is declared on the fedimint app because its companion has no
manifest, and the gate keys on port, not container.
Credential paths were NOT exposed and are verified so: /lnd-connect-info,
the /proxy/lnd/ passthrough, container logs and every RPC method through
these screens all return 401 unauthenticated. What leaked was the page
shell.
Also fixes the delivery gap that would have made this unshippable: only
bitcoin-ui, lnd-ui and electrs-ui were ever rsynced to
/opt/archipelago/docker, so edits to fips-ui and fedimint-ui reached nodes
through no path at all. All five now sync; the two whose rebuilds the
daemon owns are synced without being handed to container-specs.
Every remaining undeclared port is now declared with a stated reason —
gated: botfights 9100, router 8084, pine 10380; exempt with rationale:
fedimint consensus 8173/8174, gateway 8176/9737, netbird 8086/8087 (TLS +
own auth, and enrolled devices cannot hold a session), pine TLS 10381,
lightning-stack REST 8091 (macaroon, mirrors lnd). Zero undeclared ports
remain across all 56 manifests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mint HTTP failures (swap/melt/mint-quote) were surfacing raw JSON bodies
like {"detail":"proofs already spent","code":11001} straight to the
user. Add a translator for the NUT-02/03/04/05 transaction-validation
error codes (10001-11017, 12001-12003; see
https://github.com/cashubtc/nuts/blob/main/error_codes.md) and layer it
onto the mint_client bail sites via anyhow context, so the top-level
message is actionable while the raw status/body stays available via
{:#} for logs. receive_token now surfaces the real reason (e.g. "This
ecash has already been redeemed") instead of a generic "Failed to
receive any proofs from token" when every mint in a token fails.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Leads with what changes for the operator: app screens now require the node
password across LAN, Tailscale, mesh and Tor; the wallet/protocol ports that
must stay open stayed open; the mesh leak found during on-node verification;
nodes repairing their own legacy containers; and the signing-key rotation.
Known gaps disclosed, including the eleven still-undeclared ports and that
non-browser clients will now meet the login page.
The new block uses <strong> rather than the literal ** markers in earlier
entries, which render as asterisks in the modal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DO NOT MERGE INTO A RELEASE SIGNED WITH THE NEW KEY. See below.
The previous release root (z6Mkkid…q7ur, pinned 2026-07-02) was exposed
in a chat transcript and is treated as compromised. It signs both OTA
manifests and the app catalog, so anyone holding it could sign updates
the fleet would install.
Pins the new key in trust::anchor and moves EXPECTED_DID in all three
signing/publishing scripts.
ORDERING IS CRITICAL — nodes pin the OLD key:
* The release CARRYING this commit must be signed with the OLD key.
That is the only signature a node running the previous binary will
accept, and it is what installs the binary pinning the new key.
* Only the release AFTER that may be signed with the new key.
* Signing this release with the new key makes every node reject it,
ending OTA fleet-wide and requiring hands-on recovery per node.
sign-catalog.sh moves in the same commit, so the app catalog must also be
re-signed with the new key once this ships, or nodes accept the binary
and reject the catalog.
Key verified before pinning: the hex and the did:key are the same
keypair, checked with a base58 decoder round-tripped against the previous
known-good pair. An earlier candidate hex (cb830e13…) was rejected
because it decoded to a different DID than the one supplied — pinning it
would have made every node reject every future update.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught verifying the gate fixes on archi-dev-box: [fips0-ULA]:32838
answered HTTP 200 straight from nbxplorer with no credential. The
catalog declares that port auth: local — host-local by intent, pinned to
loopback, the gate deliberately keeps its hands off — but the mesh relay
bridges a STATIC port list to 127.0.0.1, so it republished it to the
whole mesh. Same bug class as the Tor onion gap: a transport that
converges on the app loopback without consulting the declaration.
PortMap now records declared-local ports and the relay withholds them
(tearing down an existing bridge if a catalog refresh newly declares
one), alongside the declared-gated withhold. Undeclared ports keep
todays behaviour — silence is not an instruction in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
