The catalog's default version was {"version": "latest", image:
bitcoin-knots:29.3.knots20260508, default: true}, so a fresh install — or
anyone selecting "latest" — silently got the BIP110/RDTS build. Meanwhile
the manifest's own :latest tag resolves to 29.3.knots20260210 in the
registry, so the two paths installed different daemons.
That matters because 20260508 does not merely enable RDTS, it HALTS until
an operator sets consensusrules=rdts. Node 100.64.204.114 runs it and is
frozen at block 961,692 — blocks AND headers static across 30s, 11 peers,
unpruned — while reporting initialblockdownload=false and
verificationprogress 0.99998, i.e. "synced". The two nodes on 20260210 sit
at the tip, 961,706. Re-pointing a tag could have done that to every node
at once.
Pinned in all three places that would otherwise re-introduce a moving tag:
the manifest, image-versions.sh, and the catalog VERSIONS table. The
"latest" pseudo-version is removed outright — a moving default is not
appropriate for a consensus-critical app — and the default is now the
explicit non-RDTS 29.3.knots20260210. 20260508 remains selectable, so
adopting RDTS stays possible but becomes deliberate.
Verified: both tags' daemons read via `bitcoind --version` — :latest and
:29.3.knots20260210 are both v29.3.knots20260210, so this pin is
behaviour-neutral on nodes already running latest.
NOT DONE HERE: releases/app-catalog.json still carries the old signed
entry. It must be regenerated and re-signed by the operator before it
takes effect — catalog entries override on-disk manifests (origin-wins).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the registry host across 86 files: 309 references, covering all 40
app manifests, the orchestrator and container crates, the release and catalog
scripts, both demo-images workflows, the ISO builder, demo-deploy, and the
frontend marketplace data.
Verified the domain actually serves the registry before rewriting anything,
rather than assuming the web host implies the registry:
- TLS verifies clean, HTTP/2 on the web root
- an anonymous token grants a manifest fetch (HTTP 200) with no credentials
- skopeo inspect --no-creds resolves an image and lists its tags
That last check is the one that matters: an outside developer with no account
can now pull, which was the functional blocker for publishing at all.
Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop
crossing the network in the clear.
Deliberately NOT rewritten:
- The public FIPS anchor on port 8444. It is a functional network endpoint
every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded
seeds than to leaked infrastructure. The domain does resolve to the same
host, so it could become a hostname, but that adds a DNS dependency to the
path used precisely when things are broken. Worth a deliberate decision,
not a side effect of this change.
- The companion APK on port 2100. The domain returns 404 for that path, so
rewriting it would swap a working URL for a broken one. The Releases page
does serve (200), which is where the plan already wants those binaries.
- releases/app-catalog.json, releases/manifest.json and release-manifest.json.
These carry `signature` and `signed_by`; editing their contents invalidates
the signature and the fleet refuses artifacts that fail verification. They
were rewritten in a first pass and reverted — they must be regenerated and
re-signed through the signing ceremony instead, which needs the mnemonic.
So the catalog still advertises the old host until that ceremony runs. Nodes
resolve images through the signed catalog, not the on-disk manifests, so this
commit alone does not change what a node pulls.
Verified: archipelago-container 75/75; every manifest still parses with a
top-level app block; no signed artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bitcoin app vanished from updated nodes: the container exited instantly
with 'sh: Syntax error: "fi" unexpected'. My 1.7.124 change added an
explanatory comment INSIDE the manifest's folded YAML scalar (>-), where
'#' is not a comment — it is literal text that reaches the shell. Folding
joins lines with spaces, so the comment swallowed the 'if ... then' while
the more-indented echo survived as its own line, leaving an orphan 'fi'.
bitcoind never ran, the container exited, and the app disappeared from the
UI because detection is container-based.
Explanations now live above the '- >-' line where YAML really treats them
as comments. The loopback-conf tolerance (-allowignoredconf=1) is unchanged
and still needed.
Adds scripts/check-manifest-shell.py to the release gate: it runs 'sh -n'
over every embedded manifest script and rejects '#' inside these scalars.
Nothing validated this shell before — no YAML parse or Rust test could have
caught it, and it only failed on the node, after signing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portainer: nodes have been running :latest — which is 2.39.1 — while the
manifest pinned 2.19.4 from two years ago. The port migration recreated the
container onto that old pin and Portainer refused to start: it migrates a
database forward, never backward, so an existing install died with 'schema
version does not align' and My Apps showed 'app is not responding'
(100.82.34.38). 2.39.1 published as an immutable tag and pinned forward, so
existing databases keep working and older ones migrate up.
Bitcoin: complements PR #131. That removes the code which kept writing a
datadir bitcoin.conf; -allowignoredconf=1 additionally makes an existing
one non-fatal, so a node already carrying the file recovers on restart
instead of crash-looping until something reinstalls it.
App gate login: rebuilt against the dashboard's own design — rotating
intro backgrounds served from the gate, the glass panel, the Archipelago
mark in its gradient ring, the app's icon as a My Apps tile, and the glass
button. Crucially it no longer sends X-Frame-Options: DENY, which made
every gated app render as unreachable inside My Apps' embedded frame;
frame-ancestors expresses 'only this node may frame me', which
X-Frame-Options cannot.
OTA origin: primary mirror is now source.archipelago-foundation.org over
TLS instead of a bare IP on plaintext. The IP stays as an automatic
fallback for nodes whose DNS or clock is broken — both break TLS, and the
signature, not the transport, is what establishes trust.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both found while setting up the on-node test, and both fail silently in
the same direction — the gate reports success while protecting nothing,
which is the exact failure the module was written to prevent.
1. Loopback-pinned ports were skipped entirely.
`identity.rs` dropped any port whose manifest sets `bind: 127.0.0.1`,
reasoning that a loopback publish is not externally reachable. But
`listener.rs` requires loopback-pinning as the PRECONDITION for gating —
while an app holds 0.0.0.0:<port> the kernel will not let the gate bind
that port at all. So the two contradicted each other: pinning an app, the
one action that lets the gate take over, was also what removed it from
the gated set. Completing the entire migration would have gated nothing,
and GateStatus would have reported zero unprotected ports while doing it.
`bind` cannot carry this decision, because two unrelated intentions
produce an identical loopback publish: Bitcoin's RPC 8332 is pinned so
the LAN CANNOT reach it (fronting it would newly expose it on every host
address, behind a login but exposed where it deliberately was not),
whereas a migrated app is pinned precisely so the gate CAN. Inferring
from `bind` breaks one or the other, so the intent is now declared:
`PortAuth::Local` means the first case. The three ports that are
host-local by intent (bitcoin-core/knots 8332, aiui 5180 — all already
`bind: 127.0.0.1`) say so, and a loopback publish with `auth: session`
stays gated. A test pins that property.
2. The port map was never refreshed.
`AppGate::refresh()` existed, was documented as making catalog changes
apply without a restart, and was called by nothing. The map was built
once in `new()`, so an app installed while the daemon runs would never be
gated — and would never appear in `unprotected` either, so the node would
report itself fully enforced while serving a brand-new app to anyone who
asked. The sweep now refreshes before classifying.
Tests: 22/22 appgate, 73/73 archipelago-container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for the app gate (item 1): before anything can enforce
authentication on app ports, the node has to know which ports are
*supposed* to be reachable without it.
`PortMapping` grows `auth` (PortAuth::Session | None, defaulting to
Session) and `auth_rationale`. The default is deliberately the protected
one. Every app port on this node answered with no credential at all over
LAN, Tailscale, Tor and the FIPS mesh alike — reproduced 2026-08-03 —
precisely because exposure was what you got by saying nothing. Inverting
the default means a new app is protected unless its manifest argues for
an exemption.
Validation makes the argument mandatory: `auth: none` without a
rationale is rejected, and so is a rationale without `auth: none` (that
combination means the author wrote an exemption and did not get one —
shipping it silently would leave them believing otherwise).
17 ports across 12 apps are declared exempt, each with its reason. They
are the ports that cannot sit behind an HTTP login page at all: Lightning
p2p (BOLT-8 noise), LND gRPC/REST and CLN gRPC (macaroon / mutual TLS —
Zeus and remote wallets dial these directly), Bitcoin p2p gossip,
electrum wire protocol, Wyoming voice streams, git-over-SSH, and the UDP
discovery protocols (mDNS, SSDP, STUN). Everything else — 39 published
ports — now defaults to gated.
Bitcoin's RPC 8332 is deliberately NOT exempted: it is already
`bind: 127.0.0.1`, so the gate never sees it, and claiming an exemption
it does not need would put a line in the audit list that means nothing.
If the loopback bind is ever dropped, it fails closed.
Two corpus tests keep this honest: every shipped manifest must parse
under the new rules, and the exempt set is pinned at 17 so any change to
the node's unauthenticated surface has to be a deliberate edit.
Tests: 73/73 archipelago-container, workspace builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The creds-off-argv script used printf "rpcuser=%s" in the manifest's
custom_args; quadlet copies Exec= into the generated service's ExecStart,
where systemd expands %s (user's shell) at load time — bitcoind's rpc.conf
came out as rpcuser=/bin/bash and every RPC consumer got 401s on quadlet
nodes (framework-pt: bitcoin-status, HA block-height sensor, LND chain RPC).
Two-layer fix: quadlet.rs now escapes % -> %% in Exec/Entrypoint/HealthCmd/
Environment emission (with regression test), and the bitcoin manifests write
rpc.conf with plain echo so the catalog also heals nodes still running older
binaries. Release catalog regenerated with the fixed scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Wallet scan modal: fallback buttons no longer flash on open (spinner while
the camera auto-starts, video hidden until live — also kills the Android
WebView play-glyph), decode rate 10/s -> 4/s (preview visibly lagged on
phones), clear messages for BOLT12 offers and LNURL/lightning addresses
(unsupported by the LND backend; full LNURL-pay is a queued feature),
zero-amount invoices in BIP21 URIs prefill from amount=.
- Pine 1.2.0: launch-page copy reflects the auto-seeded HA wiring (only
speaker pairing stays manual), adds the ask-Archy hint and the
silent-answers power-cycle troubleshooting tip (firmware FIFO bug).
- bitcoin-core/knots: rpcuser/rpcpassword move from bitcoind argv (world-
readable in host ps) to a 0600 config file inside the container tmpfs;
the salted txrelay rpcauth hash stays on argv. Catalog regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three stacked bugs made "switch version" silently fail / crash-loop, and
the data-access mismatch corrupted a node's index during recovery attempts.
Backend renderer:
- sync_quadlet_unit ignored the per-app pinned version and re-rendered the
quadlet with the manifest's :latest every reconcile tick, reverting any
switch. Factor the install-time catalog/pin resolution into a shared
resolve_catalog_image() and call it in BOTH install_fresh and
sync_quadlet_unit.
- The renderer folded manifest `entrypoint: ["sh","-lc"]` into Exec=, which
only worked when the image entrypoint was a passthrough shell wrapper. The
versioned images use ENTRYPOINT ["bitcoind"], so Exec=sh -lc ... became
`bitcoind sh -lc ...` and crash-looped. Emit a real Entrypoint= override;
exec_changed now also compares Entrypoint=.
Images:
- Build all bitcoin images (Core + Knots, every version) as container-root
(USER removed) like the legacy :latest image. Chain data is owned by the
data_uid (container uid 102); root reads it via CAP_DAC_OVERRIDE (granted in
the manifest). A non-root USER (the previous uid 1000) can't read existing
chain data → "Error initializing block database". Still fully rootless:
container-root maps to the unprivileged host service user.
Catalog:
- bitcoin-knots versions[]: 29.3.knots20260508/20260507/20260210 +
29.2.knots20251110, "latest" tracking newest.
- bitcoin-core versions[]: add 29.2 + a "latest" entry. All images rebuilt
root and published to the mirror.
Frontend:
- AppSidebar version dropdown: rename the latest option to "Always use the
latest version" (no v prefix), fix right padding, and guarantee the current
selection matches a real option (was rendering blank).
- New InstallVersionModal: full-screen version chooser shown from the App
Store / Discover install button for multi-version apps (Bitcoin Knots/Core),
app icon + "Install <name>", latest pre-selected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Snapshots the in-flight hardening work so subsequent reconcile/Quadlet
phases land on a clean before/after diff.
Changes:
- core/container/src/podman_client.rs: image_uses_insecure_registry()
whitelist for the OVH (146.59.87.168:3000) and legacy Hetzner
(23.182.128.160:3000) HTTP mirrors; podman_network_settings() lifts
custom networks into the Networks map so containers can join them.
- core/archipelago/src/container/prod_orchestrator.rs:
ensure_container_network() creates per-manifest networks on demand;
apply_data_uid() now goes through host_sudo for mkdir -p + chown so
bind-mount roots get created and chowned without password prompts.
- core/archipelago/src/api/rpc/package/{install,update,stacks}.rs:
podman pull adds --tls-verify=false only for whitelisted registries.
- core/archipelago/src/bootstrap.rs: removes stale dev-mode systemd
override on startup (live nodes carried it from old installers).
- core/archipelago/src/config.rs: ignore ARCHIPELAGO_DEV_MODE in prod
binaries — it had been silently rerouting volumes to /tmp.
- apps/bitcoin-{core,knots}/manifest.yml: locate bitcoind at runtime
so image-layout differences don't break entrypoint.
- scripts/app-catalog-image-smoke-test.py: production catalog/image
smoke test that probes a target node before users click Install.
- .gitignore: cover .codex, .pnpm-store, __pycache__, *.bak.
Removes filebrowser.rs.bak and two stale catalog.json.bak files
(verified identical to live counterparts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
