Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The previous commit on this branch copied cuprated's raw
--generate-config defaults (stdout=info, file=debug, max_log_files=7)
verbatim. Turns out that's the wrong reference: compared against
ssmithx@archy-dev-pa:/home/ssmithx/cuprate/Cuprated.toml — the actual
dev config this app was built and tested against — file logging is
meant to run at "info" with 14 rotated files, not the binary's raw
"debug"/7. Confirmed live on amishparadise: podman logs (stdout) was
already clean at info, but the on-disk file log
(.local/share/cuprate/logs/<date>) was flooding with per-peer DEBUG
gossip (~400KB in 2 minutes) because it inherited the binary default
instead of the intended one.
Left the resource-tuning knobs in the reference config (8GB
target_max_memory, tokio/rayon thread counts, P2P connection counts,
explicit reader_threads) out of this file — those were sized for
ssmithx's dev box and don't apply here; this manifest's
target_max_memory already stays deliberately under the container's
4Gi memory_limit.
Regenerated releases/app-catalog.json (still unsigned).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
apps/cuprate/manifest.yml only ever wrote network/target_max_memory/
rpc.restricted.enable into Cuprated.toml, so the [tracing.stdout] and
[tracing.file] tables were silently absent — cuprated still applied
its built-in info/debug/7 defaults, but nothing on disk showed it.
Verified live on amishparadise 2026-09-01: the deployed 5-line file
had no [tracing] section at all, and the level was only discoverable
by running `cuprated --generate-config` and diffing.
Add both tables to the manifest's files[].content with the same
values cuprated already defaults to, so every new install ships a
Cuprated.toml an operator can actually read and tune. overwrite:false
means already-deployed nodes (amishparadise included) keep their
existing file untouched — this only changes what fresh installs get.
Regenerated releases/app-catalog.json (unsigned) to embed the updated
manifest; needs scripts/sign-catalog.sh before it's authoritative for
the fleet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
Ollama's embedded manifest failed the typed parse (memory_limit wants a
string) so the catalog overlay was skipped for it; AdGuard Home's
conventional :3000 collided with Grafana's. The release gate now runs
the host-port collision test (repo_app_manifests_have_no_host_port_collisions)
so this class can never ship untested again.
Store-listing components are filtered via the shared serviceNames canon;
these four never earn a tile: MorphOS server is old, the Web5 DID wallet
and CryptPad are untested, Lightning Stack is an untracked upstream
bundle (LND covers it).
Nginx Proxy Manager, Tailscale, Ollama, CryptPad, and AdGuard Home now
carry full manifests: the app gate fronts their web ports (TLS on the
same port, node login where appropriate), installs run through the
orchestrator, and pins live in the signed catalog. Tailscale mirrors its
legacy shape exactly (userspace networking, web console on 8240, plain
HTTP for the gate to front). Ollama stays loopback-only — the
assistant's local model backend, not a web app.
Retires the four already-removed apps for good (FIPS, Nostr VPN,
Routstr, Penpot pins dropped from image-versions.sh, the generator map,
and image_versions.rs), fixes Cuprate's duplicated metadata block that
strict YAML parsers reject, and updates the port-inventory review gates
for the new open (3 own-login consoles) and exempt (2 DNS) ports.
Pin bumps (all verified pullable from their public registries before
editing, so none can become an image-not-found on a node):
strfry 1.1.1 -> 1.1.2 (dockurr/strfry, direct pull)
btcpay-server 2.4.2 -> 2.4.3 (docker.io/btcpayserver, direct pull)
netbird (nginx) 1.31.3-alpine -> 1.31.4-alpine
pine (nginx) 1.31.3-alpine -> 1.31.4-alpine
image-versions.sh moved in lockstep for BTCPAY_IMAGE — it is the baseline
the update badge compares against. Held back deliberately, per the risk
policy from the Aug-17 pass: gitea (four minors of DB migrations),
portainer (six minors), filebrowser (2.27 -> 2.63), fedimint/gateway
(0.8 -> 0.12, real migrations), lnd (money-critical), netbird-server/
netbird-dashboard (0.x, must move in lockstep), and everything with a
major jump or a data migration.
Cuprate also gets its curated store entry (category money, tier optional,
icon, repo) — same shape as the Alby Hub / phoenixd entries — synced
through generate-app-catalog.py into both store catalogs and the
app-session config. The fips launch-port list is unchanged (Cuprate has
no UI port; the generated file round-trips to the committed bytes after
cargo fmt).
Three further bumps are prepared and parked on the
app-bumps-mirror-pending branch, blocked only on a registry-push token:
vaultwarden 1.37.2-alpine, archy-nbxplorer 2.6.11, home-assistant
2026.8.3 — all mirror-backed, and the push credential on record for the
lfg2025 namespace is dead.
Drift gate: check-app-catalog-drift.py --release --strict clean
(31 store entries, 0 drift, 0 missing). appSessionConfig tests 7/7.
Five apps had no app.upstream block, so nothing could ever tell us
when their pins fell behind upstream:
barkd gitlab ark-bitcoin/bark (GitLab-only project)
immich-postgres ghcr immich-app/postgres (image exists only on ghcr.io)
indeedhub-minio github minio/minio
pine-whisper dockerhub rhasspy/wyoming-whisper
lightning-stack manual — no public listing exists for
lightninglabs/lightning-stack anywhere (docker.io,
ghcr.io, github.com all checked), so it is tracked by hand
This adds two fetchers to scripts/check-upstream-releases.py to reach the
first two: latest_gitlab (GitLab releases API; strips the project-name
tag prefix, e.g. bark-0.6.2 -> 0.6.2) and latest_ghcr (anonymous pull
token + tags/list, the same handshake a docker pull performs).
Live-verified after the change:
barkd 0.3.0 -> 0.6.2 (bump gated on ark_client.rs REST compat)
immich-postgres 14-vectorchord0.4.3-pgvectors0.2.0 -> 17-vectorchord0.4.3-pgvector0.8.0
indeedhub-minio RELEASE.2024-11-07T00-52-20Z -> latest (date-opaque: UNCOMPARABLE, shown for hand comparison)
pine-whisper 3.4.1 -> 3.6.0 (tuned-args revision needs re-basing, not just a pin move)
Offline coverage check: 59 apps, 0 untracked.
The manifest bound cuprated's unrestricted RPC (full node control) to
0.0.0.0 inside the container with
i_know_what_im_doing_allow_public_unrestricted_rpc = true, relying on
ports[].bind: 127.0.0.1 to keep it private. That only restricts the HOST
side. Verified live on archi-dev-box 2026-08-22: a peer container got a
valid unauthenticated get_info off container port 18081 — and still did
after cuprate was moved to its own network, because podman bridges route
to each other unless created with --opt isolate=true, which the
orchestrator's auto-create does not pass. Every app on the node could
therefore drive full node control with no credential.
The PR justified this as the pattern bitcoin-knots already uses, but
knots writes rpcuser/rpcpassword from generated secrets, so a 0.0.0.0
bind there still is not control without credentials. cuprated has no RPC
authentication at all, so the two are not equivalent.
Unrestricted RPC is now left at cuprated's own default — container
loopback only, published nowhere, reachable by nothing — which is what
upstream intends by refusing a non-local bind without an explicit
override. Restricted RPC (the safe-for-public subset wallets use) and p2p
are unchanged, and health_check moves to 18089 since 18184 is gone.
Re-verified after the change: peer container gets connection refused on
18081 (exit 7), restricted RPC and the health endpoint still answer, the
node still syncs, validator APPROVED, 76/76 container tests pass
including the unauthenticated-port canary (still 28 — an auth: local
port was removed, not an auth: none one).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Full-node daemon: P2P + Monero's own restricted RPC (the safe-for-public
subset wallets use as a "remote node") are auth:none like bitcoin/electrumx's
equivalents; unrestricted RPC (full node control) stays gated auth:local.
readonly_root works cleanly since the upstream image is FROM scratch with
ownership fixed at build time — no runtime chown/setuid needed, unlike
bitcoin-knots/core.
Verified locally end-to-end before committing: built the upstream Dockerfile,
confirmed the generated Cuprated.toml against `cuprated --generate-config`/
`--dry-run`, and ran the real image with the manifest's exact ports/volumes —
including discovering that cuprated's own 127.0.0.1-default RPC bind is
unreachable through a published host port and needs to bind 0.0.0.0
internally with ports[].bind:127.0.0.1 doing the actual restriction, the
same pattern bitcoin-knots' RPC port already uses in this repo.
Bumps the unauthenticated_ports_are_all_accounted_for canary (26 -> 28) for
cuprate's two auth:none ports, per that test's own review-before-updating
contract.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
With registry push access, the 24 mirror-backed apps stopped being
blocked. Ten images are now mirrored (single-platform amd64, matching
the existing convention) and their pins moved:
alby-hub v1.23.0 -> v1.24.0
mempool-frontend v3.0.1 -> v3.3.1 (mempool, archy-mempool-web)
mempool-backend v3.0.0 -> v3.3.1
fedimintd v0.10.0 -> v0.10.1
gatewayd v0.10.0 -> v0.10.1
nostr-rs-relay 0.9.0 -> 0.10.0
portainer 2.39.1 -> 2.39.6
vaultwarden 1.30.0-alpine -> 1.37.1-alpine
jellyfin 10.8.13 -> 10.11.11
home-assistant 2026.7.3 -> 2026.8.2
Every one verified pullable from our mirror after copying, so none can
become an image-not-found on a node. image-versions.sh moved in lockstep
— it is the baseline the update badge compares against when the catalog
does not cover an app, and leaving it behind would have kept advertising
an update that had already been applied.
Chosen by risk, not by count: these are patch/minor bumps with no data
migration. The ones held back are held for a reason each — Postgres
15->18 and 16->18 refuse to start on an older cluster, Redis 7->8,
Valkey 7->9, Nextcloud 29->32 must go one major at a time, plus
uptime-kuma 1->2, grafana 10->13, electrumx 1->2, photoprism, and
core-lightning's three years of schema migrations. Those are each a
migration plan, not a pin edit. LND (v0.18.4 -> v0.21.2) is held
separately: it is only a minor bump by version but it migrates its
channel database irreversibly, and this box holds real funds.
Note the checker still reports several of these as behind, and that is
correct: it reads the *catalog* pin, which is what nodes actually act on.
These land when the catalog is regenerated and re-signed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 33 apps behind upstream, these five pull straight from a public
registry, so their targets exist already and the bump is real work rather
than a promise:
strfry 1.0.4 -> 1.1.1
netbird (nginx) 1.27-alpine -> 1.31.3-alpine
pine (nginx) 1.27-alpine -> 1.31.3-alpine
pine-piper 2.2.2 -> 2.4.2
nostr-rs-relay 0.8.9 -> 0.10.0
All five targets verified present upstream with skopeo before editing, so
none of these can turn into an image-not-found on a node.
Deliberately NOT bumped here, though they are also direct-pull:
core-lightning (v23.08 -> v26.06, ~3 years of schema migrations), gitea
(four minors of DB migrations), and netbird-server/netbird-dashboard —
which have to move in lockstep and carry their own migrations. Those are
each a piece of work, not a line edit.
The other 24 are blocked on something else entirely: their images live in
our mirror and none of the upgrade targets have been mirrored yet, so a
pin bump alone would break every install. That needs registry push
credentials.
These take effect when the catalog is regenerated and re-signed — the
catalog overrides on-disk manifests, so editing here changes nothing on a
node until the signing ceremony.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes offer an update when the signed catalog pins something newer than
what's running, and that machinery is fine. The missing step was the one
before it: nothing told *us* when upstream shipped. A pin could sit at
fedimintd v0.10.0 for months while every node in the fleet correctly and
confidently reported "up to date".
The reason nothing could tell us is that a manifest records only our
mirror — `source.archipelago-foundation.org/lfg2025/fedimintd:v0.10.0`
says nothing about the project it was mirrored from. So this adds an
optional `app.upstream` block naming the real source, and a script that
asks each one what it has released.
Running it answers the question that prompted this. Of 58 apps, 28 are
behind, including LND v0.18.4-beta against v0.21.2-beta, Bitcoin Core
28.4 against 31.1, and fedimintd/gatewayd v0.10.0 against v0.10.1.
Two choices worth stating. An app with no `upstream` block is reported
as UNTRACKED rather than skipped — a silent skip is how this stayed
invisible, and before this commit all 58 were silently skipped. And a
suggestion prefers our own tag variant: telling someone pinned to
`postgres:16.13-alpine` that the newest tag is `18.6-trixie` is true and
useless, because swapping the base image is a different decision from
bumping a version.
Five apps are deliberately left untracked (barkd, immich-postgres,
indeedhub-minio, lightning-stack, pine-whisper): I could not establish
their upstream with confidence, and a wrong `repo` produces a confident
wrong verdict, which is worse than an honest gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some apps carry a complete account system and are broken by an upstream
challenge: git clients speak basic-auth (not browser cookies), and a
BTCPay checkout link handed to a customer must open for that customer.
Both were behind the gate's login page — the "non-browser clients need an
access token" gap disclosed in five consecutive releases.
- New manifest port policy `auth: open`: the daemon still fronts the port
exactly like `gated` (loopback pin, external binds, frame-header fixes,
app-down retry page, Tor upstream) but serves it without the login
challenge. Requires auth_rationale, same burden of proof as `none`.
Gitea 3001 and BTCPay 23000 declare it.
- Runtime operator override per app (security.set-app-gate → app-configs/
<id>.json "gateEnabled"), surfaced as Settings → app → Access control.
Wins over the manifest in both directions and applies on the next
request — no restart, and it works today on catalog-covered apps whose
signed manifest still says `gated`.
- The gate resolves policy per-request from the live port map, so a
toggle takes effect without waiting for the 60s rebind sweep. "Off"
never releases the port: gated apps are loopback-pinned, so releasing
would strand them, not open them.
- security.app-gate-status now reports gate_enabled + any override.
- New guard test pins the `auth: open` set (both entries reviewed); the
`auth: none` count moves 25 → 26, absorbing pre-existing drift from the
phoenixd onboarding (loopback JSON API with its own generated password).
- Docs: the manifest spec's ports row documented only host/container/
protocol — bind, auth, auth_rationale and session_passthrough were
undocumented. Added a full "Ports & the app gate" section plus a
developer-guide entry telling app authors to enforce their own auth
regardless, since the operator can flip the gate either way.
Verified live on archi-dev-box from an external address: gated → 401 gate
page; override off → Gitea 200 own page, BTCPay 302 to its own login,
git-over-HTTP info/refs 200; override on → 401 again; clear → default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Small-disk nodes (<1000 GB data volume) keep the same dynamic
prune-vs-archival logic but now retain ~50 GB of recent blocks instead
of the bare 550 MB minimum. Takes effect for catalog-covered installs
at the next catalog regeneration + signing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
phoenixd: the orchestrator treats bind paths containing a dot as file
mounts and never creates their source dir, so the image's default
/phoenix/.phoenix target crash-looped the unit (statfs: no such file).
Datadir moved to /data via PHOENIX_DATADIR; data_uid 1000:1000 matches the
image's phoenix user — without it phoenixd dies on phoenix.conf
'Permission denied'. Both verified end-to-end on archi-dev-box: orch
install OK, seed.dat + db on the host, authenticated /getinfo answers.
alby-hub: launch flips to embedded — pairs with the gate change that
neutralizes upstream frame blocking.
Dev guide: iframe embedding rules (who blocks framing and why the gate
may strip it; when open_in_new_tab is legitimate; test in the embedded
session, never a tab).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both images mirrored to the Foundation registry. Alby Hub: gated web UI
on 8087, LDK data under /var/lib/archipelago/alby-hub. phoenixd: headless
loopback API on 9740 (own password auth), seed dir preserved under
/var/lib/archipelago/phoenixd. Not yet in the signed catalog — disk
manifests only, pending install verification on archi-dev-box.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
archipelago
ssmithxandClaude Sonnet 5