[Rule 3 — blocking] bootstrap.rs installs systemd units from the runtime
payload at image-recipe/configs/, but create-release-manifest.sh copies only
archipelago-doctor.service and .timer into that directory. The new
archipelago-host-secrets-audit.service would therefore never exist on any
node: bootstrap looks for it, `src.exists()` is false, and it silently
installs nothing. No error, no log line — the whole deployed-node half of
10-04 would have been inert on arrival.
Two enumerations of the same list in two languages in two files is the drift
that caused it, so the loop now carries a KEEP IN SYNC pointer naming the
array in bootstrap.rs, and the redundant `if [ -f doctor.service ] || [ -f
doctor.timer ]` wrapper is gone — the per-unit `-f` test inside the loop
already does that job, and the wrapper would have skipped the whole block on
a tree that had the new unit but not the doctor ones.
Outside 10-04's declared files_modified. Taken because the alternative was to
ship a deliverable that cannot reach its target and file the gap as a
follow-up. Staged by path; no other agent had uncommitted work in this file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Measures EXPOSURE (auth.isOnboardingComplete) and SESSION ENFORCEMENT
(seed.status) separately; the audit's C-6 probed with seed.status alone,
which is not allowlisted and returns 401 by design, so its "Fail: 200"
criterion could never fire.
- Read-only by construction: methods come from a fixed READONLY_METHODS
array, never from an argument; the one mutating request is behind
--destructive with a red disposable-nodes-only banner.
- The refusal check uses the published BIP-39 all-abandon/art test vector,
so no real key material is ever generated, handled or printed.
- No node address, onion address or credential embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six code paths configured the Lightning gateway with a bcrypt hash committed
to this repository — and one deploy path with a plaintext password literal —
whenever the per-install secret was missing. Anyone holding a copy of the repo
held the admin credential for every gateway that ever took a fallback.
container::secrets now owns the credential end to end: ensure_gateway_credential
(idempotent, delegates to ensure_one's bcrypt arm) and gateway_bcrypt_hash,
which returns Err when the secret is missing/empty and when the stored value is
on the KNOWN_DEFAULT_GATEWAY_HASHES denylist — so this codebase cannot hand
back the compromised value even to a node already carrying it.
get_app_config was widened to Result so a credential-less install cannot reach
podman run at all; configure_fedimint_lnd takes the resolved hash instead of
re-reading with its own fallback. The four shell paths stop generating
credentials entirely (dropping the htpasswd host dependency) and skip container
creation with a printed reason rather than substituting anything.
Naming converges on the manifest's fedimint-gateway-hash/.pw, with legacy
fedimint-gateway-password values copied forward rather than regenerated so no
node loses a working unique credential. Plan 01-16 owns rotation of installs
already carrying the default.
Verified: cargo build clean; cargo test -p archipelago 999 passed (2
boot_reconciler timing tests failed under concurrent load, green in isolation,
untouched by this diff); bash -n clean on all five scripts; the compromised
literal now appears exactly once in the tree, as the denylist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ships botfight repo commits f5f57e6, c162d5e, 41f1b93, 10d4209:
- Fixed 6 instances of the same trust-a-client-supplied-pubkey pattern
across auth/payments/queue routes, two of them critical: GET
/winnings/:botId had no auth at all and leaked live spendable Cashu
bearer tokens; POST /connect-wallet let anyone redirect a victim
bot's future payouts to an attacker's wallet by pubkey (public by
design in nostr).
- Added owner-reachable AI-answer settings (existing bots, not just at
creation) and a "claim your winnings" UI (Cashu payouts were minted
server-side but had no frontend consumer at all until now).
Co-Authored-By: Claude <noreply@anthropic.com>
Bumps to botfights:1.2.9, which carries: Cashu token payment wired as the
primary entry-fee UX in WalletConnect.vue (was built server-side already
but never called from any UI), Lightning/NWC demoted to secondary, and a
fix so anonymous poll-mode bots (not just nostr-authenticated humans) can
use ranked/staked fights — join-ranked previously required a pubkey
unconditionally, silently locking out the entire AI-agent audience.
No archy-side manifest changes beyond the version/image bump.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix_tor_permissions() exact-matched stat's output against the literal
string "700", but stat -c '%a' omits leading zeros so Tor's own
setgid HiddenServiceDir mode (2700) never matched. Every ~5-minute
doctor run "fixed" it back to 700, restarted Tor, and Tor immediately
set 2700 again — so Tor never survived long enough to build a usable
consensus/HSDir cache, breaking the mesh's Tor fallback entirely.
- Compare only the last 3 mode digits (owner/group/other), which is
the property that actually matters, so 700 and 2700 both pass while
750/707/2755 etc. are still corrected.
- Add a 30-minute restart backoff (timestamp file under
/var/lib/archipelago/) so no future condition can reproduce a
restart storm even if the fix fires repeatedly.
- Log clearly in both directions: a debug-level no-op line when a
directory is already correct, and an explicit "was NOT fully
denied" line when a real fix is applied, plus a line when a restart
is skipped by the backoff.
Verified statically against temp directories (2700 accepted with 0
restarts; 750/707/2755 corrected with exactly 1 restart; a second
real fix inside the backoff window logs a skip instead of
restarting). No live Tor/doctor/systemd unit was touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bumps to botfights:1.2.8, which carries: the CSP img-src fix (nostr profile
pictures come from user-supplied kind:0 metadata URLs on arbitrary domains
— img-src was locked to 'self'/data:/blob: with no https:, so every
external profile picture rendered as a broken image), and discoverability
fixes for the new AI-answer feature (poll mode is now the default
connection mode, matching BOTFIGHTS.md's own documented default, and the
AI-answer section is expanded by default instead of collapsed behind an
extra click).
Canonical arena on VPS2 already rolled to 1.2.8 directly (docker compose
pull/up) ahead of this catalog publish, since it's a separate deployment
from the archy-catalog-driven per-node install path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bumps to botfights:1.2.7, which carries the botfight repo's new
"let BotFights answer for me" feature (ca5b634) — an operator can paste an
Anthropic or OpenAI API key so this node's BotFights server answers fight
challenges automatically for a poll-mode bot, no external script needed.
Storage/security follows the same pattern this node already uses for its
own AIUI Anthropic key (system.settings.set "claude_api_key" in
core/archipelago/src/api/rpc/system/handlers.rs): 0600 file, never echoed
back. Also carries the nostr-provider.js 404 fix, DocsPage/round-jump
fixes, and the HomePage "Latest Bouts" short-viewport visibility fix from
the prior 1.2.4-1.2.6 iterations that were built and tested locally but
not yet pushed through the signed-catalog path.
No archy-side manifest changes beyond the version/image bump — 1.2.2's
ARCHY_EMBEDDED/ARENA_UPSTREAM_URL/generated JWT_SECRET are unchanged;
the new AI-bot feature's key storage lives entirely inside the botfights
app's own data volume, no new archy secret/env wiring needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deploy-to-target.sh rsyncs with --delete to TARGET_DIR=/home/archipelago/archy,
which is a symlink to /home/archipelago/Projects/archy. archi-dev-box is this
same machine over loopback SSH, so deploying from the main checkout is a no-op
(source and destination resolve identically) — but deploying from a git worktree
nested under it mirrored that worktree onto the main checkout and deleted
everything else: ~1810 tracked files, the deploying worktree itself mid-run, a
running dev server, and two concurrent sessions' uncommitted work.
Guard compares /etc/machine-id across the SSH hop and, when source and
destination are on the same host, refuses if either path contains the other.
Identical paths still pass, so normal local deploys are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bumps to botfights:1.2.3, which carries the JoinBoutPage.vue fix from the
botfight repo (commit 603e09b): the poll/webhook mode picker on the bot
setup step now visibly reacts when clicked — a colored banner in the guide
viewer and a prepended line in the copied prompt point at "Option A:
Polling Bot" or "Option B: Webhook Bot" within the single unified doc
(BOT-02), instead of silently refetching the same file with no visible
change. No manifest/env changes beyond the version/image bump — 1.2.2's
ARCHY_EMBEDDED/ARENA_UPSTREAM_URL/generated JWT_SECRET are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1.2.x's auth-hardening work added Hono secureHeaders() with a default
X-Frame-Options: SAMEORIGIN, which unconditionally blocked the Archipelago
node dashboard's iframe (different origin by port) — a real regression
versus 1.1.0, which never sent this header. Fixed upstream in the botfight
repo (commit 8eb27ed): X-Frame-Options is now conditional on ARCHY_EMBEDDED,
disabled only for the first-party node-embedded instance.
apps/botfights/manifest.yml: image/version -> 1.2.2, adds
ARCHY_EMBEDDED=1 to environment, drops the interim
metadata.launch.open_in_new_tab workaround (no longer needed — the app can
now be framed). app-catalog/catalog.json, scripts/image-versions.sh,
neode-ui/public/catalog.json bumped in lockstep via
scripts/generate-app-catalog.py. core/archipelago/src/fips/app_ports.rs
regenerated (formatting only, same port set).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apps/botfights/manifest.yml: image tag 1.1.0 -> 1.2.1, adds container.generated_secrets
(botfights-jwt-secret, kind hex32) + secret_env (JWT_SECRET) so a fresh install no longer
crash-loops on the 1.2.x image's jwt.ts import-time throw when JWT_SECRET is unset under
NODE_ENV=production (D-04/BOT-04). Adds ARENA_UPSTREAM_URL=https://botfights.archipelago-foundation.org
to turn on shared public arena federation by default (BOT-03/D-03), with an in-manifest comment
documenting the per-node opt-out (remove the line to run standalone).
app-catalog/catalog.json + scripts/image-versions.sh bumped in lockstep to keep the legacy
catalog and the image-version drift checker consistent with the manifest.
releases/app-catalog.json intentionally NOT committed in this change — it has been
regenerated locally (scripts/generate-app-catalog.sh) with the new manifest embedded, but is
unsigned. It is committed separately (09-06 Task 3) only after the release-root signing
ceremony, so no unsigned catalog state ever lands even transiently on a branch this repo's
mirrors could serve to nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.7.117 broke Reticulum mesh on OTA-only fleet nodes two ways: the
update swaps only the backend binary and frontend tarball, so nodes
kept a stale archy-reticulum-daemon whose argparse exits on the new
--enable-transport flag (mesh session died on every spawn — confirmed
on framework-pt), and they never had archy-rnodeconf at all, so the
in-app Flash LoRa flow failed with a bare "No such file or directory".
Four-part fix:
- The Rust supervisor probes `daemon --help` and only passes
--enable-transport when the daemon advertises it; unsupported daemons
run edge-only exactly as pre-1.7.117 (tested against stub daemons
both ways + missing-binary fail-safe).
- Both PyInstaller tools ride the frontend tarball's runtime payload
(radio-tools/) and bootstrap.rs promotes them to /usr/local/bin on
startup when bytes differ — the first OTA path that ever updates
them. create-release.sh now rebuilds them every release and the
manifest script hard-fails if they're missing.
- The ISO bundles archy-rnodeconf alongside the daemon (it never did).
- Flash LoRa reports "tool not installed — update the node" instead of
the bare spawn error when rnodeconf is absent everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both sides independently fixed the serial-alias dedup and the ESP32
boot-reset races; kept the branch's defer-to-auto-detect for unpinned
preferred paths (single probe pass per cycle) on top of main's
advert-name threading, Reticulum name propagation and radio-first
routing. Modal keeps main's 'Set Recommended' naming + probe progress
bar alongside the branch's in-app firmware flasher step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed OTA manifest (release root verified). Also: create-release-manifest
tarball perms check made SIGPIPE-proof so releases cut on macOS too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The companion opens catalog apps by direct port over the mesh; the
fips0 default-deny baseline blocked every one of them (apps 'stuck'
from the phone, 2026-07-26). Core now writes a second fips.d drop-in
from the generated catalog launch-port list on every install/upgrade.
Service/RPC ports (bitcoind 8332, LND 10009, Tor) stay closed.
generate-app-catalog.py emits the Rust port list alongside the TS one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- scripts/build-iso-release.sh: single gated command from signed release
to tested ISO (preflight version/signature parity, release gate harness,
strict catalog drift, full Rust suite, artifact version checks, build,
mount smoke, best-effort QEMU boot)
- scripts/iso-smoke-test.sh: standalone mount-level ISO verification incl.
stale-binary version assertion inside the payload
- .gitea/workflows/build-iso.yml: resurrected ISO CI as dispatch-only job
calling the gated orchestrator (old one was stranded in _archived/)
- tests/release/run.sh: catalog drift now runs --strict (was silently
always-pass)
- test-iso-qemu.sh: headless mode used -append without -kernel, which
QEMU rejects; use -display none so -serial file: capture works
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--erase-all hit the exact same "ROM does not support function erase_flash"
error as a standalone erase_flash — confirmed live: esptool's --erase-all
is implemented as the same full-chip-erase command, not a per-sector loop,
so it needs the stub just as much. The real fix isn't finding another way
around the ROM bootloader's limitations — it's restoring the stub Debian's
package is missing.
Fetched the exact missing file (stub_flasher_32s3.json) from the matching
upstream esptool release tag and installed it directly on archy-x250-dev;
verified live with a read-only `flash_id` that stub mode now loads
("Uploading stub... Running stub..."). Removed --no-stub from flash.rs now
that normal stub-loader mode works correctly (faster and fully-featured vs.
the ROM-only fallback). scripts/self-update.sh now fetches and installs
this same file automatically whenever it's missing, so this isn't a
one-off manual fix tied to a single node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pine-openwakeword manifest (wyoming-openwakeword 2.1.0, :10400, /custom model
dir for the future Yo Archy model). Pine 1.3.0: launcher page gains a live
node-status card fed by /api/pine/status via a same-origin nginx proxy, and
copy for the new intents / Claude fallback / mesh announcements. Catalogs,
app-session config, drift ids regenerated; release catalog embeds 56
manifests (unsigned until the ceremony). Framework PT test plan in docs/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HA 2024.1's Wyoming integration predates ping/pong keepalives. The PineVoice
satellite firmware (wyoming 1.7.2) drops connections idle for ~8s, so HA
looped on 'Satellite disconnected' forever and the speaker never worked.
HA 2026.7.3 pings satellites and holds the connection (verified on
framework-pt: link stays ESTABLISHED, wake word reaches Assist).
Catalog regenerated with the bump — needs the signing ceremony before the
origin publish.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Package the PineVoice/Wyoming local voice assistant as ONE store app "Pine"
that runs three rootless containers:
- pine-whisper — Wyoming faster-whisper STT (docker.io/rhasspy/wyoming-whisper
3.4.1, model base-int8/en), publishes :10300
- pine-piper — Wyoming Piper TTS (2.2.2, voice en_GB-alba-medium),
publishes :10200
- pine — nginx launcher serving a self-contained setup/status page
(WiFi provisioning + Home Assistant wiring instructions), the Open target
The two engines publish their Wyoming ports so Home Assistant reaches them via
host.containers.internal. All rootless (cap-drop=ALL, no_new_privileges),
manifest-driven — no hardcoded podman/installer. The engines are internal stack
members (added to the drift-check INTERNAL_MANIFEST_IDS allowlist); only "pine"
carries a catalog entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v1.7.104 shipped ISOs whose units crash-looped (nostr-relay 3s loop,
archipelago-diag 203/EXEC) because build-time extraction failures were mere
warnings and the cached rootfs never tracked its recipe:
- hash the rootfs-defining region of the build script and rebuild when it
changes (stale cache shipped ISOs without wpasupplicant/iw/rfkill)
- refuse to build without nvpn / nostr-rs-relay unless
ALLOW_MISSING_VPN_BINARIES=1
- ConditionPathExists on nostr-relay/nostr-vpn/diag units so a stripped
image skips them instead of crash-looping
- post-install test: every enabled archipelago*/nostr* unit must have an
existing ExecStart payload
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.198 crash-looped 236x on a truncated OTA binary with a good backup
sitting in update-backup/ — verify_pending_update() can't help when
the new binary never runs. New scripts/ota-crash-guard.sh runs as
root from ExecStartPre: while the post-OTA pending-verify marker
exists it counts start attempts, and after 5 failures restores the
backup binary (atomic rename), replaces the marker with an
update-rolled-back.json tombstone, and lets the service come back
on the previous version.
Wiring: fresh ISOs get the ExecStartPre line in the unit file;
existing nodes get a drop-in installed by apply_update's runtime
component step on their next OTA. '+-' prefix so a missing or
failing guard can never block the service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh offline installs came up without fedimint-clientd and filebrowser
(they only appeared after connecting internet). Three root causes:
- archipelago-load-images.service ran 'podman load' as root, but every
container runs rootless as archipelago — bundled images landed in
root's storage where the rootless runtime can't see them, so all
container creation silently depended on registry pulls. The loader now
loads into the archipelago user's storage (with linger + runtime-dir
wait + system migrate).
- The unbundled ISO bundled only filebrowser.tar; fmcd (fedimint-clientd)
is a baseline first-boot app too and is now part of the unbundled core
bundle.
- first-boot's pull_with_fallback always hit the network; it now uses an
already-loaded local image first and skips the pull entirely.
Also: fedimint-clientd added to the UI's hardcoded curated-app fallback
list (it was missing when all catalog fetches fail offline), plus its
INSTALLED_ALIASES entry, and the stale fmcd bundling comment in
image-versions.sh corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- archipelago-first-boot-secrets.service regenerates the self-signed TLS
keypair (device hostname in SAN) and all SSH host keys on the installed
system's first boot, before ssh/nginx start — the squashfs bakes one
key set at build time, so every flashed device shared them (§F 🔴)
- staging-first swaps: a failed regeneration keeps the baked keys rather
than leaving the device keyless
- the builder now emits <iso>.sha256 next to the ISO, and
scripts/sign-iso-checksums.sh signs {artifact,sha256,size} as a JSON
doc with the release-root ceremony (verify: ceremony verify) (§F 🟠)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflict resolution: keep the branch's CPUQuota=200% (75% was the proven
cause of choppy HDMI audio — the file's own comment says so; the 75% on main
was an accidental regression bundled into an unrelated UI commit) and keep
main's newer MemoryMax=2800M/MemoryHigh=2200M tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Debian's podman only Recommends catatonit; nodes installed before
install-podman.sh gained the dependency fail any init-enabled container
deploy (observed on shorty-s deploying sites via Portainer). Doctor
Fix 13 installs it via apt when absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A store on a node whose LND has only unannounced channels mints BOLT11
invoices with no route hints when lightningPrivateRouteHints is off, and
external wallets fail with "no way to pay this invoice" (hit on shorty-s
2026-07-10 with a Blink payer). Hints cost nothing on public channels, so
the doctor now flips the flag on for every store on each run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The netns-egress repair only stopped/started containers, which reuses the
existing netns — its holders (aardvark-dns, podman pause) survive, so a
missing pasta tap was never rebuilt. On shorty-s (2026-07-10) this cycled all
35 containers every timer run for ~an hour, bouncing bitcoind/LND/BTCPay the
whole time, with 'egress still broken after cycle' after every pass.
Now the cycle kills the netns holders, runs podman system migrate, and clears
/run/user/<uid>/containers/networks so the first container start recreates
pasta + aardvark-dns from scratch (the sequence that actually fixed the node).
A failure counter (/var/lib/archipelago/doctor-netns-cycle-failures) stops the
fleet-wide cycling after 3 consecutive failed rebuilds until egress is
observed healthy again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- create-release.sh: assert the freshly built frontend dist actually embeds
the bumped version before packaging — the ui-dist-version guard in
tests/release/run.sh is behind --with-build, which the release path never
passes, so a silently no-opped npm build could ship a stale dist with a
valid sha256. Verified against the real dist (passes on current version,
trips on a missing one).
- publish-release-assets.sh: verify published assets by downloading and
comparing sha256 against the manifest, not just Content-Length — a
size-correct/content-wrong mirror asset now fails the publish gate.
Verified live against the v1.7.99-alpha assets on the vps2 mirror.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RNS's own rnodeconf utility (frequency/bandwidth/spreading-factor/coding-rate
read+write, firmware signature verification, board bootstrap) has been the
one tool that reliably diagnoses real RNode hardware — it's what finally
proved two live nodes were silently running at different spreading factors
(SF5 vs SF10, invisible from any of our own probe/logging code, and the
actual reason two correctly-flashed radios could detect each other's RF but
never decode a packet). Every node should have it, not just whichever one an
agent happened to build a throwaway venv on to debug a specific incident.
- reticulum-daemon/build.sh: also PyInstaller-package archy-rnodeconf
alongside the existing archy-reticulum-daemon, same --collect-submodules/
-d noarchive flags (same RNS.Interfaces __all__-glob requirement applies).
- scripts/deploy-to-target.sh: actually wire both packaged binaries into the
live deploy path (neither was wired in before — a pre-existing gap noted
in docs/RETICULUM-TRANSPORT-PROGRESS.md; this is why reticulum-daemon
previously only worked via manual dev-venv setup on rsync-deployed nodes,
not the ISO-imaged ones). Non-fatal on build/deploy failure — archipelago
already falls back to its dev-venv path if the packaged binary is absent.
Also installs the missing `python3.<minor>-venv` package when needed
(same "ensurepip not available" gap hit manually twice this session).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
check_for_updates now fetches the manifest as raw JSON and runs
trust::verify_detached before parsing: a tampered or wrong-signer
signature rejects the mirror outright, and unsigned manifests are
offered for MANUAL apply only — the 3 AM auto-apply scheduler refuses
them, closing the unattended remote-root hole (§A of the 1.8.0
hardening plan). UpdateState gains manifest_signed so the UI can
surface authenticity.
Publisher side: create-release.sh signs the manifest during the
release (ceremony, mnemonic via TTY/env only), publish-release-assets
hard-refuses to ship an unsigned manifest (grep + new 'ceremony
verify' cryptographic gate), and scripts/sign-manifest.sh covers
re-signing outside a release run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin RELEASE_ROOT_PUBKEY_HEX from the 2026-07-02 release-root signing ceremony
(signer did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur) so nodes verify
the publisher identity of the app-catalog. Sign releases/app-catalog.json in place.
Fix two floats that made the catalog unsignable: archy-btcpay-db manifest version
-> string, fedimint-clientd cpu_limit 0.25 -> 1 (u32). Add scripts/sign-catalog.sh
helper, the 1.8.0 release-hardening plan/tracker, and the commit-and-push project
rule in CLAUDE.md.
Backward-compatible: old binaries still accept the signed catalog; the pinned-anchor
binary ships in the next build/OTA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Routes ALSA's "default" device through PulseAudio/PipeWire. Was a manual,
untracked fix living only on archy-x250-exp — a reprovision or fresh image
would silently lose it, same failure mode that already bit the GPU-flags
and CPUQuota fixes. Wired into deploy-to-target.sh (both the primary --live
path and the .198/.253 secondary-copy path) and deploy-tailscale.sh, mirroring
the existing 99-mesh-radio.rules udev sync pattern (diff-check, copy if changed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
reticulum-daemon/.venv (a local Python virtualenv bundling PyInstaller +
esptool + Qt hooks, several hundred MB) was also being synced to deploy
targets uncached -- same class of bug as the releases/ exclude just added.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
releases/ (the local repo's own historical build artifacts -- dozens of
versioned binaries + frontend tarballs, 7-10GB) was never excluded, so every
deploy synced it to the target's root disk. Filled .198 (29GB disk) to 100%
mid-deploy and .228 to 100% right after a "successful" deploy -- the target
node never needs its own copy of the release archive, only the built
binary+frontend actually get installed into system paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- generate-app-catalog.sh: VERSIONS map now lists the full Knots set
(29.3.knots20260508/20260507/20260210 + 29.2.knots20251110) and Core
(adds 29.2 + a `latest` entry → newest); generator forces top-level
`version` == the default entry's version (the 169ff2e2 invariant) so
regeneration is reproducible. releases/app-catalog.json regenerated.
- docs/bitcoin-version-bulletproof-rollout.md: full handoff — root causes,
fixes, current .228 state, the coordinated fleet-rollout steps (incl.
:latest repoint sequencing / fleet-safety), reindex finish procedure, and
the switch-matrix test plan.
- PRODUCTION-MASTER-PLAN.md: link the rollout doc (§6b-bis).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three stacked bugs made "switch version" silently fail / crash-loop, and
the data-access mismatch corrupted a node's index during recovery attempts.
Backend renderer:
- sync_quadlet_unit ignored the per-app pinned version and re-rendered the
quadlet with the manifest's :latest every reconcile tick, reverting any
switch. Factor the install-time catalog/pin resolution into a shared
resolve_catalog_image() and call it in BOTH install_fresh and
sync_quadlet_unit.
- The renderer folded manifest `entrypoint: ["sh","-lc"]` into Exec=, which
only worked when the image entrypoint was a passthrough shell wrapper. The
versioned images use ENTRYPOINT ["bitcoind"], so Exec=sh -lc ... became
`bitcoind sh -lc ...` and crash-looped. Emit a real Entrypoint= override;
exec_changed now also compares Entrypoint=.
Images:
- Build all bitcoin images (Core + Knots, every version) as container-root
(USER removed) like the legacy :latest image. Chain data is owned by the
data_uid (container uid 102); root reads it via CAP_DAC_OVERRIDE (granted in
the manifest). A non-root USER (the previous uid 1000) can't read existing
chain data → "Error initializing block database". Still fully rootless:
container-root maps to the unprivileged host service user.
Catalog:
- bitcoin-knots versions[]: 29.3.knots20260508/20260507/20260210 +
29.2.knots20251110, "latest" tracking newest.
- bitcoin-core versions[]: add 29.2 + a "latest" entry. All images rebuilt
root and published to the mirror.
Frontend:
- AppSidebar version dropdown: rename the latest option to "Always use the
latest version" (no v prefix), fix right padding, and guarantee the current
selection matches a real option (was rendering blank).
- New InstallVersionModal: full-screen version chooser shown from the App
Store / Discover install button for multi-version apps (Bitcoin Knots/Core),
app icon + "Install <name>", latest pre-selected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The knots versions[] marked 29.3.knots20260508 as default while the
top-level catalog version is the floating 'latest' tag — violating the
generator's own invariant (default:true MUST equal the top-level version
so selecting it un-pins / tracks latest). Live effect via package.versions:
catalog_default_version='latest' so the UI-highlighted default actually
PINS+recreates (opposite of un-pin) and 'latest' was unreachable from the
Version & Updates card.
Add a 'latest' default entry (== the manifest's floating tag) and keep
29.3.knots20260508 as a pinnable option. Verified on .228: package.versions
now returns default=latest with 2 selectable versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On NAT'd nodes that can reach the iroh federation neither directly nor
via iroh's public relays, fmcd's embedded iroh networking enters a
relay/hole-punch reconnect hot-loop that pegs its entire CPU allotment
indefinitely (observed ~1 core sustained for 4 days on a Tailscale node,
while LAN nodes that reach the guardian directly stay <3%). fmcd 0.8.0
exposes no iroh/relay knobs, so:
- fmcd-run now samples fmcd's own CPU and restarts it when it stays near
its allotment for ~15 min (a restart demonstrably clears the stuck iroh
state; real work is bursty and never flat-pegs a core for minutes).
- Lower cpu_limit 1 -> 0.25 core so a stuck instance can't starve the
node (steady-state is <3% of a core; joins are brief).
Ships as fmcd:0.8.1 (launcher-only rebuild, same fmcd binary). Bumped the
image pin + cpu_limit in the manifest, image-versions.sh, the embedded
catalog manifest (releases/app-catalog.json), and the UI catalogs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ssmithxandClaude Sonnet 5
