Two problems with running the gate against a mid-initial-sync electrumx:
1. The (now honest) protocol probe can only fail — ElectrumX serves no
sessions until it has caught up to its daemon, so the failure names a
state nobody can act on.
2. Worse, the destructive stop/start/restart tests actively destroy sync
progress: electrumx flushes its DB cache at 1GB, i.e. rarely, and every
restart discards all unflushed work back to the last flush. This node
spent 8d14h in initial sync largely because gate runs and reboots kept
taking hours of progress away — it restarted at 06:16 and resumed from
959,774, the same height it had reported hours earlier.
The suite now detects initial sync POSITIVELY — a fresh (<30 min)
"our height: N daemon: M" line from electrumx's own log, gap > 10 — and
skips the probe and the four destructive tests with the gap named:
# skip electrumx initial sync in progress (1672 blocks behind) — ...
This is not the container-absent skip trap fixed earlier: absence of the
log line means "unknown" and the tests run and fail honestly. Validated
against the live mid-sync node: all four guards fired with the real gap;
on a synced node the line shows gap 0-1 and everything runs.
Unblocks the release gate from waiting hours on a sync it was itself
prolonging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both electrumx tests did socket.create_connection(("127.0.0.1", 50001)) and
called it proof the service was up. It is not: that is the HOST-published
port, and podman's port forwarder completes the TCP handshake even when
nothing inside the container is listening.
The cost was real. Both tests were green while mempool-api sat in a hard
ECONNREFUSED loop against electrumx:50001 from inside archy-net — the
operator's "mempool doesn't connect to bitcoin". Measured side by side:
127.0.0.1:50001 from the host OPEN, electrumx:50001 from a sibling container
REFUSED.
The cause is not a misconfiguration — mempool-api's CORE_RPC_HOST,
ELECTRUM_HOST and ports are all correct. electrumx has not finished its
initial sync (DB height 959,774 vs chain tip 961,706, ~1,932 blocks, 8d 14h
elapsed) and ElectrumX does not serve sessions until it has caught up.
Now both tests do a server.version round-trip and require a JSON result, so a
forwarder that accepts-and-drops fails. Verified against the live node: the
new probe FAILS today where the old one passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two fixes from the first full 125-test run (121 ok / 4 not ok).
1. The Quadlet unit test still asserted Restart=on-failure for backends,
which 8908fb4f deliberately changed. All 32 units on the node are
Restart=always; only archy-btcpay-db showed up because `fail` aborts at
the first offender. The old rationale — "backends need on-failure so an
operator `systemctl stop` stays stopped" — was wrong twice over:
systemd never applies Restart= to a unit stopped via `systemctl stop`,
and because quadlet renders --rm, a cleanly-exited container is deleted
and on-failure never brings it back (bitcoind exits 0 on SIGTERM, so
backends vanished). Verified on-device before that change: `podman stop
bitcoin-knots` came back in 12s, a dashboard stop stayed stopped 90s.
Now asserts Restart=always, with the corrected reasoning in-place.
2. all-apps-lifecycle cycled ten apps back-to-back and manufactured the
contention that then failed later tests: load went 10.8 at preflight to
17.6 mid-loop, and btcpay's recovery stretched from 52s on a quiet box
to 216s and then 512s, defeating waits of 180s and 300s. Widening waits
cannot win when the run sets the load they must survive, so the loop now
drains between apps — waits for load1 under 2x nproc, capped at 90s.
ARCHY_APP_SETTLE_SECS=0 disables; ARCHY_APP_SETTLE_LOAD overrides.
Verified: run.sh use-quadlet-backends-install → 6/6 (was 1 failure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both failed a gate run on a healthy node.
bitcoin-receive already tolerated WALLET_LOCKED, but its 180s window
starts when the TEST starts and the lnd restart that locks the wallet can
land partway in. On 2026-08-08 the restart hit 65s in and the wallet
unlocked at 2m25s (journal: lnd.service started 20:11:05, "wallet has
been unlocked without a time limit" 20:13:48) — 48s after the deadline
expired. The daemon's own unlock budget is ~10 min because opening the
channel and graph dbs takes minutes on a loaded box, so 180s was stricter
than the thing under test. Now 420s, ARCHY_LND_UNLOCK_SECS.
btcpay's start wait was 180s, but stopping btcpay DELETES the container
(quadlet renders --rm), so package.start is a full dotnet recreate rather
than a container start. Measured 52s on a quiet box; it exceeded 180s
during a gate run on the same node at load ~11. Now 300s,
ARCHY_BTCPAY_START_SECS.
Neither change masks a lifecycle fault: both paths were verified by hand
to complete correctly, just slower than the assertion allowed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop/start/restart loop carefully accumulates per-app failures into
$fails and prints them, but the three rpc_result calls were bare. Under
bats' errexit a bare call ends the test immediately, so the summary that
names the app never ran.
On 2026-08-08 that turned a single transient error into an unattributable
failure: the test died at package.stop with no indication which of the
ten targets was involved. It was mempool, and the identical call returned
{"status":"stopping"} by hand a few minutes later.
Each call now records <id>:<phase>-rpc and moves to the next app, so the
run reports what actually broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ui-coverage.bats and ui-probes.bash hardcoded https:// while the rest of
the harness builds URLs from ARCHY_SCHEME via lib/rpc.bash. That made the
suite unrunnable on a node serving the dashboard over http.
On archi-dev-box :443 is bound to the Tailscale / WireGuard / LAN
interface addresses but NOT to loopback, while :80 is bound on 0.0.0.0.
So five probes failed with "curl failed (network/timeout)" against
endpoints that were serving 200 the whole time — http://127.0.0.1/,
/catalog.json, /app/lnd/, /app/electrumx/ and /app/mempool/ all verified
200 by hand.
Default stays https, so nodes that already exercise the TLS path keep
doing so. Test titles drop the hardcoded scheme, since they no longer
describe which one ran.
Verified: ARCHY_SCHEME=http ./run.sh ui-coverage → 9/9 (1 skip),
previously 5 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gate run on a box at load ~14 failed five times over, and every failure
read "could not create a container" — searxng:start, package.start
btcpay-server, 3x electrumx — never a lifecycle fault. That sent two
separate sessions hunting a phantom host-wide cgroup failure. It was
contention.
Measured on the 4-core node: at load ~14 podman runs 9-16 processes deep
and healthchecks time out 3-8/min; at load ~3.7, podman ~1 and zero
timeouts. Restoring four containers whose HealthTimeout equals their
HealthInterval, unchanged, made the box *better* once load fell — so the
config is a latent hazard, not the cause here.
Preflight now checks, once, before iteration 1:
- aardvark-dns is singular (duplicates desync name resolution)
- load1 is under nproc+1, waiting up to 15 min for a spike to pass
- podman can actually create a container, 3/3
Deliberately NOT checked: the count of "Failed to create container" in
the journal. Those lines come from healthcheck exec churn and post-boot
settling, never reach 0 on a busy node, and gating on them would block
the gate forever. The probe proves creation positively instead.
Escape hatches: ARCHY_PREFLIGHT=0, ARCHY_MAX_LOAD, ARCHY_PREFLIGHT_SECS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 1 items 3 and 5.
Item 3 turned out to be far narrower than the plan's "93 files" once each hit
was classified rather than bulk-replaced. Sanitized only genuine operator
identifiers:
- FIPS test fixtures and a pine_ha comment carried real node LAN addresses ->
RFC 5737 TEST-NET-1, the convention already used elsewhere in this repo.
- Real tailnet addresses in fips/endpoints.rs, mock-backend.js and the mesh
test runner -> the base of the CGNAT range, obviously synthetic.
- Incident comments in appgate/mod.rs and apps/fedimint/manifest.yml named a
specific node; the role is what carries the meaning, so the address is gone.
- CHANGELOG.md held five real addresses in published release notes — the most
exposed of the lot.
Deliberately NOT touched, because the plan's item-3 list is over-broad and
following it literally would break working code:
- 192.168.1.1 / .254, 192.168.0.0/16 and 100.64.0.0/10 are generic router
defaults, RFC1918 classification in backup_rpc, and CGNAT range logic in
pine_ha / CompanionIntroOverlay. Not leaked infra.
- `tx1138` is listed as a hostname to scrub but is two live things: the
user-facing default block explorer (`DEFAULT_TX_EXPLORER`) and
`RETIRED_TX1138_HOST`, the migration constant whose entire job is stripping
that retired registry from existing nodes' saved mirror lists. Scrubbing
either breaks a feature. The plan needs this correction.
- Android's `192.168.1.100` strings are UI placeholder text.
Item 5: added *.key, *.pem, id_rsa*, *.sqlite, *.db to .gitignore, with a
negation for core/archipelago/src/appgate/testdata/*.key. Checked those first —
they are documented throwaway TLS fixtures compiled in via include_bytes!, not
node identity — and the negation stops the new rule silently dropping them if
they are ever regenerated. Verified both directions: fixtures not ignored, a
stray key elsewhere caught.
Verified: residual grep for real infra addresses is clean; audit-secrets.sh
still 5/5; app-catalog drift 0 (the fedimint edit is a YAML comment, which does
not survive parsing into the signed catalog); 44/44 fips tests pass with the
rewritten assertion fixtures.
Note: these test runs shared the working tree with another agent's in-flight
LND work, which was present but unstaged and is not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two concrete, verifiable documentation gaps from the open-source review:
- docs/ had no index. Adds docs/README.md grouping the 60-odd published docs by
task — getting started, architecture, app development, design docs, ADRs,
security, roadmap — in the bitcoin/bitcoin doc/ style the plan called for.
Every link in it resolves (checked). The top-level README now points at it as
the front door rather than duplicating the list.
- ROADMAP.md and tests/lifecycle/TESTING.md linked docs/multinode-testing-plan.md,
which moved to local-only (it is a fleet node inventory, not published). Those
references now describe the scope split in prose instead of pointing at a file
that is not in the public tree.
The index is honest about what is missing: it names the three app-developer
docs the plan flagged as gaps (quadlet compilation, container lifecycle,
secrets materialisation) as not-yet-written, and points at the authoritative
code for each rather than pretending they exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app store offered "update to 2.3.9" on a node already running 2.4.2 — the
release that fixes an actively exploited 2FA bypass. Taking it would have
rolled the node back onto the vulnerable version.
Root cause: available_update_for_images compared tags for inequality only.
Same repo + different tag meant "update available", with no ordering. Every
version claim upstream of it can go stale — the signed catalog, a legacy
catalog entry, the image-versions.sh baseline pin — and any one of them
lagging turned into a backwards Update button.
Guard added: when both tags parse as dotted-numeric versions, a lower pinned
version is never offered. Tags that cannot be ordered (RELEASE.2024-11-07…,
14-vectorchord0.4.3) keep the previous behaviour rather than silently losing
updates. This makes stale data fail safe, which matters more than any single
pin being correct.
Four sources still named 2.3.9, three of them able to act on it:
- releases/app-catalog.json — a LEGACY `btcpay` entry, distinct from
`btcpay-server`, carrying a concrete 2.3.9 image. catalog_primary_image
treats that as authoritative, so this is what drove the button. Fixed, but
held back from this commit: it needs re-signing.
- scripts/image-versions.sh — the baseline pin used when the catalog does not
cover an app.
- stacks.rs — the legacy BTCPay installer, twice. The fallback install path
would have deployed 2.3.9 outright.
- neode-ui curatedApps/marketplaceData and public/catalog.json — the store's
displayed version, hardcoded rather than read from the catalog, which is why
it still showed 2.3.9 after the update landed.
Audited every other installer for the same shape. The remaining literals are
the immich stack, which currently agrees with its manifests; hits in
set_config.rs and app_catalog.rs are test fixtures. To keep it that way,
scripts/check-installer-image-pins.py asserts that any installer literal
naming the same repository as an app manifest carries the same tag, and runs
blocking in CI. Verified it catches a simulated revert to 2.3.9.
Tests: 13/13 in image_versions including the exact BTCPay case, a genuine
upgrade still offered, equal versions silent, prerelease suffixes ordered on
their numbers, and opaque tags unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the registry host across 86 files: 309 references, covering all 40
app manifests, the orchestrator and container crates, the release and catalog
scripts, both demo-images workflows, the ISO builder, demo-deploy, and the
frontend marketplace data.
Verified the domain actually serves the registry before rewriting anything,
rather than assuming the web host implies the registry:
- TLS verifies clean, HTTP/2 on the web root
- an anonymous token grants a manifest fetch (HTTP 200) with no credentials
- skopeo inspect --no-creds resolves an image and lists its tags
That last check is the one that matters: an outside developer with no account
can now pull, which was the functional blocker for publishing at all.
Plain-HTTP references become HTTPS in the same pass, so OTA downloads stop
crossing the network in the clear.
Deliberately NOT rewritten:
- The public FIPS anchor on port 8444. It is a functional network endpoint
every node dials to bootstrap the mesh — closer to Bitcoin Core's hardcoded
seeds than to leaked infrastructure. The domain does resolve to the same
host, so it could become a hostname, but that adds a DNS dependency to the
path used precisely when things are broken. Worth a deliberate decision,
not a side effect of this change.
- The companion APK on port 2100. The domain returns 404 for that path, so
rewriting it would swap a working URL for a broken one. The Releases page
does serve (200), which is where the plan already wants those binaries.
- releases/app-catalog.json, releases/manifest.json and release-manifest.json.
These carry `signature` and `signed_by`; editing their contents invalidates
the signature and the fleet refuses artifacts that fail verification. They
were rewritten in a first pass and reverted — they must be regenerated and
re-signed through the signing ceremony instead, which needs the mnemonic.
So the catalog still advertises the old host until that ceremony runs. Nodes
resolve images through the signed catalog, not the on-disk manifests, so this
commit alone does not change what a node pulls.
Verified: archipelago-container 75/75; every manifest still parses with a
top-level app block; no signed artifact modified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keeps the dev and test tooling an outside contributor would want, and takes
our node addresses out of it.
Scripts that silently defaulted to one of our nodes now require an explicit
host and exit 2 without one: smoke-test.sh, trust-archipelago-cert.sh,
dev-container-test.sh (which also derives its RPC and health URLs from the
SSH target instead of a second hardcoded copy), and image-recipe/dev-branding.sh.
A default that points at a machine the user does not own is worse than no
default: it fails confusingly, or reaches a stranger's device.
Usage examples, mock data and test fixtures move to the RFC 5737
documentation range (192.0.2.0/24). CGNAT test values stay inside
100.64.0.0/10 so the range-check semantics they exercise still hold, and
192.168.1.0/.1/.254 are left alone — those are gateway logic and UI
placeholders, not our addresses.
Playwright and the perf spec defaulted their baseURL to one of our nodes;
they now default to localhost:8100, the local dev server.
Removed neode-ui APP_URLS entirely. It is dead code — exported, never
imported — and it pinned fedimint's *prod* launch URL to 192.168.1.228:8175.
Had anything consumed it, every user's node would have tried to reach an
address that on their LAN is either nothing or someone else's machine.
Deleting beats sanitizing dead config.
Verified: frontend 868/868 vitest across 108 files; archipelago-container
75/75; mesh tests 9/9; audit-secrets 5/5. Zero node addresses and zero node
names remain in tracked files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Untracking the ops docs broke every reference to them. Repoints or removes
those references across README, architecture, ROADMAP, the archive index,
lifecycle TESTING, the hardening plan, and the security docs — pointing at
the issue tracker where a live task list was meant, and dropping the entry
entirely where it only existed to link an internal file.
Also fixes two pre-existing broken links found by validating every relative
link in the tracked docs:
- README linked docs/OPEN_SOURCE_READINESS.md, which never existed
(underscores vs hyphens).
- reticulum-daemon/README.md linked a local Claude session plan at
../../.claude/plans/enchanted-strolling-rocket.md — outside the repo, and
a path that would have shipped publicly pointing at nothing.
All relative links in tracked markdown now resolve: 0 broken.
audit-secrets.sh still 5/5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operations docs move out of git entirely rather than being sanitized. They
stay on disk for local use and are gitignored, so the Phase 6 export (which
takes HEAD) can never carry them. 15 files: the fleet runbook, hotfix
process, node inventories, internal trackers, session handoffs, the key
rotation/signing-posture records, and the open-source plan itself.
For the docs that remain public, infra identifiers are replaced with things
that are better documentation rather than placeholders: curl examples now
use `archipelago.local`, the product's own mDNS name, so a reader can run
them as-is instead of substituting an address that was never theirs.
Deliberately NOT scrubbed, both verified as functional rather than leaked:
- `tx1138.com` is the shipped default block explorer (DEFAULT_TX_EXPLORER in
useTxExplorer.ts, surfaced in WalletSettingsModal). Product behavior.
- `git.tx1138.com` in core/container/{image_policy,registry}.rs is a retired-
registry constant the code matches on to strip stale entries from legacy
node configs. Removing it would break migration for older nodes.
- `192.168.1.254` in bulletproof-containers.md is the LAN gateway in a podman
bug description, and `192.168.1.x` in user-walkthrough.md is already generic.
Whether a personal domain should be the shipped explorer default in a public
product is a separate product question, not a security one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scrubs the fleet SSH/UI password from every tracked file (22 occurrences)
and removes inline credentials from the code paths that used them.
Docs and trackers keep the surrounding context — these are published under
docs/history/ per the open-source plan — with the literals replaced by
<FLEET_PW> / <FLEET_PW_ALT> so the "two variants exist" detail survives
without the values.
Three of the eight files were in .planning/ and were NOT in the plan's
enumerated list; the reworked audit-secrets.sh found them.
Code changes:
- neode-ui/test-openwrt.mjs: node URL and password come from ARCHY_NODE_URL /
ARCHY_NODE_PW; the SSH target derives from the URL instead of a hardcoded
tailnet IP; exits 2 when unset.
- scripts/run-post-install-tests.sh: drops the built-in "testpass123!"
default and adds --password-stdin; refuses to run unauthenticated instead
of silently trying a known password. --phase1-only still needs no password.
- .gitea/workflows/post-install-tests.yml: sshpass with an inline literal
replaced by key auth (NODE_SSH_KEY secret); password comes from the
NODE_UI_PASSWORD secret and is piped over stdin rather than argv, so it
stays out of the node's process list and the job log. Default target IP
removed.
scripts/audit-secrets.sh now reports 5/5 pass, 0 fail.
Note: rotation of the exposed credentials is deliberately deferred to the
pre-publish gate and is NOT done by this commit — these values are still
live. See Phase 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
63 main commits since the fork point — gate cookie-strip fix, named-volume
create fix, appgate catalog classification, RNode error surfacing — merged
so 13-14/13-15 on-device verification runs against current production code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bitcoin app vanished from updated nodes: the container exited instantly
with 'sh: Syntax error: "fi" unexpected'. My 1.7.124 change added an
explanatory comment INSIDE the manifest's folded YAML scalar (>-), where
'#' is not a comment — it is literal text that reaches the shell. Folding
joins lines with spaces, so the comment swallowed the 'if ... then' while
the more-indented echo survived as its own line, leaving an orphan 'fi'.
bitcoind never ran, the container exited, and the app disappeared from the
UI because detection is container-based.
Explanations now live above the '- >-' line where YAML really treats them
as comments. The loopback-conf tolerance (-allowignoredconf=1) is unchanged
and still needed.
Adds scripts/check-manifest-shell.py to the release gate: it runs 'sh -n'
over every embedded manifest script and rejects '#' inside these scalars.
Nothing validated this shell before — no YAML parse or Rust test could have
caught it, and it only failed on the node, after signing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds assert_safe_same_host_deploy(local_src, remote_dst) to lib/common.sh:
pure, no SSH inside it, callable directly from a test with fixed inputs.
Returns 0 only when the two already-resolved paths are equal; refuses
(non-zero, message naming both paths + the 2026-07-31 incident) on any
other same-host mismatch.
This closes a real gap in the 2026-07-31 incident's original fix: the old
guard's two `case` blocks refused only containment (source-in-destination
or destination-in-source). A SIBLING directory — for example this very
worktree, archy-phase13, deploying onto TARGET_DIR's resolved symlink
target (archy, the main checkout) — is neither contained by nor containing
of the destination, so the old guard let it through and `rsync --delete`
would have mirrored the sibling onto the main checkout, deleting everything
the sibling lacks. Found while retargeting deploy-to-target.sh for D-19,
not a D-19 effect itself.
deploy-to-target.sh's guard block now calls assert_safe_same_host_deploy
instead of the two inline containment-only case blocks (old logic removed,
not left dead alongside the new call).
tests/production-quality/deploy-guard-same-host.sh pins all five
<behavior> cases (identical/contained/containing/sibling/unrelated)
against the function with no SSH, no rsync, no real deploy — including the
exact archy-phase13-vs-archy pair as the sibling-directory regression pin.
Manually confirmed non-vacuous: flipping the sibling fixture's expectation
to "allow" makes the test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds tests/production-quality/aiui-proxy-closed.sh (follows lnd-cors-test.sh's
shape) and deploys+runs it against a real, genuinely remote node
(archy-x250-dev3, operator-approved deviation from archi-dev-box — see
SUMMARY key-decisions for why).
Confirmed on the node: unauthenticated /aiui/api/claude/v1/messages and
/aiui/api/ollama/api/tags both 401; claude-api-proxy sidecar unit gone;
nothing listens on :3142; the second key ledger (claude-api-proxy.env) is
gone. Along the way, root-caused and worked around a real deploy-topology
gap — the daemon self-heals nginx config from a second, stale on-node
template copy on every restart, silently reverting a hand-patched fix.
One finding is reported honestly rather than tuned away: deleted
/aiui/api/openrouter/ returns 200/405 via this app's SPA catch-all, not the
plan's literal 404 — the relay is structurally gone (zero proxy_pass to
openrouter.ai), but the exact status code doesn't match the acceptance
criterion. Left open for a human decision, per this task's own instruction
not to force a probe to pass.
This is Task 3 of a checkpoint:human-verify plan with gate="blocking". The
positive-path browser check and the openrouter-finding disposition remain
for a human; this executor does not self-approve the gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parsing each half back proves each is well-formed; it never proves they
belong together. A key from one generation beside a cert from another
passes both individual parse checks, gets blessed, and then nginx refuses
to start at the exact moment the marker claims first boot succeeded.
gen_tls() now extracts the public key from each half and compares them
before the swap, and needs_tls() applies the same check to what is already
installed, so a mismatched pair that reached disk some other way (an older
build, a half-finished manual edit) is repaired instead of quietly
breaking nginx. Extraction subsumes parsing, so this replaces the separate
-noout parse checks rather than adding to them.
Kept deliberately in step with regenerate_tls_cert() in
core/archipelago/src/api/rpc/system/handlers.rs, which does the same
comparison on the running node after a rename.
Test harness: the openssl stub keypair now carries the generation it came
from, and STUB_OPENSSL_MISMATCH emits a cert from a different one — the
pair that passes both parse checks and still breaks nginx. New case 9
covers both directions: fail closed when the mismatch arises during
generation, repair exactly once when found already on disk, and no spin
on the run after either. 9/9 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure fail-closed cannot catch, because generation SUCCEEDS.
This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long
before time has synced. `openssl req -x509` stamps notBefore from whatever the
clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be
years out: clock ahead -> clients reject it as "not yet valid", a harder failure
than the usual self-signed warning; clock behind -> notAfter is already in the
past once time syncs. The completion marker was then set and never revisited —
a node permanently serving a cert nothing accepts.
Finding 1, reported rather than assumed: this image does NOT use
systemd-timesyncd. It installs and enables chrony, and chrony-wait.service —
the unit that is Before=time-sync.target — is not enabled. So time-sync.target
is inert here and ordering After= it would buy nothing. Enabling chrony-wait to
make it meaningful would stall boot behind NTP on a node with no network, and
these nodes are routinely offline at first boot. Not deadlocking boot outranks
cert-date elegance, so the ordering is deliberately left alone.
Fixed locally instead, in two parts:
1. Backdate notBefore by 24h so ordinary skew between node and client cannot
invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and
the rootfs is debian:trixie which ships it, but the capability is PROBED,
not assumed — guessing wrong would fail every attempt and brick the node,
the exact outcome all of this exists to prevent. Without the flags we simply
do not backdate and rule 2 still covers the dangerous case.
2. Refuse to bless a cert dated by a clock outside a plausible window
(2026-01-01 .. 2056-01-01). The material stays installed so the node is
usable and sshd comes up, but the bad dates are recorded as
failed=cert-dates and the cert is regenerated automatically once time syncs.
Generation is now driven by need rather than by "is the marker absent", and
ConditionPathExists=! is removed from the unit so a node that already completed
can still be re-examined — skipping the unit is precisely how such a node stays
broken forever. The script exits in milliseconds when everything is fine.
Anti-spin is one condition: a date-driven regeneration happens ONLY when the
clock is currently plausible. A node whose clock is still wrong re-checks and
mints nothing.
Regression caught while writing this: driving generation purely by content made
needs_ssh() false whenever any host key existed, which would have left an
image-baked fleet-shared key in place forever — F-03 reopened. The marker check
is back in both needs_ functions and case 1 (which prestages a baked key and
asserts it was replaced) is what caught it.
Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin
directions. Controls: blessing regardless of clock reddens only case 8
(run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8
(SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially
passed against a broken guard because the assertion compared certificate dates,
and a re-mint under a frozen clock produces a byte-identical notBefore — the
assertion now counts mints, which is the only thing that distinguishes "left
alone" from "regenerated again".
Not covered here: nodes already deployed from earlier ISOs never receive this
script (it is installed by the installer, not by OTA), so fleet remediation for
them remains 10-04/OTA work in core/**, which is held by other executors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was unquoted, so the build
shell performed command substitution on the Dockerfile body. Any backtick in a
Dockerfile COMMENT was executed on the build host and its output spliced into
the generated file. Six comments did this. One of them ran
`systemctl start archipelago-fips.service` against the build machine on every
ISO build; the others were harmless only by accident of being command-not-found.
Fixes the class, not the six instances. The delimiter is now quoted, so the
body is emitted verbatim and a future backticked comment is inert. Verified the
boundary by line range first: the other backticked comments in this file
(:264, :809, :1188, :1289, :1506, :1605, :3597, :3651) are ordinary shell
comments outside any unquoted heredoc and were never at risk — they are
untouched.
The body needs exactly four build-time values and they are all package names
(LINUX_IMAGE_PKG, GRUB_EFI_PKG, GRUB_EFI_SIGNED_PKG, GRUB_PC_PKG), on four
consecutive lines. So quoting was practical: the heredoc is split into
DOCKERFILE_HEAD and DOCKERFILE_TAIL, both quoted, with a single explicit printf
interpolating those four names between them. Escapes that existed only because
the heredoc was unquoted are undone in the same pass: six trailing `\\` become
`\` (Docker line continuations) and four `\$` become `$` (RUN arguments reach
the shell verbatim — Docker does not substitute variables in RUN).
Verified by rendering the generated Dockerfile before and after with the same
inputs and diffing them normalised (continuations joined, whitespace
collapsed). Both are 190 normalised lines and the ONLY differences are the six
comments regaining their text — every instruction is byte-identical. Before:
"# the archipelago backend calls" / after: "# the archipelago backend calls
`systemctl start archipelago-fips.service`".
Test: case 7 asserts every heredoc writing Dockerfile.rootfs has a quoted
delimiter, and when one is not, reports which body lines would execute. The
assertion is on the delimiter, not on backticks — with quoting a backticked
comment is legal and six of them are back in the body on purpose, so flagging
backticks would flag a non-bug and fail on the very comments this restored.
This bug is invisible to `bash -n`; an instance of it introduced earlier in
this plan hung a syntactically-clean build for two minutes before being caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.
Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.
Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.
Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.
Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.
Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
marker set, failed units restarted. Control: dead-end on a node that already
failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
-> only case 6 red, naming the line.
Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first-boot per-device secret regeneration was fail-open: both branches
logged a warning and continued, and `touch "$MARKER"` ran unconditionally
outside both `if` blocks. Combined with the unit's ConditionPathExists=! and
the script's own marker fast-path, one transient failure left that node on the
image-wide shared SSH host key and TLS private key permanently and silently —
and the ISO is a published artefact, so every downloader holds those keys.
- Retry each generator 3 times with backoff (D-05), so a transient first-boot
condition recovers inside the same boot instead of being terminal.
- Write the completion marker ONLY when both TLS and SSH succeeded, so a
failed boot leaves the unit eligible to run again on the next boot.
- On terminal failure: durable record at
/var/lib/archipelago/first-boot-secrets.failed naming which generator
failed, plus console + logger + stderr, and exit 1 so the unit lands in
`failed` rather than `active`. The record is cleared on a later success.
- Add FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF seams. Unset in
production the behaviour is byte-identical; set, they let the fail-closed
property be asserted rather than claimed.
- Order the unit After=systemd-random-seed.service (no-op today, correct if a
seed file is ever baked).
- State the operational trade in the script header: after the rootfs strip, a
terminal failure means no SSH and no TLS and needs the physical console.
That was chosen deliberately over running on fleet-shared keys.
tests/first-boot-secrets/run-tests.sh extracts the shipped heredoc body from
the builder and drives it against a temp root with stubbed generators: both
succeed, openssl fails every attempt, ssh-keygen fails twice then succeeds.
Moving the marker touch back outside the success branch makes case 2 fail with
MARKER-SET-ON-FAILURE, which is the regression this pins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/mesh/run-mesh-tests.sh: cheap-first layers — 116 Rust mesh unit
tests, the daemon --selftest (now also asserting the announce app_data
wire contract and the set_name verb), and opt-in live assertions against
a running node (radio connected, named, mesh.refresh/broadcast, no
ARCHY-blob peer names). Verified green on archi-dev-box and archy-x250-dev.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- scripts/build-iso-release.sh: single gated command from signed release
to tested ISO (preflight version/signature parity, release gate harness,
strict catalog drift, full Rust suite, artifact version checks, build,
mount smoke, best-effort QEMU boot)
- scripts/iso-smoke-test.sh: standalone mount-level ISO verification incl.
stale-binary version assertion inside the payload
- .gitea/workflows/build-iso.yml: resurrected ISO CI as dispatch-only job
calling the gated orchestrator (old one was stranded in _archived/)
- tests/release/run.sh: catalog drift now runs --strict (was silently
always-pass)
- test-iso-qemu.sh: headless mode used -append without -kernel, which
QEMU rejects; use -display none so -serial file: capture works
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run B failed the same probe with LND_ERROR (gRPC 'waiting to start' maps
there) — the 7b77462d window only matched WALLET_LOCKED, one of four
phases a cascade-restarted lnd passes through. Retry every code except
LND_WALLET_UNINITIALIZED (no wallet — never self-heals) within a bounded
180s window; persistent failures still fail after it. lnd verified
healthy minutes after both failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate test 24 probed lnd.newaddress moments after the bitcoin bounce
cascade-restarted lnd; the auto-unlocker was still retrying against a
starting gRPC, so the probe saw LND_WALLET_LOCKED on a node that unlocks
itself seconds later (verified on .228 — same request succeeds post-run).
Treat WALLET_LOCKED as settling with a bounded 120s retry, mirroring the
e21f3baf settle window for tests 123/124; every other error still fails
fast, and a genuinely stuck-locked wallet still fails after the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mempool-api exits at startup while electrumx is catching up to the daemon
(7-day initial-sync ETA on .228), flapping through 'activating' for ~2 min
after each lifecycle cycle before systemd heals it. The instant asserts
probed inside that window both iterations. Poll up to 180s for
converges-to-active — a genuine crash-loop still fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wyoming-piper/wyoming-whisper are being integrated and are not yet part of
the platform contract; their hand-staged units (companion-style
Restart=always) failed gate test 121 on .228. Exclude wyoming-* from
backend_quadlet_units() until their packaging lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backend_quadlet_units() enumerates every *.container file on disk and the
active-state tests asserted each unit active + container running. A
user-stopped app keeps its unit file (e.g. the inactive half of the
bitcoin-core/bitcoin-knots multi-version pair after tonight's crash-loop
wave left core's unit on disk), so its inactive service / absent container
is the CORRECT state — gate run E false-failed on it (tests 123/124,
.228 2026-07-09). Honour the orchestrator's user-stopped.json intent
marker and skip those units in both tests.
Verified on .228: bitcoin-core's lingering unit now exempted; suite parses
(bats --count = 6) and the two fixed tests pass user-stopped filtering.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lnd.bats retry loop had no per-exec timeout — a wedged lnd RPC (chain-blind
after bitcoin-knots was recreated under it) hung one podman exec, and the
whole gate, indefinitely (.228, 30+ min at test 75). timeout 10 per attempt
keeps the intended ~4min bound. Tracker: new §C item for the root cause —
apps cache their backend container's IP; a backend recreate must cascade a
dependent restart/alert or lnd goes chain-blind silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- multinode suites no longer carry real node passwords as defaults: *_PW
vars are required, auto-loaded from git-ignored tests/multinode/.env
(lib sources it; .env.example documents the shape). Suites fail fast
with a clear message when unset.
- node_login/node_rpc get --connect-timeout 10 --max-time 120 (override:
MULTINODE_RPC_TIMEOUT) so one hung server-side RPC can't stall the whole
suite. Verified live: login+rpc against .116 OK; unroutable node fails
in 10s instead of hanging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- operations-runbook: redact the plaintext sudo password (rotate it — it
stays in git history); replace the nonexistent test-cross-node.sh /
test-reboot-survival.sh with the real entry points (lifecycle gate,
multinode smoke, e2e/post-install scripts)
- hotfix-process: de-hardcode the v1.0.x framing, point releases at
create-release.sh + primary Gitea, correct the rollback backup paths
(/opt/archipelago/rollback + updater backup dir, not /usr/local/bin)
- troubleshooting: connectivity probe start9.com -> debian.org
- INSTALL.sh: drop removed endurain app, point at developer-guide.md
(development-setup.md never existed)
- tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session
resume block with a resolved note; release-gate item 8 reflects the
2026-07-08 version decision (1.8.0-alpha)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prior fix's loop `container_installed "$c" && echo "$c"` makes the
function's own exit status the exit status of its LAST array entry. If
that entry isn't installed on this node (e.g. required-stack-destructive's
array ends with mempool-api, absent on .5), the whole function reports
failure even though earlier entries matched fine — and under bats' set -e,
`targets="$(installed_required_containers)"` then aborts the test outright.
required-stack.bats got lucky (its array happens to end with an installed
container) but has the identical latent bug. Caught live on .5's iteration
3 of the multinode-pass gate run. Add explicit `return 0`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of bug as required-stack.bats: hardcoded required_containers
included mempool/mempool-api unconditionally, so a node without the
mempool stack (e.g. .5) hard-fails restarting a container that was never
installed, and waits out full 180-240s timeouts probing endpoints that
will never come up. Likely explains .5's abnormally long (2216s) iteration
1 runtime during the current multinode-pass run. Same skip-if-absent fix
as the prior commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live during the .5 multinode-pass run: this suite was hardcoded to
.116's exact app bundle (including the mempool stack), so any node missing
an app hard-failed instead of skipping — and a missing local fail() helper
(present in 3 sibling bats files, absent here) masked the real error as
"command not found" (exit 127). Add the same skip-if-absent idiom already
used in mempool.bats per-app, and define fail() locally like the others.
Verified: skips cleanly on .116 (no bitcoin-knots here), still exercises
real checks for apps that are installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
docs/UNIFIED-TASK-TRACKER.md replaces hunting across SESSION-1.8.0-OTA-PROGRESS.md
and PRODUCTION-MASTER-PLAN.md for "what's left" — fastest/simplest tasks first.
Verified against live code/nodes rather than trusting doc text: several previously
"open" items (bind-dir chown, netbird legacy installer, launch-port fallback,
archival-bitcoin manifest field, progress-UI monotonicity, all-apps coverage,
fedimint test coverage, changelog backfill, portainer image pin, grafana quadlet
activation) turned out already shipped or non-issues, and are closed out here.
TESTING.md's release-gate checklist updated to match reality (cargo warnings,
5x gate, changelog already green; multinode/backend-default-flip/tag genuinely open).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two Phase-3 bats suites used `fail` (a bats-assert helper) but bats-assert
isn't installed on the alpha fleet (only bats-core), so every tripped
assertion crashed with `fail: command not found` (status 127) instead of
reporting a real pass/fail. Define the same minimal `fail() { echo ...;
return 1; }` the other suites already use (see mempool.bats). Without this
the gates were silently non-functional.
Also rewrite the obsolete "HealthCmd= implies Notify=healthy" assertion in
use-quadlet-backends-install.bats. Phase 3.4's Notify=healthy was
deliberately reverted: gating `systemctl start` on health hung boot
reconciliation for dependency-waiting apps (fedimint idles until Bitcoin
IBD; lnd until macaroon unlock), leaving units stuck "deactivating". The
renderer now emits HealthCmd= for Podman's health state but TimeoutStartSec=0
and NO Notify=healthy (quadlet.rs render() + contains_stale_health_gate()).
The test now asserts the current invariant: no backend unit gates start on
health.
Verified on the .228 canary node (ARCHIPELAGO_USE_QUADLET_BACKENDS=1):
use-quadlet-backends-install 6/6, backend-survives-archipelago-restart 3/3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fresh Meshtastic radios ship region-UNSET (RF-silent) and on mismatched
channels, so nodes only ever saw themselves. Bring them to MeshCore parity
using the official Meshtastic admin API:
- Auto-provision LoRa region (set_config, AdminMessage field 34) from a new
mesh-config `lora_region` (e.g. EU_868) when the radio's region differs.
- Auto-provision a shared primary channel (set_channel, field 33) with a
PSK derived deterministically from channel_name, so every node converges on
one mesh — the parity equivalent of MeshCore's named "archipelago" channel.
- Read current region/channel from want_config; only write when different
(no reboot loop); cap attempts so a radio that won't persist can't loop.
- Active NodeInfo advert scaffolding + aggressive serial drain.
Verified on .116+.228: region+channel persist, discovery works (both see each
other as named reachable contacts), bidirectional RF + sending confirmed.
Receiving in the running driver is still under diagnosis (instrumentation added).
Also removes the unwanted `meshtastic` daemon app from the registry (it was
never meant to be a container — native driver provides system-level support):
deletes apps/meshtastic + catalog entries (app-catalog, neode-ui, releases) +
test refs. Meshtastic stays native, like MeshCore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Active counterpart to the read-only all-apps-matrix.bats: drives
stop/start/restart for every installed app and, under
ARCHY_ALLOW_CASCADE_DESTRUCTIVE, a FULL teardown (uninstall →
no-ghost → reinstall) — the broad coverage F needs beyond the ~8 core
suites. App set is discovered from My Apps ∩ the node catalog; reinstall
spec comes from catalog.json {dockerImage, containerConfig}.
PROTECTED by default (never cycled or torn down): bitcoin*/electrum*
(expensive resync) AND lnd/btcpay*/fedimint* (teardown = irreversible
wallet/channel/guardian loss). The user asked to protect only
bitcoin+electrum; the wallet apps are added for safety and can be
removed via ARCHY_MATRIX_PROTECT. Heavy + destructive → a supervised
pass, not folded into run-gate. Validated on .228: discovery excludes
the 6 protected installed apps; lifecycle tier cycles a single app
(botfights) stop/start/restart green; teardown gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run-gate.sh ran only the DESTRUCTIVE tier; the cascade-uninstall suite
(uninstall→no-ghost→reinstall, the #13/#14/uninstall-hang regression
guard) existed but was never enabled by the gate. Add an opt-in single
cascade pass after the 5× loop (ARCHY_GATE_CASCADE=1, requires
ARCHY_ALLOW_DESTRUCTIVE=1), counted into the pass/fail tally. Kept out
of the 5× loop deliberately — uninstall/reinstall every iteration would
balloon runtime and re-pull images; one pass guards the class. Default
gate behavior unchanged. Validated: cascade-uninstall.bats 7/7 on .228.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 5x destructive gate on heavy nodes false-failed on transient windows
during stack recovery, not real regressions:
- immich.bats: lan_address port-publish probe 30s -> 90s. The postgres->redis
->server (DB migrations on boot) stack can take >30s to republish :2283 after
a churn-induced recreate; destructive-tier immich tests already allow 180-240s.
- mempool.bats: orphan-container check now polls to steady state (<=30s) instead
of a single-shot count, which caught a recreated member briefly visible
alongside its replacement mid-reconcile.
- run-gate.sh: settle cap 180s -> 300s and also gate on immich's :2283 when
installed, so the next iteration's read-only probe doesn't race a still-
recovering stack. Settle returns the instant every probe is green.
A genuinely unexposed/orphaned/unhealthy app still fails these checks; they only
absorb the transient recreate window under sustained churn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-app suites cover ~8 core apps in depth; nothing covered the ~30 others
(jellyfin, vaultwarden, penpot, nextcloud, grafana, …). all-apps-matrix.bats
derives the app set from server.get-state package-data (no hardcoded list) and
asserts baseline health across EVERY installed app:
- settles to a non-transitional state within a window (the #13/#14 stuck-ghost
class, generalized fleet-wide — installing/removing that never settles)
- not in error/failed
- reports a recognized (non-garbage) state
- every running UI app (manifest ui=="true") exposes a non-null lan-address
(the immich/port-drift unreachable-UI failure, generalized to all UI apps)
Read-only, so it joins run.sh/run-gate.sh on every node and grows coverage as
nodes install more apps. Verified 5/5 on .228 (17 apps) and .116 (20 apps).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 5x gate is DESTRUCTIVE-only and never exercised uninstall/reinstall — where
the worst field bugs lived (#13 app ghosting in My Apps after uninstall, #14
reinstall stalling on stale state). New cascade-uninstall.bats drives the full
teardown path on a throwaway app (default grafana, precondition-skips if already
installed so it can't destroy real data) and asserts:
- fresh install reaches running via a truthful, non-silent progression
- uninstall makes the entry DISAPPEAR from server.get-state package-data
(the literal My Apps map) — no ghost, no stuck uninstall stage
- container + (on-node) data dir are gone
- reinstall returns to running
- node left as found
Opt-in via ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1; not yet folded into the canonical
gate. Verified 7/7 against .228.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It called bats-assert's `fail` (not loaded in this file) → "fail:
command not found"/127, masking the real reason. Emit+return instead,
bump the cold-restart RPC window 60s→120s (block-index reload), and
note a node mid-IBD legitimately can't serve getinfo (environmental
precondition, not a product regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
