Header claimed the whole 2026-04-22 plan "has been implemented". The
architecture was adopted, but checking each item against the tree:
- The `core/archipelago/src/reconcile/` module the doc lays out in detail —
desired.rs / current.rs / diff.rs / apply.rs / derived.rs / backoff.rs — was
never created. The reconciler shipped as container/boot_reconciler.rs +
container/prod_orchestrator.rs instead.
- FM2's named fix `reconcile::derived::render_bitcoin_conf` does not exist. The
drift was eliminated a different way: bitcoind runs with an explicit `-conf`
derived from secrets each start, and stale datadir configs are removed.
- FM1/FM3 are partial — companion UIs are Quadlet units, main app containers are
not, since `use_quadlet_backends` still defaults false. The "v1.7.48+ full
reconcile module / main containers become Quadlet units" step has not happened.
- **FM6 was never implemented.** There is no podman corrupt-state probe and no
`system renumber` recovery anywhere in the tree. The 2026-04 failure that made
a registry node unreachable would still require manual SSH today — which is
precisely the "zero-manual-intervention" target this doc opens with.
FM4 and FM5 did ship as described.
Replaced the blanket claim with a per-item table so the doc stays useful as
incident history without reading as a description of the code, and noted that
the unit path throughout says /etc/containers/systemd/ while units are actually
written to ~/.config/containers/systemd/ (the path is rootless).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This 2026-04 plan has been implemented, but it still read as an active plan
("implementation started"), linked private agent-memory paths, and ended with a
stale "To resume" work block naming fleet nodes, dated fleet state, and the next
file to edit.
- Header now marks it a historical design record and points at
container-lifecycle.md for the current behaviour.
- Removed the two private ~/.claude/.../memory/ references from the header and
the entire "To resume" section (private paths, node numbers, 2026-04-22 fleet
snapshot — none of it belongs in a public design doc).
- Genericized the one remaining node-number reference in the incident narrative.
The valuable content — the six failure modes and the reconciler reasoning that
answered them — is kept intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operations docs move out of git entirely rather than being sanitized. They
stay on disk for local use and are gitignored, so the Phase 6 export (which
takes HEAD) can never carry them. 15 files: the fleet runbook, hotfix
process, node inventories, internal trackers, session handoffs, the key
rotation/signing-posture records, and the open-source plan itself.
For the docs that remain public, infra identifiers are replaced with things
that are better documentation rather than placeholders: curl examples now
use `archipelago.local`, the product's own mDNS name, so a reader can run
them as-is instead of substituting an address that was never theirs.
Deliberately NOT scrubbed, both verified as functional rather than leaked:
- `tx1138.com` is the shipped default block explorer (DEFAULT_TX_EXPLORER in
useTxExplorer.ts, surfaced in WalletSettingsModal). Product behavior.
- `git.tx1138.com` in core/container/{image_policy,registry}.rs is a retired-
registry constant the code matches on to strip stale entries from legacy
node configs. Removing it would break migration for older nodes.
- `192.168.1.254` in bulletproof-containers.md is the LAN gateway in a podman
bug description, and `192.168.1.x` in user-walkthrough.md is already generic.
Whether a personal domain should be the shipped explorer default in a public
product is a separate product question, not a security one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes failure mode FM5 from docs/bulletproof-containers.md: the v1.7.38 +
v1.7.39 rollouts left every affected node on an unreachable UI (nginx 500)
with no recovery path short of SSH. This release adds a self-check
guardrail to the update flow.
What changed:
- apply_update() writes a pending-verify marker with old+new version and
a 150s deadline immediately before scheduling the service restart.
- verify_pending_update() runs from main.rs startup. If the marker is
present and within its freshness window, the new binary waits 15s for
nginx + backend to settle, then probes https://127.0.0.1/ every 5s for
up to 90s (self-signed certs accepted).
- On any probe success within the window, the marker is cleared and
nothing else happens.
- On window-exhaust, the new binary:
1. Moves the broken /opt/archipelago/web-ui to web-ui.failed.<ts>
(quarantined, not deleted, so we can post-mortem).
2. Restores web-ui.bak on top of web-ui.
3. Calls rollback_update() to restore the previous binary.
4. Updates state.current_version to reflect the rollback.
5. systemctl --no-block restart archipelago so the OLD binary boots.
- Markers older than 10 minutes are treated as stale and cleared without
probing, so a crashed-during-startup marker from weeks ago cannot
spontaneously roll back a healthy node on a later reboot.
- rollback_update() binary copy now goes through host_sudo instead of
tokio::fs::copy, so it escapes the service's ProtectSystem=strict
mount namespace. Without this, the rollback silently failed with
EROFS on /usr/local/bin and orphaned the rollback - the exact
opposite of what auto-rollback is for.
Tests: 4 new unit tests in update::tests covering marker round-trip,
absent-marker noop, no-panic on verify_pending_update with nothing to
verify, and an invariant assert that the 90s probe window stays below
the 600s stale threshold. All passing.
Side fix: scripts/create-release-manifest.sh was dying with exit 141
(SIGPIPE from tar tvzf pipe head pipe awk) under set -euo pipefail.
Replaced with a single awk NR==1 that doesn't short-circuit the upstream
pipe, so the release-build flow is idempotent again.
