archy/docs/security/KEY-02-FLEET-ROTATION.md

KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)

Phase 10 plan 10-04. Companion to docs/security/KEY-02-ROOTFS-EVIDENCE.md, which covers the build half (10-03).

10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion marker on a failed run. That does nothing for nodes already in the field, which is exactly where the exposure sits — a node that hit the old fail-open path is running the SSH host key and TLS private key that every downloader of that ISO also holds, and it will never try again.

This document records the two human decisions that govern the deployed half.

---

D-06 rotation trigger

Chosen option: detect-report-then-apply — recorded 2026-08-02.

Verbatim option id as written in 10-04-PLAN.md: detect-report-then-apply ("Detect and report on boot; rotate only when an operator runs the script with an explicit apply flag").

Why

Rotating an SSH host key is one-way. Every known_hosts entry for that node breaks, on every machine that has ever connected to it, and the old private key is destroyed by the swap. The fleet is reached over Tailscale for day-to-day work and several nodes are remote — .228 is at a remote site and is in real use (CLAUDE.md). auto-on-boot would fire that rotation on many nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the new fingerprints. A node whose only access path is SSH and whose tooling pins the host key becomes unreachable until someone clears the entry; a rotation that fails partway on a remote node needs physical console access to recover, which for .228 means a site visit.

Against that, the cost of detect-report-then-apply is that exposure persists on any node whose operator does not act. That cost is bounded by making the verdict visible: detection runs at boot on every node and the verdict reaches system.stats, so an exposed node shows up in the dashboard without shell access. The exposure becomes measured rather than assumed, and the list of nodes still to rotate is a fact on a screen rather than a guess.

This also matches the project's standing policy that changes are verified on the dev pair (archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, feedback_dev_pair_before_ota). A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it has been observed on the dev pair it has already run everywhere.

What this decision binds

• scripts/security/host-secrets-audit.sh defaults to --detect, which is read-only.
• --apply without --yes prints its plan and exits 0 having touched nothing, so a mistyped invocation is inert.
• image-recipe/configs/archipelago-host-secrets-audit.service ships in detect-only mode. It contains no apply path. Making the boot unit rotate would require editing the unit, which is a deliberate act, not a default.
• --apply --yes refuses to do anything unless the detect pass returned shared. A node whose verdict is per-node cannot have its keys rotated by this script even by explicit command — the guard against "operator runs it on the wrong node" is structural, not procedural.

Consequence recorded honestly

Any node whose verdict comes back shared and which is never revisited stays exposed indefinitely. The mitigation is the visibility, not the automation. The list under "Nodes with a shared verdict, deliberately not rotated" below exists so that no such node is quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.

---

How a node decides

Four on-disk signals, evaluated in this precedence order by scripts/security/host-secrets-audit.sh --detect. Every verdict carries the evidence strings that produced it, and each evidence string names the file it was read from.

#	Signal	Source
1	mtime of each host key / the TLS key against the first-boot anchor	/var/lib/archipelago/.secrets-regenerated, falling back to /root/.luks-archipelago.key then /etc/machine-id
2	The fail-open fingerprint: marker present and a WARNING: line in the first-boot log	/var/log/archipelago-first-boot-secrets.log
3	10-03's durable failure record	/var/lib/archipelago/first-boot-secrets.failed
4	Rootfs provenance	/opt/archipelago/rootfs-identity-stripped

Verdicts: per-node, shared, fail-closed-missing, unknown.

per-node is never reported on the strength of an absent signal. With no anchor at all the verdict is unknown, and while a durable failure record stands the verdict is unknown rather than per-node — the node's own generator most recently reported failure, so a clean-looking mtime is not evidence of success.

Signal 4 changes the meaning of missing material rather than adding to the shared/per-node question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an absent host key is a fail-closed state (generation never succeeded), not a shared one.

---

C-3 — per-node host key and TLS uniqueness

Audit checklist item C-3 (docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md §855), described there as "the highest-value check here".

Status: PARTIAL — see the table below

---

Nodes with a shared verdict, deliberately not rotated

None recorded yet.

Any node that reports shared and is not rotated in the same session MUST be added here with the date and the reason, so that the standing consequence of detect-report-then-apply is a visible list rather than an assumption.

Node label	Date detected	Why not rotated	Owner
—	—	—	—

---

Operator runbook — rotating one node

Run this from a session you are willing to lose, on one node at a time. Never on .228 or any node in real use without arranging access recovery first.

# 1. Detect. Read-only; safe on any node, including production.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
cat /var/lib/archipelago/host-secrets-audit.json

# 2. Dry run. Prints the plan, touches nothing, exits 0.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply

# 3. Rotate. Only proceeds if the verdict is `shared`.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes

# 4. WITHOUT closing that session, prove it survived:
echo still-here

# 5. From a second terminal, expect a host-key mismatch warning. That is the
#    correct outcome. Update known_hosts against the fingerprints printed by
#    step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
#    blindly accepting whatever is offered.
ssh-keygen -R <node>
ssh <node>

# 6. The web UI will present a new self-signed cert. A fresh browser trust
#    prompt is expected and is the correct outcome.

The script reloads sshd rather than restarting it. A reload re-execs the listener while already-forked session children keep running, which is why the operator's own SSH session survives its own rotation. restart would kill it, and on a remote node with no console that is unrecoverable.

Old fingerprints are written to /var/lib/archipelago/host-key-rotation.json before the swap, so an operator who loses access anyway can still identify what changed.