# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half) Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the build half (10-03). 10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion marker on a failed run. That does **nothing** for nodes already in the field, which is exactly where the exposure sits — a node that hit the old fail-open path is running the SSH host key and TLS private key that every downloader of that ISO also holds, and it will never try again. This document records the two human decisions that govern the deployed half. --- ## D-06 rotation trigger **Chosen option: `detect-report-then-apply`** — recorded 2026-08-02. Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`** ("Detect and report on boot; rotate only when an operator runs the script with an explicit apply flag"). ### Why Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every machine that has ever connected to it, and the old private key is destroyed by the swap. The fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the new fingerprints. A node whose only access path is SSH and whose tooling pins the host key becomes unreachable until someone clears the entry; a rotation that fails partway on a remote node needs physical console access to recover, which for `.228` means a site visit. Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose operator does not act. That cost is bounded by making the verdict **visible**: detection runs at boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the dashboard without shell access. The exposure becomes measured rather than assumed, and the list of nodes still to rotate is a fact on a screen rather than a guess. This also matches the project's standing policy that changes are verified on the dev pair (archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`). A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it has been observed on the dev pair it has already run everywhere. ### What this decision binds - `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only. - `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped invocation is inert. - `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode. It contains no apply path. Making the boot unit rotate would require editing the unit, which is a deliberate act, not a default. - `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose verdict is `per-node` cannot have its keys rotated by this script even by explicit command — the guard against "operator runs it on the wrong node" is structural, not procedural. ### Consequence recorded honestly Any node whose verdict comes back `shared` and which is never revisited stays exposed indefinitely. The mitigation is the visibility, not the automation. The list under "Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept. --- ## How a node decides Four on-disk signals, evaluated in this precedence order by `scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings that produced it, and each evidence string names the file it was read from. | # | Signal | Source | |---|---|---| | 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` | | 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` | | 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` | | 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` | Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`. **`per-node` is never reported on the strength of an absent signal.** With no anchor at all the verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather than `per-node` — the node's own generator most recently reported failure, so a clean-looking mtime is not evidence of success. Signal 4 changes the meaning of missing material rather than adding to the shared/per-node question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an absent host key is a **fail-closed** state (generation never succeeded), not a shared one. --- ## C-3 — per-node host key and TLS uniqueness Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described there as "the highest-value check here". ### Status: PARTIAL — see the table below --- ## Nodes with a `shared` verdict, deliberately not rotated None recorded yet. Any node that reports `shared` and is not rotated in the same session MUST be added here with the date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible list rather than an assumption. | Node label | Date detected | Why not rotated | Owner | |---|---|---|---| | — | — | — | — | --- ## Operator runbook — rotating one node Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or any node in real use without arranging access recovery first. ```bash # 1. Detect. Read-only; safe on any node, including production. sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect cat /var/lib/archipelago/host-secrets-audit.json # 2. Dry run. Prints the plan, touches nothing, exits 0. sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply # 3. Rotate. Only proceeds if the verdict is `shared`. sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes # 4. WITHOUT closing that session, prove it survived: echo still-here # 5. From a second terminal, expect a host-key mismatch warning. That is the # correct outcome. Update known_hosts against the fingerprints printed by # step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by # blindly accepting whatever is offered. ssh-keygen -R ssh # 6. The web UI will present a new self-signed cert. A fresh browser trust # prompt is expected and is the correct outcome. ``` The script reloads sshd rather than restarting it. A reload re-execs the listener while already-forked session children keep running, which is why the operator's own SSH session survives its own rotation. `restart` would kill it, and on a remote node with no console that is unrecoverable. Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the swap, so an operator who loses access anyway can still identify what changed.