archy/docs/security/KEY-02-FLEET-ROTATION.md

# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)

Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
build half (10-03).

10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
TLS private key that every downloader of that ISO also holds, and it will never try again.

This document records the two human decisions that govern the deployed half.

---

## D-06 rotation trigger

**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.

Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
flag").

### Why

Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
machine that has ever connected to it, and the old private key is destroyed by the swap. The
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
node needs physical console access to recover, which for `.228` means a site visit.

Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
of nodes still to rotate is a fact on a screen rather than a guess.

This also matches the project's standing policy that changes are verified on the dev pair
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
has been observed on the dev pair it has already run everywhere.

### What this decision binds

- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
  invocation is inert.
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
  It contains no apply path. Making the boot unit rotate would require editing the unit, which is
  a deliberate act, not a default.
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
  verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
  the guard against "operator runs it on the wrong node" is structural, not procedural.

### Consequence recorded honestly

Any node whose verdict comes back `shared` and which is never revisited stays exposed
indefinitely. The mitigation is the visibility, not the automation. The list under
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.

---

## How a node decides

Four on-disk signals, evaluated in this precedence order by
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
that produced it, and each evidence string names the file it was read from.

| # | Signal | Source |
|---|---|---|
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |

Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.

**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
mtime is not evidence of success.

Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.

---

## C-3 — per-node host key and TLS uniqueness

Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
there as "the highest-value check here".

### Status: PARTIAL — see the table below

<!-- gsd:c3-evidence -->

---

## Nodes with a `shared` verdict, deliberately not rotated

None recorded yet.

Any node that reports `shared` and is not rotated in the same session MUST be added here with the
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
list rather than an assumption.

| Node label | Date detected | Why not rotated | Owner |
|---|---|---|---|
| — | — | — | — |

---

## Operator runbook — rotating one node

Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
any node in real use without arranging access recovery first.

```bash
# 1. Detect. Read-only; safe on any node, including production.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
cat /var/lib/archipelago/host-secrets-audit.json

# 2. Dry run. Prints the plan, touches nothing, exits 0.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply

# 3. Rotate. Only proceeds if the verdict is `shared`.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes

# 4. WITHOUT closing that session, prove it survived:
echo still-here

# 5. From a second terminal, expect a host-key mismatch warning. That is the
#    correct outcome. Update known_hosts against the fingerprints printed by
#    step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
#    blindly accepting whatever is offered.
ssh-keygen -R <node>
ssh <node>

# 6. The web UI will present a new self-signed cert. A fresh browser trust
#    prompt is expected and is the correct outcome.
```

The script reloads sshd rather than restarting it. A reload re-execs the listener while
already-forked session children keep running, which is why the operator's own SSH session
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
unrecoverable.

Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
swap, so an operator who loses access anyway can still identify what changed.