archy/docs/security/KEY-02-FLEET-ROTATION.md

# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)

Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
build half (10-03).

10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
TLS private key that every downloader of that ISO also holds, and it will never try again.

This document records the two human decisions that govern the deployed half.

---

## D-06 rotation trigger

**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.

Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
flag").

### Why

Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
machine that has ever connected to it, and the old private key is destroyed by the swap. The
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
node needs physical console access to recover, which for `.228` means a site visit.

Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
of nodes still to rotate is a fact on a screen rather than a guess.

This also matches the project's standing policy that changes are verified on the dev pair
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
has been observed on the dev pair it has already run everywhere.

### What this decision binds

- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
  invocation is inert.
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
  It contains no apply path. Making the boot unit rotate would require editing the unit, which is
  a deliberate act, not a default.
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
  verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
  the guard against "operator runs it on the wrong node" is structural, not procedural.

### Consequence recorded honestly

Any node whose verdict comes back `shared` and which is never revisited stays exposed
indefinitely. The mitigation is the visibility, not the automation. The list under
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.

---

## How a node decides

Four on-disk signals, evaluated in this precedence order by
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
that produced it, and each evidence string names the file it was read from.

| # | Signal | Source |
|---|---|---|
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |

Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.

**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
mtime is not evidence of success.

Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.

---

## C-3 — per-node host key and TLS uniqueness

Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
there as "the highest-value check here".

### Status: **FAILED — with finding.** Recorded 2026-08-02.

> **This section names live fleet nodes that are still running shared key material.
> Review it before this repository is made public** (`docs/OPEN-SOURCE-READINESS-PLAN.md`).
> Digests below are truncated; the fingerprints of public keys are public data — every SSH
> handshake offers them — but there is no reason to make a target list convenient.

**Three distinct live fleet nodes share all three of their SSH host keys. Two of those three
also share their TLS certificate, and therefore their TLS private key.** This is not a
theoretical exposure: it is F-03 in production, today.

#### Method

Gathered **remotely and read-only** — no node was logged into, nothing was written to any node,
nothing was rotated. Host keys came from `ssh-keyscan`, which is what every SSH client does
before it decides whether to trust a host, and certificates from an anonymous TLS handshake:

```bash
ssh-keyscan -T 6 <node> | ssh-keygen -lf -
openssl s_client -connect <node>:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 -subject
```

This is a deliberately weaker instrument than the checklist's on-node commands, and it was chosen
because it needs no access and can therefore cover the whole reachable fleet rather than two
nodes. What it can prove is exactly the FAIL condition: *any fingerprint appearing on two nodes*.

#### Result

| Node label | SSH host keys (ECDSA/ED25519/RSA, truncated) | TLS cert sha256 (truncated) | Cert CN |
|---|---|---|---|
| `archipelago-1`    | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archy-x250-beta`  | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archipelago`      | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `7C:6B:CD:98…` | `austin-sapien` |
| `archipelago-5`    | `/bmgd6jS…` / `SpaNfLLf…` / `hhVFABi3…` | `95:FE:EB:C7…` | `archipelago.local` |
| `archi-dev-box`    | `8hFU7QGM…` / `GAxNAcgX…` / `Tv7AfaVp…` | (no :443 listener) | — |
| `archy-dev-pa`     | `JtD/RM0a…` / `XD2A5OVL…` / `esIBpbWk…` | not probed | — |
| `framework-pt`     | `oicpsj3Y…` / `zxA1/kRU…` / `oxi+tMli…` | `88:85:CE:CC…` | `framework-pt` |
| `shorty-s` (`.228`) | `YVsgrv8M…` / `D/5n851i…` / `YMFLUerk…` | `4D:98:D4:9B…` | `shorty-s` |

Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`,
`quantumterminal`.

#### That the three are genuinely different machines, not one host seen three times

The obvious alternative explanation for identical host keys is a single machine registered on the
tailnet more than once. Ruled out:

- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled`
  instance serves one tailnet identity, so three simultaneously-live addresses are three hosts.
- `tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers
  from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago`
  answer from `45.20.199.86` on different source ports — a different continent for the first,
  and two distinct machines behind one NAT for the other two.
- They are owned by different tailnet accounts.

#### Why `archipelago` has a different TLS cert but the same SSH keys

Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a
node that was **renamed** through `server.set-name`, which re-mints the TLS cert via
`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else.

This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that
a node's key material is per-node.** Any renamed node gets a unique certificate for free while
its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS
fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable
signal, and this is why the audit script treats the two classes separately and reports which one
is shared rather than issuing a single node-level verdict.

#### What this does NOT establish — UNVERIFIED

| Claim | Status | Evidence still needed |
|---|---|---|
| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. |
| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. |
| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. |
| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. |
| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. |

#### Consequence

`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**.
Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys,
and for the first two, their TLS private key as well — enough for undetectable SSH host
impersonation and transparent MITM of the web UI.

None of them was rotated as part of this verification, and that is deliberate: this checkpoint
verifies, it does not remediate, and remediating a node inside a verification task is how a
verification task takes a node offline. They are recorded below.

---

## Nodes with a `shared` verdict, deliberately not rotated

Any node that reports `shared` and is not rotated in the same session MUST be added here with the
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
list rather than an assumption.

| Node label | Date detected | Why not rotated | Next step |
|---|---|---|---|
| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. |
| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. |
| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. |

**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use;
the rotation is one-way and every existing entry for them dies with it. Sequencing that is an
operator decision, which is the whole content of D-06.

---

## Operator runbook — rotating one node

Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
any node in real use without arranging access recovery first.

```bash
# 1. Detect. Read-only; safe on any node, including production.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
cat /var/lib/archipelago/host-secrets-audit.json

# 2. Dry run. Prints the plan, touches nothing, exits 0.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply

# 3. Rotate. Only proceeds if the verdict is `shared`.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes

# 4. WITHOUT closing that session, prove it survived:
echo still-here

# 5. From a second terminal, expect a host-key mismatch warning. That is the
#    correct outcome. Update known_hosts against the fingerprints printed by
#    step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
#    blindly accepting whatever is offered.
ssh-keygen -R <node>
ssh <node>

# 6. The web UI will present a new self-signed cert. A fresh browser trust
#    prompt is expected and is the correct outcome.
```

The script reloads sshd rather than restarting it. A reload re-execs the listener while
already-forked session children keep running, which is why the operator's own SSH session
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
unrecoverable.

Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
swap, so an operator who loses access anyway can still identify what changed.