Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
build half (10-03).
10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
TLS private key that every downloader of that ISO also holds, and it will never try again.
This document records the two human decisions that govern the deployed half.
---
## D-06 rotation trigger
**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.
Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
flag").
### Why
Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
machine that has ever connected to it, and the old private key is destroyed by the swap. The
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
node needs physical console access to recover, which for `.228` means a site visit.
Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
of nodes still to rotate is a fact on a screen rather than a guess.
This also matches the project's standing policy that changes are verified on the dev pair
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
has been observed on the dev pair it has already run everywhere.
### What this decision binds
-`scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
-`--apply`**without**`--yes` prints its plan and exits 0 having touched nothing, so a mistyped
invocation is inert.
-`image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
It contains no apply path. Making the boot unit rotate would require editing the unit, which is
a deliberate act, not a default.
-`--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
the guard against "operator runs it on the wrong node" is structural, not procedural.
### Consequence recorded honestly
Any node whose verdict comes back `shared` and which is never revisited stays exposed
indefinitely. The mitigation is the visibility, not the automation. The list under
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.
---
## How a node decides
Four on-disk signals, evaluated in this precedence order by
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
that produced it, and each evidence string names the file it was read from.
| # | Signal | Source |
|---|---|---|
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`,
`quantumterminal`.
#### That the three are genuinely different machines, not one host seen three times
The obvious alternative explanation for identical host keys is a single machine registered on the
tailnet more than once. Ruled out:
- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled`
instance serves one tailnet identity, so three simultaneously-live addresses are three hosts.
-`tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers
from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago`
answer from `45.20.199.86` on different source ports — a different continent for the first,
and two distinct machines behind one NAT for the other two.
- They are owned by different tailnet accounts.
#### Why `archipelago` has a different TLS cert but the same SSH keys
Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a
node that was **renamed** through `server.set-name`, which re-mints the TLS cert via
`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else.
This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that
a node's key material is per-node.** Any renamed node gets a unique certificate for free while
its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS
fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable
signal, and this is why the audit script treats the two classes separately and reports which one
is shared rather than issuing a single node-level verdict.
#### What this does NOT establish — UNVERIFIED
| Claim | Status | Evidence still needed |
|---|---|---|
| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. |
| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. |
| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. |
| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. |
| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. |
#### Consequence
`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**.
Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys,
and for the first two, their TLS private key as well — enough for undetectable SSH host
impersonation and transparent MITM of the web UI.
None of them was rotated as part of this verification, and that is deliberate: this checkpoint
verifies, it does not remediate, and remediating a node inside a verification task is how a
verification task takes a node offline. They are recorded below.
| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. |
| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. |
| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. |
**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use;
the rotation is one-way and every existing entry for them dies with it. Sequencing that is an
operator decision, which is the whole content of D-06.
