diff --git a/docs/security/KEY-02-FLEET-ROTATION.md b/docs/security/KEY-02-FLEET-ROTATION.md index 4d3a867a..1b2c515e 100644 --- a/docs/security/KEY-02-FLEET-ROTATION.md +++ b/docs/security/KEY-02-FLEET-ROTATION.md @@ -95,23 +95,113 @@ absent host key is a **fail-closed** state (generation never succeeded), not a s Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described there as "the highest-value check here". -### Status: PARTIAL — see the table below +### Status: **FAILED — with finding.** Recorded 2026-08-02. - +> **This section names live fleet nodes that are still running shared key material. +> Review it before this repository is made public** (`docs/OPEN-SOURCE-READINESS-PLAN.md`). +> Digests below are truncated; the fingerprints of public keys are public data — every SSH +> handshake offers them — but there is no reason to make a target list convenient. + +**Three distinct live fleet nodes share all three of their SSH host keys. Two of those three +also share their TLS certificate, and therefore their TLS private key.** This is not a +theoretical exposure: it is F-03 in production, today. + +#### Method + +Gathered **remotely and read-only** — no node was logged into, nothing was written to any node, +nothing was rotated. Host keys came from `ssh-keyscan`, which is what every SSH client does +before it decides whether to trust a host, and certificates from an anonymous TLS handshake: + +```bash +ssh-keyscan -T 6 | ssh-keygen -lf - +openssl s_client -connect :443 /dev/null \ + | openssl x509 -noout -fingerprint -sha256 -subject +``` + +This is a deliberately weaker instrument than the checklist's on-node commands, and it was chosen +because it needs no access and can therefore cover the whole reachable fleet rather than two +nodes. What it can prove is exactly the FAIL condition: *any fingerprint appearing on two nodes*. + +#### Result + +| Node label | SSH host keys (ECDSA/ED25519/RSA, truncated) | TLS cert sha256 (truncated) | Cert CN | +|---|---|---|---| +| `archipelago-1` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` | +| `archy-x250-beta` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` | +| `archipelago` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `7C:6B:CD:98…` | `austin-sapien` | +| `archipelago-5` | `/bmgd6jS…` / `SpaNfLLf…` / `hhVFABi3…` | `95:FE:EB:C7…` | `archipelago.local` | +| `archi-dev-box` | `8hFU7QGM…` / `GAxNAcgX…` / `Tv7AfaVp…` | (no :443 listener) | — | +| `archy-dev-pa` | `JtD/RM0a…` / `XD2A5OVL…` / `esIBpbWk…` | not probed | — | +| `framework-pt` | `oicpsj3Y…` / `zxA1/kRU…` / `oxi+tMli…` | `88:85:CE:CC…` | `framework-pt` | +| `shorty-s` (`.228`) | `YVsgrv8M…` / `D/5n851i…` / `YMFLUerk…` | `4D:98:D4:9B…` | `shorty-s` | + +Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`, +`quantumterminal`. + +#### That the three are genuinely different machines, not one host seen three times + +The obvious alternative explanation for identical host keys is a single machine registered on the +tailnet more than once. Ruled out: + +- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled` + instance serves one tailnet identity, so three simultaneously-live addresses are three hosts. +- `tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers + from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago` + answer from `45.20.199.86` on different source ports — a different continent for the first, + and two distinct machines behind one NAT for the other two. +- They are owned by different tailnet accounts. + +#### Why `archipelago` has a different TLS cert but the same SSH keys + +Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a +node that was **renamed** through `server.set-name`, which re-mints the TLS cert via +`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else. + +This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that +a node's key material is per-node.** Any renamed node gets a unique certificate for free while +its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS +fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable +signal, and this is why the audit script treats the two classes separately and reports which one +is shared rather than issuing a single node-level verdict. + +#### What this does NOT establish — UNVERIFIED + +| Claim | Status | Evidence still needed | +|---|---|---| +| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. | +| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. | +| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. | +| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. | +| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. | + +#### Consequence + +`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**. +Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys, +and for the first two, their TLS private key as well — enough for undetectable SSH host +impersonation and transparent MITM of the web UI. + +None of them was rotated as part of this verification, and that is deliberate: this checkpoint +verifies, it does not remediate, and remediating a node inside a verification task is how a +verification task takes a node offline. They are recorded below. --- ## Nodes with a `shared` verdict, deliberately not rotated -None recorded yet. - Any node that reports `shared` and is not rotated in the same session MUST be added here with the date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible list rather than an assumption. -| Node label | Date detected | Why not rotated | Owner | +| Node label | Date detected | Why not rotated | Next step | |---|---|---|---| -| — | — | — | — | +| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. | +| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. | +| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. | + +**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use; +the rotation is one-way and every existing entry for them dies with it. Sequencing that is an +operator decision, which is the whole content of D-06. ---