From 96dba73a1621181a0cd904ed559dbb4ca8d78fef Mon Sep 17 00:00:00 2001 From: archipelago Date: Sun, 2 Aug 2026 14:20:53 -0400 Subject: [PATCH] docs(10-04): record D-06 rotation trigger as detect-report-then-apply MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 1 of 10-04 is a blocking decision checkpoint, rated one-way: rotating a node's SSH host key invalidates every known_hosts entry for it fleet-wide and the old private key is destroyed by the swap. Chosen: detect-report-then-apply. auto-on-boot would fire simultaneous known_hosts breakage across the fleet during an OTA with no operator holding the new fingerprints, and a rotation that fails partway on a remote node (.228 is at a remote site and in real use) needs physical console access. It also cannot be dev-paired, which contradicts the standing verify-on-the-dev-pair- first policy — by the time it has been observed on the dev pair it has already run everywhere. The cost of the chosen option — exposure persists on any node nobody revisits — is bounded by making the verdict visible in system.stats rather than by automation, and by keeping a list in this document of every node that reported `shared` and was deliberately not rotated. Records what the decision binds: detect-only default, --apply inert without --yes, the boot unit carries no apply path, and --apply --yes refuses on any node whose verdict is not `shared`. Co-Authored-By: Claude Opus 5 (1M context) --- docs/security/KEY-02-FLEET-ROTATION.md | 154 +++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 docs/security/KEY-02-FLEET-ROTATION.md diff --git a/docs/security/KEY-02-FLEET-ROTATION.md b/docs/security/KEY-02-FLEET-ROTATION.md new file mode 100644 index 00000000..4d3a867a --- /dev/null +++ b/docs/security/KEY-02-FLEET-ROTATION.md @@ -0,0 +1,154 @@ +# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half) + +Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the +build half (10-03). + +10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into +the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion +marker on a failed run. That does **nothing** for nodes already in the field, which is exactly +where the exposure sits — a node that hit the old fail-open path is running the SSH host key and +TLS private key that every downloader of that ISO also holds, and it will never try again. + +This document records the two human decisions that govern the deployed half. + +--- + +## D-06 rotation trigger + +**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02. + +Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`** +("Detect and report on boot; rotate only when an operator runs the script with an explicit apply +flag"). + +### Why + +Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every +machine that has ever connected to it, and the old private key is destroyed by the swap. The +fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at +a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many +nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the +new fingerprints. A node whose only access path is SSH and whose tooling pins the host key +becomes unreachable until someone clears the entry; a rotation that fails partway on a remote +node needs physical console access to recover, which for `.228` means a site visit. + +Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose +operator does not act. That cost is bounded by making the verdict **visible**: detection runs at +boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the +dashboard without shell access. The exposure becomes measured rather than assumed, and the list +of nodes still to rotate is a fact on a screen rather than a guess. + +This also matches the project's standing policy that changes are verified on the dev pair +(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`). +A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it +has been observed on the dev pair it has already run everywhere. + +### What this decision binds + +- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only. +- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped + invocation is inert. +- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode. + It contains no apply path. Making the boot unit rotate would require editing the unit, which is + a deliberate act, not a default. +- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose + verdict is `per-node` cannot have its keys rotated by this script even by explicit command — + the guard against "operator runs it on the wrong node" is structural, not procedural. + +### Consequence recorded honestly + +Any node whose verdict comes back `shared` and which is never revisited stays exposed +indefinitely. The mitigation is the visibility, not the automation. The list under +"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is +quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept. + +--- + +## How a node decides + +Four on-disk signals, evaluated in this precedence order by +`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings +that produced it, and each evidence string names the file it was read from. + +| # | Signal | Source | +|---|---|---| +| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` | +| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` | +| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` | +| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` | + +Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`. + +**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the +verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather +than `per-node` — the node's own generator most recently reported failure, so a clean-looking +mtime is not evidence of success. + +Signal 4 changes the meaning of missing material rather than adding to the shared/per-node +question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an +absent host key is a **fail-closed** state (generation never succeeded), not a shared one. + +--- + +## C-3 — per-node host key and TLS uniqueness + +Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described +there as "the highest-value check here". + +### Status: PARTIAL — see the table below + + + +--- + +## Nodes with a `shared` verdict, deliberately not rotated + +None recorded yet. + +Any node that reports `shared` and is not rotated in the same session MUST be added here with the +date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible +list rather than an assumption. + +| Node label | Date detected | Why not rotated | Owner | +|---|---|---|---| +| — | — | — | — | + +--- + +## Operator runbook — rotating one node + +Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or +any node in real use without arranging access recovery first. + +```bash +# 1. Detect. Read-only; safe on any node, including production. +sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect +cat /var/lib/archipelago/host-secrets-audit.json + +# 2. Dry run. Prints the plan, touches nothing, exits 0. +sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply + +# 3. Rotate. Only proceeds if the verdict is `shared`. +sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes + +# 4. WITHOUT closing that session, prove it survived: +echo still-here + +# 5. From a second terminal, expect a host-key mismatch warning. That is the +# correct outcome. Update known_hosts against the fingerprints printed by +# step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by +# blindly accepting whatever is offered. +ssh-keygen -R +ssh + +# 6. The web UI will present a new self-signed cert. A fresh browser trust +# prompt is expected and is the correct outcome. +``` + +The script reloads sshd rather than restarting it. A reload re-execs the listener while +already-forked session children keep running, which is why the operator's own SSH session +survives its own rotation. `restart` would kill it, and on a remote node with no console that is +unrecoverable. + +Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the +swap, so an operator who loses access anyway can still identify what changed.