docs(10-04): record D-06 rotation trigger as detect-report-then-apply
Task 1 of 10-04 is a blocking decision checkpoint, rated one-way: rotating a node's SSH host key invalidates every known_hosts entry for it fleet-wide and the old private key is destroyed by the swap. Chosen: detect-report-then-apply. auto-on-boot would fire simultaneous known_hosts breakage across the fleet during an OTA with no operator holding the new fingerprints, and a rotation that fails partway on a remote node (.228 is at a remote site and in real use) needs physical console access. It also cannot be dev-paired, which contradicts the standing verify-on-the-dev-pair- first policy — by the time it has been observed on the dev pair it has already run everywhere. The cost of the chosen option — exposure persists on any node nobody revisits — is bounded by making the verdict visible in system.stats rather than by automation, and by keeping a list in this document of every node that reported `shared` and was deliberately not rotated. Records what the decision binds: detect-only default, --apply inert without --yes, the boot unit carries no apply path, and --apply --yes refuses on any node whose verdict is not `shared`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
527f602322
commit
96dba73a16
@@ -0,0 +1,154 @@
|
||||
# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)
|
||||
|
||||
Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
|
||||
build half (10-03).
|
||||
|
||||
10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
|
||||
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
|
||||
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
|
||||
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
|
||||
TLS private key that every downloader of that ISO also holds, and it will never try again.
|
||||
|
||||
This document records the two human decisions that govern the deployed half.
|
||||
|
||||
---
|
||||
|
||||
## D-06 rotation trigger
|
||||
|
||||
**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.
|
||||
|
||||
Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
|
||||
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
|
||||
flag").
|
||||
|
||||
### Why
|
||||
|
||||
Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
|
||||
machine that has ever connected to it, and the old private key is destroyed by the swap. The
|
||||
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
|
||||
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
|
||||
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
|
||||
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
|
||||
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
|
||||
node needs physical console access to recover, which for `.228` means a site visit.
|
||||
|
||||
Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
|
||||
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
|
||||
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
|
||||
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
|
||||
of nodes still to rotate is a fact on a screen rather than a guess.
|
||||
|
||||
This also matches the project's standing policy that changes are verified on the dev pair
|
||||
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
|
||||
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
|
||||
has been observed on the dev pair it has already run everywhere.
|
||||
|
||||
### What this decision binds
|
||||
|
||||
- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
|
||||
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
|
||||
invocation is inert.
|
||||
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
|
||||
It contains no apply path. Making the boot unit rotate would require editing the unit, which is
|
||||
a deliberate act, not a default.
|
||||
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
|
||||
verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
|
||||
the guard against "operator runs it on the wrong node" is structural, not procedural.
|
||||
|
||||
### Consequence recorded honestly
|
||||
|
||||
Any node whose verdict comes back `shared` and which is never revisited stays exposed
|
||||
indefinitely. The mitigation is the visibility, not the automation. The list under
|
||||
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
|
||||
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.
|
||||
|
||||
---
|
||||
|
||||
## How a node decides
|
||||
|
||||
Four on-disk signals, evaluated in this precedence order by
|
||||
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
|
||||
that produced it, and each evidence string names the file it was read from.
|
||||
|
||||
| # | Signal | Source |
|
||||
|---|---|---|
|
||||
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
|
||||
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
|
||||
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
|
||||
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |
|
||||
|
||||
Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.
|
||||
|
||||
**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
|
||||
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
|
||||
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
|
||||
mtime is not evidence of success.
|
||||
|
||||
Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
|
||||
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
|
||||
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.
|
||||
|
||||
---
|
||||
|
||||
## C-3 — per-node host key and TLS uniqueness
|
||||
|
||||
Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
|
||||
there as "the highest-value check here".
|
||||
|
||||
### Status: PARTIAL — see the table below
|
||||
|
||||
<!-- gsd:c3-evidence -->
|
||||
|
||||
---
|
||||
|
||||
## Nodes with a `shared` verdict, deliberately not rotated
|
||||
|
||||
None recorded yet.
|
||||
|
||||
Any node that reports `shared` and is not rotated in the same session MUST be added here with the
|
||||
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
|
||||
list rather than an assumption.
|
||||
|
||||
| Node label | Date detected | Why not rotated | Owner |
|
||||
|---|---|---|---|
|
||||
| — | — | — | — |
|
||||
|
||||
---
|
||||
|
||||
## Operator runbook — rotating one node
|
||||
|
||||
Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
|
||||
any node in real use without arranging access recovery first.
|
||||
|
||||
```bash
|
||||
# 1. Detect. Read-only; safe on any node, including production.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
|
||||
cat /var/lib/archipelago/host-secrets-audit.json
|
||||
|
||||
# 2. Dry run. Prints the plan, touches nothing, exits 0.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply
|
||||
|
||||
# 3. Rotate. Only proceeds if the verdict is `shared`.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes
|
||||
|
||||
# 4. WITHOUT closing that session, prove it survived:
|
||||
echo still-here
|
||||
|
||||
# 5. From a second terminal, expect a host-key mismatch warning. That is the
|
||||
# correct outcome. Update known_hosts against the fingerprints printed by
|
||||
# step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
|
||||
# blindly accepting whatever is offered.
|
||||
ssh-keygen -R <node>
|
||||
ssh <node>
|
||||
|
||||
# 6. The web UI will present a new self-signed cert. A fresh browser trust
|
||||
# prompt is expected and is the correct outcome.
|
||||
```
|
||||
|
||||
The script reloads sshd rather than restarting it. A reload re-execs the listener while
|
||||
already-forked session children keep running, which is why the operator's own SSH session
|
||||
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
|
||||
unrecoverable.
|
||||
|
||||
Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
|
||||
swap, so an operator who loses access anyway can still identify what changed.
|
||||
Reference in New Issue
Block a user