lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 4 Actions Packages Projects Releases 72 Wiki Activity

docs(10-04): complete fleet host-secret detection and rotation plan

Browse Source 
SUMMARY for 10-04, plus three WINDOWS.md entries (11-13) so the unverified
items stay visible at ship time: the rotation never exercised on real
hardware, host_secrets never observed in a live system.stats, and the C-3
finding itself — three live nodes still on shared SSH host keys, two of them
also sharing a TLS private key, none of them rotated.

STATE.md and ROADMAP.md deliberately not touched: both carry other agents'
uncommitted work in this shared tree and the orchestrator owns them for this
wave.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-02 15:11:58 -04:00
co-authored by Claude Opus 5
parent a806a658a6
commit 689c4cca1a
2 changed files with 588 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/WINDOWS.md
+42 -3
View File
@@ -1,10 +1,10 @@
---
schema_version: 1
open_count: 9
open_count: 12
waived_count: 0
fixed_count: 1
total_count: 10
last_updated: 2026-07-31T10:56:26.933Z
total_count: 13
last_updated: 2026-08-02T19:07:40.522Z
---
# Broken Windows Ledger
@@ -25,6 +25,9 @@ last_updated: 2026-07-31T10:56:26.933Z
| 8 | 02 | deviation | neode-ui/src/views/web5/Web5.vue | | Web5 revisit-ms regression (566->709->1329ms, zero overlap across 3 runs) despite confirmed instance survival; confirmed phase-2-caused split-signal cost, not fixed (deploy blocked this session) | open | | 2026-07-31T10:56:26.570Z | |
| 9 | 02 | deviation | neode-ui/src/views/AppDetails.vue | | AppDetails revisit-ms regression (1204->1510->2668ms across 3 runs); confirmed phase-2-caused split-signal cost, not fixed (deploy blocked this session) | open | | 2026-07-31T10:56:26.751Z | |
| 10 | 02 | deviation | neode-ui/src/views/server/OpenWrtGateway.vue | | OpenWrtGateway revisit-ms regression (663.5->1148->1460ms across 3 runs); confirmed phase-2-caused split-signal cost, not fixed (deploy blocked this session) | open | | 2026-07-31T10:56:26.933Z | |
| 11 | 10 | unrun-verify | docs/security/KEY-02-FLEET-ROTATION.md | | C-3 FAILED: archipelago-1, archy-x250-beta and archipelago share all three SSH host keys; the first two also share their TLS private key. Not rotated — needs an operator-driven --apply --yes per node. | open | | 2026-08-02T19:07:39.861Z | |
| 12 | 10 | unrun-verify | scripts/security/host-secrets-audit.sh | | Rotation never exercised on real hardware: that 'systemctl reload ssh' keeps the operator's own forked session alive is proven only by design, not by observation. Needs --apply --yes on one disposable node from a session the operator is willing to lose. | open | | 2026-08-02T19:07:40.217Z | |
| 13 | 10 | unrun-verify | core/archipelago/src/api/rpc/system/handlers.rs | | system.stats host_secrets never observed on a real node — proven against the file contract in unit tests only. Needs a build carrying 10-04 deployed to the dev pair, then a system.stats call. | open | | 2026-08-02T19:07:40.522Z | |
````json
[
@@ -147,6 +150,42 @@ last_updated: 2026-07-31T10:56:26.933Z
"reason": "",
"recorded_at": "2026-07-31T10:56:26.933Z",
"resolved_at": null
},
{
"id": 11,
"kind": "unrun-verify",
"phase": "10",
"file": "docs/security/KEY-02-FLEET-ROTATION.md",
"line": null,
"description": "C-3 FAILED: archipelago-1, archy-x250-beta and archipelago share all three SSH host keys; the first two also share their TLS private key. Not rotated — needs an operator-driven --apply --yes per node.",
"status": "open",
"reason": "",
"recorded_at": "2026-08-02T19:07:39.861Z",
"resolved_at": null
},
{
"id": 12,
"kind": "unrun-verify",
"phase": "10",
"file": "scripts/security/host-secrets-audit.sh",
"line": null,
"description": "Rotation never exercised on real hardware: that 'systemctl reload ssh' keeps the operator's own forked session alive is proven only by design, not by observation. Needs --apply --yes on one disposable node from a session the operator is willing to lose.",
"status": "open",
"reason": "",
"recorded_at": "2026-08-02T19:07:40.217Z",
"resolved_at": null
},
{
"id": 13,
"kind": "unrun-verify",
"phase": "10",
"file": "core/archipelago/src/api/rpc/system/handlers.rs",
"line": null,
"description": "system.stats host_secrets never observed on a real node — proven against the file contract in unit tests only. Needs a build carrying 10-04 deployed to the dev pair, then a system.stats call.",
"status": "open",
"reason": "",
"recorded_at": "2026-08-02T19:07:40.522Z",
"resolved_at": null
}
]
````
.planning/phases/10-key-material-hardening/10-04-SUMMARY.md
+546
View File
@@ -0,0 +1,546 @@
---
phase: 10-key-material-hardening
plan: 04
subsystem: fleet-host-secrets
tags: [security, ssh-host-keys, tls, systemd, ota, rotation, bash, rust, f-03]
requires:
- "10-03: fail-closed first-boot secret generation, the /opt/archipelago/rootfs-identity-stripped provenance marker, and the /var/lib/archipelago/first-boot-secrets.failed durable failure record this script keys off"
provides:
- "scripts/security/host-secrets-audit.sh — on-node detection of image-baked host secrets from the node's own disk alone, and a guarded one-time rotation"
- "Four-signal detection with a fixed precedence and per-verdict evidence strings, each naming the file it was read from"
- "Verdicts per-node | shared | fail-closed-missing | unknown — per-node is never inferred from an absent signal"
- "Per-key-class sharedness: SSH and TLS are judged and rotated independently, because a renamed node has a unique cert and shared host keys"
- "Access-preserving rotation: stage everything, abort before any swap, record old fingerprints first, TLS before SSH, mv-onto-path rather than rm-then-mv, reload sshd never restart"
- "archipelago-host-secrets-audit.service — detect-only boot unit delivered by the existing OTA runtime-asset promotion"
- "system.stats host_secrets object — the verdict visible without shell access"
- "tests/first-boot-secrets/rotation-tests.sh — 8-case harness through the HOST_SECRETS_ROOT seam, with four negative controls"
- "docs/security/KEY-02-FLEET-ROTATION.md — D-06's recorded decision, the C-3 result, and the not-yet-rotated register"
affects: [ota-runtime-assets, systemd, system.stats, sshd, nginx-tls, release-packaging]
tech-stack:
added: []
patterns:
- "HOST_SECRETS_ROOT path-prefix seam, the same move FIRST_BOOT_SECRETS_ROOT makes in 10-03. Every property worth testing here is negative or ordering — 'touches nothing', 'aborts before any swap', 'records old fingerprints BEFORE the swap' — and none of them is assertable without the ability to force the failure."
- "Ordering asserted by observation, not by content. The systemctl stub records whether the rotation record existed AT THE MOMENT of the first reload. Comparing fingerprints proves the right values were written; only this proves they were written first."
- "Judge and remediate per key class, never per node. A node renamed via server.set-name has a freshly-minted TLS cert and untouched image-baked SSH host keys; a node-level verdict would call it clean."
- "Precedence over accumulation: missing material can never be shared material, so the missing check runs first; direct evidence (the fail-open log line) outranks inference from timestamps."
key-files:
created:
- scripts/security/host-secrets-audit.sh
- image-recipe/configs/archipelago-host-secrets-audit.service
- tests/first-boot-secrets/rotation-tests.sh
- docs/security/KEY-02-FLEET-ROTATION.md
modified:
- core/archipelago/src/bootstrap.rs
- core/archipelago/src/api/rpc/system/handlers.rs
- scripts/create-release-manifest.sh
key-decisions:
- "D-06 resolved as detect-report-then-apply. auto-on-boot would fire simultaneous fleet-wide known_hosts breakage during an OTA with no operator holding the new fingerprints, and — the argument that settled it — it cannot be dev-paired: by the time the behaviour has been observed on the dev pair it has already run everywhere, which contradicts the project's standing verify-on-the-dev-pair-first policy. The cost of the chosen option (exposure persists on any node nobody revisits) is bounded by visibility in system.stats and by a written register of nodes not yet rotated, not by automation."
- "--apply writes NOTHING — not even its own verdict file — until --yes. 'Touches nothing' is worth being able to state without a footnote, and 'except for one file it rewrites' is what the footnote would have been. This also turns the dry-run test into an exact whole-tree comparison rather than one with a carve-out."
- "--apply --yes refuses unless the verdict is `shared`. The guard against running it on the wrong node is structural rather than procedural — on a per-node node the command is inert even when typed deliberately and confirmed."
- "Host keys are replaced by mv onto the existing path, not rm-then-mv. rm-then-mv opens a window in which the node has zero host keys on disk; sshd restarting into that window is unrecoverable remotely. Stale key types the new set does not include are removed only AFTER every staged key has landed — leaving an ssh_host_dsa_key behind would leave shared material behind."
- "TLS is swapped before SSH. A dead web UI is recoverable over SSH; dead SSH on a remote node is not. Do the recoverable one first so a failure between the two leaves the recoverable path intact."
- "reload sshd, never restart — stated in the script as the single most important line in the file. A reload re-execs the listener while already-forked session children keep running, so the operator survives their own rotation. The harness fails outright, before any case runs, if `systemctl restart ssh` ever appears in the file."
- "A third sanctioned key producer is created, and said so loudly rather than quietly. Producer 1 (the ISO builder) is not present on a deployed node; producer 2 (TlsMaterial::regenerate) does TLS only and nothing in the daemon has ever rotated an SSH host key. The script header names all three and pins their shared parameters (rsa:2048, 3650 days, same subject and SAN, stage-parse-pair-check-swap) so they cannot drift apart."
- "The verdict is never allowed to be optimistic. No anchor -> unknown; a standing first-boot-secrets.failed record -> unknown even when every mtime looks clean. T-10-37 is that a false per-node verdict leaves an exposed node looking clean, which is strictly worse than no verdict."
- "system.stats carries the verdict and the evidence but NOT the fingerprints. They are public data, so this is not confidentiality — it is that a payload polled every few seconds should not carry digests an operator already on the node can read from disk. A unit test fails if a future edit forwards the whole file."
requirements-completed: []
coverage:
- id: D1
description: "A deployed node determines from its own disk alone whether its SSH host keys and TLS key are image-baked or per-node (D-06)"
requirement: KEY-02
verification:
- kind: unit
ref: "tests/first-boot-secrets/rotation-tests.sh cases 1-5 — per-node, shared-by-mtime, shared-by-fail-open-fingerprint, fail-closed-missing, unknown"
status: pass
- kind: other
ref: "Real run on archi-dev-box: `sudo scripts/security/host-secrets-audit.sh --detect --json` -> per-node, anchored on /etc/machine-id, and its three fingerprints match an independent ssh-keyscan of the same host exactly"
status: pass
human_judgment: false
- id: D2
description: "The verdict is surfaced beyond a log file — it appears in system.stats so it is visible without shell access"
requirement: KEY-02
verification:
- kind: unit
ref: "handlers.rs::host_secrets_tests — 4/4: absent file -> unknown, unparseable file -> unknown, recorded verdict+evidence surfaced, rotated_at only when a rotation was recorded, fingerprints deliberately absent"
status: pass
- kind: other
ref: "Never observed on a real node — needs a build carrying this plan deployed to the dev pair, then a system.stats call"
status: blocked
human_judgment: true
- id: D3
description: "Rotation never loses remote access mid-flight: staging then atomic swap, reload rather than restart, new fingerprints recorded where an operator can read them"
requirement: KEY-02
verification:
- kind: unit
ref: "rotation-tests.sh case 7a — old fingerprints on disk at the moment of the first reload (ordering observed, not inferred), keys replaced, `reload ssh` present and `restart ssh` absent from the systemctl log, verdict re-derived to per-node"
status: pass
- kind: unit
ref: "rotation-tests.sh case 7b — with the SSH generator failing after TLS staging succeeded, the whole tree is byte-identical, no rotation record is written, and not one service is reloaded"
status: pass
- kind: other
ref: "That a reload keeps the operator's own forked SSH session alive is proven by design, not by observation. UNVERIFIED on hardware."
status: blocked
human_judgment: true
- id: D4
description: "Rotation does not happen by accident: detect-only default, --apply inert without --yes, and the trigger is a resolved human decision (D-06)"
requirement: KEY-02
verification:
- kind: unit
ref: "rotation-tests.sh case 6 — --apply without --yes exits 0 and not one byte of the tree changes, including the state dir; dry-run output warns that it is one-way"
status: pass
- kind: other
ref: "docs/security/KEY-02-FLEET-ROTATION.md ## D-06 rotation trigger records `detect-report-then-apply` verbatim with the date and the reason; the shipped unit contains no apply path"
status: pass
human_judgment: false
- id: D5
description: "Two real nodes flashed from the same ISO are proven to have distinct SSH host-key and TLS fingerprints (audit C-3)"
requirement: KEY-02
verification:
- kind: other
ref: "docs/security/KEY-02-FLEET-ROTATION.md ## C-3 — **FAILED with finding**. Three distinct live nodes share all three SSH host keys; two also share their TLS private key. Gathered read-only via ssh-keyscan + anonymous TLS handshake; distinctness of the hosts confirmed via tailscale ping endpoints."
status: fail
- kind: other
ref: "Same-ISO provenance for those three nodes is UNVERIFIED — not required for the FAIL, but needed to bound how many other downloads carry the same keys"
status: blocked
human_judgment: true
- id: D6
description: "The OTA runtime-asset promotion actually delivers both the script and the unit to a fleet node"
requirement: KEY-02
verification:
- kind: other
ref: "replace_dir_from_runtime uses `cp -a src/. tmp` then `cp -a tmp/. dest` — recursive, so scripts/security/ rides along; the chmod sweep is `find dest -type f -name '*.sh' -exec chmod 755` with no -maxdepth, so the script lands executable. Read, not assumed."
status: pass
- kind: other
ref: "The unit had to be added to create-release-manifest.sh as well — bootstrap would have found nothing and installed nothing, silently. Neither half exercised end-to-end; needs a real release build and an OTA."
status: blocked
human_judgment: true
duration: 3h
completed: 2026-08-02
status: complete
---
# Phase 10 Plan 04: Fleet host-secret detection and guarded rotation — Summary
Every deployed node can now say, from its own disk alone, whether it is running the SSH host keys
and TLS private key baked into its ISO — the ones every downloader of that ISO also holds — and
can be fixed once, by an operator, without losing remote access in the middle. Closes the
deployed half of audit finding **F-03**.
> **C-3 FAILED, and that is the most important line in this document.** Three live fleet nodes —
> `archipelago-1`, `archy-x250-beta` and `archipelago` — share all three SSH host key
> fingerprints. Two of them also share their TLS certificate, and therefore their TLS private
> key. F-03 is not theoretical on this fleet. **None was rotated**; all three are registered in
> `docs/security/KEY-02-FLEET-ROTATION.md` with the reason and the next step.
## Commits
| Commit | Task | What |
|---|---|---|
| `96dba73a` | 1 | D-06 recorded as `detect-report-then-apply`, with what the decision binds |
| `0ed9334f` | 2 | The audit script, the boot unit, the OTA wiring, the `system.stats` field, the 8-case harness |
| `373c3bb3` | 2 (deviation) | Ship the unit in the OTA runtime payload — without this the whole plan was inert on arrival |
| `a806a658` | 3 | C-3 result: FAILED, with the finding, the method, and everything it does not establish |
Nothing was pushed, tagged, built or deployed. No node was logged into, written to, or rotated.
## Task 1 — D-06
**`detect-report-then-apply`**, recorded verbatim in `docs/security/KEY-02-FLEET-ROTATION.md`
under `## D-06 rotation trigger`.
The argument that settled it is not the one the plan anticipated. Both options were weighed on
blast radius, but the decisive point is that **`auto-on-boot` cannot be dev-paired**. This
project's standing policy is that nothing reaches the fleet before it is verified on
archi-dev-box + x250-dev. A rotation that fires unattended on the first boot after an OTA has, by
the time you have watched it happen on the dev pair, already happened everywhere. There is no
observation point before the irreversible act.
Its cost is real and is written down rather than softened: any node whose operator does not act
stays exposed indefinitely. It is bounded by making the verdict *visible* — `system.stats`, so an
exposed node shows up without shell access — and by a written register of every node that
reported `shared` and was not rotated. That register now has three entries in it, added by this
plan's own verification.
## Task 2 — detection, rotation, delivery, surfacing
### How a node decides
Four signals in a **fixed precedence**, which matters more than the signals do:
1. **Missing material can never be shared material.** Checked first. On a 10-03-or-later node
(`/opt/archipelago/rootfs-identity-stripped` present) an absent host key means generation never
succeeded — `fail-closed-missing`, a materially different verdict, and rotation is not the
remedy. Without the provenance marker the material is still absent, and the evidence says so
rather than guessing.
2. **The fail-open fingerprint outranks timestamps**, because it is direct evidence rather than
inference: `.secrets-regenerated` present plus a `WARNING:` line in
`/var/log/archipelago-first-boot-secrets.log`. The two literal strings the pre-10-03 script
emitted (`WARNING: TLS regeneration failed, keeping baked key` and
`WARNING: ssh-keygen -A failed, keeping baked host keys`) also say *which class* survived, so
the rotation narrows to it. An unrecognised `WARNING:` widens to both rather than guessing.
3. **mtime against a first-boot anchor** — `.secrets-regenerated`, falling back to
`/root/.luks-archipelago.key` (written by the installer with `dd if=/dev/urandom`) and then
`/etc/machine-id`. A key more than 300s *older* than the anchor carries the image build time.
4. **The durable failure record** (`first-boot-secrets.failed`) can only ever *withhold* a
verdict, never grant one.
**`per-node` is never inferred from an absent signal.** No anchor → `unknown`. A standing failure
record → `unknown`, even when every mtime looks clean. That is T-10-37: a false `per-node` leaves
an exposed node looking clean, which is strictly worse than no verdict.
Every verdict carries the evidence strings that produced it, each naming the file it was read
from, and the provenance signal is recorded on every run regardless of verdict because it changes
what the other signals *mean*.
### Judged per key class, not per node
This turned out to matter, and the C-3 scan is what proved it — see the `archipelago` finding
below. `SSH_SHARED` and `TLS_SHARED` are tracked separately through detection and into rotation,
so a node with a unique cert and shared host keys has only its host keys rotated.
### The rotation, and why the order is specified
1. **Stage everything first.** Both the TLS pair and the full host-key set are generated into
staging before anything live is touched, and any generation failure aborts with the tree
untouched. A partial rotation is the failure mode that loses a node.
2. **Record the OLD fingerprints before the swap.** After the swap the old material is gone and
unrecoverable; an operator who loses access anyway can still identify what changed.
3. **TLS, then reload nginx.** A dead web UI is recoverable over SSH. The converse is not. Do the
recoverable one first.
4. **SSH, then `systemctl reload ssh` — never restart.** A reload re-execs the listener while
already-forked session children keep running, so the operator survives their own rotation.
Host keys are replaced by `mv` **onto** the existing path rather than `rm` then `mv`: the
rm-then-mv shape opens a window in which the node has zero host keys on disk, and sshd
restarting into that window is unrecoverable remotely. Stale key types the new set does not
include are removed only after every staged key has landed.
5. **New fingerprints to the record, to stdout and to `/dev/console`** (guarded), then the detect
pass re-runs so the verdict file reflects the post-rotation state.
### Safety gates, in order of how likely each is to be the one that saves a node
- `--detect` is the default and is read-only.
- `--apply` without `--yes` writes **nothing at all**, not even its own verdict file.
- `--apply --yes` **refuses unless the verdict is `shared`.** On a `per-node` node the command is
inert even when typed deliberately and confirmed. This is the guard against running it on the
wrong node, and it is structural rather than procedural.
- The shipped unit contains no apply path at all, and says in a comment that adding one is a
decision rather than a configuration change.
- `ExecStart=-` on the unit: a failed audit must never fail a boot.
### A third key producer, declared
Producer 1 is `gen_tls()`/`gen_ssh()` in the ISO builder; producer 2 is `TlsMaterial::regenerate`
in `handlers.rs`. Neither can do this job: producer 1 is not present on a deployed node, and
producer 2 does TLS only — nothing in the daemon has ever rotated an SSH host key. So a third
exists, and the script header names all three and pins what they must keep in common (rsa:2048,
3650 days, the same subject and SAN set, stage → parse both halves → prove they are a pair →
swap) rather than leaving that to be rediscovered. `tls_pair_matches()` is carried over verbatim
in intent from `dad40c23`.
### Delivery — `replace_dir_from_runtime` confirmed by reading, not assumed
The plan asked for this to be confirmed rather than assumed. It was:
`replace_dir_from_runtime` does `cp -a "$src/." "$tmp"` then `cp -a "$tmp/." "$dest"` — both
recursive, so `scripts/security/` rides along with the rest of `scripts/`. The executable sweep
is `find "$dest" -type f -name '*.sh' -exec chmod 755 {} +` with no `-maxdepth`, so the script
lands executable at `/opt/archipelago/scripts/security/host-secrets-audit.sh`.
The unit is added to the existing `for unit in [...]` array and enabled with `--now`, so the
verdict lands with the OTA rather than at the next reboot.
### Surfacing
`handle_system_stats` gains a `host_secrets` object read from the on-disk verdict. Three
properties, because `system.stats` is in `CACHEABLE_METHODS` and the dashboard polls it: it never
errors (absent, truncated or unparseable all yield `{"verdict":"unknown"}` — and *every* fleet
node is in the absent case until the OTA lands, so that is the common path, not the edge one);
it is two small file reads with no process spawn; and it carries no fingerprints. A unit test
fails if a future edit forwards the whole file.
### Harness — 8 cases, all green
```
host-secrets-audit.sh: 567 lines; bash -n clean
sshd handling: reload present, restart absent
PASS: host keys newer than the anchor -> per-node, JSON written, nothing else changed
PASS: host keys 30 days older than the anchor -> shared, evidence names both key classes
PASS: marker plus a WARNING: line -> shared, with both signals in evidence, despite per-node mtimes
PASS: identity-stripped rootfs with no host keys -> fail-closed-missing, not shared
PASS: no first-boot anchor -> unknown, never per-node
PASS: --apply without --yes -> exits 0 and not one byte of the tree changes
PASS: --apply --yes -> old fingerprints recorded BEFORE the swap, keys replaced, sshd reloaded not restarted, verdict re-derived
PASS: generation failure -> aborts before any swap; live keys byte-identical, no service reloaded
──────── host-secrets-audit summary ────────
passed: 8 failed: 0
```
Case 3 is dated so that the mtime signal alone would say `per-node`; if it passes it is because
signal 2 fired. Case 7b forces the SSH generator to fail *after* TLS staging succeeded — the exact
interleaving in which a naive implementation has already swapped the TLS pair.
**Ordering is asserted by observation, not by content.** Comparing the recorded old fingerprints
against the pre-rotation keys proves the right values were written; it cannot prove they were
written *first*. The `systemctl` stub therefore records, alongside each call, whether the rotation
record existed at that moment. The first reload happens after the first swap, so `rotjson=yes` on
that line is the ordering fact.
### Negative controls — each reddens exactly one case
Each defect was reintroduced, the suite run, and the defect reverted.
```
Control A — the dry run writes its own verdict file ("one harmless file"):
FAIL: --apply without --yes -> STATE-DIR-CHANGED
passed: 7 failed: 1
Control B — old fingerprints recorded after the swap instead of before:
FAIL: --apply --yes -> OLD-FINGERPRINTS-NOT-RECORDED-BEFORE-THE-SWAP[reload nginx rotjson=no]
passed: 7 failed: 1
Control C — a failed SSH generation tolerated instead of aborting:
FAIL: generation failure -> exit-zero-on-aborted-rotation
LIVE-MATERIAL-CHANGED-ON-AN-ABORTED-ROTATION
rotation-record-written-for-a-rotation-that-never-happened
no-loud-abort-on-stderr reloaded-a-service-during-an-aborted-rotation
passed: 7 failed: 1
Control D — per-node claimed with no anchor at all:
FAIL: no first-boot anchor -> verdict=per-node CLAIMED-PER-NODE-WITHOUT-EVIDENCE
passed: 7 failed: 1
```
Control B is the one worth noting: it reddens *only* because of the ordering observation. Every
content-based assertion in case 7a still passes against that defect, because the right
fingerprints do end up in the file — just too late to be of any use to someone who has lost
access.
Control C also exposed a bug in my own harness (below).
### Rust
```
running 4 tests
test ...host_secrets_tests::verdict_is_unknown_when_the_audit_file_is_absent ... ok
test ...host_secrets_tests::rotated_at_is_surfaced_only_when_a_rotation_was_recorded ... ok
test ...host_secrets_tests::verdict_is_unknown_when_the_audit_file_is_unparseable ... ok
test ...host_secrets_tests::recorded_verdict_and_evidence_are_surfaced ... ok
test result: ok. 4 passed; 0 failed
```
`CARGO_INCREMENTAL=0 cargo build -p archipelago` succeeds. `cargo clippy -p archipelago` produces
**zero** diagnostics for `bootstrap.rs` and `system/handlers.rs`. Three warnings exist elsewhere
in the crate (`federation/handlers.rs` unused import, `mesh/flash.rs` unused assignment,
`package/dependencies.rs` dead const) — all pre-existing, all in other agents' files, none
touched.
`shellcheck` is **not installed** on this machine, so `shellcheck -S error` was not run. Recorded
rather than skipped silently. `bash -n` is clean on both new shell files.
### Real run on this node
```
$ sudo scripts/security/host-secrets-audit.sh --detect --json
host-secrets: per-node — this node's SSH host keys and TLS key were generated here.
{
"verdict": "per-node",
"checked_at": "2026-08-02T18:57:08Z",
"evidence": ["provenance: /opt/archipelago/rootfs-identity-stripped absent — this rootfs
predates the 10-03 identity strip, so baked material is possible",
"anchor: /etc/machine-id (machine-id, populated on this node's first boot), mtime
2026-04-09T18:25:45Z",
"per-node: every SSH host key and the TLS key is newer than the anchor, so all of it was
generated on this node"],
...
}
$ ls -l /var/lib/archipelago/host-secrets-audit.json
-rw-r--r-- 1 root root 959 ...
```
archi-dev-box was installed from Debian directly, not flashed from the ISO, so it has no
`.secrets-regenerated` marker and no first-boot log — it exercises the third fallback anchor. Its
three fingerprints match an **independent** `ssh-keyscan` of the same host exactly, which is the
only cheap cross-check available that the script's fingerprint extraction is correct against real
tools.
`--apply` was never run outside a temp root, on this or any other machine.
## Task 3 — C-3: **FAILED, with finding**
### What was found
Three distinct live fleet nodes present byte-identical ECDSA, ED25519 **and** RSA host key
fingerprints. Two of them also present the same TLS certificate, so they share the TLS private
key.
| Node | SSH host keys | TLS cert | Cert CN |
|---|---|---|---|
| `archipelago-1` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archy-x250-beta` | **same three** | **same cert** | `archipelago` |
| `archipelago` | **same three** | `7C:6B:CD:98…` | `austin-sapien` |
`archipelago-5`, `archi-dev-box`, `archy-dev-pa`, `framework-pt` and `shorty-s` (`.228`) are each
distinct from every other node and from each other. `archy-x250-dev`, `archy-x250-pa`,
`archy-x250-r2` and `quantumterminal` were unreachable and are UNVERIFIED.
### Method, and why it is not the checklist's method
Gathered **remotely and read-only**: `ssh-keyscan -T 6 <node> | ssh-keygen -lf -`, and an
anonymous TLS handshake for the certificate. No node was logged into, nothing was written, nothing
was rotated.
This is a weaker instrument than C-3's on-node commands — it cannot read `.secrets-regenerated`,
the first-boot log, or the ISO provenance. It was chosen because it needs no access and therefore
covers the whole reachable fleet rather than two nodes, and because it is sufficient for the FAIL
condition, which is *any fingerprint appearing on two nodes*.
### Ruling out the boring explanation
Identical host keys are also what you would see if one machine were registered on the tailnet
three times. Ruled out: all three answered a live TCP connection on port 22 within the same
minute (one `tailscaled` serves one identity, so three simultaneously-live addresses are three
hosts), they are owned by different tailnet accounts, and `tailscale ping` resolves them to
different physical endpoints — `archy-x250-beta` answers from a different continent than the
other two, which answer from the same NAT on different source ports.
### The finding inside the finding
`archipelago` has a **unique TLS cert and shared SSH host keys**. Its cert CN is `austin-sapien`,
not the image default — the signature of a node renamed through `server.set-name`, which re-mints
the certificate via `regenerate_tls_cert()` so the SAN matches, and touches nothing else.
**TLS uniqueness is therefore not evidence that a node's key material is per-node.** Any renamed
node gets a unique certificate for free while its SSH host keys stay exactly as the image shipped
them. Had C-3 been checked on certificates alone, this node would have looked clean. This is the
concrete justification for judging and reporting the two key classes separately rather than
issuing one node-level verdict — a design choice made before the scan, and vindicated by it.
### Deliberately not rotated
All three are registered in `docs/security/KEY-02-FLEET-ROTATION.md` under
"Nodes with a `shared` verdict, deliberately not rotated", with the reason and the next step.
A checkpoint that remediates is a checkpoint that takes a node offline; `archy-x250-beta` in
particular is reached over a DERP relay from another continent and is the least recoverable node
in the set.
## Deviations from Plan
### 1. [Rule 3 — Blocking] The unit could never have reached a node (`373c3bb3`)
- **Found during:** Task 2, tracing the delivery path end to end rather than trusting the plan's
key_link.
- **Issue:** `bootstrap.rs` installs units from `image-recipe/configs/` **inside the OTA runtime
payload**, but `scripts/create-release-manifest.sh` copies only `archipelago-doctor.service`
and `.timer` into that directory. `archipelago-host-secrets-audit.service` would never have
existed on any node — `src.exists()` false, install skipped, **no error and no log line**. The
entire deployed-node half of this plan would have shipped inert, and nothing would have said so.
- **Fix:** added the unit to that loop. The redundant
`if [ -f doctor.service ] || [ -f doctor.timer ]` wrapper was removed at the same time — the
per-unit `-f` test inside the loop already does that job, and the wrapper would have skipped the
whole block on a tree carrying the new unit but not the doctor ones. A `KEEP IN SYNC` comment
now names the array in `bootstrap.rs`, since two enumerations of one list in two languages in
two files is what caused this.
- **Scope:** `scripts/create-release-manifest.sh` is **outside this plan's `files_modified`**.
Taken because the alternative was to ship a deliverable that cannot reach its target and file
the gap as a follow-up. Staged by path; `git status --porcelain` confirmed no other agent had
uncommitted work in that file.
### 2. [Rule 1 — Bug in my own harness] `set -o pipefail` swallowed the summary
- **Found during:** negative control C.
- **Issue:** the failure-reporting path does `diff <(…) <(…) | head -10`. `diff` exits 1 when it
finds differences, and under `set -o pipefail` that aborted the whole harness — so a case that
failed *by changing the tree* killed the run before the summary line and before the remaining
cases. Control A did not expose it, because its failure was a state-dir comparison rather than
a tree diff, so `diff` exited 0.
- **Why it matters more than it looks:** the suppressed case is the one that detects a live
rotation having modified files it should not have. A harness that dies silently on its most
serious failure mode is worse than one that reports it noisily.
- **Fix:** `|| true` on both reporting pipelines, with a comment naming the cause. Control C was
re-run afterwards and the harness now exits 1 with the summary intact.
### 3. [Rule 2 — Correctness] `--apply` writes nothing at all, not just "nothing live"
- **Issue:** the natural implementation runs the detect pass and writes the verdict file before
branching on mode, so `--apply` without `--yes` rewrites one file. Defensible, and it makes
"touches nothing" a claim with a footnote.
- **Fix:** the write moved inside the `--detect` branch. `--apply` is now read-only in every path
that does not reach a real rotation, and case 6 became an exact whole-tree comparison rather
than one with a carve-out. Control A pins it.
### 4. [Rule 2 — Access preservation] `mv` onto the path instead of `rm` then `mv`
- **Issue:** the plan says "never delete a key without a successfully staged replacement in
hand", which the ISO builder's `gen_ssh` satisfies with `rm -f` then `mv`. On a deployed node
that still opens a window — small, but real — in which `/etc/ssh` holds zero host keys.
- **Fix:** each staged key is `mv`'d **onto** its live path (a `rename(2)`, so atomic per key, and
the directory is never empty), and only afterwards are key types the new set does not include
removed — because leaving a stale `ssh_host_dsa_key` would leave shared material behind, which
is the entire point of rotating.
## Known Stubs
None. No placeholder values, no TODOs, no unwired code paths. Every path in the script is reached
by at least one harness case.
## Threat Flags
None. No new network endpoint, no new auth path, no schema change at a trust boundary. The script
performs no network I/O and takes no input from the network; delivery reuses the existing,
already-trusted `run_runtime_assets` path and adds no new trust source (T-10-34). No package was
installed and no crate was added (T-10-SC: accept).
`system.stats` gains a field on an already-authenticated method (T-10-35: accept) and deliberately
carries no fingerprints.
## UNVERIFIED — exact evidence needed
Nothing below was observed. None of it is claimed as verified anywhere in this plan's output.
| # | Item | Evidence needed |
|---|---|---|
| 1 | **A rotation preserves the operator's own SSH session.** The single most important behavioural claim in the plan, and it is proven by design only. | On ONE disposable node, from a session you are willing to lose: `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes`; then, WITHOUT closing it, `echo still-here`; then a SECOND connection showing the expected host-key mismatch; then `cat /var/lib/archipelago/host-key-rotation.json` showing both old and new. The harness proves ordering and the abort path; it cannot prove `systemctl reload ssh` keeps a forked child alive. |
| 2 | **`host_secrets` reaches `system.stats` on a real node.** | A build carrying this plan deployed to the dev pair (archi-dev-box + x250-dev), then a `system.stats` call, then the same call after a rotation to confirm `verdict` flips to `per-node` and `rotated_at` appears. Proven against the file contract in unit tests only. |
| 3 | **The OTA actually delivers script and unit.** | A real `scripts/create-release-manifest.sh` run, then `tar -tf` the frontend tarball for `archipelago-runtime/scripts/security/host-secrets-audit.sh` and `archipelago-runtime/image-recipe/configs/archipelago-host-secrets-audit.service`; then on a node after the OTA: `ls -l /opt/archipelago/scripts/security/host-secrets-audit.sh` (expect mode 755) and `systemctl status archipelago-host-secrets-audit.service`. |
| 4 | **The audit script's own verdict on the three shared-key nodes.** Predicted `shared`; predicted is not observed. | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on `archipelago-1`, `archy-x250-beta` and `archipelago`, plus `cat /var/lib/archipelago/host-secrets-audit.json`. Needs the OTA, or the script hand-staged. |
| 5 | **Same-ISO provenance for those three nodes.** Not needed for the C-3 FAIL, but it bounds how many other downloads carry the same keys. | On-node: `ls -l /opt/archipelago/rootfs-identity-stripped`, `cat /var/lib/archipelago/.secrets-regenerated`, `grep -i warning /var/log/archipelago-first-boot-secrets.log`, plus whatever build id the installer recorded. |
| 6 | **The four unreachable nodes** (`archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`, `quantumterminal`). | Re-run the C-3 scan when they are online. `archy-x250-dev` is half the dev pair and has been offline 2 days. |
| 7 | **The `/dev/console` leg of the rotation shout.** | A real node or a VM console. Cannot be exercised in a temp root — the same limitation 10-03 recorded for its failure shout. |
| 8 | **`shellcheck -S error` on both new shell files.** | `shellcheck` is not installed on this machine. Install it and run it. |
| 9 | **`systemctl enable --now` behaviour of the new unit.** | A node. `systemctl is-enabled archipelago-host-secrets-audit.service` after an OTA. |
Items 1, 2 and 4 are also recorded in `.planning/WINDOWS.md` (entries 11–13) so they remain
visible at ship time.
## Self-Check
- `scripts/security/host-secrets-audit.sh` — FOUND, mode 755, 567 lines, `bash -n` clean,
contains `HOST_SECRETS_ROOT`
- `image-recipe/configs/archipelago-host-secrets-audit.service` — FOUND
- `tests/first-boot-secrets/rotation-tests.sh` — FOUND, mode 755, 458 lines, exits 0 with 8 PASS
- `docs/security/KEY-02-FLEET-ROTATION.md` — FOUND, contains `## D-06 rotation trigger` and
`## C-3 — per-node host key and TLS uniqueness`
- `grep -c 'archipelago-host-secrets-audit' core/archipelago/src/bootstrap.rs` → **7** (≥1 required)
- `grep -c 'host_secrets' core/archipelago/src/api/rpc/system/handlers.rs` → **11**
- `grep -c 'host-secrets-audit.json'` → 1 in `handlers.rs` (via `HOST_SECRETS_AUDIT_FILE`), 1 in
the script — the key_link holds on both ends
- `grep -n 'systemctl reload ssh'` → line 415; `grep -c 'systemctl restart ssh'` → **0**
- 10-03's harness re-run and still **9/9 green**; `image-recipe/_archived/build-auto-installer-iso.sh`
was not modified by this plan
- Commits `96dba73a`, `0ed9334f`, `373c3bb3`, `a806a658` — all FOUND
- `git diff` on the two shared Rust files inspected hunk by hunk before staging: additions only,
all within `host_secrets` / the audit unit. No file belonging to plans 10-02, 10-06 or 01-18
(`credentials/store.rs`, `device_tokens.rs`, `main.rs`, `seed.rs`, `session.rs`,
`storage_crypto.rs`, `entropy.rs`) was staged in any commit
- `.planning/STATE.md` and `.planning/ROADMAP.md` deliberately **not** updated — both carry other
agents' uncommitted work in this shared tree, and the orchestrator owns them for this wave
- Nothing pushed, per the execution brief
## Self-Check: PASSED