archipelagoandClaude Opus 5
2efab5f219
Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.
Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.
Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.
Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.
Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.
Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
marker set, failed units restarted. Control: dead-end on a node that already
failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
-> only case 6 red, naming the line.
Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
209 lines
9.1 KiB
Markdown
209 lines
9.1 KiB
Markdown
# KEY-02 — build-host evidence for the rootfs identity strip
|
|
|
|
**Audit item:** C-4 of `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` (§868), which
|
|
belongs to finding **F-03** (fail-open, never-retried first-boot secret regeneration over
|
|
a fleet-shared rootfs).
|
|
|
|
**Status: ⛔ UNVERIFIED — awaiting a run on a real ISO build host.**
|
|
|
|
The code change is committed and unit-tested; the tar listing that proves its effect on a
|
|
real build has not been produced yet, because it requires a build host with podman/docker
|
|
and enough disk for a full rootfs rebuild. Do not read anything below the "Result" heading
|
|
as a passing check until it is filled in.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Builder commit (Task 1) | `21043096` — fail-closed first-boot regeneration |
|
|
| Builder commit (Task 2) | `408b328c` — rootfs identity strip |
|
|
| Builder commit (follow-up) | single-producer unification, build-time generator assertion, self-heal timer |
|
|
| Builder file | `image-recipe/_archived/build-auto-installer-iso.sh` (LIVE; `image-recipe/build-debian-iso.sh` execs it) |
|
|
| Build host | _to be recorded_ |
|
|
| Date run | _to be recorded_ |
|
|
| RECIPE_HASH observed | _to be recorded — read it from the stamp file, see the caveat below_ |
|
|
|
|
---
|
|
|
|
## The expectation is deliberately INVERTED relative to the audit
|
|
|
|
This is the single most important thing to understand when comparing this document with the
|
|
audit, and the reason it is stated before the commands rather than after.
|
|
|
|
The audit's C-4 entry says:
|
|
|
|
> **Expected:** SSH host keys and the TLS key **present** (they are baked — see
|
|
> `build-auto-installer-iso.sh:345`, `:463-469`), `random-seed` **absent**, `machine-id`
|
|
> absent or zero-length. Anything else changes F-03's severity.
|
|
|
|
That expectation described the **broken** state the audit found, and recording it was how the
|
|
audit measured the size of F-03. Phase 10 plan 10-03 Task 2 removed that material. So:
|
|
|
|
**After this change, the audit's stated expectation is the FAILURE condition.** If SSH host
|
|
keys or the TLS private key still appear in the tar, the strip layer did not run — most
|
|
likely because a cached `archipelago-rootfs.tar` was reused. That is not a regression in the
|
|
check; it is the check working.
|
|
|
|
The two negative findings the audit recorded are unchanged and must still hold:
|
|
`var/lib/systemd/random-seed` absent, `etc/machine-id` absent or zero-length.
|
|
|
|
---
|
|
|
|
## Commands to run
|
|
|
|
Run all of these **on the build host**, from the repo root, on a checkout that contains
|
|
commits `21043096` and `408b328c`.
|
|
|
|
### 1. Force a full rebuild
|
|
|
|
The strip layer lives inside the `RECIPE_HASH` region (between the `# STEP 1: Build complete
|
|
root filesystem` and `# STEP 2: Build minimal installer` markers), so the hash changes and the
|
|
cached tar is invalidated automatically. `--rebuild` is passed anyway so that a stale tar
|
|
cannot mask the result for any reason:
|
|
|
|
```bash
|
|
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild
|
|
```
|
|
|
|
`UNBUNDLED=1` is mandatory per `CLAUDE.md` and project memory — the default env silently
|
|
builds the wrong full-bundle variant.
|
|
|
|
### 2. List the identity artefacts in the shipped tar
|
|
|
|
`WORK_DIR` is `image-recipe/build/auto-installer`, so:
|
|
|
|
```bash
|
|
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
|
|
| grep -E 'etc/ssh/ssh_host|etc/machine-id|var/lib/systemd/random-seed|archipelago/ssl/archipelago'
|
|
```
|
|
|
|
### 3. Expected result after this plan
|
|
|
|
- **no** `etc/ssh/ssh_host_*` entries at all
|
|
- **no** `etc/archipelago/ssl/archipelago.key` and **no** `archipelago.crt`
|
|
(the `etc/archipelago/ssl/` **directory** must still be present — the first-boot staging
|
|
swap needs somewhere to land)
|
|
- **no** `var/lib/systemd/random-seed`
|
|
- `etc/machine-id` present with size **0**, or absent. Either satisfies "not shared"; record
|
|
which one was actually observed rather than generalising.
|
|
|
|
Note on the TLS keypair specifically: it is now absent for two independent reasons, not one.
|
|
The Dockerfile no longer generates it at all (that layer was removed so there is a single
|
|
producer), *and* the strip layer still deletes it as belt-and-braces in case a future layer
|
|
starts baking one. Seeing it present therefore means both defences were bypassed.
|
|
|
|
### 4. Confirm the provenance file rode along
|
|
|
|
```bash
|
|
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs-identity-stripped
|
|
```
|
|
|
|
Expected: one entry, `opt/archipelago/rootfs-identity-stripped`. Its absence means the strip
|
|
layer did not execute and the whole check is void.
|
|
|
|
### 5. Confirm the regeneration path and its self-heal timer are still shipped
|
|
|
|
This is the brick check, and it is not optional. A stripped rootfs whose first-boot
|
|
generation script failed to ship would leave every flashed node with no SSH host key and
|
|
nothing to create one. The timer is part of the same check: without it, a node whose
|
|
generators fail every in-boot retry has no unattended way back.
|
|
|
|
```bash
|
|
ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \
|
|
image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \
|
|
image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer
|
|
```
|
|
|
|
Expected: all three present, `first-boot-secrets.sh` executable.
|
|
|
|
### 5b. Confirm the build-time generator assertion actually ran
|
|
|
|
The rootfs build fails outright if `openssl` or `ssh-keygen` is missing or non-executable,
|
|
because that is the one way first-boot generation can fail deterministically — retries and
|
|
reboots would never fix it, so it must never reach a node. A successful build therefore
|
|
already proves the generators are present, and the build log says so:
|
|
|
|
```bash
|
|
grep 'first-boot secret generators present' <build log>
|
|
```
|
|
|
|
If you did not capture the log, assert it against the tar instead:
|
|
|
|
```bash
|
|
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
|
|
| grep -E 'usr/bin/(openssl|ssh-keygen)$'
|
|
```
|
|
|
|
Expected: both present and mode `-rwxr-xr-x`.
|
|
|
|
### 6. Record the RECIPE_HASH the builder actually used
|
|
|
|
```bash
|
|
cat image-recipe/build/auto-installer/archipelago-rootfs.recipe.sha256
|
|
```
|
|
|
|
**Caveat — do not compute this hash from the repo file.** `image-recipe/build-debian-iso.sh`
|
|
copies the archived builder to a temp path and rewrites its relative paths before exec'ing it,
|
|
and `RECIPE_HASH` hashes `"$0"` — the rewritten copy. The hashed region contains 35 such
|
|
rewritten path expressions, and `SCRIPT_DIR` is substituted with an absolute path, so the hash
|
|
is specific to the build host and checkout location. For reference, hashing the region of the
|
|
committed repo file directly gives `d2dc4df5427fe73d48227aab08cdf6debfe8dd554e6b18e3718f8d37ea9d675c`,
|
|
which is **expected to differ** from the stamp above.
|
|
|
|
---
|
|
|
|
## Result
|
|
|
|
_Paste the raw output of steps 2, 4, 5 and 6 here, then set the status at the top of this
|
|
document to VERIFIED with the date and build-host label._
|
|
|
|
```text
|
|
(pending — not yet run on a build host)
|
|
```
|
|
|
|
**Verdict:** _pending_
|
|
|
|
---
|
|
|
|
## What this does and does not prove
|
|
|
|
**Proves (once run):** the rootfs tar extracted verbatim onto every disk flashed from the ISO
|
|
carries no SSH host key, no TLS private key and no populated machine-id — so a first-boot
|
|
regeneration failure degrades to "no key, the service refuses to start" rather than
|
|
"fleet-shared key, silently", which is the substance of F-03.
|
|
|
|
**Does not prove:** that two nodes flashed from the same ISO actually end up with different
|
|
keys. That is audit item **C-3** (§779) and needs two physical machines; it remains
|
|
separately UNVERIFIED. C-4 is a build-host check only.
|
|
|
|
### Guidance for C-3: SSH and TLS are now equally sharp signals
|
|
|
|
An earlier revision of this document said SSH host keys were the sharper divergence signal for
|
|
C-3, because the installer had a per-install TLS fallback that would produce a differing cert
|
|
even if first-boot generation had failed. **That asymmetry no longer exists.**
|
|
|
|
There is now exactly one producer of each secret — `gen_tls()` and `gen_ssh()` inside
|
|
`first-boot-secrets.sh` — and no other code in the ISO build creates either. The Dockerfile no
|
|
longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for
|
|
C-3, treat both the same way:
|
|
|
|
```bash
|
|
# on each node
|
|
ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf -
|
|
openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256
|
|
```
|
|
|
|
**Pass:** both fingerprints differ between the two nodes. **Fail:** either matches — a matching
|
|
TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have
|
|
been explained away by the fallback.
|
|
|
|
Also check, on each node, that the run actually succeeded rather than merely being quiet:
|
|
|
|
```bash
|
|
ls -l /var/lib/archipelago/.secrets-regenerated # present on a healthy node
|
|
cat /var/lib/archipelago/first-boot-secrets.failed 2>&1 # absent on a healthy node
|
|
systemctl status archipelago-first-boot-secrets.timer # enabled; the self-heal path
|
|
```
|
|
|
|
The audit's original C-3 fail condition — a `WARNING:` line in the log alongside an existing
|
|
marker — can no longer occur by construction: the marker is only written when both generators
|
|
succeeded. If you ever see that combination, the fix has been reverted.
|