archy/docs/security/KEY-02-ROOTFS-EVIDENCE.md

# KEY-02 — build-host evidence for the rootfs identity strip

**Audit item:** C-4 of `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` (§868), which
belongs to finding **F-03** (fail-open, never-retried first-boot secret regeneration over
a fleet-shared rootfs).

**Status: ⛔ UNVERIFIED — awaiting a run on a real ISO build host.**

The code change is committed and unit-tested; the tar listing that proves its effect on a
real build has not been produced yet, because it requires a build host with podman/docker
and enough disk for a full rootfs rebuild. Do not read anything below the "Result" heading
as a passing check until it is filled in.

| Field | Value |
|---|---|
| Builder commit (Task 1) | `21043096` — fail-closed first-boot regeneration |
| Builder commit (Task 2) | `408b328c` — rootfs identity strip |
| Builder commit (follow-up) | single-producer unification, build-time generator assertion, self-heal timer |
| Builder file | `image-recipe/_archived/build-auto-installer-iso.sh` (LIVE; `image-recipe/build-debian-iso.sh` execs it) |
| Build host | _to be recorded_ |
| Date run | _to be recorded_ |
| RECIPE_HASH observed | _to be recorded — read it from the stamp file, see the caveat below_ |

---

## The expectation is deliberately INVERTED relative to the audit

This is the single most important thing to understand when comparing this document with the
audit, and the reason it is stated before the commands rather than after.

The audit's C-4 entry says:

> **Expected:** SSH host keys and the TLS key **present** (they are baked — see
> `build-auto-installer-iso.sh:345`, `:463-469`), `random-seed` **absent**, `machine-id`
> absent or zero-length. Anything else changes F-03's severity.

That expectation described the **broken** state the audit found, and recording it was how the
audit measured the size of F-03. Phase 10 plan 10-03 Task 2 removed that material. So:

**After this change, the audit's stated expectation is the FAILURE condition.** If SSH host
keys or the TLS private key still appear in the tar, the strip layer did not run — most
likely because a cached `archipelago-rootfs.tar` was reused. That is not a regression in the
check; it is the check working.

The two negative findings the audit recorded are unchanged and must still hold:
`var/lib/systemd/random-seed` absent, `etc/machine-id` absent or zero-length.

---

## Commands to run

Run all of these **on the build host**, from the repo root, on a checkout that contains
commits `21043096` and `408b328c`.

### 1. Force a full rebuild

The strip layer lives inside the `RECIPE_HASH` region (between the `# STEP 1: Build complete
root filesystem` and `# STEP 2: Build minimal installer` markers), so the hash changes and the
cached tar is invalidated automatically. `--rebuild` is passed anyway so that a stale tar
cannot mask the result for any reason:

```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild
```

`UNBUNDLED=1` is mandatory per `CLAUDE.md` and project memory — the default env silently
builds the wrong full-bundle variant.

### 2. List the identity artefacts in the shipped tar

`WORK_DIR` is `image-recipe/build/auto-installer`, so:

```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
  | grep -E 'etc/ssh/ssh_host|etc/machine-id|var/lib/systemd/random-seed|archipelago/ssl/archipelago'
```

### 3. Expected result after this plan

- **no** `etc/ssh/ssh_host_*` entries at all
- **no** `etc/archipelago/ssl/archipelago.key` and **no** `archipelago.crt`
  (the `etc/archipelago/ssl/` **directory** must still be present — the first-boot staging
  swap needs somewhere to land)
- **no** `var/lib/systemd/random-seed`
- `etc/machine-id` present with size **0**, or absent. Either satisfies "not shared"; record
  which one was actually observed rather than generalising.

Note on the TLS keypair specifically: it is now absent for two independent reasons, not one.
The Dockerfile no longer generates it at all (that layer was removed so there is a single
producer), *and* the strip layer still deletes it as belt-and-braces in case a future layer
starts baking one. Seeing it present therefore means both defences were bypassed.

### 4. Confirm the provenance file rode along

```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs-identity-stripped
```

Expected: one entry, `opt/archipelago/rootfs-identity-stripped`. Its absence means the strip
layer did not execute and the whole check is void.

### 5. Confirm the regeneration path and its self-heal timer are still shipped

This is the brick check, and it is not optional. A stripped rootfs whose first-boot
generation script failed to ship would leave every flashed node with no SSH host key and
nothing to create one. The timer is part of the same check: without it, a node whose
generators fail every in-boot retry has no unattended way back.

```bash
ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \
      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \
      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer
```

Expected: all three present, `first-boot-secrets.sh` executable.

### 5b. Confirm the build-time generator assertion actually ran

The rootfs build fails outright if `openssl` or `ssh-keygen` is missing or non-executable,
because that is the one way first-boot generation can fail deterministically — retries and
reboots would never fix it, so it must never reach a node. A successful build therefore
already proves the generators are present, and the build log says so:

```bash
grep 'first-boot secret generators present' <build log>
```

If you did not capture the log, assert it against the tar instead:

```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
  | grep -E 'usr/bin/(openssl|ssh-keygen)$'
```

Expected: both present and mode `-rwxr-xr-x`.

### 6. Record the RECIPE_HASH the builder actually used

```bash
cat image-recipe/build/auto-installer/archipelago-rootfs.recipe.sha256
```

**Caveat — do not compute this hash from the repo file.** `image-recipe/build-debian-iso.sh`
copies the archived builder to a temp path and rewrites its relative paths before exec'ing it,
and `RECIPE_HASH` hashes `"$0"` — the rewritten copy. The hashed region contains 35 such
rewritten path expressions, and `SCRIPT_DIR` is substituted with an absolute path, so the hash
is specific to the build host and checkout location. For reference, hashing the region of the
committed repo file directly gives `d2dc4df5427fe73d48227aab08cdf6debfe8dd554e6b18e3718f8d37ea9d675c`,
which is **expected to differ** from the stamp above.

---

## Result

_Paste the raw output of steps 2, 4, 5 and 6 here, then set the status at the top of this
document to VERIFIED with the date and build-host label._

```text
(pending — not yet run on a build host)
```

**Verdict:** _pending_

---

## What this does and does not prove

**Proves (once run):** the rootfs tar extracted verbatim onto every disk flashed from the ISO
carries no SSH host key, no TLS private key and no populated machine-id — so a first-boot
regeneration failure degrades to "no key, the service refuses to start" rather than
"fleet-shared key, silently", which is the substance of F-03.

**Does not prove:** that two nodes flashed from the same ISO actually end up with different
keys. That is audit item **C-3** (§779) and needs two physical machines; it remains
separately UNVERIFIED. C-4 is a build-host check only.

### Guidance for C-3: SSH and TLS are now equally sharp signals

An earlier revision of this document said SSH host keys were the sharper divergence signal for
C-3, because the installer had a per-install TLS fallback that would produce a differing cert
even if first-boot generation had failed. **That asymmetry no longer exists.**

There is now exactly one producer of each secret — `gen_tls()` and `gen_ssh()` inside
`first-boot-secrets.sh` — and no other code in the ISO build creates either. The Dockerfile no
longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for
C-3, treat both the same way:

```bash
# on each node
ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf -
openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256
```

**Pass:** both fingerprints differ between the two nodes. **Fail:** either matches — a matching
TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have
been explained away by the fallback.

Also check, on each node, that the run actually succeeded rather than merely being quiet:

```bash
ls -l /var/lib/archipelago/.secrets-regenerated          # present on a healthy node
cat /var/lib/archipelago/first-boot-secrets.failed 2>&1  # absent on a healthy node
systemctl status archipelago-first-boot-secrets.timer    # enabled; the self-heal path
```

The audit's original C-3 fail condition — a `WARNING:` line in the log alongside an existing
marker — can no longer occur by construction: the marker is only written when both generators
succeeded. If you ever see that combination, the fix has been reverted.