fix(10-03): unify secret generation to a single producer + self-heal (F-03)

Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.

Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
  deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
  the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.

Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.

Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.

Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.

Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
  Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
  marker set, failed units restarted. Control: dead-end on a node that already
  failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
  -> only case 6 red, naming the line.

Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/security/KEY-02-ROOTFS-EVIDENCE.md

@@ -15,6 +15,7 @@ as a passing check until it is filled in.
 |---|---|
 | Builder commit (Task 1) | `21043096` — fail-closed first-boot regeneration |
 | Builder commit (Task 2) | `408b328c` — rootfs identity strip |
+| Builder commit (follow-up) | single-producer unification, build-time generator assertion, self-heal timer |
 | Builder file | `image-recipe/_archived/build-auto-installer-iso.sh` (LIVE; `image-recipe/build-debian-iso.sh` execs it) |
 | Build host | _to be recorded_ |
 | Date run | _to be recorded_ |
@@ -84,6 +85,11 @@ tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
 - `etc/machine-id` present with size **0**, or absent. Either satisfies "not shared"; record
   which one was actually observed rather than generalising.
 
+Note on the TLS keypair specifically: it is now absent for two independent reasons, not one.
+The Dockerfile no longer generates it at all (that layer was removed so there is a single
+producer), *and* the strip layer still deletes it as belt-and-braces in case a future layer
+starts baking one. Seeing it present therefore means both defences were bypassed.
+
 ### 4. Confirm the provenance file rode along
 
 ```bash
@@ -93,18 +99,40 @@ tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs-
 Expected: one entry, `opt/archipelago/rootfs-identity-stripped`. Its absence means the strip
 layer did not execute and the whole check is void.
 
-### 5. Confirm the regeneration path is still shipped
+### 5. Confirm the regeneration path and its self-heal timer are still shipped
 
 This is the brick check, and it is not optional. A stripped rootfs whose first-boot
-regeneration script failed to ship would leave every flashed node with no SSH host key and
-nothing to create one.
+generation script failed to ship would leave every flashed node with no SSH host key and
+nothing to create one. The timer is part of the same check: without it, a node whose
+generators fail every in-boot retry has no unattended way back.
 
 ```bash
 ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \
-      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service
+      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \
+      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer
 ```
 
-Expected: both present, `first-boot-secrets.sh` executable.
+Expected: all three present, `first-boot-secrets.sh` executable.
+
+### 5b. Confirm the build-time generator assertion actually ran
+
+The rootfs build fails outright if `openssl` or `ssh-keygen` is missing or non-executable,
+because that is the one way first-boot generation can fail deterministically — retries and
+reboots would never fix it, so it must never reach a node. A successful build therefore
+already proves the generators are present, and the build log says so:
+
+```bash
+grep 'first-boot secret generators present' <build log>
+```
+
+If you did not capture the log, assert it against the tar instead:
+
+```bash
+tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
+  | grep -E 'usr/bin/(openssl|ssh-keygen)$'
+```
+
+Expected: both present and mode `-rwxr-xr-x`.
 
 ### 6. Record the RECIPE_HASH the builder actually used
 
@@ -146,9 +174,35 @@ regeneration failure degrades to "no key, the service refuses to start" rather t
 keys. That is audit item **C-3** (§779) and needs two physical machines; it remains
 separately UNVERIFIED. C-4 is a build-host check only.
 
-**Also worth recording when C-3 is run:** the installer writes a per-install TLS keypair as a
-fallback (`build-auto-installer-iso.sh`, the "Ensure SSL cert exists for nginx HTTPS" block).
-That fallback fires on every install now that the rootfs ships without a keypair. It is
-generated separately on each target machine, so it is per node and does not reopen F-03 — but
-it does mean the two nodes' *TLS* certs will differ even if first-boot regeneration failed on
-both. SSH host keys have no such fallback, so they are the sharper signal for C-3.
+### Guidance for C-3: SSH and TLS are now equally sharp signals
+
+An earlier revision of this document said SSH host keys were the sharper divergence signal for
+C-3, because the installer had a per-install TLS fallback that would produce a differing cert
+even if first-boot generation had failed. **That asymmetry no longer exists.**
+
+There is now exactly one producer of each secret — `gen_tls()` and `gen_ssh()` inside
+`first-boot-secrets.sh` — and no other code in the ISO build creates either. The Dockerfile no
+longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for
+C-3, treat both the same way:
+
+```bash
+# on each node
+ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf -
+openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256
+```
+
+**Pass:** both fingerprints differ between the two nodes. **Fail:** either matches — a matching
+TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have
+been explained away by the fallback.
+
+Also check, on each node, that the run actually succeeded rather than merely being quiet:
+
+```bash
+ls -l /var/lib/archipelago/.secrets-regenerated          # present on a healthy node
+cat /var/lib/archipelago/first-boot-secrets.failed 2>&1  # absent on a healthy node
+systemctl status archipelago-first-boot-secrets.timer    # enabled; the self-heal path
+```
+
+The audit's original C-3 fail condition — a `WARNING:` line in the log alongside an existing
+marker — can no longer occur by construction: the marker is only written when both generators
+succeeded. If you ever see that combination, the fix has been reverted.

image-recipe/_archived/build-auto-installer-iso.sh

@@ -456,17 +456,41 @@ RUN ln -sf /etc/nginx/sites-available/archipelago /etc/nginx/sites-enabled/archi
 # Install nginx snippets (PWA config, HTTPS app proxies)
 COPY snippets/ /etc/nginx/snippets/
 
-# Generate self-signed SSL certificate for HTTPS (PWA install + mic/camera
-# access both require a secure context). SAN covers the install-time default
-# hostname -- server.set-name regenerates this with the new hostname's SAN
-# if a node is renamed.
-RUN mkdir -p /etc/archipelago/ssl && \
-    openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
-    -keyout /etc/archipelago/ssl/archipelago.key \
-    -out /etc/archipelago/ssl/archipelago.crt \
-    -subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=archipelago" \
-    -addext "subjectAltName=DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" && \
-    chmod 600 /etc/archipelago/ssl/archipelago.key
+# The self-signed HTTPS keypair is NOT generated here (audit F-03).
+#
+# It used to be: this layer ran "openssl req" and baked one keypair into the
+# shared image, which meant every node flashed from one ISO — and everyone who
+# downloaded the ISO — held the same TLS private key. The strip layer at the
+# end of this Dockerfile would delete it again anyway, so generating it here
+# now only creates a SECOND piece of code that can mint a TLS key with its own
+# accounting. There is exactly one producer of this keypair, and it is
+# first-boot-secrets.sh, which retries and reports.
+#
+# The ssl directory is created so that producer's staging swap has somewhere
+# to land.
+RUN mkdir -p /etc/archipelago/ssl
+
+# Fail the BUILD if the rootfs cannot generate per-device secrets.
+#
+# The one realistic way first-boot generation fails on every retry is a missing
+# generator binary, and that failure is deterministic, not transient — retries
+# and reboots will never fix it. A node in the field must not be where we
+# discover it. openssl and openssh-server are both in the package list above
+# (and openssh-server hard-depends openssh-client, which ships ssh-keygen), so
+# today this assertion is cheap insurance rather than a fix. It earns its place
+# by turning a silent fleet-wide brick into a loud build failure the first time
+# anyone edits that package list.
+RUN set -e; \
+    for bin in /usr/bin/openssl /usr/bin/ssh-keygen; do \
+        if [ ! -x "\$bin" ]; then \
+            echo "FATAL: \$bin missing or not executable in the rootfs." >&2; \
+            echo "first-boot-secrets.sh cannot generate per-device SSH host keys" >&2; \
+            echo "or the TLS keypair without it, and that failure is permanent." >&2; \
+            echo "Restore openssl / openssh-server in the package list above." >&2; \
+            exit 1; \
+        fi; \
+    done; \
+    echo "first-boot secret generators present: openssl, ssh-keygen"
 
 # Create archipelago systemd service
 COPY archipelago.service /etc/systemd/system/archipelago.service
@@ -586,27 +610,29 @@ RUN apt-get clean && \
 # identity-shaped left in here is therefore held by every node AND by every
 # person who downloaded the ISO.
 #
-# Two things get baked without anyone asking for them:
-#   - Debian's openssh-server postinst generates /etc/ssh/ssh_host_* at package
-#     install time, i.e. inside this container build.
-#   - the "openssl req" layer above writes /etc/archipelago/ssl/archipelago.key.
 # NOTE: this heredoc is UNQUOTED, so backticks here are command substitution
 # and would run at build time. Never put backticks in these comments.
-# Plus /etc/machine-id, which systemd populates during the build and which
-# correlates every node flashed from one ISO.
 #
-# archipelago-first-boot-secrets.service recreates all of this per device on
-# first boot. The point of removing it HERE is to change what a regeneration
-# failure costs: with the material stripped, a failure degrades to "no key, the
-# service refuses to start" instead of "fleet-shared key, silently" — which is
-# the whole of F-03. That makes fail-closed structural rather than procedural.
+# The TLS keypair is no longer generated in this Dockerfile at all (see the
+# note where that layer used to be). What still gets baked without anyone
+# asking for it is Debian's openssh-server postinst, which generates
+# /etc/ssh/ssh_host_* at package install time — i.e. inside this container
+# build — plus /etc/machine-id, which systemd populates during the build and
+# which correlates every node flashed from one ISO. This layer removes both.
+#
+# archipelago-first-boot-secrets.service recreates all of it per device on
+# first boot, retrying on a timer until it succeeds. The point of removing it
+# HERE is to change what a generation failure costs: with the material
+# stripped, a failure degrades to "no key, the service refuses to start"
+# instead of "fleet-shared key, silently" — which is the whole of F-03. That
+# makes fail-closed structural rather than procedural.
 #
 # This must stay the LAST layer: anything that installs packages after it can
-# reintroduce host keys. Keep the /etc/archipelago/ssl directory itself so the
-# first-boot script's staging swap has somewhere to land. The "openssl req"
-# layer above is deliberately left in place — it keeps proving openssl is
-# present and keeps the SAN template next to the code that uses it; this layer
-# is what makes the shipped output non-shared.
+# reintroduce host keys. The rm of the TLS keypair is kept as belt-and-braces
+# even though nothing in this build creates one any more — if a future layer
+# starts baking a cert, this catches it. Keep the /etc/archipelago/ssl
+# directory itself so the first-boot script's staging swap has somewhere to
+# land.
 RUN rm -f /etc/ssh/ssh_host_* && \
     rm -f /etc/archipelago/ssl/archipelago.key /etc/archipelago/ssl/archipelago.crt && \
     mkdir -p /etc/archipelago/ssl && \
@@ -1654,6 +1680,36 @@ RemainAfterExit=yes
 WantedBy=multi-user.target
 SECRETSSERVICE
 
+# Self-heal timer. Fail-closed governs SERVING (never present a key we did not
+# generate); this timer governs RECOVERING (never dead-end a node).
+#
+# Without it, a node whose generators failed all their in-boot retries would sit
+# with no SSH host key and no TLS key until somebody walked to it with a
+# keyboard. With it, a transient cause that later clears — a full disk that gets
+# freed, a pool that eventually seeds — repairs the node unattended.
+#
+# The service's own ConditionPathExists=! is what stops this: once the marker
+# exists, every subsequent trigger is a no-op that systemd records as success,
+# so the timer costs nothing on a healthy node and needs no separate teardown.
+# This uses systemd's own facilities on purpose; a sleep loop inside the script
+# would hold a oneshot open for hours and hide the failure from systemctl.
+cat > "$WORK_DIR/archipelago-first-boot-secrets.timer" <<'SECRETSTIMER'
+[Unit]
+Description=Retry per-device secret generation until it succeeds
+Documentation=man:archipelago-first-boot-secrets.service(8)
+
+[Timer]
+# First retry shortly after boot has settled — by then the disk, the entropy
+# pool and any late-mounting filesystem have had a chance to become healthy.
+OnBootSec=5min
+OnUnitActiveSec=15min
+AccuracySec=30s
+Unit=archipelago-first-boot-secrets.service
+
+[Install]
+WantedBy=timers.target
+SECRETSTIMER
+
 cat > "$WORK_DIR/first-boot-secrets.sh" <<'SECRETSSCRIPT'
 #!/bin/bash
 # Create this device's own TLS keypair and SSH host keys on first boot.
@@ -1666,26 +1722,41 @@ cat > "$WORK_DIR/first-boot-secrets.sh" <<'SECRETSSCRIPT'
 # host keys, the TLS keypair and machine-id out of the shared image. THIS
 # SCRIPT IS THE ONLY THING THAT CREATES THEM. That is deliberate.
 #
-# The operational consequence, in plain words: if regeneration fails every
-# retry, this node has no SSH host key, so sshd will not start and the node
-# cannot be reached over SSH — recovery requires the physical console. Nothing
-# else on the install path creates host keys, so that outcome is certain.
-# (TLS is softer: the installer writes a per-node fallback keypair with the
-# generic CN=archipelago SAN, so the web UI usually still comes up. That
-# fallback is per install, never image-wide, so it does not reopen F-03.)
+# SINGLE PRODUCER. gen_tls() below is the only code anywhere in the ISO build
+# that creates /etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() is the only
+# code that creates /etc/ssh/ssh_host_*. The Dockerfile no longer bakes a
+# keypair and the installer's old "ensure SSL cert exists" fallback is gone.
+# That is the actual lesson of F-03: the bug was never "a second attempt to
+# create a key exists", it was that failure was silent and the marker lied
+# about it. A second producer is dangerous precisely because it has its own
+# accounting — its own idea of success, its own (absent) retry policy, its own
+# (absent) failure record. One producer means one place that can fail, one
+# place that retries, one place that reports.
 #
-# That cost was accepted on purpose. The behaviour it replaces was worse: log
-# a warning, set the completion marker anyway, and run forever on the SSH host
-# key and TLS private key that every downloader of the ISO also holds — which
-# is undetectable host impersonation and transparent MITM of the web UI, on a
-# node whose operator has no idea.
+# FAIL CLOSED applies to SERVING: if generation fails, no key exists, so sshd
+# and the nginx TLS listener refuse to start. They never come up on a
+# placeholder, a zero-length file, or a key from anywhere else. gen_tls swaps
+# into place only after openssl has parsed both halves back, so a truncated or
+# corrupt artefact is never what a service reads.
 #
-# So: the completion marker is written ONLY when both generators succeeded. A
-# failed boot leaves the marker absent, which leaves the unit's
-# ConditionPathExists=! satisfied, so the whole thing runs again on the next
-# boot. Each generator is retried with backoff first, so a transient first-boot
-# condition (slow entropy pool, momentarily full disk) recovers without needing
-# a reboot at all.
+# SELF-HEAL applies to RECOVERING, and it is a different thing: a failure must
+# never dead-end the node. Three attempts with backoff inside the boot, then
+# archipelago-first-boot-secrets.timer retries every 15 minutes, and every
+# subsequent boot retries too — all because the marker is never written on
+# failure. A transient cause that later clears (a full disk that gets freed, a
+# pool that eventually seeds) repairs the node with nobody at a console. On
+# success the script restarts whatever refused to start, so recovery is
+# complete rather than pending-a-reboot.
+#
+# The only failure that survives all of that is a deterministic one — a missing
+# generator binary — and the rootfs build asserts openssl and ssh-keygen are
+# present and executable, so the build fails rather than the fleet.
+#
+# What this replaces was worse in every direction: log a warning, set the
+# completion marker anyway, never retry, and run forever on the SSH host key and
+# TLS private key that every downloader of the ISO also holds — undetectable
+# host impersonation and transparent MITM of the web UI, on a node whose
+# operator has no idea.
 #
 # Testability seam: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path. It is
 # unset in production — the expansion is empty and behaviour is identical to a
@@ -1754,10 +1825,16 @@ retry() {
     return 1
 }
 
-# 1. Self-signed TLS: fresh keypair with this device's hostname in the SAN
-#    (server.set-name regenerates again if the node is renamed later).
-#    Generated to .new and swapped only on success, so the node is never left
-#    holding half a keypair.
+# THE SINGLE PRODUCER of this node's TLS keypair. No other code in the ISO
+# build creates /etc/archipelago/ssl/archipelago.{key,crt}. Do not add one; add
+# a caller of this instead. tests/first-boot-secrets/run-tests.sh case 6 fails
+# the build's test suite if a second producer appears.
+#
+# Fresh keypair with this device's hostname in the SAN (server.set-name
+# regenerates again if the node is renamed later). Generated to .new, PARSED
+# BACK, and only then swapped in — so nginx can never be handed a truncated,
+# zero-length or half-written artefact, which is the "never serve with a bogus
+# key" half of fail-closed.
 gen_tls() {
     mkdir -p "$SSL_DIR" || return 1
     rm -f "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.crt.new"
@@ -1768,7 +1845,10 @@ gen_tls() {
         -addext "subjectAltName=DNS:${NODE_NAME},DNS:${NODE_NAME}.local,DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" \
         >> "$LOG" 2>&1 \
         && [ -s "$SSL_DIR/archipelago.key.new" ] \
-        && [ -s "$SSL_DIR/archipelago.crt.new" ]; then
+        && [ -s "$SSL_DIR/archipelago.crt.new" ] \
+        && openssl pkey -noout -in "$SSL_DIR/archipelago.key.new" >> "$LOG" 2>&1 \
+        && openssl x509 -noout -in "$SSL_DIR/archipelago.crt.new" >> "$LOG" 2>&1; then
+        chmod 600 "$SSL_DIR/archipelago.key.new"
         mv "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.key" \
             && mv "$SSL_DIR/archipelago.crt.new" "$SSL_DIR/archipelago.crt" || return 1
         chmod 600 "$SSL_DIR/archipelago.key"
@@ -1778,7 +1858,10 @@ gen_tls() {
     return 1
 }
 
-# 2. SSH host keys: generate a full fresh set in staging, then swap.
+# THE SINGLE PRODUCER of this node's SSH host keys. Same rule as gen_tls: no
+# second producer anywhere, add a caller instead.
+#
+# Generates a full fresh set in staging, then swaps.
 gen_ssh() {
     local staging
     staging=$(mktemp -d) || return 1
@@ -1795,6 +1878,31 @@ gen_ssh() {
     return 1
 }
 
+# Hand the new material to whatever consumes it.
+#
+# Two different situations, and getting this wrong is what would turn the timer
+# into theatre:
+#   - First boot. We are ordered Before= these units, so they have not started
+#     yet. try-reload-or-restart is a no-op on an inactive unit, which is
+#     exactly right — they will start on their own moments later and read the
+#     keys we just wrote.
+#   - Self-heal, minutes or hours later. The unit already tried to start with no
+#     key and is sitting in `failed`. try-reload-or-restart would be a no-op
+#     there too, which would leave the node broken with valid keys on disk — the
+#     recovery would be "complete" but the service still down. So a failed unit
+#     is explicitly restarted.
+# --no-block because on first boot we are inside a unit these services are
+# ordered after; a blocking start could deadlock the boot transaction.
+refresh_consumer() {
+    local unit="$1"
+    if systemctl is-failed --quiet "$unit" 2>/dev/null; then
+        log "$unit is failed (it started without a key); restarting it"
+        systemctl --no-block restart "$unit" >> "$LOG" 2>&1 || true
+    else
+        systemctl try-reload-or-restart "$unit" >> "$LOG" 2>&1 || true
+    fi
+}
+
 log "regenerating per-device secrets"
 
 TLS_OK=0
@@ -1803,13 +1911,13 @@ SSH_OK=0
 if retry "TLS keypair regeneration" gen_tls; then
     TLS_OK=1
     log "TLS keypair regenerated (CN=${NODE_NAME})"
-    systemctl try-reload-or-restart nginx >> "$LOG" 2>&1 || true
+    refresh_consumer nginx
 fi
 
 if retry "SSH host key regeneration" gen_ssh; then
     SSH_OK=1
     log "SSH host keys regenerated"
-    systemctl try-reload-or-restart ssh >> "$LOG" 2>&1 || true
+    refresh_consumer ssh
 fi
 
 if [ "$TLS_OK" -eq 1 ] && [ "$SSH_OK" -eq 1 ]; then
@@ -1821,8 +1929,12 @@ if [ "$TLS_OK" -eq 1 ] && [ "$SSH_OK" -eq 1 ]; then
     exit 0
 fi
 
-# Fail closed. Deliberately NO marker: its absence is what lets the unit run
-# again on the next boot.
+# Fail closed on serving, self-heal on recovering.
+#
+# Deliberately NO marker. Its absence is the entire retry mechanism: it keeps
+# the unit's ConditionPathExists=! satisfied, so archipelago-first-boot-secrets
+# .timer re-runs this in 15 minutes and every subsequent boot re-runs it too.
+# This is a failure, not a dead end — do not "fix" it by writing the marker.
 WHICH=""
 [ "$TLS_OK" -eq 0 ] && WHICH="TLS"
 [ "$SSH_OK" -eq 0 ] && WHICH="${WHICH:+$WHICH and }SSH"
@@ -1831,15 +1943,16 @@ WHICH=""
     echo "failed=$WHICH"
     echo "tls_ok=$TLS_OK"
     echo "ssh_ok=$SSH_OK"
-    echo "detail=per-device secret regeneration failed after all retries; the completion marker was NOT set, so this unit runs again on the next boot"
+    echo "detail=per-device secret generation failed after all retries; the completion marker was NOT set, so archipelago-first-boot-secrets.timer retries in 15 minutes and every subsequent boot retries too"
 } > "$FAILED"
-shout "ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's $WHICH key material. Refusing to continue — the affected services will not start. Record: $FAILED  Log: $LOG"
+shout "ARCHIPELAGO: could not generate this device's $WHICH key material. The affected services will NOT start rather than run on a key we did not generate. Retrying automatically every 15 minutes and on every boot. Record: $FAILED  Log: $LOG"
 exit 1
 SECRETSSCRIPT
 
 chmod +x "$WORK_DIR/first-boot-secrets.sh"
 cp "$WORK_DIR/first-boot-secrets.sh" "$ARCH_DIR/scripts/"
 cp "$WORK_DIR/archipelago-first-boot-secrets.service" "$ARCH_DIR/scripts/"
+cp "$WORK_DIR/archipelago-first-boot-secrets.timer" "$ARCH_DIR/scripts/"
 
 # Ship the mesh-radio udev rule at the media root — the embedded installer
 # searches "$BOOT_MEDIA/99-mesh-radio.rules" first, but nothing ever staged
@@ -2764,6 +2877,11 @@ if [ -d "$BOOT_MEDIA/archipelago/container-images" ]; then
     if [ -f "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.service" ]; then
         cp "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.service" /mnt/target/etc/systemd/system/
     fi
+    # The self-heal timer. Without it a node whose generators fail every in-boot
+    # retry has no unattended way back — it would need someone at the console.
+    if [ -f "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.timer" ]; then
+        cp "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.timer" /mnt/target/etc/systemd/system/
+    fi
     if [ -f "$BOOT_MEDIA/archipelago/scripts/setup-tor.sh" ]; then
         cp "$BOOT_MEDIA/archipelago/scripts/setup-tor.sh" /mnt/target/opt/archipelago/scripts/
         chmod +x /mnt/target/opt/archipelago/scripts/setup-tor.sh
@@ -3430,28 +3548,37 @@ RemainAfterExit=yes
 WantedBy=multi-user.target
 DIAGSVC
 
-# Ensure SSL cert exists for nginx HTTPS.
+# NO install-time TLS certificate is generated here, on purpose (audit F-03,
+# phase 10 plan 10-03).
 #
-# Since the F-03 identity-strip layer this is no longer a rarely-taken safety
-# net — the rootfs ships with NO TLS keypair, so this branch fires on every
-# install. That is fine and deliberate: the installer runs separately on each
-# target machine, so the key it writes is per node, not image-wide. It uses the
-# generic CN=archipelago SAN; archipelago-first-boot-secrets.service replaces it
-# on first boot with one carrying this device's actual hostname.
+# There used to be an "ensure SSL cert exists for nginx HTTPS" block that ran
+# its own "chroot /mnt/target openssl req ..." whenever the target had no cert.
+# Before the identity strip it almost never fired; after the strip it would
+# have fired on EVERY install.
 #
-# Consequence worth knowing: if first-boot regeneration fails, the web UI still
-# has *a* per-node cert from here, whereas SSH has nothing at all (nothing
-# recreates host keys at install time) and sshd will refuse to start.
-if [ ! -f /mnt/target/etc/archipelago/ssl/archipelago.crt ]; then
-    mkdir -p /mnt/target/etc/archipelago/ssl
-    chroot /mnt/target openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
-        -keyout /etc/archipelago/ssl/archipelago.key \
-        -out /etc/archipelago/ssl/archipelago.crt \
-        -subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=archipelago" \
-        -addext "subjectAltName=DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" 2>/dev/null
-    chmod 600 /mnt/target/etc/archipelago/ssl/archipelago.key
-    echo "   Generated self-signed SSL certificate"
-fi
+# The problem with it was never that a second attempt to create a key existed.
+# It was that it was a second PRODUCER: its own openssl invocation, its own idea
+# of success, no retry policy, no failure record, no marker discipline. F-03 was
+# a silent-failure bug, and a second producer with its own accounting is exactly
+# how silent failures happen. So the fix is to unify, not to add another
+# fallback and not to leave the node with no way back.
+#
+# What replaces it: archipelago-first-boot-secrets.service is the single
+# producer of /etc/archipelago/ssl/archipelago.{key,crt}, exactly as it is the
+# single producer of /etc/ssh/ssh_host_*, and it retries with backoff in-boot,
+# on archipelago-first-boot-secrets.timer every 15 minutes, and on every boot
+# until it succeeds. A machine that is merely busy repairs itself unattended; a
+# machine that is genuinely broken says so loudly on the console and in
+# /var/lib/archipelago/first-boot-secrets.failed instead of quietly serving a
+# key nobody audited.
+#
+# The deterministic way this could never succeed — a missing openssl or
+# ssh-keygen — is caught at BUILD time by the assertion layer in STEP 1, so it
+# cannot reach a node at all.
+#
+# Do not reintroduce a producer here. If you need a key earlier, call the same
+# generator. tests/first-boot-secrets/run-tests.sh case 6 fails if a second
+# independent key-creating path appears in this file.
 
 # Enable linger for rootless podman (containers survive logout)
 mkdir -p /mnt/target/var/lib/systemd/linger
@@ -3518,6 +3645,14 @@ chroot /mnt/target systemctl enable archipelago.service 2>/dev/null || true
 chroot /mnt/target systemctl enable nginx.service 2>/dev/null || true
 chroot /mnt/target systemctl enable archipelago-load-images.service 2>/dev/null || true
 chroot /mnt/target systemctl enable archipelago-first-boot-secrets.service 2>/dev/null || true
+# Self-heal timer: if this does not get enabled, a node whose secret generation
+# fails every retry has no unattended way back. chroot systemctl enable can fail
+# silently, so fall back to writing the symlink by hand rather than trusting
+# `|| true` to have done anything.
+chroot /mnt/target systemctl enable archipelago-first-boot-secrets.timer 2>/dev/null || \
+  { mkdir -p /mnt/target/etc/systemd/system/timers.target.wants && \
+    ln -sf /etc/systemd/system/archipelago-first-boot-secrets.timer \
+           /mnt/target/etc/systemd/system/timers.target.wants/archipelago-first-boot-secrets.timer 2>/dev/null || true; }
 chroot /mnt/target systemctl enable archipelago-setup-tor.service 2>/dev/null || true
 chroot /mnt/target systemctl enable archipelago-first-boot-containers.service 2>/dev/null || true
 chroot /mnt/target systemctl enable archipelago-kiosk.service 2>/dev/null || true

tests/first-boot-secrets/run-tests.sh

@@ -62,17 +62,41 @@ fi
 # systemctl / logger calls hit these instead of the real tools. Behaviour is
 # driven by env vars the stubs read at call time.
 #
-#   STUB_OPENSSL_MODE     ok | fail
-#   STUB_SSHKEYGEN_MODE   ok | fail | fail-twice   (fail-twice uses a counter file)
-#   STUB_COUNTER_DIR      where the counter file lives
+#   STUB_OPENSSL_MODE            ok | fail        (fail affects `req` only —
+#                                                  `pkey`/`x509` validation still
+#                                                  works, so a failed generation
+#                                                  cannot be mistaken for a failed
+#                                                  validation)
+#   STUB_SSHKEYGEN_MODE          ok | fail | fail-twice   (uses a counter file)
+#   STUB_COUNTER_DIR             where the counter file lives
+#   STUB_SYSTEMCTL_FAILED_UNITS  units `systemctl is-failed` should report failed
+#   STUB_SYSTEMCTL_LOG           file the systemctl stub appends its args to
 make_stubs() {
     local dir="$1"
     mkdir -p "$dir"
 
     cat > "$dir/openssl" <<'STUB'
 #!/bin/bash
-# Stub openssl: honours -keyout/-out so the script's staging-then-swap and its
-# non-empty checks are exercised for real.
+# Stub openssl. Honours -keyout/-out so the script's staging-then-swap and its
+# non-empty checks are exercised for real, and implements the `pkey`/`x509`
+# parse-back validation the generator does before it swaps.
+sub="${1:-}"
+case "$sub" in
+    pkey|x509)
+        # Validation: succeed iff the file exists and is non-empty. This is what
+        # lets the harness prove a truncated artefact is never swapped in.
+        f=""
+        while [ $# -gt 0 ]; do
+            case "$1" in
+                -in) f="$2"; shift 2 ;;
+                *)   shift ;;
+            esac
+        done
+        [ -n "$f" ] && [ -s "$f" ] || exit 1
+        exit 0
+        ;;
+esac
+
 [ "${STUB_OPENSSL_MODE:-ok}" = "fail" ] && exit 1
 keyout="" out=""
 while [ $# -gt 0 ]; do
@@ -119,8 +143,23 @@ done
 exit 0
 STUB
 
-    # Neither of these must be allowed to touch the host during a test run.
-    printf '#!/bin/bash\nexit 0\n' > "$dir/systemctl"
+    cat > "$dir/systemctl" <<'STUB'
+#!/bin/bash
+# Stub systemctl. Records every invocation so the harness can prove the
+# self-heal path actually restarts a unit that failed for want of a key, and
+# reports is-failed honestly so first-boot and self-heal take different paths.
+[ -n "${STUB_SYSTEMCTL_LOG:-}" ] && echo "$*" >> "$STUB_SYSTEMCTL_LOG"
+if [ "${1:-}" = "is-failed" ]; then
+    unit="${!#}"
+    for u in ${STUB_SYSTEMCTL_FAILED_UNITS:-}; do
+        [ "$u" = "$unit" ] && exit 0
+    done
+    exit 1
+fi
+exit 0
+STUB
+
+    # Must not be allowed to touch the host journal during a test run.
     printf '#!/bin/bash\nexit 0\n' > "$dir/logger"
 
     chmod +x "$dir"/openssl "$dir"/ssh-keygen "$dir"/systemctl "$dir"/logger
@@ -130,23 +169,37 @@ STUBS="$WORK/stubs"
 make_stubs "$STUBS"
 
 # ── Runner ────────────────────────────────────────────────────────────────
-# Runs the script against a fresh temp root. Echoes the exit status; the
-# caller asserts on it plus the resulting filesystem state.
+# run_case <name> <openssl_mode> <sshkeygen_mode> [prestage] [reuse]
+#
+#   prestage  baked    — root already holds the shared keys, i.e. a pre-strip
+#                        rootfs. Assertions can then prove the swap replaced them.
+#             stripped — root holds no key material at all, i.e. the rootfs this
+#                        build actually ships. This is the state in which "no key
+#                        may appear from anywhere but the generator" is testable.
+#   reuse     1        — do not wipe the root or the counters; continue from the
+#                        previous run against the same node. Models a reboot or a
+#                        timer-triggered retry.
 CASE_ROOT=""
 CASE_RC=0
+CASE_SYSTEMCTL_LOG=""
 run_case() {
     local name="$1" openssl_mode="$2" sshkeygen_mode="$3"
+    local prestage="${4:-baked}" reuse="${5:-0}"
     CASE_ROOT="$WORK/root-$name"
-    rm -rf "$CASE_ROOT"
-    mkdir -p "$CASE_ROOT/var/lib/archipelago" "$CASE_ROOT/var/log" \
-             "$CASE_ROOT/etc/ssh" "$CASE_ROOT/etc/archipelago/ssl"
-    # A pre-existing baked host key + TLS key, i.e. the pre-strip rootfs state:
-    # the assertions below then also show the swap actually replaced them.
-    echo "BAKED-SHARED-HOST-KEY" > "$CASE_ROOT/etc/ssh/ssh_host_rsa_key"
-    echo "BAKED-SHARED-TLS-KEY"  > "$CASE_ROOT/etc/archipelago/ssl/archipelago.key"
+    CASE_SYSTEMCTL_LOG="$WORK/$name.systemctl"
 
-    rm -f "$WORK/counters-$name/ssh-keygen.count"
-    mkdir -p "$WORK/counters-$name"
+    if [ "$reuse" != "1" ]; then
+        rm -rf "$CASE_ROOT"
+        mkdir -p "$CASE_ROOT/var/lib/archipelago" "$CASE_ROOT/var/log" \
+                 "$CASE_ROOT/etc/ssh" "$CASE_ROOT/etc/archipelago/ssl"
+        if [ "$prestage" = "baked" ]; then
+            echo "BAKED-SHARED-HOST-KEY" > "$CASE_ROOT/etc/ssh/ssh_host_rsa_key"
+            echo "BAKED-SHARED-TLS-KEY"  > "$CASE_ROOT/etc/archipelago/ssl/archipelago.key"
+        fi
+        rm -f "$WORK/counters-$name/ssh-keygen.count"
+        mkdir -p "$WORK/counters-$name"
+        : > "$CASE_SYSTEMCTL_LOG"
+    fi
 
     set +e
     env PATH="$STUBS:$PATH" \
@@ -155,6 +208,8 @@ run_case() {
         STUB_OPENSSL_MODE="$openssl_mode" \
         STUB_SSHKEYGEN_MODE="$sshkeygen_mode" \
         STUB_COUNTER_DIR="$WORK/counters-$name" \
+        STUB_SYSTEMCTL_LOG="$CASE_SYSTEMCTL_LOG" \
+        STUB_SYSTEMCTL_FAILED_UNITS="${STUB_SYSTEMCTL_FAILED_UNITS:-}" \
         bash "$SCRIPT" > "$WORK/$name.out" 2> "$WORK/$name.err"
     CASE_RC=$?
     set -e
@@ -217,6 +272,98 @@ else
     bad "ssh-keygen fails twice then succeeds ->$c3"; fail_detail ssh-flaky
 fi
 
+# ── Case 4: TLS fails every attempt on a STRIPPED root -> no key at all ──
+# Case 2 proves the marker is not set. This proves the stronger property that
+# replaced the installer's TLS fallback: on the rootfs we actually ship, a
+# failed generation leaves NO key, from any source. If anything ever mints a
+# key outside gen_tls — an install-time fallback, a placeholder, a zero-length
+# touch to keep nginx happy — this is the case that goes red.
+run_case tls-fail-stripped fail ok stripped
+c4=""
+[ "$CASE_RC" -ne 0 ] || c4="$c4 exit-zero-on-failure"
+[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] && c4="$c4 MARKER-SET-ON-FAILURE"
+[ -e "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" ] && c4="$c4 TLS-KEY-EXISTS-AFTER-FAILURE"
+[ -e "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] && c4="$c4 TLS-CRT-EXISTS-AFTER-FAILURE"
+[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] || c4="$c4 no-failure-record"
+grep -q 'failed=.*TLS' "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" 2>/dev/null \
+    || c4="$c4 failure-record-does-not-name-TLS"
+ls "$CASE_ROOT"/etc/archipelago/ssl/*.new >/dev/null 2>&1 && c4="$c4 dotnew-leftover"
+if [ -z "$c4" ]; then
+    ok "TLS fails every attempt on a stripped root -> NO key, NO marker, non-zero exit, record names TLS"
+else
+    bad "TLS fails every attempt on a stripped root ->$c4"; fail_detail tls-fail-stripped
+fi
+
+# ── Case 5: self-heal — a failed run, then a later run that succeeds ─────
+# The case that proves a node is not permanently dead. Run 1 is a machine whose
+# generator fails every retry; run 2 is the same machine minutes later, once the
+# transient cause cleared, driven by archipelago-first-boot-secrets.timer. It
+# must end with the key present and the marker set, with no human at a console.
+# Run 2 also declares nginx/ssh already `failed` — they tried to start without a
+# key — so the run must actively restart them, not just reload. A "recovery"
+# that leaves the services down is not a recovery.
+#
+# Run 1 asserts only enough to establish the precondition (it really did fail,
+# and it left the node eligible to retry). Whether a key exists after a failure
+# is case 4's job — asserting it here too would make a single defect light up
+# two cases and blunt the signal.
+run_case self-heal fail ok stripped
+c5=""
+[ "$CASE_RC" -ne 0 ] || c5="$c5 run1-exit-zero"
+[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] && c5="$c5 run1-marker-set"
+[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] || c5="$c5 run1-no-failure-record"
+
+STUB_SYSTEMCTL_FAILED_UNITS="nginx ssh" run_case self-heal ok ok stripped 1
+[ "$CASE_RC" -eq 0 ] || c5="$c5 run2-exit-nonzero"
+[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] || c5="$c5 run2-marker-missing"
+[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" ] || c5="$c5 run2-key-missing"
+[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] || c5="$c5 run2-crt-missing"
+[ -s "$CASE_ROOT/etc/ssh/ssh_host_ed25519_key" ] || c5="$c5 run2-ssh-key-missing"
+[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] && c5="$c5 run2-stale-failure-record"
+grep -q 'restart nginx' "$CASE_SYSTEMCTL_LOG" 2>/dev/null || c5="$c5 run2-did-not-restart-failed-nginx"
+if [ -z "$c5" ]; then
+    ok "self-heal: failed run then a later successful run -> key present, marker set, failed units restarted"
+else
+    bad "self-heal ->$c5"; fail_detail self-heal
+fi
+
+# ── Case 6: single-producer invariant ────────────────────────────────────
+# The regression that would silently recreate F-03 is not a broken assertion —
+# it is somebody adding a second, well-meaning place that mints a key. A second
+# producer brings its own idea of success, its own absent retry policy and its
+# own absent failure record, and that is what made F-03 silent.
+#
+# So: every executable key-creating invocation in the builder must live inside
+# the first-boot-secrets.sh heredoc, i.e. inside gen_tls/gen_ssh. Comments are
+# exempt (they discuss the history); binary-existence checks are not matched
+# because they do not carry a key-creating subcommand.
+c6=""
+SS_START=$(grep -n '^cat > "\$WORK_DIR/first-boot-secrets.sh" <<.SECRETSSCRIPT.$' "$BUILDER" | cut -d: -f1)
+SS_END=$(awk -v s="$SS_START" 'NR>s && /^SECRETSSCRIPT$/ { print NR; exit }' "$BUILDER")
+if [ -z "$SS_START" ] || [ -z "$SS_END" ]; then
+    c6="$c6 could-not-locate-generator-heredoc"
+else
+    PRODUCERS=$(grep -nE 'openssl[[:space:]]+req|ssh-keygen[[:space:]]+-A|ssh-keygen[[:space:]]+-t' "$BUILDER" \
+                | grep -vE '^[0-9]+:[[:space:]]*#' || true)
+    while IFS= read -r line; do
+        [ -z "$line" ] && continue
+        ln=${line%%:*}
+        if [ "$ln" -lt "$SS_START" ] || [ "$ln" -gt "$SS_END" ]; then
+            c6="$c6 SECOND-PRODUCER-at-line-$ln"
+        fi
+    done <<< "$PRODUCERS"
+    # Sanity: the one producer we expect must actually be in there, otherwise an
+    # empty result would pass this case vacuously.
+    echo "$PRODUCERS" | grep -q 'openssl[[:space:]]*req' || c6="$c6 no-tls-producer-found-at-all"
+    echo "$PRODUCERS" | grep -q 'ssh-keygen' || c6="$c6 no-ssh-producer-found-at-all"
+fi
+if [ -z "$c6" ]; then
+    ok "single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh"
+else
+    bad "single-producer invariant ->$c6"
+    echo "    generator heredoc spans lines $SS_START-$SS_END of $BUILDER"
+fi
+
 # ── Summary ───────────────────────────────────────────────────────────────
 echo
 echo "──────── first-boot-secrets summary ────────"