diff --git a/docs/security/KEY-02-ROOTFS-EVIDENCE.md b/docs/security/KEY-02-ROOTFS-EVIDENCE.md index bf3b3dec..7e0b06f1 100644 --- a/docs/security/KEY-02-ROOTFS-EVIDENCE.md +++ b/docs/security/KEY-02-ROOTFS-EVIDENCE.md @@ -15,6 +15,7 @@ as a passing check until it is filled in. |---|---| | Builder commit (Task 1) | `21043096` — fail-closed first-boot regeneration | | Builder commit (Task 2) | `408b328c` — rootfs identity strip | +| Builder commit (follow-up) | single-producer unification, build-time generator assertion, self-heal timer | | Builder file | `image-recipe/_archived/build-auto-installer-iso.sh` (LIVE; `image-recipe/build-debian-iso.sh` execs it) | | Build host | _to be recorded_ | | Date run | _to be recorded_ | @@ -84,6 +85,11 @@ tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \ - `etc/machine-id` present with size **0**, or absent. Either satisfies "not shared"; record which one was actually observed rather than generalising. +Note on the TLS keypair specifically: it is now absent for two independent reasons, not one. +The Dockerfile no longer generates it at all (that layer was removed so there is a single +producer), *and* the strip layer still deletes it as belt-and-braces in case a future layer +starts baking one. Seeing it present therefore means both defences were bypassed. + ### 4. Confirm the provenance file rode along ```bash @@ -93,18 +99,40 @@ tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs- Expected: one entry, `opt/archipelago/rootfs-identity-stripped`. Its absence means the strip layer did not execute and the whole check is void. -### 5. Confirm the regeneration path is still shipped +### 5. Confirm the regeneration path and its self-heal timer are still shipped This is the brick check, and it is not optional. A stripped rootfs whose first-boot -regeneration script failed to ship would leave every flashed node with no SSH host key and -nothing to create one. +generation script failed to ship would leave every flashed node with no SSH host key and +nothing to create one. The timer is part of the same check: without it, a node whose +generators fail every in-boot retry has no unattended way back. ```bash ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \ - image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service + image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \ + image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer ``` -Expected: both present, `first-boot-secrets.sh` executable. +Expected: all three present, `first-boot-secrets.sh` executable. + +### 5b. Confirm the build-time generator assertion actually ran + +The rootfs build fails outright if `openssl` or `ssh-keygen` is missing or non-executable, +because that is the one way first-boot generation can fail deterministically — retries and +reboots would never fix it, so it must never reach a node. A successful build therefore +already proves the generators are present, and the build log says so: + +```bash +grep 'first-boot secret generators present' +``` + +If you did not capture the log, assert it against the tar instead: + +```bash +tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \ + | grep -E 'usr/bin/(openssl|ssh-keygen)$' +``` + +Expected: both present and mode `-rwxr-xr-x`. ### 6. Record the RECIPE_HASH the builder actually used @@ -146,9 +174,35 @@ regeneration failure degrades to "no key, the service refuses to start" rather t keys. That is audit item **C-3** (§779) and needs two physical machines; it remains separately UNVERIFIED. C-4 is a build-host check only. -**Also worth recording when C-3 is run:** the installer writes a per-install TLS keypair as a -fallback (`build-auto-installer-iso.sh`, the "Ensure SSL cert exists for nginx HTTPS" block). -That fallback fires on every install now that the rootfs ships without a keypair. It is -generated separately on each target machine, so it is per node and does not reopen F-03 — but -it does mean the two nodes' *TLS* certs will differ even if first-boot regeneration failed on -both. SSH host keys have no such fallback, so they are the sharper signal for C-3. +### Guidance for C-3: SSH and TLS are now equally sharp signals + +An earlier revision of this document said SSH host keys were the sharper divergence signal for +C-3, because the installer had a per-install TLS fallback that would produce a differing cert +even if first-boot generation had failed. **That asymmetry no longer exists.** + +There is now exactly one producer of each secret — `gen_tls()` and `gen_ssh()` inside +`first-boot-secrets.sh` — and no other code in the ISO build creates either. The Dockerfile no +longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for +C-3, treat both the same way: + +```bash +# on each node +ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf - +openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256 +``` + +**Pass:** both fingerprints differ between the two nodes. **Fail:** either matches — a matching +TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have +been explained away by the fallback. + +Also check, on each node, that the run actually succeeded rather than merely being quiet: + +```bash +ls -l /var/lib/archipelago/.secrets-regenerated # present on a healthy node +cat /var/lib/archipelago/first-boot-secrets.failed 2>&1 # absent on a healthy node +systemctl status archipelago-first-boot-secrets.timer # enabled; the self-heal path +``` + +The audit's original C-3 fail condition — a `WARNING:` line in the log alongside an existing +marker — can no longer occur by construction: the marker is only written when both generators +succeeded. If you ever see that combination, the fix has been reverted. diff --git a/image-recipe/_archived/build-auto-installer-iso.sh b/image-recipe/_archived/build-auto-installer-iso.sh index 0f75c922..e6ada1e8 100755 --- a/image-recipe/_archived/build-auto-installer-iso.sh +++ b/image-recipe/_archived/build-auto-installer-iso.sh @@ -456,17 +456,41 @@ RUN ln -sf /etc/nginx/sites-available/archipelago /etc/nginx/sites-enabled/archi # Install nginx snippets (PWA config, HTTPS app proxies) COPY snippets/ /etc/nginx/snippets/ -# Generate self-signed SSL certificate for HTTPS (PWA install + mic/camera -# access both require a secure context). SAN covers the install-time default -# hostname -- server.set-name regenerates this with the new hostname's SAN -# if a node is renamed. -RUN mkdir -p /etc/archipelago/ssl && \ - openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \ - -keyout /etc/archipelago/ssl/archipelago.key \ - -out /etc/archipelago/ssl/archipelago.crt \ - -subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=archipelago" \ - -addext "subjectAltName=DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" && \ - chmod 600 /etc/archipelago/ssl/archipelago.key +# The self-signed HTTPS keypair is NOT generated here (audit F-03). +# +# It used to be: this layer ran "openssl req" and baked one keypair into the +# shared image, which meant every node flashed from one ISO — and everyone who +# downloaded the ISO — held the same TLS private key. The strip layer at the +# end of this Dockerfile would delete it again anyway, so generating it here +# now only creates a SECOND piece of code that can mint a TLS key with its own +# accounting. There is exactly one producer of this keypair, and it is +# first-boot-secrets.sh, which retries and reports. +# +# The ssl directory is created so that producer's staging swap has somewhere +# to land. +RUN mkdir -p /etc/archipelago/ssl + +# Fail the BUILD if the rootfs cannot generate per-device secrets. +# +# The one realistic way first-boot generation fails on every retry is a missing +# generator binary, and that failure is deterministic, not transient — retries +# and reboots will never fix it. A node in the field must not be where we +# discover it. openssl and openssh-server are both in the package list above +# (and openssh-server hard-depends openssh-client, which ships ssh-keygen), so +# today this assertion is cheap insurance rather than a fix. It earns its place +# by turning a silent fleet-wide brick into a loud build failure the first time +# anyone edits that package list. +RUN set -e; \ + for bin in /usr/bin/openssl /usr/bin/ssh-keygen; do \ + if [ ! -x "\$bin" ]; then \ + echo "FATAL: \$bin missing or not executable in the rootfs." >&2; \ + echo "first-boot-secrets.sh cannot generate per-device SSH host keys" >&2; \ + echo "or the TLS keypair without it, and that failure is permanent." >&2; \ + echo "Restore openssl / openssh-server in the package list above." >&2; \ + exit 1; \ + fi; \ + done; \ + echo "first-boot secret generators present: openssl, ssh-keygen" # Create archipelago systemd service COPY archipelago.service /etc/systemd/system/archipelago.service @@ -586,27 +610,29 @@ RUN apt-get clean && \ # identity-shaped left in here is therefore held by every node AND by every # person who downloaded the ISO. # -# Two things get baked without anyone asking for them: -# - Debian's openssh-server postinst generates /etc/ssh/ssh_host_* at package -# install time, i.e. inside this container build. -# - the "openssl req" layer above writes /etc/archipelago/ssl/archipelago.key. # NOTE: this heredoc is UNQUOTED, so backticks here are command substitution # and would run at build time. Never put backticks in these comments. -# Plus /etc/machine-id, which systemd populates during the build and which -# correlates every node flashed from one ISO. # -# archipelago-first-boot-secrets.service recreates all of this per device on -# first boot. The point of removing it HERE is to change what a regeneration -# failure costs: with the material stripped, a failure degrades to "no key, the -# service refuses to start" instead of "fleet-shared key, silently" — which is -# the whole of F-03. That makes fail-closed structural rather than procedural. +# The TLS keypair is no longer generated in this Dockerfile at all (see the +# note where that layer used to be). What still gets baked without anyone +# asking for it is Debian's openssh-server postinst, which generates +# /etc/ssh/ssh_host_* at package install time — i.e. inside this container +# build — plus /etc/machine-id, which systemd populates during the build and +# which correlates every node flashed from one ISO. This layer removes both. +# +# archipelago-first-boot-secrets.service recreates all of it per device on +# first boot, retrying on a timer until it succeeds. The point of removing it +# HERE is to change what a generation failure costs: with the material +# stripped, a failure degrades to "no key, the service refuses to start" +# instead of "fleet-shared key, silently" — which is the whole of F-03. That +# makes fail-closed structural rather than procedural. # # This must stay the LAST layer: anything that installs packages after it can -# reintroduce host keys. Keep the /etc/archipelago/ssl directory itself so the -# first-boot script's staging swap has somewhere to land. The "openssl req" -# layer above is deliberately left in place — it keeps proving openssl is -# present and keeps the SAN template next to the code that uses it; this layer -# is what makes the shipped output non-shared. +# reintroduce host keys. The rm of the TLS keypair is kept as belt-and-braces +# even though nothing in this build creates one any more — if a future layer +# starts baking a cert, this catches it. Keep the /etc/archipelago/ssl +# directory itself so the first-boot script's staging swap has somewhere to +# land. RUN rm -f /etc/ssh/ssh_host_* && \ rm -f /etc/archipelago/ssl/archipelago.key /etc/archipelago/ssl/archipelago.crt && \ mkdir -p /etc/archipelago/ssl && \ @@ -1654,6 +1680,36 @@ RemainAfterExit=yes WantedBy=multi-user.target SECRETSSERVICE +# Self-heal timer. Fail-closed governs SERVING (never present a key we did not +# generate); this timer governs RECOVERING (never dead-end a node). +# +# Without it, a node whose generators failed all their in-boot retries would sit +# with no SSH host key and no TLS key until somebody walked to it with a +# keyboard. With it, a transient cause that later clears — a full disk that gets +# freed, a pool that eventually seeds — repairs the node unattended. +# +# The service's own ConditionPathExists=! is what stops this: once the marker +# exists, every subsequent trigger is a no-op that systemd records as success, +# so the timer costs nothing on a healthy node and needs no separate teardown. +# This uses systemd's own facilities on purpose; a sleep loop inside the script +# would hold a oneshot open for hours and hide the failure from systemctl. +cat > "$WORK_DIR/archipelago-first-boot-secrets.timer" <<'SECRETSTIMER' +[Unit] +Description=Retry per-device secret generation until it succeeds +Documentation=man:archipelago-first-boot-secrets.service(8) + +[Timer] +# First retry shortly after boot has settled — by then the disk, the entropy +# pool and any late-mounting filesystem have had a chance to become healthy. +OnBootSec=5min +OnUnitActiveSec=15min +AccuracySec=30s +Unit=archipelago-first-boot-secrets.service + +[Install] +WantedBy=timers.target +SECRETSTIMER + cat > "$WORK_DIR/first-boot-secrets.sh" <<'SECRETSSCRIPT' #!/bin/bash # Create this device's own TLS keypair and SSH host keys on first boot. @@ -1666,26 +1722,41 @@ cat > "$WORK_DIR/first-boot-secrets.sh" <<'SECRETSSCRIPT' # host keys, the TLS keypair and machine-id out of the shared image. THIS # SCRIPT IS THE ONLY THING THAT CREATES THEM. That is deliberate. # -# The operational consequence, in plain words: if regeneration fails every -# retry, this node has no SSH host key, so sshd will not start and the node -# cannot be reached over SSH — recovery requires the physical console. Nothing -# else on the install path creates host keys, so that outcome is certain. -# (TLS is softer: the installer writes a per-node fallback keypair with the -# generic CN=archipelago SAN, so the web UI usually still comes up. That -# fallback is per install, never image-wide, so it does not reopen F-03.) +# SINGLE PRODUCER. gen_tls() below is the only code anywhere in the ISO build +# that creates /etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() is the only +# code that creates /etc/ssh/ssh_host_*. The Dockerfile no longer bakes a +# keypair and the installer's old "ensure SSL cert exists" fallback is gone. +# That is the actual lesson of F-03: the bug was never "a second attempt to +# create a key exists", it was that failure was silent and the marker lied +# about it. A second producer is dangerous precisely because it has its own +# accounting — its own idea of success, its own (absent) retry policy, its own +# (absent) failure record. One producer means one place that can fail, one +# place that retries, one place that reports. # -# That cost was accepted on purpose. The behaviour it replaces was worse: log -# a warning, set the completion marker anyway, and run forever on the SSH host -# key and TLS private key that every downloader of the ISO also holds — which -# is undetectable host impersonation and transparent MITM of the web UI, on a -# node whose operator has no idea. +# FAIL CLOSED applies to SERVING: if generation fails, no key exists, so sshd +# and the nginx TLS listener refuse to start. They never come up on a +# placeholder, a zero-length file, or a key from anywhere else. gen_tls swaps +# into place only after openssl has parsed both halves back, so a truncated or +# corrupt artefact is never what a service reads. # -# So: the completion marker is written ONLY when both generators succeeded. A -# failed boot leaves the marker absent, which leaves the unit's -# ConditionPathExists=! satisfied, so the whole thing runs again on the next -# boot. Each generator is retried with backoff first, so a transient first-boot -# condition (slow entropy pool, momentarily full disk) recovers without needing -# a reboot at all. +# SELF-HEAL applies to RECOVERING, and it is a different thing: a failure must +# never dead-end the node. Three attempts with backoff inside the boot, then +# archipelago-first-boot-secrets.timer retries every 15 minutes, and every +# subsequent boot retries too — all because the marker is never written on +# failure. A transient cause that later clears (a full disk that gets freed, a +# pool that eventually seeds) repairs the node with nobody at a console. On +# success the script restarts whatever refused to start, so recovery is +# complete rather than pending-a-reboot. +# +# The only failure that survives all of that is a deterministic one — a missing +# generator binary — and the rootfs build asserts openssl and ssh-keygen are +# present and executable, so the build fails rather than the fleet. +# +# What this replaces was worse in every direction: log a warning, set the +# completion marker anyway, never retry, and run forever on the SSH host key and +# TLS private key that every downloader of the ISO also holds — undetectable +# host impersonation and transparent MITM of the web UI, on a node whose +# operator has no idea. # # Testability seam: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path. It is # unset in production — the expansion is empty and behaviour is identical to a @@ -1754,10 +1825,16 @@ retry() { return 1 } -# 1. Self-signed TLS: fresh keypair with this device's hostname in the SAN -# (server.set-name regenerates again if the node is renamed later). -# Generated to .new and swapped only on success, so the node is never left -# holding half a keypair. +# THE SINGLE PRODUCER of this node's TLS keypair. No other code in the ISO +# build creates /etc/archipelago/ssl/archipelago.{key,crt}. Do not add one; add +# a caller of this instead. tests/first-boot-secrets/run-tests.sh case 6 fails +# the build's test suite if a second producer appears. +# +# Fresh keypair with this device's hostname in the SAN (server.set-name +# regenerates again if the node is renamed later). Generated to .new, PARSED +# BACK, and only then swapped in — so nginx can never be handed a truncated, +# zero-length or half-written artefact, which is the "never serve with a bogus +# key" half of fail-closed. gen_tls() { mkdir -p "$SSL_DIR" || return 1 rm -f "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.crt.new" @@ -1768,7 +1845,10 @@ gen_tls() { -addext "subjectAltName=DNS:${NODE_NAME},DNS:${NODE_NAME}.local,DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" \ >> "$LOG" 2>&1 \ && [ -s "$SSL_DIR/archipelago.key.new" ] \ - && [ -s "$SSL_DIR/archipelago.crt.new" ]; then + && [ -s "$SSL_DIR/archipelago.crt.new" ] \ + && openssl pkey -noout -in "$SSL_DIR/archipelago.key.new" >> "$LOG" 2>&1 \ + && openssl x509 -noout -in "$SSL_DIR/archipelago.crt.new" >> "$LOG" 2>&1; then + chmod 600 "$SSL_DIR/archipelago.key.new" mv "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.key" \ && mv "$SSL_DIR/archipelago.crt.new" "$SSL_DIR/archipelago.crt" || return 1 chmod 600 "$SSL_DIR/archipelago.key" @@ -1778,7 +1858,10 @@ gen_tls() { return 1 } -# 2. SSH host keys: generate a full fresh set in staging, then swap. +# THE SINGLE PRODUCER of this node's SSH host keys. Same rule as gen_tls: no +# second producer anywhere, add a caller instead. +# +# Generates a full fresh set in staging, then swaps. gen_ssh() { local staging staging=$(mktemp -d) || return 1 @@ -1795,6 +1878,31 @@ gen_ssh() { return 1 } +# Hand the new material to whatever consumes it. +# +# Two different situations, and getting this wrong is what would turn the timer +# into theatre: +# - First boot. We are ordered Before= these units, so they have not started +# yet. try-reload-or-restart is a no-op on an inactive unit, which is +# exactly right — they will start on their own moments later and read the +# keys we just wrote. +# - Self-heal, minutes or hours later. The unit already tried to start with no +# key and is sitting in `failed`. try-reload-or-restart would be a no-op +# there too, which would leave the node broken with valid keys on disk — the +# recovery would be "complete" but the service still down. So a failed unit +# is explicitly restarted. +# --no-block because on first boot we are inside a unit these services are +# ordered after; a blocking start could deadlock the boot transaction. +refresh_consumer() { + local unit="$1" + if systemctl is-failed --quiet "$unit" 2>/dev/null; then + log "$unit is failed (it started without a key); restarting it" + systemctl --no-block restart "$unit" >> "$LOG" 2>&1 || true + else + systemctl try-reload-or-restart "$unit" >> "$LOG" 2>&1 || true + fi +} + log "regenerating per-device secrets" TLS_OK=0 @@ -1803,13 +1911,13 @@ SSH_OK=0 if retry "TLS keypair regeneration" gen_tls; then TLS_OK=1 log "TLS keypair regenerated (CN=${NODE_NAME})" - systemctl try-reload-or-restart nginx >> "$LOG" 2>&1 || true + refresh_consumer nginx fi if retry "SSH host key regeneration" gen_ssh; then SSH_OK=1 log "SSH host keys regenerated" - systemctl try-reload-or-restart ssh >> "$LOG" 2>&1 || true + refresh_consumer ssh fi if [ "$TLS_OK" -eq 1 ] && [ "$SSH_OK" -eq 1 ]; then @@ -1821,8 +1929,12 @@ if [ "$TLS_OK" -eq 1 ] && [ "$SSH_OK" -eq 1 ]; then exit 0 fi -# Fail closed. Deliberately NO marker: its absence is what lets the unit run -# again on the next boot. +# Fail closed on serving, self-heal on recovering. +# +# Deliberately NO marker. Its absence is the entire retry mechanism: it keeps +# the unit's ConditionPathExists=! satisfied, so archipelago-first-boot-secrets +# .timer re-runs this in 15 minutes and every subsequent boot re-runs it too. +# This is a failure, not a dead end — do not "fix" it by writing the marker. WHICH="" [ "$TLS_OK" -eq 0 ] && WHICH="TLS" [ "$SSH_OK" -eq 0 ] && WHICH="${WHICH:+$WHICH and }SSH" @@ -1831,15 +1943,16 @@ WHICH="" echo "failed=$WHICH" echo "tls_ok=$TLS_OK" echo "ssh_ok=$SSH_OK" - echo "detail=per-device secret regeneration failed after all retries; the completion marker was NOT set, so this unit runs again on the next boot" + echo "detail=per-device secret generation failed after all retries; the completion marker was NOT set, so archipelago-first-boot-secrets.timer retries in 15 minutes and every subsequent boot retries too" } > "$FAILED" -shout "ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's $WHICH key material. Refusing to continue — the affected services will not start. Record: $FAILED Log: $LOG" +shout "ARCHIPELAGO: could not generate this device's $WHICH key material. The affected services will NOT start rather than run on a key we did not generate. Retrying automatically every 15 minutes and on every boot. Record: $FAILED Log: $LOG" exit 1 SECRETSSCRIPT chmod +x "$WORK_DIR/first-boot-secrets.sh" cp "$WORK_DIR/first-boot-secrets.sh" "$ARCH_DIR/scripts/" cp "$WORK_DIR/archipelago-first-boot-secrets.service" "$ARCH_DIR/scripts/" +cp "$WORK_DIR/archipelago-first-boot-secrets.timer" "$ARCH_DIR/scripts/" # Ship the mesh-radio udev rule at the media root — the embedded installer # searches "$BOOT_MEDIA/99-mesh-radio.rules" first, but nothing ever staged @@ -2764,6 +2877,11 @@ if [ -d "$BOOT_MEDIA/archipelago/container-images" ]; then if [ -f "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.service" ]; then cp "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.service" /mnt/target/etc/systemd/system/ fi + # The self-heal timer. Without it a node whose generators fail every in-boot + # retry has no unattended way back — it would need someone at the console. + if [ -f "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.timer" ]; then + cp "$BOOT_MEDIA/archipelago/scripts/archipelago-first-boot-secrets.timer" /mnt/target/etc/systemd/system/ + fi if [ -f "$BOOT_MEDIA/archipelago/scripts/setup-tor.sh" ]; then cp "$BOOT_MEDIA/archipelago/scripts/setup-tor.sh" /mnt/target/opt/archipelago/scripts/ chmod +x /mnt/target/opt/archipelago/scripts/setup-tor.sh @@ -3430,28 +3548,37 @@ RemainAfterExit=yes WantedBy=multi-user.target DIAGSVC -# Ensure SSL cert exists for nginx HTTPS. +# NO install-time TLS certificate is generated here, on purpose (audit F-03, +# phase 10 plan 10-03). # -# Since the F-03 identity-strip layer this is no longer a rarely-taken safety -# net — the rootfs ships with NO TLS keypair, so this branch fires on every -# install. That is fine and deliberate: the installer runs separately on each -# target machine, so the key it writes is per node, not image-wide. It uses the -# generic CN=archipelago SAN; archipelago-first-boot-secrets.service replaces it -# on first boot with one carrying this device's actual hostname. +# There used to be an "ensure SSL cert exists for nginx HTTPS" block that ran +# its own "chroot /mnt/target openssl req ..." whenever the target had no cert. +# Before the identity strip it almost never fired; after the strip it would +# have fired on EVERY install. # -# Consequence worth knowing: if first-boot regeneration fails, the web UI still -# has *a* per-node cert from here, whereas SSH has nothing at all (nothing -# recreates host keys at install time) and sshd will refuse to start. -if [ ! -f /mnt/target/etc/archipelago/ssl/archipelago.crt ]; then - mkdir -p /mnt/target/etc/archipelago/ssl - chroot /mnt/target openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \ - -keyout /etc/archipelago/ssl/archipelago.key \ - -out /etc/archipelago/ssl/archipelago.crt \ - -subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=archipelago" \ - -addext "subjectAltName=DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" 2>/dev/null - chmod 600 /mnt/target/etc/archipelago/ssl/archipelago.key - echo " Generated self-signed SSL certificate" -fi +# The problem with it was never that a second attempt to create a key existed. +# It was that it was a second PRODUCER: its own openssl invocation, its own idea +# of success, no retry policy, no failure record, no marker discipline. F-03 was +# a silent-failure bug, and a second producer with its own accounting is exactly +# how silent failures happen. So the fix is to unify, not to add another +# fallback and not to leave the node with no way back. +# +# What replaces it: archipelago-first-boot-secrets.service is the single +# producer of /etc/archipelago/ssl/archipelago.{key,crt}, exactly as it is the +# single producer of /etc/ssh/ssh_host_*, and it retries with backoff in-boot, +# on archipelago-first-boot-secrets.timer every 15 minutes, and on every boot +# until it succeeds. A machine that is merely busy repairs itself unattended; a +# machine that is genuinely broken says so loudly on the console and in +# /var/lib/archipelago/first-boot-secrets.failed instead of quietly serving a +# key nobody audited. +# +# The deterministic way this could never succeed — a missing openssl or +# ssh-keygen — is caught at BUILD time by the assertion layer in STEP 1, so it +# cannot reach a node at all. +# +# Do not reintroduce a producer here. If you need a key earlier, call the same +# generator. tests/first-boot-secrets/run-tests.sh case 6 fails if a second +# independent key-creating path appears in this file. # Enable linger for rootless podman (containers survive logout) mkdir -p /mnt/target/var/lib/systemd/linger @@ -3518,6 +3645,14 @@ chroot /mnt/target systemctl enable archipelago.service 2>/dev/null || true chroot /mnt/target systemctl enable nginx.service 2>/dev/null || true chroot /mnt/target systemctl enable archipelago-load-images.service 2>/dev/null || true chroot /mnt/target systemctl enable archipelago-first-boot-secrets.service 2>/dev/null || true +# Self-heal timer: if this does not get enabled, a node whose secret generation +# fails every retry has no unattended way back. chroot systemctl enable can fail +# silently, so fall back to writing the symlink by hand rather than trusting +# `|| true` to have done anything. +chroot /mnt/target systemctl enable archipelago-first-boot-secrets.timer 2>/dev/null || \ + { mkdir -p /mnt/target/etc/systemd/system/timers.target.wants && \ + ln -sf /etc/systemd/system/archipelago-first-boot-secrets.timer \ + /mnt/target/etc/systemd/system/timers.target.wants/archipelago-first-boot-secrets.timer 2>/dev/null || true; } chroot /mnt/target systemctl enable archipelago-setup-tor.service 2>/dev/null || true chroot /mnt/target systemctl enable archipelago-first-boot-containers.service 2>/dev/null || true chroot /mnt/target systemctl enable archipelago-kiosk.service 2>/dev/null || true diff --git a/tests/first-boot-secrets/run-tests.sh b/tests/first-boot-secrets/run-tests.sh index 3363d895..9157acbe 100755 --- a/tests/first-boot-secrets/run-tests.sh +++ b/tests/first-boot-secrets/run-tests.sh @@ -62,17 +62,41 @@ fi # systemctl / logger calls hit these instead of the real tools. Behaviour is # driven by env vars the stubs read at call time. # -# STUB_OPENSSL_MODE ok | fail -# STUB_SSHKEYGEN_MODE ok | fail | fail-twice (fail-twice uses a counter file) -# STUB_COUNTER_DIR where the counter file lives +# STUB_OPENSSL_MODE ok | fail (fail affects `req` only — +# `pkey`/`x509` validation still +# works, so a failed generation +# cannot be mistaken for a failed +# validation) +# STUB_SSHKEYGEN_MODE ok | fail | fail-twice (uses a counter file) +# STUB_COUNTER_DIR where the counter file lives +# STUB_SYSTEMCTL_FAILED_UNITS units `systemctl is-failed` should report failed +# STUB_SYSTEMCTL_LOG file the systemctl stub appends its args to make_stubs() { local dir="$1" mkdir -p "$dir" cat > "$dir/openssl" <<'STUB' #!/bin/bash -# Stub openssl: honours -keyout/-out so the script's staging-then-swap and its -# non-empty checks are exercised for real. +# Stub openssl. Honours -keyout/-out so the script's staging-then-swap and its +# non-empty checks are exercised for real, and implements the `pkey`/`x509` +# parse-back validation the generator does before it swaps. +sub="${1:-}" +case "$sub" in + pkey|x509) + # Validation: succeed iff the file exists and is non-empty. This is what + # lets the harness prove a truncated artefact is never swapped in. + f="" + while [ $# -gt 0 ]; do + case "$1" in + -in) f="$2"; shift 2 ;; + *) shift ;; + esac + done + [ -n "$f" ] && [ -s "$f" ] || exit 1 + exit 0 + ;; +esac + [ "${STUB_OPENSSL_MODE:-ok}" = "fail" ] && exit 1 keyout="" out="" while [ $# -gt 0 ]; do @@ -119,8 +143,23 @@ done exit 0 STUB - # Neither of these must be allowed to touch the host during a test run. - printf '#!/bin/bash\nexit 0\n' > "$dir/systemctl" + cat > "$dir/systemctl" <<'STUB' +#!/bin/bash +# Stub systemctl. Records every invocation so the harness can prove the +# self-heal path actually restarts a unit that failed for want of a key, and +# reports is-failed honestly so first-boot and self-heal take different paths. +[ -n "${STUB_SYSTEMCTL_LOG:-}" ] && echo "$*" >> "$STUB_SYSTEMCTL_LOG" +if [ "${1:-}" = "is-failed" ]; then + unit="${!#}" + for u in ${STUB_SYSTEMCTL_FAILED_UNITS:-}; do + [ "$u" = "$unit" ] && exit 0 + done + exit 1 +fi +exit 0 +STUB + + # Must not be allowed to touch the host journal during a test run. printf '#!/bin/bash\nexit 0\n' > "$dir/logger" chmod +x "$dir"/openssl "$dir"/ssh-keygen "$dir"/systemctl "$dir"/logger @@ -130,23 +169,37 @@ STUBS="$WORK/stubs" make_stubs "$STUBS" # ── Runner ──────────────────────────────────────────────────────────────── -# Runs the script against a fresh temp root. Echoes the exit status; the -# caller asserts on it plus the resulting filesystem state. +# run_case [prestage] [reuse] +# +# prestage baked — root already holds the shared keys, i.e. a pre-strip +# rootfs. Assertions can then prove the swap replaced them. +# stripped — root holds no key material at all, i.e. the rootfs this +# build actually ships. This is the state in which "no key +# may appear from anywhere but the generator" is testable. +# reuse 1 — do not wipe the root or the counters; continue from the +# previous run against the same node. Models a reboot or a +# timer-triggered retry. CASE_ROOT="" CASE_RC=0 +CASE_SYSTEMCTL_LOG="" run_case() { local name="$1" openssl_mode="$2" sshkeygen_mode="$3" + local prestage="${4:-baked}" reuse="${5:-0}" CASE_ROOT="$WORK/root-$name" - rm -rf "$CASE_ROOT" - mkdir -p "$CASE_ROOT/var/lib/archipelago" "$CASE_ROOT/var/log" \ - "$CASE_ROOT/etc/ssh" "$CASE_ROOT/etc/archipelago/ssl" - # A pre-existing baked host key + TLS key, i.e. the pre-strip rootfs state: - # the assertions below then also show the swap actually replaced them. - echo "BAKED-SHARED-HOST-KEY" > "$CASE_ROOT/etc/ssh/ssh_host_rsa_key" - echo "BAKED-SHARED-TLS-KEY" > "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" + CASE_SYSTEMCTL_LOG="$WORK/$name.systemctl" - rm -f "$WORK/counters-$name/ssh-keygen.count" - mkdir -p "$WORK/counters-$name" + if [ "$reuse" != "1" ]; then + rm -rf "$CASE_ROOT" + mkdir -p "$CASE_ROOT/var/lib/archipelago" "$CASE_ROOT/var/log" \ + "$CASE_ROOT/etc/ssh" "$CASE_ROOT/etc/archipelago/ssl" + if [ "$prestage" = "baked" ]; then + echo "BAKED-SHARED-HOST-KEY" > "$CASE_ROOT/etc/ssh/ssh_host_rsa_key" + echo "BAKED-SHARED-TLS-KEY" > "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" + fi + rm -f "$WORK/counters-$name/ssh-keygen.count" + mkdir -p "$WORK/counters-$name" + : > "$CASE_SYSTEMCTL_LOG" + fi set +e env PATH="$STUBS:$PATH" \ @@ -155,6 +208,8 @@ run_case() { STUB_OPENSSL_MODE="$openssl_mode" \ STUB_SSHKEYGEN_MODE="$sshkeygen_mode" \ STUB_COUNTER_DIR="$WORK/counters-$name" \ + STUB_SYSTEMCTL_LOG="$CASE_SYSTEMCTL_LOG" \ + STUB_SYSTEMCTL_FAILED_UNITS="${STUB_SYSTEMCTL_FAILED_UNITS:-}" \ bash "$SCRIPT" > "$WORK/$name.out" 2> "$WORK/$name.err" CASE_RC=$? set -e @@ -217,6 +272,98 @@ else bad "ssh-keygen fails twice then succeeds ->$c3"; fail_detail ssh-flaky fi +# ── Case 4: TLS fails every attempt on a STRIPPED root -> no key at all ── +# Case 2 proves the marker is not set. This proves the stronger property that +# replaced the installer's TLS fallback: on the rootfs we actually ship, a +# failed generation leaves NO key, from any source. If anything ever mints a +# key outside gen_tls — an install-time fallback, a placeholder, a zero-length +# touch to keep nginx happy — this is the case that goes red. +run_case tls-fail-stripped fail ok stripped +c4="" +[ "$CASE_RC" -ne 0 ] || c4="$c4 exit-zero-on-failure" +[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] && c4="$c4 MARKER-SET-ON-FAILURE" +[ -e "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" ] && c4="$c4 TLS-KEY-EXISTS-AFTER-FAILURE" +[ -e "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] && c4="$c4 TLS-CRT-EXISTS-AFTER-FAILURE" +[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] || c4="$c4 no-failure-record" +grep -q 'failed=.*TLS' "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" 2>/dev/null \ + || c4="$c4 failure-record-does-not-name-TLS" +ls "$CASE_ROOT"/etc/archipelago/ssl/*.new >/dev/null 2>&1 && c4="$c4 dotnew-leftover" +if [ -z "$c4" ]; then + ok "TLS fails every attempt on a stripped root -> NO key, NO marker, non-zero exit, record names TLS" +else + bad "TLS fails every attempt on a stripped root ->$c4"; fail_detail tls-fail-stripped +fi + +# ── Case 5: self-heal — a failed run, then a later run that succeeds ───── +# The case that proves a node is not permanently dead. Run 1 is a machine whose +# generator fails every retry; run 2 is the same machine minutes later, once the +# transient cause cleared, driven by archipelago-first-boot-secrets.timer. It +# must end with the key present and the marker set, with no human at a console. +# Run 2 also declares nginx/ssh already `failed` — they tried to start without a +# key — so the run must actively restart them, not just reload. A "recovery" +# that leaves the services down is not a recovery. +# +# Run 1 asserts only enough to establish the precondition (it really did fail, +# and it left the node eligible to retry). Whether a key exists after a failure +# is case 4's job — asserting it here too would make a single defect light up +# two cases and blunt the signal. +run_case self-heal fail ok stripped +c5="" +[ "$CASE_RC" -ne 0 ] || c5="$c5 run1-exit-zero" +[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] && c5="$c5 run1-marker-set" +[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] || c5="$c5 run1-no-failure-record" + +STUB_SYSTEMCTL_FAILED_UNITS="nginx ssh" run_case self-heal ok ok stripped 1 +[ "$CASE_RC" -eq 0 ] || c5="$c5 run2-exit-nonzero" +[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] || c5="$c5 run2-marker-missing" +[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" ] || c5="$c5 run2-key-missing" +[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] || c5="$c5 run2-crt-missing" +[ -s "$CASE_ROOT/etc/ssh/ssh_host_ed25519_key" ] || c5="$c5 run2-ssh-key-missing" +[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] && c5="$c5 run2-stale-failure-record" +grep -q 'restart nginx' "$CASE_SYSTEMCTL_LOG" 2>/dev/null || c5="$c5 run2-did-not-restart-failed-nginx" +if [ -z "$c5" ]; then + ok "self-heal: failed run then a later successful run -> key present, marker set, failed units restarted" +else + bad "self-heal ->$c5"; fail_detail self-heal +fi + +# ── Case 6: single-producer invariant ──────────────────────────────────── +# The regression that would silently recreate F-03 is not a broken assertion — +# it is somebody adding a second, well-meaning place that mints a key. A second +# producer brings its own idea of success, its own absent retry policy and its +# own absent failure record, and that is what made F-03 silent. +# +# So: every executable key-creating invocation in the builder must live inside +# the first-boot-secrets.sh heredoc, i.e. inside gen_tls/gen_ssh. Comments are +# exempt (they discuss the history); binary-existence checks are not matched +# because they do not carry a key-creating subcommand. +c6="" +SS_START=$(grep -n '^cat > "\$WORK_DIR/first-boot-secrets.sh" <<.SECRETSSCRIPT.$' "$BUILDER" | cut -d: -f1) +SS_END=$(awk -v s="$SS_START" 'NR>s && /^SECRETSSCRIPT$/ { print NR; exit }' "$BUILDER") +if [ -z "$SS_START" ] || [ -z "$SS_END" ]; then + c6="$c6 could-not-locate-generator-heredoc" +else + PRODUCERS=$(grep -nE 'openssl[[:space:]]+req|ssh-keygen[[:space:]]+-A|ssh-keygen[[:space:]]+-t' "$BUILDER" \ + | grep -vE '^[0-9]+:[[:space:]]*#' || true) + while IFS= read -r line; do + [ -z "$line" ] && continue + ln=${line%%:*} + if [ "$ln" -lt "$SS_START" ] || [ "$ln" -gt "$SS_END" ]; then + c6="$c6 SECOND-PRODUCER-at-line-$ln" + fi + done <<< "$PRODUCERS" + # Sanity: the one producer we expect must actually be in there, otherwise an + # empty result would pass this case vacuously. + echo "$PRODUCERS" | grep -q 'openssl[[:space:]]*req' || c6="$c6 no-tls-producer-found-at-all" + echo "$PRODUCERS" | grep -q 'ssh-keygen' || c6="$c6 no-ssh-producer-found-at-all" +fi +if [ -z "$c6" ]; then + ok "single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh" +else + bad "single-producer invariant ->$c6" + echo " generator heredoc spans lines $SS_START-$SS_END of $BUILDER" +fi + # ── Summary ─────────────────────────────────────────────────────────────── echo echo "──────── first-boot-secrets summary ────────"