fix(10-03): fail closed on first-boot secret regeneration failure (F-03)
The first-boot per-device secret regeneration was fail-open: both branches logged a warning and continued, and `touch "$MARKER"` ran unconditionally outside both `if` blocks. Combined with the unit's ConditionPathExists=! and the script's own marker fast-path, one transient failure left that node on the image-wide shared SSH host key and TLS private key permanently and silently — and the ISO is a published artefact, so every downloader holds those keys. - Retry each generator 3 times with backoff (D-05), so a transient first-boot condition recovers inside the same boot instead of being terminal. - Write the completion marker ONLY when both TLS and SSH succeeded, so a failed boot leaves the unit eligible to run again on the next boot. - On terminal failure: durable record at /var/lib/archipelago/first-boot-secrets.failed naming which generator failed, plus console + logger + stderr, and exit 1 so the unit lands in `failed` rather than `active`. The record is cleared on a later success. - Add FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF seams. Unset in production the behaviour is byte-identical; set, they let the fail-closed property be asserted rather than claimed. - Order the unit After=systemd-random-seed.service (no-op today, correct if a seed file is ever baked). - State the operational trade in the script header: after the rootfs strip, a terminal failure means no SSH and no TLS and needs the physical console. That was chosen deliberately over running on fleet-shared keys. tests/first-boot-secrets/run-tests.sh extracts the shipped heredoc body from the builder and drives it against a temp root with stubbed generators: both succeed, openssl fails every attempt, ssh-keygen fails twice then succeeds. Moving the marker touch back outside the success branch makes case 2 fail with MARKER-SET-ON-FAILURE, which is the regression this pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
c502ff0e0a
commit
210430967d
@@ -1601,6 +1601,11 @@ cat > "$WORK_DIR/archipelago-first-boot-secrets.service" <<'SECRETSSERVICE'
|
||||
Description=Regenerate per-device secrets (TLS key, SSH host keys)
|
||||
DefaultDependencies=no
|
||||
After=local-fs.target
|
||||
# No random-seed file is baked into the rootfs today (verified by the entropy
|
||||
# audit's C-4 tar listing), so this ordering is a no-op right now. It is here
|
||||
# so that if one is ever introduced, the pool is credited before this unit —
|
||||
# the first consumer of entropy on a freshly-flashed machine — draws from it.
|
||||
After=systemd-random-seed.service
|
||||
Before=ssh.service nginx.service archipelago.service
|
||||
ConditionPathExists=!/var/lib/archipelago/.secrets-regenerated
|
||||
|
||||
@@ -1615,53 +1620,182 @@ SECRETSSERVICE
|
||||
|
||||
cat > "$WORK_DIR/first-boot-secrets.sh" <<'SECRETSSCRIPT'
|
||||
#!/bin/bash
|
||||
# Replace image-baked secrets with device-unique ones on first boot.
|
||||
# Never leaves the device without working keys: new material is generated
|
||||
# to a staging path first and only swapped in on success.
|
||||
# Create this device's own TLS keypair and SSH host keys on first boot.
|
||||
#
|
||||
# ── FAIL CLOSED — read this before changing anything below (audit F-03) ──
|
||||
#
|
||||
# The rootfs tar is byte-identical on every node flashed from one ISO, and the
|
||||
# ISO is a published artefact. It therefore no longer carries any identity
|
||||
# material: the rootfs Dockerfile in STEP 1 of this builder strips the SSH
|
||||
# host keys, the TLS keypair and machine-id out of the shared image. THIS
|
||||
# SCRIPT IS THE ONLY THING THAT CREATES THEM. That is deliberate.
|
||||
#
|
||||
# The operational consequence, in plain words: if regeneration fails every
|
||||
# retry, this node has no SSH host key and no TLS key. sshd will not start and
|
||||
# the nginx TLS listener will not start, so the node cannot be reached over
|
||||
# the network and recovery requires the physical console.
|
||||
#
|
||||
# That cost was accepted on purpose. The behaviour it replaces was worse: log
|
||||
# a warning, set the completion marker anyway, and run forever on the SSH host
|
||||
# key and TLS private key that every downloader of the ISO also holds — which
|
||||
# is undetectable host impersonation and transparent MITM of the web UI, on a
|
||||
# node whose operator has no idea.
|
||||
#
|
||||
# So: the completion marker is written ONLY when both generators succeeded. A
|
||||
# failed boot leaves the marker absent, which leaves the unit's
|
||||
# ConditionPathExists=! satisfied, so the whole thing runs again on the next
|
||||
# boot. Each generator is retried with backoff first, so a transient first-boot
|
||||
# condition (slow entropy pool, momentarily full disk) recovers without needing
|
||||
# a reboot at all.
|
||||
#
|
||||
# Testability seam: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path. It is
|
||||
# unset in production — the expansion is empty and behaviour is identical to a
|
||||
# script with the paths hard-coded — and set to a temp dir by
|
||||
# tests/first-boot-secrets/run-tests.sh, which is what makes the fail-closed
|
||||
# property assertable instead of merely claimed.
|
||||
set -u
|
||||
|
||||
LOG=/var/log/archipelago-first-boot-secrets.log
|
||||
MARKER=/var/lib/archipelago/.secrets-regenerated
|
||||
ROOT="${FIRST_BOOT_SECRETS_ROOT:-}"
|
||||
|
||||
# Waits between attempts at one generator. The attempt count is the number of
|
||||
# entries; the wait after the final attempt is skipped, because a failed last
|
||||
# attempt is terminal and there is nothing left to wait for. With the default
|
||||
# 3-entry list that means 3 attempts at t=0s, t=2s and t=10s, and the trailing
|
||||
# 20 is the ceiling that applies if the list is ever lengthened. Tests override
|
||||
# this with zeros so the suite does not sleep.
|
||||
BACKOFF="${FIRST_BOOT_SECRETS_BACKOFF:-2 8 20}"
|
||||
|
||||
LOG="$ROOT/var/log/archipelago-first-boot-secrets.log"
|
||||
MARKER="$ROOT/var/lib/archipelago/.secrets-regenerated"
|
||||
FAILED="$ROOT/var/lib/archipelago/first-boot-secrets.failed"
|
||||
CONSOLE="$ROOT/dev/console"
|
||||
SSL_DIR="$ROOT/etc/archipelago/ssl"
|
||||
SSH_DIR="$ROOT/etc/ssh"
|
||||
|
||||
[ -f "$MARKER" ] && exit 0
|
||||
mkdir -p /var/lib/archipelago
|
||||
mkdir -p "$ROOT/var/lib/archipelago" "$ROOT/var/log"
|
||||
NODE_NAME=$(hostname 2>/dev/null || echo archipelago)
|
||||
|
||||
echo "$(date): regenerating per-device secrets" >> "$LOG"
|
||||
log() { echo "$(date): $*" >> "$LOG"; }
|
||||
|
||||
# A terminal failure must be impossible to miss: journal, console and stderr,
|
||||
# on top of the durable on-disk record. Every channel is guarded so that a
|
||||
# missing /dev/console (test root, or an early boot without one) cannot itself
|
||||
# make the failure path fail.
|
||||
shout() {
|
||||
log "$*"
|
||||
if command -v logger >/dev/null 2>&1; then
|
||||
logger -t archipelago-first-boot-secrets "$*" 2>/dev/null || true
|
||||
fi
|
||||
if [ -w "$CONSOLE" ]; then
|
||||
printf '%s\n' "$*" | tee -a "$CONSOLE" >/dev/null 2>&1 || true
|
||||
fi
|
||||
printf '%s\n' "$*" >&2
|
||||
}
|
||||
|
||||
# retry <label> <command...> — run the command up to N times with backoff.
|
||||
retry() {
|
||||
local label="$1"; shift
|
||||
local -a waits
|
||||
# shellcheck disable=SC2206
|
||||
waits=($BACKOFF)
|
||||
local n=${#waits[@]}
|
||||
local i=1
|
||||
while [ "$i" -le "$n" ]; do
|
||||
if "$@"; then
|
||||
[ "$i" -gt 1 ] && log "$label succeeded on attempt $i of $n"
|
||||
return 0
|
||||
fi
|
||||
log "$label attempt $i of $n failed"
|
||||
if [ "$i" -lt "$n" ]; then
|
||||
sleep "${waits[$((i-1))]}" 2>/dev/null || true
|
||||
fi
|
||||
i=$((i+1))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# 1. Self-signed TLS: fresh keypair with this device's hostname in the SAN
|
||||
# (server.set-name regenerates again if the node is renamed later)
|
||||
mkdir -p /etc/archipelago/ssl
|
||||
if openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout /etc/archipelago/ssl/archipelago.key.new \
|
||||
-out /etc/archipelago/ssl/archipelago.crt.new \
|
||||
-subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=${NODE_NAME}" \
|
||||
-addext "subjectAltName=DNS:${NODE_NAME},DNS:${NODE_NAME}.local,DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" \
|
||||
>> "$LOG" 2>&1; then
|
||||
mv /etc/archipelago/ssl/archipelago.key.new /etc/archipelago/ssl/archipelago.key
|
||||
mv /etc/archipelago/ssl/archipelago.crt.new /etc/archipelago/ssl/archipelago.crt
|
||||
chmod 600 /etc/archipelago/ssl/archipelago.key
|
||||
echo "$(date): TLS keypair regenerated (CN=${NODE_NAME})" >> "$LOG"
|
||||
# (server.set-name regenerates again if the node is renamed later).
|
||||
# Generated to .new and swapped only on success, so the node is never left
|
||||
# holding half a keypair.
|
||||
gen_tls() {
|
||||
mkdir -p "$SSL_DIR" || return 1
|
||||
rm -f "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.crt.new"
|
||||
if openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout "$SSL_DIR/archipelago.key.new" \
|
||||
-out "$SSL_DIR/archipelago.crt.new" \
|
||||
-subj "/C=XX/ST=Bitcoin/L=Node/O=Archipelago/CN=${NODE_NAME}" \
|
||||
-addext "subjectAltName=DNS:${NODE_NAME},DNS:${NODE_NAME}.local,DNS:archipelago,DNS:archipelago.local,DNS:localhost,IP:127.0.0.1" \
|
||||
>> "$LOG" 2>&1 \
|
||||
&& [ -s "$SSL_DIR/archipelago.key.new" ] \
|
||||
&& [ -s "$SSL_DIR/archipelago.crt.new" ]; then
|
||||
mv "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.key" \
|
||||
&& mv "$SSL_DIR/archipelago.crt.new" "$SSL_DIR/archipelago.crt" || return 1
|
||||
chmod 600 "$SSL_DIR/archipelago.key"
|
||||
return 0
|
||||
fi
|
||||
rm -f "$SSL_DIR/archipelago.key.new" "$SSL_DIR/archipelago.crt.new"
|
||||
return 1
|
||||
}
|
||||
|
||||
# 2. SSH host keys: generate a full fresh set in staging, then swap.
|
||||
gen_ssh() {
|
||||
local staging
|
||||
staging=$(mktemp -d) || return 1
|
||||
mkdir -p "$staging/etc/ssh"
|
||||
if ssh-keygen -A -f "$staging" >> "$LOG" 2>&1 \
|
||||
&& ls "$staging"/etc/ssh/ssh_host_*_key >/dev/null 2>&1; then
|
||||
mkdir -p "$SSH_DIR"
|
||||
rm -f "$SSH_DIR"/ssh_host_*
|
||||
mv "$staging"/etc/ssh/ssh_host_* "$SSH_DIR"/
|
||||
rm -rf "$staging"
|
||||
return 0
|
||||
fi
|
||||
rm -rf "$staging"
|
||||
return 1
|
||||
}
|
||||
|
||||
log "regenerating per-device secrets"
|
||||
|
||||
TLS_OK=0
|
||||
SSH_OK=0
|
||||
|
||||
if retry "TLS keypair regeneration" gen_tls; then
|
||||
TLS_OK=1
|
||||
log "TLS keypair regenerated (CN=${NODE_NAME})"
|
||||
systemctl try-reload-or-restart nginx >> "$LOG" 2>&1 || true
|
||||
else
|
||||
rm -f /etc/archipelago/ssl/archipelago.key.new /etc/archipelago/ssl/archipelago.crt.new
|
||||
echo "$(date): WARNING: TLS regeneration failed, keeping baked key" >> "$LOG"
|
||||
fi
|
||||
|
||||
# 2. SSH host keys: generate a full fresh set in staging, then swap
|
||||
STAGING=$(mktemp -d)
|
||||
mkdir -p "$STAGING/etc/ssh"
|
||||
if ssh-keygen -A -f "$STAGING" >> "$LOG" 2>&1 && ls "$STAGING"/etc/ssh/ssh_host_*_key >/dev/null 2>&1; then
|
||||
rm -f /etc/ssh/ssh_host_*
|
||||
mv "$STAGING"/etc/ssh/ssh_host_* /etc/ssh/
|
||||
echo "$(date): SSH host keys regenerated" >> "$LOG"
|
||||
if retry "SSH host key regeneration" gen_ssh; then
|
||||
SSH_OK=1
|
||||
log "SSH host keys regenerated"
|
||||
systemctl try-reload-or-restart ssh >> "$LOG" 2>&1 || true
|
||||
else
|
||||
echo "$(date): WARNING: ssh-keygen -A failed, keeping baked host keys" >> "$LOG"
|
||||
fi
|
||||
rm -rf "$STAGING"
|
||||
|
||||
touch "$MARKER"
|
||||
echo "$(date): per-device secrets done" >> "$LOG"
|
||||
if [ "$TLS_OK" -eq 1 ] && [ "$SSH_OK" -eq 1 ]; then
|
||||
# Clear any alarm left by an earlier boot that failed and then recovered,
|
||||
# so a healthy node does not carry a stale failure record forever.
|
||||
rm -f "$FAILED"
|
||||
touch "$MARKER"
|
||||
log "per-device secrets done"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Fail closed. Deliberately NO marker: its absence is what lets the unit run
|
||||
# again on the next boot.
|
||||
WHICH=""
|
||||
[ "$TLS_OK" -eq 0 ] && WHICH="TLS"
|
||||
[ "$SSH_OK" -eq 0 ] && WHICH="${WHICH:+$WHICH and }SSH"
|
||||
{
|
||||
echo "timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date)"
|
||||
echo "failed=$WHICH"
|
||||
echo "tls_ok=$TLS_OK"
|
||||
echo "ssh_ok=$SSH_OK"
|
||||
echo "detail=per-device secret regeneration failed after all retries; the completion marker was NOT set, so this unit runs again on the next boot"
|
||||
} > "$FAILED"
|
||||
shout "ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's $WHICH key material. Refusing to continue — the affected services will not start. Record: $FAILED Log: $LOG"
|
||||
exit 1
|
||||
SECRETSSCRIPT
|
||||
|
||||
chmod +x "$WORK_DIR/first-boot-secrets.sh"
|
||||
|
||||
Executable
+225
@@ -0,0 +1,225 @@
|
||||
#!/bin/bash
|
||||
# Regression harness for the first-boot per-device secret regeneration script
|
||||
# (audit finding F-03, phase 10 / KEY-02).
|
||||
#
|
||||
# What this pins, and why it exists at all: the script used to `touch` its
|
||||
# completion marker unconditionally, outside both success branches, so one
|
||||
# transient failure at first boot left the node running the ISO-wide shared
|
||||
# SSH host key and TLS private key forever, silently. The property that must
|
||||
# never regress is therefore negative — "on failure the marker is NOT created"
|
||||
# — and a negative property is only assertable if the failure can be forced.
|
||||
# So the generators are stubbed and the script is driven against a temp root
|
||||
# through the FIRST_BOOT_SECRETS_ROOT seam.
|
||||
#
|
||||
# The script under test is not a file in this repo: it is a heredoc inside
|
||||
# image-recipe/_archived/build-auto-installer-iso.sh (which is LIVE —
|
||||
# image-recipe/build-debian-iso.sh execs it). The harness extracts the heredoc
|
||||
# body between the SECRETSSCRIPT delimiters so it is testing the bytes that
|
||||
# actually ship, not a copy that can drift.
|
||||
#
|
||||
# Usage: bash tests/first-boot-secrets/run-tests.sh
|
||||
# Exit 0 only if all three cases PASS.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
BUILDER="$REPO/image-recipe/_archived/build-auto-installer-iso.sh"
|
||||
|
||||
WORK=$(mktemp -d)
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
PASS_COUNT=0
|
||||
FAIL_COUNT=0
|
||||
|
||||
ok() { echo "PASS: $1"; PASS_COUNT=$((PASS_COUNT + 1)); }
|
||||
bad() { echo "FAIL: $1"; FAIL_COUNT=$((FAIL_COUNT + 1)); }
|
||||
|
||||
# ── Step 0: extract the script under test and syntax-check it ────────────
|
||||
[ -f "$BUILDER" ] || { echo "FAIL: builder not found at $BUILDER"; exit 1; }
|
||||
|
||||
SCRIPT="$WORK/first-boot-secrets.sh"
|
||||
awk '/^cat > "\$WORK_DIR\/first-boot-secrets.sh" <<.SECRETSSCRIPT.$/ { f = 1; next }
|
||||
f && /^SECRETSSCRIPT$/ { f = 0 }
|
||||
f { print }' \
|
||||
"$BUILDER" > "$SCRIPT"
|
||||
|
||||
if [ ! -s "$SCRIPT" ]; then
|
||||
echo "FAIL: could not extract the first-boot-secrets.sh heredoc from the builder"
|
||||
echo " (did the SECRETSSCRIPT delimiter or the cat> line change?)"
|
||||
exit 1
|
||||
fi
|
||||
chmod +x "$SCRIPT"
|
||||
|
||||
if bash -n "$SCRIPT"; then
|
||||
echo "extracted $(wc -l < "$SCRIPT") lines from the builder; bash -n clean"
|
||||
else
|
||||
echo "FAIL: extracted script does not parse"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Stubs ─────────────────────────────────────────────────────────────────
|
||||
# A stub dir is prepended to PATH so the script's openssl / ssh-keygen /
|
||||
# systemctl / logger calls hit these instead of the real tools. Behaviour is
|
||||
# driven by env vars the stubs read at call time.
|
||||
#
|
||||
# STUB_OPENSSL_MODE ok | fail
|
||||
# STUB_SSHKEYGEN_MODE ok | fail | fail-twice (fail-twice uses a counter file)
|
||||
# STUB_COUNTER_DIR where the counter file lives
|
||||
make_stubs() {
|
||||
local dir="$1"
|
||||
mkdir -p "$dir"
|
||||
|
||||
cat > "$dir/openssl" <<'STUB'
|
||||
#!/bin/bash
|
||||
# Stub openssl: honours -keyout/-out so the script's staging-then-swap and its
|
||||
# non-empty checks are exercised for real.
|
||||
[ "${STUB_OPENSSL_MODE:-ok}" = "fail" ] && exit 1
|
||||
keyout="" out=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-keyout) keyout="$2"; shift 2 ;;
|
||||
-out) out="$2"; shift 2 ;;
|
||||
*) shift ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$keyout" ] && printf -- '-----BEGIN PRIVATE KEY-----\nstub\n-----END PRIVATE KEY-----\n' > "$keyout"
|
||||
[ -n "$out" ] && printf -- '-----BEGIN CERTIFICATE-----\nstub\n-----END CERTIFICATE-----\n' > "$out"
|
||||
exit 0
|
||||
STUB
|
||||
|
||||
cat > "$dir/ssh-keygen" <<'STUB'
|
||||
#!/bin/bash
|
||||
# Stub ssh-keygen -A: writes a host-key set into <-f dir>/etc/ssh, matching
|
||||
# the real tool's layout, which is what the script globs for.
|
||||
mode="${STUB_SSHKEYGEN_MODE:-ok}"
|
||||
counter="${STUB_COUNTER_DIR:-/tmp}/ssh-keygen.count"
|
||||
|
||||
n=$(cat "$counter" 2>/dev/null || echo 0)
|
||||
n=$((n + 1))
|
||||
echo "$n" > "$counter"
|
||||
|
||||
case "$mode" in
|
||||
fail) exit 1 ;;
|
||||
fail-twice) [ "$n" -le 2 ] && exit 1 ;;
|
||||
esac
|
||||
|
||||
root=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-f) root="$2"; shift 2 ;;
|
||||
*) shift ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$root" ] || exit 1
|
||||
mkdir -p "$root/etc/ssh"
|
||||
for t in rsa ecdsa ed25519; do
|
||||
printf -- '-----BEGIN OPENSSH PRIVATE KEY-----\nstub-%s\n' "$t" > "$root/etc/ssh/ssh_host_${t}_key"
|
||||
printf -- 'ssh-%s AAAAstub stub@archipelago\n' "$t" > "$root/etc/ssh/ssh_host_${t}_key.pub"
|
||||
done
|
||||
exit 0
|
||||
STUB
|
||||
|
||||
# Neither of these must be allowed to touch the host during a test run.
|
||||
printf '#!/bin/bash\nexit 0\n' > "$dir/systemctl"
|
||||
printf '#!/bin/bash\nexit 0\n' > "$dir/logger"
|
||||
|
||||
chmod +x "$dir"/openssl "$dir"/ssh-keygen "$dir"/systemctl "$dir"/logger
|
||||
}
|
||||
|
||||
STUBS="$WORK/stubs"
|
||||
make_stubs "$STUBS"
|
||||
|
||||
# ── Runner ────────────────────────────────────────────────────────────────
|
||||
# Runs the script against a fresh temp root. Echoes the exit status; the
|
||||
# caller asserts on it plus the resulting filesystem state.
|
||||
CASE_ROOT=""
|
||||
CASE_RC=0
|
||||
run_case() {
|
||||
local name="$1" openssl_mode="$2" sshkeygen_mode="$3"
|
||||
CASE_ROOT="$WORK/root-$name"
|
||||
rm -rf "$CASE_ROOT"
|
||||
mkdir -p "$CASE_ROOT/var/lib/archipelago" "$CASE_ROOT/var/log" \
|
||||
"$CASE_ROOT/etc/ssh" "$CASE_ROOT/etc/archipelago/ssl"
|
||||
# A pre-existing baked host key + TLS key, i.e. the pre-strip rootfs state:
|
||||
# the assertions below then also show the swap actually replaced them.
|
||||
echo "BAKED-SHARED-HOST-KEY" > "$CASE_ROOT/etc/ssh/ssh_host_rsa_key"
|
||||
echo "BAKED-SHARED-TLS-KEY" > "$CASE_ROOT/etc/archipelago/ssl/archipelago.key"
|
||||
|
||||
rm -f "$WORK/counters-$name/ssh-keygen.count"
|
||||
mkdir -p "$WORK/counters-$name"
|
||||
|
||||
set +e
|
||||
env PATH="$STUBS:$PATH" \
|
||||
FIRST_BOOT_SECRETS_ROOT="$CASE_ROOT" \
|
||||
FIRST_BOOT_SECRETS_BACKOFF="0 0 0" \
|
||||
STUB_OPENSSL_MODE="$openssl_mode" \
|
||||
STUB_SSHKEYGEN_MODE="$sshkeygen_mode" \
|
||||
STUB_COUNTER_DIR="$WORK/counters-$name" \
|
||||
bash "$SCRIPT" > "$WORK/$name.out" 2> "$WORK/$name.err"
|
||||
CASE_RC=$?
|
||||
set -e
|
||||
}
|
||||
|
||||
fail_detail() {
|
||||
echo " exit=$CASE_RC root=$CASE_ROOT"
|
||||
echo " stderr: $(head -c 300 "$WORK/$1.err" 2>/dev/null)"
|
||||
}
|
||||
|
||||
# ── Case 1: both generators succeed ──────────────────────────────────────
|
||||
run_case both-ok ok ok
|
||||
c1=""
|
||||
[ "$CASE_RC" -eq 0 ] || c1="$c1 exit-nonzero"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] || c1="$c1 marker-missing"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] && c1="$c1 stale-failure-record"
|
||||
[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" ] || c1="$c1 tls-key-missing"
|
||||
[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] || c1="$c1 tls-crt-missing"
|
||||
grep -q BAKED-SHARED-TLS-KEY "$CASE_ROOT/etc/archipelago/ssl/archipelago.key" && c1="$c1 tls-key-not-replaced"
|
||||
[ -s "$CASE_ROOT/etc/ssh/ssh_host_ed25519_key" ] || c1="$c1 ssh-host-key-missing"
|
||||
grep -q BAKED-SHARED-HOST-KEY "$CASE_ROOT/etc/ssh/ssh_host_rsa_key" && c1="$c1 ssh-key-not-replaced"
|
||||
ls "$CASE_ROOT"/etc/archipelago/ssl/*.new >/dev/null 2>&1 && c1="$c1 dotnew-leftover"
|
||||
if [ -z "$c1" ]; then
|
||||
ok "both generators succeed -> exit 0, marker set, keys swapped in"
|
||||
else
|
||||
bad "both generators succeed ->$c1"; fail_detail both-ok
|
||||
fi
|
||||
|
||||
# ── Case 2: openssl fails every attempt -> FAIL CLOSED ───────────────────
|
||||
# This is the case that would have passed against the old script and is the
|
||||
# whole reason this harness exists: the old code logged a warning and set the
|
||||
# marker anyway.
|
||||
run_case tls-fail fail ok
|
||||
c2=""
|
||||
[ "$CASE_RC" -ne 0 ] || c2="$c2 exit-zero-on-failure"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] && c2="$c2 MARKER-SET-ON-FAILURE"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] || c2="$c2 no-failure-record"
|
||||
grep -q 'failed=.*TLS' "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" 2>/dev/null \
|
||||
|| c2="$c2 failure-record-does-not-name-TLS"
|
||||
ls "$CASE_ROOT"/etc/archipelago/ssl/*.new >/dev/null 2>&1 && c2="$c2 dotnew-leftover"
|
||||
grep -qi 'FAILED' "$WORK/tls-fail.err" || c2="$c2 no-loud-stderr"
|
||||
if [ -z "$c2" ]; then
|
||||
ok "openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS"
|
||||
else
|
||||
bad "openssl fails every attempt ->$c2"; fail_detail tls-fail
|
||||
fi
|
||||
|
||||
# ── Case 3: ssh-keygen fails twice then succeeds -> backoff recovers ─────
|
||||
run_case ssh-flaky ok fail-twice
|
||||
c3=""
|
||||
[ "$CASE_RC" -eq 0 ] || c3="$c3 exit-nonzero"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] || c3="$c3 marker-missing"
|
||||
[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] && c3="$c3 failure-record-present"
|
||||
[ -s "$CASE_ROOT/etc/ssh/ssh_host_ed25519_key" ] || c3="$c3 ssh-host-key-missing"
|
||||
attempts=$(cat "$WORK/counters-ssh-flaky/ssh-keygen.count" 2>/dev/null || echo 0)
|
||||
[ "$attempts" -eq 3 ] || c3="$c3 expected-3-attempts-got-$attempts"
|
||||
if [ -z "$c3" ]; then
|
||||
ok "ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)"
|
||||
else
|
||||
bad "ssh-keygen fails twice then succeeds ->$c3"; fail_detail ssh-flaky
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────
|
||||
echo
|
||||
echo "──────── first-boot-secrets summary ────────"
|
||||
echo "passed: $PASS_COUNT failed: $FAIL_COUNT"
|
||||
[ "$FAIL_COUNT" -eq 0 ] || exit 1
|
||||
exit 0
|
||||
Reference in New Issue
Block a user