19082a44f0ee207ce13c95ba037d5e96f589e117
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 archipelagoandClaude Opus 5
|
0ed9334f15 |
feat(10-04): let a deployed node report — and fix — fleet-shared host keys
10-03 closed the build half of F-03: the ISO no longer bakes SSH host keys or
a TLS keypair into the shared rootfs, and first-boot regeneration fails closed.
Nodes already in the field receive none of that — the first-boot script is
installed by the installer, not shipped by OTA — so a node that hit the old
fail-open path is still running key material that every downloader of its ISO
also holds, and its completion marker guarantees it will never try again.
scripts/security/host-secrets-audit.sh decides, from the node's own disk alone,
which of those it is. Four signals in a fixed precedence: missing material can
never be shared material; the fail-open fingerprint (marker present plus the
literal `WARNING: TLS regeneration failed` / `WARNING: ssh-keygen -A failed`
lines the old script emitted) is direct evidence and outranks timestamps and
also names WHICH class survived; then key mtime against a first-boot anchor
(.secrets-regenerated, falling back to the installer's LUKS key then
machine-id). Verdicts are per-node / shared / fail-closed-missing / unknown,
and every one of them carries the evidence strings that produced it, each
naming the file it was read from.
per-node is never claimed from an absent signal. No anchor means `unknown`, and
a standing first-boot-secrets.failed record also means `unknown` — a clean
mtime is not evidence that generation succeeded. That is T-10-37: a false
per-node verdict leaves an exposed node looking clean, which is worse than no
verdict at all.
Rotation (D-06: detect-report-then-apply, recorded in
docs/security/KEY-02-FLEET-ROTATION.md):
- --detect is the default and is read-only; it always exits 0, because
detection is informational and must never fail a boot.
- --apply without --yes writes nothing at all, not even its own verdict file.
"Touches nothing" is worth being able to say without a footnote.
- --apply --yes refuses unless the verdict is `shared`, so the wrong node
cannot be rotated even deliberately.
- It stages the full replacement TLS pair AND host-key set before touching
anything live and aborts if either fails; records the OLD fingerprints
before the swap; does TLS first (a dead web UI is recoverable over SSH, the
converse is not); replaces host keys by mv-onto-the-existing-path rather
than rm-then-mv, so the directory is never momentarily empty; and RELOADS
sshd, never restarts it, so the operator's own session survives its own
rotation.
bootstrap.rs ships the boot unit through the existing run_runtime_assets
promotion and enables it --now, so the verdict lands with the OTA rather than
at the next reboot. handle_system_stats gains a host_secrets object read from
the on-disk verdict — cheap, never an error however malformed the file, and
deliberately carrying no fingerprints, because a payload polled every few
seconds does not need digests an operator on the node can already read.
tests/first-boot-secrets/rotation-tests.sh: 8 cases against temp roots through
the HOST_SECRETS_ROOT seam. Negative controls run and reverted, each reddening
exactly one case: dry run writing its verdict file (STATE-DIR-CHANGED); the
old fingerprints recorded after the swap instead of before (caught by an
ordering observation, not a content comparison — the systemctl stub records
whether the file existed at the moment of the first reload); a tolerated
generation failure leaving a half-rotated node; and `per-node` claimed with no
anchor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
archipelagoandClaude Opus 5
|
dad40c23f1 |
fix(10-03): prove the first-boot TLS key and cert are actually a pair
Parsing each half back proves each is well-formed; it never proves they belong together. A key from one generation beside a cert from another passes both individual parse checks, gets blessed, and then nginx refuses to start at the exact moment the marker claims first boot succeeded. gen_tls() now extracts the public key from each half and compares them before the swap, and needs_tls() applies the same check to what is already installed, so a mismatched pair that reached disk some other way (an older build, a half-finished manual edit) is repaired instead of quietly breaking nginx. Extraction subsumes parsing, so this replaces the separate -noout parse checks rather than adding to them. Kept deliberately in step with regenerate_tls_cert() in core/archipelago/src/api/rpc/system/handlers.rs, which does the same comparison on the running node after a rename. Test harness: the openssl stub keypair now carries the generation it came from, and STUB_OPENSSL_MISMATCH emits a cert from a different one — the pair that passes both parse checks and still breaks nginx. New case 9 covers both directions: fail closed when the mismatch arises during generation, repair exactly once when found already on disk, and no spin on the run after either. 9/9 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
archipelagoandClaude Opus 5
|
40b77e392a |
fix(10-03): don't bless a cert minted under an untrustworthy clock
The failure fail-closed cannot catch, because generation SUCCEEDS. This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long before time has synced. `openssl req -x509` stamps notBefore from whatever the clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be years out: clock ahead -> clients reject it as "not yet valid", a harder failure than the usual self-signed warning; clock behind -> notAfter is already in the past once time syncs. The completion marker was then set and never revisited — a node permanently serving a cert nothing accepts. Finding 1, reported rather than assumed: this image does NOT use systemd-timesyncd. It installs and enables chrony, and chrony-wait.service — the unit that is Before=time-sync.target — is not enabled. So time-sync.target is inert here and ordering After= it would buy nothing. Enabling chrony-wait to make it meaningful would stall boot behind NTP on a node with no network, and these nodes are routinely offline at first boot. Not deadlocking boot outranks cert-date elegance, so the ordering is deliberately left alone. Fixed locally instead, in two parts: 1. Backdate notBefore by 24h so ordinary skew between node and client cannot invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and the rootfs is debian:trixie which ships it, but the capability is PROBED, not assumed — guessing wrong would fail every attempt and brick the node, the exact outcome all of this exists to prevent. Without the flags we simply do not backdate and rule 2 still covers the dangerous case. 2. Refuse to bless a cert dated by a clock outside a plausible window (2026-01-01 .. 2056-01-01). The material stays installed so the node is usable and sshd comes up, but the bad dates are recorded as failed=cert-dates and the cert is regenerated automatically once time syncs. Generation is now driven by need rather than by "is the marker absent", and ConditionPathExists=! is removed from the unit so a node that already completed can still be re-examined — skipping the unit is precisely how such a node stays broken forever. The script exits in milliseconds when everything is fine. Anti-spin is one condition: a date-driven regeneration happens ONLY when the clock is currently plausible. A node whose clock is still wrong re-checks and mints nothing. Regression caught while writing this: driving generation purely by content made needs_ssh() false whenever any host key existed, which would have left an image-baked fleet-shared key in place forever — F-03 reopened. The marker check is back in both needs_ functions and case 1 (which prestages a baked key and asserts it was replaced) is what caught it. Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin directions. Controls: blessing regardless of clock reddens only case 8 (run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8 (SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially passed against a broken guard because the assertion compared certificate dates, and a re-mint under a frozen clock produces a byte-identical notBefore — the assertion now counts mints, which is the only thing that distinguishes "left alone" from "regenerated again". Not covered here: nodes already deployed from earlier ISOs never receive this script (it is installed by the installer, not by OTA), so fleet remediation for them remains 10-04/OTA work in core/**, which is held by other executors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
archipelagoandClaude Opus 5
|
d9b3a7d5e0 |
fix(10-03): quote the Dockerfile heredoc so comments cannot execute
`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was unquoted, so the build shell performed command substitution on the Dockerfile body. Any backtick in a Dockerfile COMMENT was executed on the build host and its output spliced into the generated file. Six comments did this. One of them ran `systemctl start archipelago-fips.service` against the build machine on every ISO build; the others were harmless only by accident of being command-not-found. Fixes the class, not the six instances. The delimiter is now quoted, so the body is emitted verbatim and a future backticked comment is inert. Verified the boundary by line range first: the other backticked comments in this file (:264, :809, :1188, :1289, :1506, :1605, :3597, :3651) are ordinary shell comments outside any unquoted heredoc and were never at risk — they are untouched. The body needs exactly four build-time values and they are all package names (LINUX_IMAGE_PKG, GRUB_EFI_PKG, GRUB_EFI_SIGNED_PKG, GRUB_PC_PKG), on four consecutive lines. So quoting was practical: the heredoc is split into DOCKERFILE_HEAD and DOCKERFILE_TAIL, both quoted, with a single explicit printf interpolating those four names between them. Escapes that existed only because the heredoc was unquoted are undone in the same pass: six trailing `\\` become `\` (Docker line continuations) and four `\$` become `$` (RUN arguments reach the shell verbatim — Docker does not substitute variables in RUN). Verified by rendering the generated Dockerfile before and after with the same inputs and diffing them normalised (continuations joined, whitespace collapsed). Both are 190 normalised lines and the ONLY differences are the six comments regaining their text — every instruction is byte-identical. Before: "# the archipelago backend calls" / after: "# the archipelago backend calls `systemctl start archipelago-fips.service`". Test: case 7 asserts every heredoc writing Dockerfile.rootfs has a quoted delimiter, and when one is not, reports which body lines would execute. The assertion is on the delimiter, not on backticks — with quoting a backticked comment is legal and six of them are back in the body on purpose, so flagging backticks would flag a non-bug and fail on the very comments this restored. This bug is invisible to `bash -n`; an instance of it introduced earlier in this plan hung a syntactically-clean build for two minutes before being caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
archipelagoandClaude Opus 5
|
2efab5f219 |
fix(10-03): unify secret generation to a single producer + self-heal (F-03)
Unify rather than delete. The defect in F-03 was never "a second attempt to
create a key exists" — it was that failure was silent and the completion marker
lied about it. A second attempt is only dangerous when it is an unaudited second
PRODUCER carrying its own idea of success, its own absent retry policy and its
own absent failure record.
Single producer. gen_tls() is now the only code in the ISO build that creates
/etc/archipelago/ssl/archipelago.{key,crt}; gen_ssh() the only code that creates
/etc/ssh/ssh_host_*. Two secondary producers are gone:
- the Dockerfile's `openssl req` layer, which baked a keypair the strip layer
deleted moments later in the same build;
- the installer's "ensure SSL cert exists for nginx HTTPS" block, which before
the strip almost never fired and after it would have fired on every install.
Proof is mechanical, not a claim: every executable `openssl req` / `ssh-keygen
-A` invocation in the builder now lives inside the generator heredoc, and the
test suite fails if one appears outside it.
Build-time assertion. The one realistic total failure is a missing generator
binary, which is deterministic — no retry or reboot fixes it. A rootfs RUN layer
now fails the build if openssl or ssh-keygen is missing or non-executable.
openssl and openssh-server are both already in the package list (and
openssh-server hard-depends openssh-client, which ships ssh-keygen), so today
this is cheap insurance; it earns its place the first time someone edits that
list.
Self-heal, never dead-end. Fail-closed governs SERVING; retry governs
RECOVERING, and they are different things. Adds
archipelago-first-boot-secrets.timer (OnBootSec=5min, OnUnitActiveSec=15min),
installed and enabled with a hand-written symlink fallback because chroot
systemctl enable can fail silently. The service's own ConditionPathExists=!
makes every trigger a no-op once the marker exists, so a healthy node pays
nothing. On success the script now restarts consumers that are in `failed` —
try-reload-or-restart is a no-op on a failed unit, so without this a recovered
node would have valid keys on disk and nginx still down.
Never serve a bogus key. gen_tls parses both halves back with `openssl pkey`
and `openssl x509` before the swap, so a truncated or half-written artefact is
never what nginx reads.
Tests: 6 cases, each with an isolated negative control (transcripts in SUMMARY).
- case 4, TLS fails every attempt on a stripped root -> no key from any source.
Control: reintroduce a fallback key creation -> only case 4 red.
- case 5, self-heal: a failed run then a later successful run -> key present,
marker set, failed units restarted. Control: dead-end on a node that already
failed -> only case 5 red.
- case 6, single-producer invariant. Control: reintroduce the installer block
-> only case 6 red, naming the line.
Residual risk, stated plainly: a machine where generation can never succeed
still ends up with no SSH and no TLS. Build-time assertion removes the
deterministic cause, retry plus timer removes the transient ones, so what
remains is genuinely broken hardware — and it says so on the console and in
/var/lib/archipelago/first-boot-secrets.failed rather than quietly serving a
key nobody audited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
archipelagoandClaude Opus 5
|
210430967d |
fix(10-03): fail closed on first-boot secret regeneration failure (F-03)
The first-boot per-device secret regeneration was fail-open: both branches logged a warning and continued, and `touch "$MARKER"` ran unconditionally outside both `if` blocks. Combined with the unit's ConditionPathExists=! and the script's own marker fast-path, one transient failure left that node on the image-wide shared SSH host key and TLS private key permanently and silently — and the ISO is a published artefact, so every downloader holds those keys. - Retry each generator 3 times with backoff (D-05), so a transient first-boot condition recovers inside the same boot instead of being terminal. - Write the completion marker ONLY when both TLS and SSH succeeded, so a failed boot leaves the unit eligible to run again on the next boot. - On terminal failure: durable record at /var/lib/archipelago/first-boot-secrets.failed naming which generator failed, plus console + logger + stderr, and exit 1 so the unit lands in `failed` rather than `active`. The record is cleared on a later success. - Add FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF seams. Unset in production the behaviour is byte-identical; set, they let the fail-closed property be asserted rather than claimed. - Order the unit After=systemd-random-seed.service (no-op today, correct if a seed file is ever baked). - State the operational trade in the script header: after the rootfs strip, a terminal failure means no SSH and no TLS and needs the physical console. That was chosen deliberately over running on fleet-shared keys. tests/first-boot-secrets/run-tests.sh extracts the shipped heredoc body from the builder and drives it against a temp root with stubbed generators: both succeed, openssl fails every attempt, ssh-keygen fails twice then succeeds. Moving the marker touch back outside the success branch makes case 2 fail with MARKER-SET-ON-FAILURE, which is the regression this pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |