The failure fail-closed cannot catch, because generation SUCCEEDS.
This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long
before time has synced. `openssl req -x509` stamps notBefore from whatever the
clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be
years out: clock ahead -> clients reject it as "not yet valid", a harder failure
than the usual self-signed warning; clock behind -> notAfter is already in the
past once time syncs. The completion marker was then set and never revisited —
a node permanently serving a cert nothing accepts.
Finding 1, reported rather than assumed: this image does NOT use
systemd-timesyncd. It installs and enables chrony, and chrony-wait.service —
the unit that is Before=time-sync.target — is not enabled. So time-sync.target
is inert here and ordering After= it would buy nothing. Enabling chrony-wait to
make it meaningful would stall boot behind NTP on a node with no network, and
these nodes are routinely offline at first boot. Not deadlocking boot outranks
cert-date elegance, so the ordering is deliberately left alone.
Fixed locally instead, in two parts:
1. Backdate notBefore by 24h so ordinary skew between node and client cannot
invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and
the rootfs is debian:trixie which ships it, but the capability is PROBED,
not assumed — guessing wrong would fail every attempt and brick the node,
the exact outcome all of this exists to prevent. Without the flags we simply
do not backdate and rule 2 still covers the dangerous case.
2. Refuse to bless a cert dated by a clock outside a plausible window
(2026-01-01 .. 2056-01-01). The material stays installed so the node is
usable and sshd comes up, but the bad dates are recorded as
failed=cert-dates and the cert is regenerated automatically once time syncs.
Generation is now driven by need rather than by "is the marker absent", and
ConditionPathExists=! is removed from the unit so a node that already completed
can still be re-examined — skipping the unit is precisely how such a node stays
broken forever. The script exits in milliseconds when everything is fine.
Anti-spin is one condition: a date-driven regeneration happens ONLY when the
clock is currently plausible. A node whose clock is still wrong re-checks and
mints nothing.
Regression caught while writing this: driving generation purely by content made
needs_ssh() false whenever any host key existed, which would have left an
image-baked fleet-shared key in place forever — F-03 reopened. The marker check
is back in both needs_ functions and case 1 (which prestages a baked key and
asserts it was replaced) is what caught it.
Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin
directions. Controls: blessing regardless of clock reddens only case 8
(run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8
(SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially
passed against a broken guard because the assertion compared certificate dates,
and a re-mint under a frozen clock produces a byte-identical notBefore — the
assertion now counts mints, which is the only thing that distinguishes "left
alone" from "regenerated again".
Not covered here: nodes already deployed from earlier ISOs never receive this
script (it is installed by the installer, not by OTA), so fleet remediation for
them remains 10-04/OTA work in core/**, which is held by other executors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
