kdump + rasdaemon on every node, per docs/kdump-rasdaemon-design.md with
the approved decisions: hang capture ON (a wedged kiosk dumps and reboots
itself instead of sitting dead), crashkernel=256M, backfill ships with
this release, phase-2 UI surfacing deferred.
Host fixups (docs/system-level-ota-design.md) are the general answer to
'deliver system-level updates OTA': curated OS packages, sysctl drop-ins,
service enablement and the GRUB crashkernel line, carried by the signed
binary and applied idempotently at startup — non-fatal by construction
(offline/locked-dpkg nodes converge on a later boot), skipped on dev
boxes and non-Debian hosts. This formalizes the polkit/audio repair
precedents into a channel with a stated policy: pinned packages and
parameter intent only, never dist-upgrade automation; the ISO bakes the
identical end state into fresh installs (next commit).
The one runtime limitation is honest: crashkernel memory can only be
reserved at boot, so the fixup writes GRUB, runs update-grub, and logs
that it takes effect on the next reboot.
tests/lifecycle/os-audit.sh gains section D — a graded baseline check:
FAIL if capture never landed, WARN if written but awaiting reboot, PASS
when reserved, policy live and rasdaemon recording. Section D runs
independently of RPC health: a wedged backend must not mask that the
node also stopped capturing evidence.
Verification: host_fixups unit tests 4/4; cargo fmt clean; full suite
runs in the release gate (create-release) and the archi-dev-box
lifecycle gate before the tag.
Design for capturing post-mortem and hardware-error evidence on fleet
nodes: kdump (crashkernel=256M, dump to /var/crash on the unencrypted
root — never the LUKS data partition, so the crash kernel never handles
key material; makedumpfile-compressed, keep-2 retention) and rasdaemon
(EDAC/ECC events into sqlite on the same root).
Deliberately phased: phase 1 = capture on the image + bootstrap backfill
for existing nodes (kernel cmdline can't travel by OTA; takes effect on
next reboot); phase 2 = a read-only system.diagnostics surface in the
UI, only after a fleet node has produced a real dump.
Four decisions flagged in the doc: hang-capture on/off (recommended ON
— a wedged kiosk is useless anyway, and this turns every freeze into
evidence + self-reboot), crashkernel size, backfill timing, and phase-2
scope. Implementation touchpoints listed (Dockerfile.rootfs,
auto-install.sh:1810 cmdline, kdump-tools config, bootstrap, lifecycle
gate assertions).
archipelago