lfg2025/archy

Files

T

archipelago cbd463e980 feat(host): crash/hardware-error capture, delivered by a new host-fixup OTA channel (#144 )

kdump + rasdaemon on every node, per docs/kdump-rasdaemon-design.md with
the approved decisions: hang capture ON (a wedged kiosk dumps and reboots
itself instead of sitting dead), crashkernel=256M, backfill ships with
this release, phase-2 UI surfacing deferred.

Host fixups (docs/system-level-ota-design.md) are the general answer to
'deliver system-level updates OTA': curated OS packages, sysctl drop-ins,
service enablement and the GRUB crashkernel line, carried by the signed
binary and applied idempotently at startup — non-fatal by construction
(offline/locked-dpkg nodes converge on a later boot), skipped on dev
boxes and non-Debian hosts. This formalizes the polkit/audio repair
precedents into a channel with a stated policy: pinned packages and
parameter intent only, never dist-upgrade automation; the ISO bakes the
identical end state into fresh installs (next commit).

The one runtime limitation is honest: crashkernel memory can only be
reserved at boot, so the fixup writes GRUB, runs update-grub, and logs
that it takes effect on the next reboot.

tests/lifecycle/os-audit.sh gains section D — a graded baseline check:
FAIL if capture never landed, WARN if written but awaiting reboot, PASS
when reserved, policy live and rasdaemon recording. Section D runs
independently of RPC health: a wedged backend must not mask that the
node also stopped capturing evidence.

Verification: host_fixups unit tests 4/4; cargo fmt clean; full suite
runs in the release gate (create-release) and the archi-dev-box
lifecycle gate before the tag.

2026-08-31 07:23:44 -04:00

4.1 KiB

Raw Blame History

System-Level OTA — host fixups

Status: Implemented (first payload shipped alongside this doc) Owner: core/archipelago/src/host_fixups.rs Related: docs/kdump-rasdaemon-design.md (first payload), CLAUDE.md invariants

The problem

The binary OTA updates the node's own software, and the signed app catalog updates apps. But the host OS — Debian packages, kernel parameters, system services — previously moved only through ISO re-installs. A node deployed a year ago can be running today's node software on a host that never gained anything the image learned since. Issue #99 (missing polkit rule on old nodes) and the audio-stack heal were each hand-carved one-off bootstrap repairs; there was no general channel and no stated policy for touching the host from the node.

The mechanism

host_fixups::ensure_host_fixups() — spawned from main.rs at startup alongside the other ensure_* heals, in the background, best-effort:

Dev-box guard — skip when /home/archipelago/archy is a symlink (contributor checkout) and when there's no dpkg (non-Debian host).
Packages — install only what's missing, from a curated, in-code list (HOST_PACKAGES), apt-get install first, one apt-get update retry, both under timeout, never fatal (offline/locked-dpkg nodes converge on a later boot).
Configuration — idempotent per-concern helpers writing root-owned config (via the existing host_sudo path): sysctl drop-ins, service defaults, GRUB cmdline, service enablement.
Reporting — every step logs what it did; failures log warnings and move on. A host fixup must never be able to stop the node from starting.

Why embedded-in-the-binary rather than fetched

Same reasoning as the tor-helper (bootstrap.rs): the signed binary OTA is the only authenticated delivery channel every node already trusts and pulls on schedule. Fixups compiled into the binary travel with a version, are reviewable in git, and can't be served to a subset of the fleet.

Policy — what may travel this channel

May	May not
Specific, pinned packages the node needs (kdump-tools, rasdaemon, …)	`dist-upgrade` or silent kernel/libc swaps — regular Debian upgrades stay with the operator
Kernel parameters via GRUB/sysctl — with the next-reboot caveat logged loudly	Anything requiring a secret, or touching LUKS key material
Service enablement + config the image also bakes in	Divergence: the ISO must converge to the SAME end state so fresh installs are a no-op
Small, reviewable, per-concern Rust functions with tests	Shell-script-of-things payloads beyond a single concern

The rule: the ISO and the fixup must express the same intent twice, in reviewable places — Dockerfile.rootfs/auto-install.sh for fresh installs, host_fixups.rs for the deployed fleet. A change that lands in one and not the other is a bug.

Kernel cmdline caveat

crashkernel= (and any future hugepages=-style reservation) only takes effect at boot: the fixup writes /etc/default/grub + update-grub and logs takes effect on the NEXT reboot. Operators reboot nodes when applying releases; no special ceremony is required beyond that, but the lifecycle gate grades this state honestly (WARN for written-but-not-yet- rebooted, FAIL for never-written — see tests/lifecycle/os-audit.sh section D).

Verification story

Unit tests pin the policy constants and script shapes (host_fixups tests in core/archipelago).
tests/lifecycle/os-audit.sh section D asserts the end state on a real node (config present, crashkernel reserved or pending reboot, hang policy live, rasdaemon active).
The lifecycle gate runs on archi-dev-box per release; the QEMU ISO smoke covers fresh installs.

Future payloads (candidates, not commitments)

unattended-upgrades posture + a default-deny host nftables ruleset (the §F hardening-plan item — needs its own design first).
Host firewall rules for mesh/WG ports.
Chronic: anything the image learns post-deploy that old nodes must converge on (the polkit and audio precedents, formalized).

4.1 KiB Raw Blame History