archipelago
cbd463e980
kdump + rasdaemon on every node, per docs/kdump-rasdaemon-design.md with the approved decisions: hang capture ON (a wedged kiosk dumps and reboots itself instead of sitting dead), crashkernel=256M, backfill ships with this release, phase-2 UI surfacing deferred. Host fixups (docs/system-level-ota-design.md) are the general answer to 'deliver system-level updates OTA': curated OS packages, sysctl drop-ins, service enablement and the GRUB crashkernel line, carried by the signed binary and applied idempotently at startup — non-fatal by construction (offline/locked-dpkg nodes converge on a later boot), skipped on dev boxes and non-Debian hosts. This formalizes the polkit/audio repair precedents into a channel with a stated policy: pinned packages and parameter intent only, never dist-upgrade automation; the ISO bakes the identical end state into fresh installs (next commit). The one runtime limitation is honest: crashkernel memory can only be reserved at boot, so the fixup writes GRUB, runs update-grub, and logs that it takes effect on the next reboot. tests/lifecycle/os-audit.sh gains section D — a graded baseline check: FAIL if capture never landed, WARN if written but awaiting reboot, PASS when reserved, policy live and rasdaemon recording. Section D runs independently of RPC health: a wedged backend must not mask that the node also stopped capturing evidence. Verification: host_fixups unit tests 4/4; cargo fmt clean; full suite runs in the release gate (create-release) and the archi-dev-box lifecycle gate before the tag.
83 lines
4.1 KiB
Markdown
83 lines
4.1 KiB
Markdown
# System-Level OTA — host fixups
|
|
|
|
Status: Implemented (first payload shipped alongside this doc)
|
|
Owner: `core/archipelago/src/host_fixups.rs`
|
|
Related: docs/kdump-rasdaemon-design.md (first payload), CLAUDE.md invariants
|
|
|
|
## The problem
|
|
|
|
The binary OTA updates the node's own software, and the signed app catalog
|
|
updates apps. But the **host OS** — Debian packages, kernel parameters,
|
|
system services — previously moved only through ISO re-installs. A node
|
|
deployed a year ago can be running today's node software on a host that
|
|
never gained anything the image learned since. Issue #99 (missing polkit
|
|
rule on old nodes) and the audio-stack heal were each hand-carved
|
|
one-off bootstrap repairs; there was no general channel and no stated
|
|
policy for touching the host from the node.
|
|
|
|
## The mechanism
|
|
|
|
`host_fixups::ensure_host_fixups()` — spawned from `main.rs` at startup
|
|
alongside the other `ensure_*` heals, in the background, best-effort:
|
|
|
|
1. **Dev-box guard** — skip when `/home/archipelago/archy` is a symlink
|
|
(contributor checkout) and when there's no dpkg (non-Debian host).
|
|
2. **Packages** — install only what's missing, from a curated, in-code
|
|
list (`HOST_PACKAGES`), `apt-get install` first, one `apt-get update`
|
|
retry, both under timeout, never fatal (offline/locked-dpkg nodes
|
|
converge on a later boot).
|
|
3. **Configuration** — idempotent per-concern helpers writing root-owned
|
|
config (via the existing `host_sudo` path): sysctl drop-ins, service
|
|
defaults, GRUB cmdline, service enablement.
|
|
4. **Reporting** — every step logs what it did; failures log warnings and
|
|
move on. A host fixup must never be able to stop the node from starting.
|
|
|
|
### Why embedded-in-the-binary rather than fetched
|
|
|
|
Same reasoning as the tor-helper (`bootstrap.rs`): the signed binary OTA
|
|
is the only authenticated delivery channel every node already trusts and
|
|
pulls on schedule. Fixups compiled into the binary travel with a version,
|
|
are reviewable in git, and can't be served to a subset of the fleet.
|
|
|
|
## Policy — what may travel this channel
|
|
|
|
| May | May not |
|
|
|---|---|
|
|
| Specific, pinned packages the node needs (kdump-tools, rasdaemon, …) | `dist-upgrade` or silent kernel/libc swaps — regular Debian upgrades stay with the operator |
|
|
| Kernel *parameters* via GRUB/sysctl — with the next-reboot caveat logged loudly | Anything requiring a secret, or touching LUKS key material |
|
|
| Service enablement + config the image also bakes in | Divergence: the ISO must converge to the SAME end state so fresh installs are a no-op |
|
|
| Small, reviewable, per-concern Rust functions with tests | Shell-script-of-things payloads beyond a single concern |
|
|
|
|
The rule: **the ISO and the fixup must express the same intent twice,
|
|
in reviewable places** — Dockerfile.rootfs/auto-install.sh for fresh
|
|
installs, `host_fixups.rs` for the deployed fleet. A change that lands in
|
|
one and not the other is a bug.
|
|
|
|
## Kernel cmdline caveat
|
|
|
|
`crashkernel=` (and any future `hugepages=`-style reservation) only takes
|
|
effect at boot: the fixup writes `/etc/default/grub` + `update-grub` and
|
|
logs `takes effect on the NEXT reboot`. Operators reboot nodes when
|
|
applying releases; no special ceremony is required beyond that, but the
|
|
lifecycle gate grades this state honestly (WARN for written-but-not-yet-
|
|
rebooted, FAIL for never-written — see `tests/lifecycle/os-audit.sh`
|
|
section D).
|
|
|
|
## Verification story
|
|
|
|
- Unit tests pin the policy constants and script shapes
|
|
(`host_fixups` tests in `core/archipelago`).
|
|
- `tests/lifecycle/os-audit.sh` section D asserts the end state on a real
|
|
node (config present, crashkernel reserved or pending reboot, hang
|
|
policy live, rasdaemon active).
|
|
- The lifecycle gate runs on archi-dev-box per release; the QEMU ISO
|
|
smoke covers fresh installs.
|
|
|
|
## Future payloads (candidates, not commitments)
|
|
|
|
- `unattended-upgrades` posture + a default-deny host nftables ruleset
|
|
(the §F hardening-plan item — needs its own design first).
|
|
- Host firewall rules for mesh/WG ports.
|
|
- Chronic: anything the image learns post-deploy that old nodes must
|
|
converge on (the polkit and audio precedents, formalized).
|