archy/docs/system-level-ota-design.md

Historical system-level OTA — narrow host fixups

Status: Historical narrow host-fixup implementation. The new protected OS path has separate candidate code and real disposable Debian/kernel transaction tests. General installed-node/release acceptance remains open, including the actual deployed data-root custody prerequisite; see the migration report for exact scope. Owner: core/archipelago/src/host_fixups.rs Related: docs/kdump-rasdaemon-design.md (first payload), CLAUDE.md invariants

Current scope clarification (2026-11-03)

The mechanism below is not a general OS transaction executor. In particular its policy explicitly excludes dist-upgrade and silent kernel/libc replacement. An ISO reinstall is not inherently necessary to install a protected updater on an existing Debian system. Where the old updater cannot establish independent root custody, a one-time installation through independently verified existing administrator access is required instead. That is an administrative migration, not a reinstall and not proof of seamless OTA support.

scripts/os_updater_bootstrap.py prepares an inert, pinned code/control generation through that administrator channel. scripts/os_updater_activate.py then delegates one-shot installation/admission and prerequisite activation to the actual protected D09 owner using a separate independently authenticated approval. Subsequent OS effects belong to the admitted D09 worker, never the bootstrap helper. Neither helper restores application/financial state. General no-reinstall support remains gated on the OS-OTA contract, immutable artifacts and disposable old-installation → new-owner → actual package/kernel transaction/recovery tests, followed by separately authorized installed-node acceptance. See docs/INPLACE-OS-OTA-MIGRATION-20261103.md for the precise support matrix.

The problem

The binary OTA updates the node's own software, and the signed app catalog updates apps. But the host OS — Debian packages, kernel parameters, system services — previously moved only through ISO re-installs. A node deployed a year ago can be running today's node software on a host that never gained anything the image learned since. Issue #99 (missing polkit rule on old nodes) and the audio-stack heal were each hand-carved one-off bootstrap repairs; there was no general channel and no stated policy for touching the host from the node.

The mechanism

host_fixups::ensure_host_fixups() — spawned from main.rs at startup alongside the other ensure_* heals, in the background, best-effort:

1. Dev-box guard — skip when /home/archipelago/archy is a symlink (contributor checkout) and when there's no dpkg (non-Debian host).
2. Packages — install only what's missing, from a curated, in-code list (HOST_PACKAGES), apt-get install first, one apt-get update retry, both under timeout, never fatal (offline/locked-dpkg nodes converge on a later boot).
3. Configuration — idempotent per-concern helpers writing root-owned config (via the existing host_sudo path): sysctl drop-ins, service defaults, GRUB cmdline, service enablement.
4. Reporting — every step logs what it did; failures log warnings and move on. A host fixup must never be able to stop the node from starting.

Why embedded-in-the-binary rather than fetched

Same reasoning as the tor-helper (bootstrap.rs): the signed binary OTA is the only authenticated delivery channel every node already trusts and pulls on schedule. Fixups compiled into the binary travel with a version, are reviewable in git, and can't be served to a subset of the fleet.

Policy — what may travel this channel

May	May not
Specific, pinned packages the node needs (kdump-tools, rasdaemon, …)	dist-upgrade or silent kernel/libc swaps — regular Debian upgrades stay with the operator
Kernel parameters via GRUB/sysctl — with the next-reboot caveat logged loudly	Anything requiring a secret, or touching LUKS key material
Service enablement + config the image also bakes in	Divergence: the ISO must converge to the SAME end state so fresh installs are a no-op
Small, reviewable, per-concern Rust functions with tests	Shell-script-of-things payloads beyond a single concern

The rule: the ISO and the fixup must express the same intent twice, in reviewable places — Dockerfile.rootfs/auto-install.sh for fresh installs, host_fixups.rs for the deployed fleet. A change that lands in one and not the other is a bug.

Kernel cmdline caveat

crashkernel= (and any future hugepages=-style reservation) only takes effect at boot: the fixup writes /etc/default/grub + update-grub and logs takes effect on the NEXT reboot. Operators reboot nodes when applying releases; no special ceremony is required beyond that, but the lifecycle gate grades this state honestly (WARN for written-but-not-yet- rebooted, FAIL for never-written — see tests/lifecycle/os-audit.sh section D).

Verification story

• Unit tests pin the policy constants and script shapes (host_fixups tests in core/archipelago).
• tests/lifecycle/os-audit.sh section D asserts the end state on a real node (config present, crashkernel reserved or pending reboot, hang policy live, rasdaemon active).
• The lifecycle gate runs on archi-dev-box per release; the QEMU ISO smoke covers fresh installs.

Future payloads (candidates, not commitments)

• unattended-upgrades posture + a default-deny host nftables ruleset (the §F hardening-plan item — needs its own design first).
• Host firewall rules for mesh/WG ports.
• Chronic: anything the image learns post-deploy that old nodes must converge on (the polkit and audio precedents, formalized).