diff --git a/docs/README.md b/docs/README.md index 982a46a6..2bb021f0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,18 +22,15 @@ disagree, the code wins and the doc is a bug. - [App Developer Guide](app-developer-guide.md) — build and package a containerized app - [App Manifest Specification](app-manifest-spec.md) — the manifest schema, field by field +- [Manifest → Quadlet unit](quadlet-compilation.md) — how a manifest compiles to a systemd-owned container unit +- [Container lifecycle](container-lifecycle.md) — the reconciler state machine: install/adopt/start/stop/self-heal +- [App secrets](secrets.md) — declaring, generating and injecting per-install credentials - [Registry-Distributed Manifests](registry-manifest-design.md) — how manifests reach nodes via the signed catalog - [Decentralized Marketplace Protocol](marketplace-protocol.md) — publishing apps via an external registry - [Bitcoin RPC Relay](bitcoin-rpc-relay.md) — letting an external wallet reach the node's Bitcoin RPC - [Companion Pairing QR](companion-pairing-qr.md) — the pairing handoff contract - [TV input inside iframe apps](tv-input-iframe-apps.md) — keyboard/gamepad routing into embedded apps -> **Known gap (tracked):** three app-developer references named in the -> open-source plan are not yet written — how a manifest compiles to a -> Quadlet/systemd unit, the container-lifecycle reconciler state machine, and -> the `generated_secrets` materialisation flow. Until they land, the source of -> truth is the code: `core/archipelago/src/container/quadlet*.rs`, -> `prod_orchestrator.rs`, and `container::secrets` respectively. ## Design docs diff --git a/docs/container-lifecycle.md b/docs/container-lifecycle.md new file mode 100644 index 00000000..2e04b7cc --- /dev/null +++ b/docs/container-lifecycle.md @@ -0,0 +1,108 @@ +# Container lifecycle + +How Archipelago keeps apps in the state you asked for — install, start, stop, +restart, uninstall — and how it self-heals without ever resurrecting something +you deliberately stopped. Source of truth: +`core/archipelago/src/container/prod_orchestrator.rs` and +`core/archipelago/src/container/boot_reconciler.rs`. + +## The model: level-triggered, not fire-and-forget + +Archipelago does not start a container and hope. A long-running **reconciler** +compares *desired state* (what the manifests and your explicit choices say +should be running) against *actual state* (what podman reports) and repairs the +difference. It is **level-triggered**: it acts on the current gap every tick, not +on a one-time event, so a container that dies, a unit that vanishes, or a reboot +that clears everything are all just "the gap is non-zero, close it". + +The reconciler is spawned once at boot (`BootReconciler`) after an initial +`adopt_existing()` pass, and runs every **30 seconds**. It finishes an in-flight +pull or build before honouring a shutdown signal — it is never interrupted +mid-operation. + +Concurrency: each app has its own async mutex guarding all mutating operations +against the reconciler, so a manual `stop` and a reconcile tick can't race, but +reconciles across different apps still run without serialising against each +other. + +## Desired state has three inputs + +For each app the reconciler asks: *should this be running right now?* The answer +comes from three durable signals, checked in this order: + +1. **Explicitly user-stopped** (`user-stopped.json`). If you stopped an app, its + id is recorded and the reconciler leaves it down — it is **not** a gap to + repair. Cleared when you start it again. This is what makes a stop *stick* + across restarts and reboots. +2. **Explicitly uninstalled** (`user-uninstalled.json`). Same idea for uninstall: + a baseline app you removed stays removed, so self-heal can't reinstall it. +3. **Otherwise, the manifest set** — every catalog/disk app that isn't stopped or + uninstalled should be running. + +Dependencies are pulled in: an app that is up requires its declared +dependencies, so they are kept up too — but a dependency you explicitly stopped +still stays stopped. + +## The operations + +All go through the orchestrator, all take the per-app lock, all are idempotent: + +| Operation | What it does | +|-------------|--------------| +| **adopt** | At boot, take ownership of a pre-existing container **by name** rather than recreating it — preserves data, ports and identity across a daemon restart. | +| **install** | Materialise secrets → ensure image (build from a local Dockerfile or use a pre-pulled image) → create and start the container (via Quadlet where enabled). | +| **start / stop** | Bring the container up/down and record the desired-state change. A stop writes the app to `user-stopped.json`. | +| **restart** | Stop then start, preserving the container's data and identity. | +| **remove** | Stop and remove the container, **preserving `/var/lib/archipelago/`, secrets, credentials and ports** — a reinstall or upgrade lands on the same data. | +| **upgrade** | Recreate at a new image while preserving data (see the version rules below). | +| **health** | Report the container's health from its declared `health_check`. | + +## Self-heal vs. respecting your choice + +The one rule that ties it together: **self-heal must never override a deliberate +stop or uninstall.** + +- A container that disappeared while its siblings run — a wedged teardown, a + reboot that cleared it — is a hole to repair, and the reconciler rebuilds it + from the durable "was running" snapshot. +- A container that is down because you stopped or uninstalled it is a *choice*, + and the reconciler leaves it alone. + +A small set of **baseline apps** are expected to exist from first boot and +self-heal when their container is missing — but the `user_stopped` / +`user_uninstalled` gates are checked first, so even a baseline app you turned off +stays off. Getting this wrong in either direction is a real bug: resurrecting a +stopped app ignores the operator, and failing to rebuild a crashed one is the +fire-and-forget failure the whole design exists to remove. + +## Migrations never destroy data + +Any recreate path — upgrade, reinstall, repair — preserves the app's data +directory, its generated secrets, its credentials, its ports, and the container +name used for adoption. An update that would roll a version *backwards* is +refused (see the version guard in `container::image_versions`): the update button +never offers a lower version than what is running, so a stale record cannot turn +into a downgrade. Version pins are honoured — a pinned app is not "updated" out +from under the operator by the catalog. + +## Inspecting lifecycle state + +Run as the archipelago service user: + +```bash +# what podman actually has +podman ps -a --format '{{.Names}}\t{{.Status}}' + +# the durable desired-state signals +cat /var/lib/archipelago/user-stopped.json +cat /var/lib/archipelago/user-uninstalled.json + +# the reconciler's decisions +journalctl --user -u archipelago | grep -iE 'reconcile|adopt|install|user.stopped' +``` + +## Related + +- [Manifest → Quadlet unit](quadlet-compilation.md) — how the unit the reconciler manages is generated +- [App secrets](secrets.md) — the `ensure_generated_secrets` tick that runs before start +- [App Manifest Specification](app-manifest-spec.md) — `health_check`, `dependencies`, `restart` fields diff --git a/docs/quadlet-compilation.md b/docs/quadlet-compilation.md new file mode 100644 index 00000000..1d79872a --- /dev/null +++ b/docs/quadlet-compilation.md @@ -0,0 +1,121 @@ +# Manifest → Quadlet unit + +How an app manifest becomes a Podman [Quadlet](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) +`.container` unit that systemd owns, where the unit lands, and how to inspect one. +Source of truth: `core/archipelago/src/container/quadlet.rs`. + +## Why Quadlet + +Containers used to be fire-and-forget `tokio::spawn` blocks. If the daemon +crashed mid-spawn or the kernel reaped a parent cgroup, the container vanished +from `podman ps` and only a manual `podman run` brought it back. Quadlet removes +that whole class of failure: the unit lives on disk, **systemd owns +start/restart, and archipelago is just the provisioner**. This is the path that +runs the companion UI containers today (`archy-bitcoin-ui`, `archy-lnd-ui`, +`archy-electrs-ui`), and the validated path being flipped to default for apps. + +## What gets generated + +`Quadlet::from_manifest(manifest, name)` translates a manifest into a unit, and +`render()` produces the file. Every unit carries a header making clear it is not +hand-edited: + +```ini +# Generated by archipelago. DO NOT EDIT. +# Edits are overwritten on the next reconcile. + +[Unit] +Description= +After=network-online.target +Wants=network-online.target +Requires=.service # one per declared dependency +After=.service + +[Container] +ContainerName= +Image= +Pull=never # image must be present locally already +Network= +User= # when the manifest pins one +DropCapability=ALL # security default +AddCapability= # only capabilities the manifest opts into +PublishPort=::/ +Environment== # non-secret env only +Secret=,type=env,target= # secrets by REFERENCE, never value +Volume=: +ReadOnly=true # when security.readonly_root +NoNewPrivileges=true # when security.no_new_privileges +HealthCmd= # from the health_check block + +[Service] +TimeoutStartSec=0 +Restart= # from the restart policy +RestartSec=10 # 10s backoff caps a crash loop + +[Install] +WantedBy=default.target +``` + +Two things to note in that mapping: + +- **Secrets go in by reference, never by value.** A `secret_env` entry renders as + `Secret=,type=env,target=`, so podman injects the value at run time + from the node's secret store. The plaintext never appears in the unit file. See + [App secrets](secrets.md). +- **`Pull=never` is deliberate.** The provisioner does not pull images from here; + the image must already be local (pre-pulled or built). A missing image surfaces + immediately instead of retrying silently behind systemd's restart loop. + +## Where units land + +Rootless, per-user, under the archipelago service user (uid 1000, with linger +enabled so the units run without an active login): + +``` +~/.config/containers/systemd/.container +``` + +Quadlet's systemd generator translates `.container` into a +`.service` unit at **daemon-reload** time. Everything is `systemctl --user` +— the system bus is never touched from this path. + +## Lifecycle: render → write → enable → disable + +The module does four things and nothing else: + +1. **render** — manifest → unit text (above). +2. **write** — `tempfile + rename` so a partially-written unit is never visible to + systemd, and `write_if_changed` compares bytes first: if the rendered unit + matches what is on disk, nothing is touched — no daemon-reload, no restart + cascade. This is what makes a reconcile tick cheap and non-disruptive. +3. **enable** — `daemon-reload` then start the `.service`. +4. **disable** — stop and remove. + +## Inspecting a unit + +Run these **as the archipelago service user** (the units are in its user bus): + +```bash +# the generated unit +cat ~/.config/containers/systemd/archy-bitcoin-ui.container + +# what systemd made of it +systemctl --user cat archy-bitcoin-ui.service +systemctl --user status archy-bitcoin-ui.service +journalctl --user -u archy-bitcoin-ui.service + +# after editing a unit by hand for debugging (it will be overwritten on reconcile) +systemctl --user daemon-reload +``` + +Because the unit is regenerated on every reconcile, the way to change a +container's shape is to change its **manifest** (and, for a catalog-covered app, +regenerate and re-sign the catalog), never to edit the `.container` file — the +`DO NOT EDIT` header is literal. + +## Related + +- [Container lifecycle](container-lifecycle.md) — the reconciler that drives this +- [App Manifest Specification](app-manifest-spec.md) — the manifest fields mapped above +- [App secrets](secrets.md) — how `Secret=` references resolve +- [ADR-001: Podman over Docker](adr/001-podman-over-docker.md) diff --git a/docs/secrets.md b/docs/secrets.md new file mode 100644 index 00000000..5a761bf3 --- /dev/null +++ b/docs/secrets.md @@ -0,0 +1,107 @@ +# App secrets + +How an app declares a secret, how Archipelago materialises it, and how it +reaches the container — with the rules a developer must not break. + +The whole point: **an app never ships a credential.** It declares the *shape* of +the secrets it needs, and the node generates a fresh, per-install value that +never leaves the node and is never logged. Source of truth: +`core/archipelago/src/container/secrets.rs` and the manifest schema in +`core/container/src/manifest.rs`. + +## The two halves + +A secret has a producer and a consumer, and they are separate manifest fields: + +- **`generated_secrets`** — *produce* a random value into a file. +- **`secret_env`** — *inject* a file's contents into the container as an env var. + +An app can use either alone. A generated secret with no consumer is just a file +on the node; a `secret_env` with no matching `generated_secrets` reads a file +that some other component (or the daemon) is expected to have written. + +## Declaring a generated secret + +```yaml +container: + generated_secrets: + - name: btcpay-db-password + kind: hex16 + - name: fedimint-gateway-hash + kind: bcrypt +``` + +`name` is a **bare filename** under the node's secrets directory +(`/var/lib/archipelago/secrets/`). It is validated at manifest-load time — no +`/`, no `..` — so a manifest cannot write outside that directory. + +`kind` chooses how the value is produced. Each kind is deterministic in *shape* +(the orchestrator knows exactly which files it will create) but random in value: + +| `kind` | Value | Files written | Use for | +|---------|-----------------------------------------|-----------------------------------|---------| +| `hex16` | 16 random bytes, lowercase hex (32 ch) | `` | service passwords, API tokens | +| `hex32` | 32 random bytes, lowercase hex (64 ch) | `` | longer keys/cookies | +| `base64`| 32 random bytes, standard base64 (44 ch)| `` | services that base64-decode their key (e.g. netbird relay `authSecret`) | +| `bcrypt`| a random password **and** its bcrypt hash| `` (hash) + `.pw` (plaintext) | server configured with a hash, client needs the plaintext | + +`bcrypt` is the only kind that writes two files: `` holds the bcrypt hash a +server is configured with, and `.pw` holds the plaintext for any client +that must authenticate against it. A `secret_env` injects whichever of the two it +references. + +## Injecting a secret into the container + +```yaml +container: + secret_env: + - key: BTCPAY_DB_PASS + secret_file: btcpay-db-password +``` + +At apply time the orchestrator reads `/var/lib/archipelago/secrets/` +and sets `` in the container's environment to its contents. The value is +never written into the manifest, the Quadlet unit, or any log line. + +## How materialisation works + +`ensure_generated_secrets()` runs on **every install and reconcile tick**, before +`secret_env` is resolved. It is idempotent and self-healing: + +1. **Fast path.** If every target file for a secret already exists, is readable + by the service user, and is non-empty, it is left untouched. A secret is + generated **once** and then persists across restarts, updates and reinstalls — + this is what makes credentials stable (migrations never regenerate a working + secret out from under a database). +2. **Self-heal.** A target file that exists but is unreadable or empty — e.g. + left root-owned by a botched earlier write — is removed and recreated, owned + by the service user. The unlink uses the secrets directory's own write bit, so + recovery needs no privilege escalation. +3. **Write.** New values are written through an atomic `0600` writer: a temp file + in the same directory, fsynced, then renamed over the target, so a reader never + sees a half-written secret and the file is only ever readable by its owner. + +Because it runs every tick and no-ops when the secret is healthy, calling it is +always safe; there is no separate "provision secrets" step to forget. + +## Rules a developer must not break + +- **Never hardcode a credential**, in the manifest or in code, even as a + fallback. A shared fallback password means everyone holding a copy of the repo + holds that credential. Declare `generated_secrets` instead. +- **Never log a secret.** `secret_env` values and the files under the secrets + directory stay out of logs, error messages and status output. +- **One canonical name.** The orchestrator, first-boot script, reconcile path and + any deploy tooling must all reference a secret by the *same* filename. A + producer writing `-password` while the consumer reads `-hash` yields a + service that authenticates against a credential nothing generated. +- **Pick the encoding the service expects.** `hex*` and `base64` decode to + different bytes; a service that base64-decodes its configured key must be given + a `base64` secret, or it will run with the wrong key material. + +## Related + +- [App Manifest Specification](app-manifest-spec.md) — the full manifest schema +- [ADR-009: Manifest-Level Container Security](adr/009-manifest-container-security.md) +- [Entropy Enforcement (KEY-05)](security/KEY-05-ENTROPY-ENFORCEMENT.md) — why secret + generation draws from an explicitly-named CSPRNG