lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 4 Actions Packages Projects Releases 71 Wiki Activity

refactor(container): move companion UIs to systemd via Quadlet

Browse Source 
Companion UI containers (archy-bitcoin-ui, archy-lnd-ui,
archy-electrs-ui) used to be launched as fire-and-forget tokio::spawn
blocks from install.rs. If archipelago crashed mid-spawn or the
container's cgroup was reaped, companions vanished from podman ps -a
and only a manual rm/run could bring them back (the .228 incident).

Now each companion is rendered as a Quadlet .container unit under
~/.config/containers/systemd/, daemon-reloaded, and started via
systemctl --user. systemd owns supervision from that point on:

- archipelago can crash, restart, or be uninstalled without touching
  any companion.
- Quadlet's Restart=always + RestartSec=10 handles container exits.
- A 30s reconcile tick in boot_reconciler enumerates expected
  companion units and re-installs any whose unit file or service
  vanished — defense-in-depth against external tampering.

New module layout:
- container/quadlet.rs: pure unit renderer + atomic write_if_changed
  + systemctl helpers (daemon_reload_user / enable_now / disable_remove
  / is_active). 6 unit tests, no I/O in the renderer.
- container/companion.rs: per-app companion specs, install/remove/
  reconcile, image presence (build local first, fall back to insecure
  registry only via image_uses_insecure_registry whitelist). 2 tests.

install.rs handle_package_install now ends with a single call to
companion::install_for(package_id), replacing 287 lines of spawn-and-
hope shellouts plus a ~120-line nginx auth-injector helper that worked
around per-node RPC password baking. The helper is gone too — the
pre-start hook renders the per-node nginx.conf to /var/lib/archipelago/
bitcoin-ui/nginx.conf and the Quadlet unit bind-mounts it read-only.

runtime.rs handle_package_uninstall now disables companions before
the container rm loop. Otherwise systemd's Restart=always would
respawn each companion within ~10s of removal.

Tests: 53 container tests pass, including 6 quadlet renderer tests
(host network, bridge network, capability set, atomic write idempotence)
and 2 companion specs (per-app companion lookup, build_unit shape).
boot_reconciler tests gain a #[cfg(test)] without_companion_stage()
flag so the paused-clock fixtures don't race the real systemctl I/O.

A bats regression test (companion-survives-archipelago-restart.bats,
gated on ARCHY_ALLOW_DESTRUCTIVE=1) asserts the .228 failure mode
cannot recur: every installed companion has a unit file, services
stay active across systemctl --user restart archipelago, and a
deleted unit file is recreated within one reconcile tick.

Net delta: +941 / -363, but the +941 is mostly tests (~440 lines)
and the new declarative layer; the imperative tokio::spawn block and
its nginx-auth helper are gone, removing two failure classes
(orphan companions on archipelago crash, and post-start exec races
under tightly-confined cgroups) that previously needed manual SSH
recovery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-05-01 10:45:07 -04:00
co-authored by Claude Opus 4.7
parent 2bf8181110
commit 23c4e7441f
9 changed files with 941 additions and 363 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/api/rpc/package/install.rs
+9 -353
View File
@@ -32,130 +32,6 @@ pub(in crate::api::rpc) async fn install_log(msg: &str) {
}
}
/// Patch the Bitcoin RPC `Authorization: Basic ...` header inside the running
/// bitcoin-ui container's nginx config and reload nginx. Authoritative
/// credential injection — runs whether the image was built locally or pulled
/// from the registry. Without this, registry images ship with whatever auth
/// header was baked at build time on the publisher's machine, which never
/// matches the per-node randomly-generated bitcoin-rpc-password.
///
/// Implementation note: this used to do `podman exec sed`, but rootless
/// podman + tightly-confined containers (--cap-drop=ALL, restricted user)
/// reject the exec because crun can't add a new process to the container's
/// cgroup ("write cgroup.procs: Permission denied"). Switched to
/// `podman cp` (storage layer, no cgroup join) + `podman kill --signal=SIGHUP`
/// (signal to existing PID 1, no new process needed). Verified on .228.
async fn inject_bitcoin_rpc_auth_into_running_container(container: &str, auth_b64: &str) {
use rand::distributions::{Alphanumeric, DistString};
let token = Alphanumeric.sample_string(&mut rand::thread_rng(), 8);
let host_path = format!("/tmp/archy-{container}-nginx.conf-{token}");
let in_container = "/etc/nginx/conf.d/default.conf";
// 1. Copy the running config out to host
let cp_out = tokio::process::Command::new("podman")
.args(["cp", &format!("{container}:{in_container}"), &host_path])
.output()
.await;
if let Err(e) = cp_out {
warn!("inject auth: podman cp out failed for {}: {}", container, e);
return;
}
if let Ok(ref o) = cp_out {
if !o.status.success() {
warn!(
"inject auth: podman cp out failed for {}: {}",
container,
String::from_utf8_lossy(&o.stderr)
);
return;
}
}
// 2. Patch the auth line on disk
let content = match tokio::fs::read_to_string(&host_path).await {
Ok(c) => c,
Err(e) => {
warn!("inject auth: read {} failed: {}", host_path, e);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
};
let mut patched_any = false;
let updated: String = content
.lines()
.map(|line| {
if line.contains("proxy_set_header Authorization") && line.contains("Basic") {
patched_any = true;
format!(
" proxy_set_header Authorization \"Basic {}\";",
auth_b64
)
} else {
line.to_string()
}
})
.collect::<Vec<_>>()
.join("\n");
if !patched_any {
warn!(
"inject auth: no Authorization line matched in {}'s nginx.conf",
container
);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
if let Err(e) = tokio::fs::write(&host_path, format!("{}\n", updated)).await {
warn!("inject auth: write back failed: {}", e);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
// 3. Copy patched config back into the container
let cp_in = tokio::process::Command::new("podman")
.args(["cp", &host_path, &format!("{container}:{in_container}")])
.output()
.await;
let _ = tokio::fs::remove_file(&host_path).await;
match cp_in {
Ok(o) if !o.status.success() => {
warn!(
"inject auth: podman cp in failed for {}: {}",
container,
String::from_utf8_lossy(&o.stderr)
);
return;
}
Err(e) => {
warn!("inject auth: podman cp in errored for {}: {}", container, e);
return;
}
_ => {}
}
// 4. Reload nginx via SIGHUP to PID 1 (no exec/cgroup join needed)
let reload = tokio::process::Command::new("podman")
.args(["kill", "--signal=SIGHUP", container])
.output()
.await;
match reload {
Ok(o) if o.status.success() => {
info!(
"Injected Bitcoin RPC auth into {} (post-start, cp+SIGHUP)",
container
);
}
Ok(o) => warn!(
"Patched nginx.conf in {} but SIGHUP failed: {}",
container,
String::from_utf8_lossy(&o.stderr)
),
Err(e) => warn!(
"Patched nginx.conf in {} but SIGHUP errored: {}",
container, e
),
}
}
impl RpcHandler {
/// Install a package from a Docker image.
/// Security: Image verification, resource limits, network isolation.
@@ -1552,235 +1428,15 @@ autopilot.active=false\n",
info!("Nextcloud trusted domains configured for {}", host_ip);
}
// Inject Bitcoin RPC auth into bitcoin-ui nginx.conf.
// Two paths because the credential is per-node and randomly generated
// at first boot, so it can't be baked into the published registry image:
// 1. Build-time: rewrite nginx.conf on disk before `podman build`.
// Only fires when /opt/archipelago/docker/bitcoin-ui exists (dev
// box or ISO that shipped the docker tree). Skipped silently in
// production where ui_builds falls through to the registry image.
// 2. Post-start: `podman exec` into the running container to patch
// nginx.conf and reload. Authoritative for both paths — runs
// regardless of how the image was built.
let bitcoin_rpc_auth_b64: Option<String> = if matches!(
package_id,
"bitcoin" | "bitcoin-core" | "bitcoin-knots"
) {
let (rpc_user, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
use base64::Engine;
let auth_b64 = base64::engine::general_purpose::STANDARD
.encode(format!("{}:{}", rpc_user, rpc_pass));
for dir in [
"/opt/archipelago/docker/bitcoin-ui",
"/home/archipelago/archy/docker/bitcoin-ui",
] {
let conf_path = format!("{}/nginx.conf", dir);
match tokio::fs::read_to_string(&conf_path).await {
Ok(content) => {
let updated = content
.replace("__BITCOIN_RPC_AUTH__", &auth_b64)
.lines()
.map(|line| {
if line.contains("proxy_set_header Authorization")
&& line.contains("Basic")
{
format!(
" proxy_set_header Authorization \"Basic {}\";",
auth_b64
)
} else {
line.to_string()
}
})
.collect::<Vec<_>>()
.join("\n");
if let Err(e) = tokio::fs::write(&conf_path, format!("{}\n", updated)).await
{
warn!(
"Failed to write {} with injected RPC auth: {}",
conf_path, e
);
} else {
info!("Injected Bitcoin RPC auth into {} (build-time)", conf_path);
}
}
Err(_) => {
debug!(
"No build-time nginx.conf at {} (will patch running container after start)",
conf_path
);
}
}
}
Some(auth_b64)
} else {
None
};
// Build and start companion UI containers for headless services.
// All UIs proxy to localhost (backend :5678 or bitcoin :8332) so they need --network=host.
let ui_builds: Vec<(&str, &str, &str)> = match package_id {
"bitcoin" | "bitcoin-core" | "bitcoin-knots" => {
vec![(
"archy-bitcoin-ui",
"/opt/archipelago/docker/bitcoin-ui",
"bitcoin-ui",
)]
}
"lnd" => {
vec![("archy-lnd-ui", "/opt/archipelago/docker/lnd-ui", "lnd-ui")]
}
"electrumx" | "electrs" | "mempool-electrs" => {
vec![(
"archy-electrs-ui",
"/opt/archipelago/docker/electrs-ui",
"electrs-ui",
)]
}
_ => vec![],
};
for (name, ui_dir, image_base) in ui_builds {
let name = name.to_string();
// Check multiple paths: /opt (production), project tree (dev)
let ui_dir = [
ui_dir.to_string(),
format!("/home/archipelago/archy/docker/{}", image_base),
format!("/home/archipelago/Projects/archy/docker/{}", image_base),
]
.into_iter()
.find(|d| std::path::Path::new(d).join("Dockerfile").exists())
.unwrap_or_else(|| ui_dir.to_string());
let image_base = image_base.to_string();
let registry = "146.59.87.168:3000/lfg2025";
let registry_image = format!("{}/{}:latest", registry, image_base);
let local_image = format!("localhost/{}:latest", image_base);
let post_start_auth = if name == "archy-bitcoin-ui" {
bitcoin_rpc_auth_b64.clone()
} else {
None
};
tokio::spawn(async move {
// Remove existing container
let _ = tokio::process::Command::new("podman")
.args(["rm", "-f", &name])
.output()
.await;
// Build locally first (templates may have injected credentials),
// fall back to registry only if no local Dockerfile exists.
let image = {
if std::path::Path::new(&ui_dir).exists() {
info!("Building {} locally from {}", name, ui_dir);
let build = tokio::process::Command::new("podman")
.args(["build", "--no-cache", "-t", &local_image, &ui_dir])
.output()
.await;
match build {
Ok(o) if o.status.success() => local_image,
Ok(o) => {
warn!(
"Failed to build {}: {}",
name,
String::from_utf8_lossy(&o.stderr)
);
return;
}
Err(e) => {
warn!("Failed to build {}: {}", name, e);
return;
}
}
} else {
// No local Dockerfile — try pulling from registry
let mut pull_cmd = tokio::process::Command::new("podman");
pull_cmd
.arg("pull")
.arg("--tls-verify=false")
.arg(&registry_image);
let pull = pull_cmd.output().await;
if pull.is_ok_and(|o| o.status.success()) {
info!("Pulled {} UI from registry", name);
registry_image.clone()
} else {
warn!("No local source or registry image for {} — skipping", name);
return;
}
}
};
// For bitcoin-ui specifically: render nginx.conf to host BEFORE
// starting the container, then bind-mount it. This is the durable
// fix for the bitcoin-rpc 401 — the per-node password is in the
// file before nginx ever opens it. Survives container recreate,
// image update, reboot, --restart=unless-stopped cycles, and
// doesn't need any post-start patching that could fail under
// tightly-confined cgroup permissions.
let mut bitcoin_ui_mount: Option<String> = None;
if name == "archy-bitcoin-ui" {
let paths = crate::container::bitcoin_ui::RenderPaths::default();
match crate::container::bitcoin_ui::render(&paths).await {
Ok(outcome) => {
bitcoin_ui_mount = Some(format!(
"{}:/etc/nginx/conf.d/default.conf:ro,Z",
paths.rendered_path.display()
));
info!(
"bitcoin-ui nginx.conf rendered ({:?}) — will bind-mount at startup",
outcome
);
}
Err(e) => warn!(
"Failed to render bitcoin-ui nginx.conf: {} — \
will fall back to post-start patch (less reliable)",
e
),
}
}
// Run with --network=host (UIs proxy to localhost backend/bitcoin)
// --user 0:0: run as root inside container (still unprivileged on host
// in rootless podman) to avoid nginx chown failures
let mut args: Vec<String> = vec![
"run".into(),
"-d".into(),
"--name".into(),
name.clone(),
"--restart=unless-stopped".into(),
"--network=host".into(),
"--user=0:0".into(),
"--cap-drop=ALL".into(),
"--cap-add=CHOWN".into(),
"--cap-add=DAC_OVERRIDE".into(),
"--cap-add=NET_BIND_SERVICE".into(),
"--cap-add=SETUID".into(),
"--cap-add=SETGID".into(),
"--memory=128m".into(),
];
if let Some(ref mount) = bitcoin_ui_mount {
args.push("-v".into());
args.push(mount.clone());
}
args.push(image.clone());
let run = tokio::process::Command::new("podman")
.args(&args)
.output()
.await;
match run {
Ok(o) if o.status.success() => {
info!("{} UI container started (host network)", name);
if let Some(ref auth) = post_start_auth {
inject_bitcoin_rpc_auth_into_running_container(&name, auth).await;
}
}
Ok(o) => warn!(
"Failed to start {}: {}",
name,
String::from_utf8_lossy(&o.stderr)
),
Err(e) => warn!("Failed to start {}: {}", name, e),
}
});
// Companion UIs (archy-bitcoin-ui, archy-lnd-ui, archy-electrs-ui)
// are now Quadlet-managed: install_for writes ~/.config/containers/
// systemd/<name>.container, daemon-reloads, and starts the generated
// .service. systemd owns supervision from there — companions survive
// archipelago crashes, restarts, and OOM kills. Per-node config
// (e.g. bitcoin-ui's nginx.conf with the live RPC auth) is rendered
// by each spec's pre_start hook and bind-mounted read-only.
for (name, err) in crate::container::companion::install_for(package_id).await {
install_log(&format!("COMPANION FAIL: {name} — {err:#}")).await;
}
}
core/archipelago/src/api/rpc/package/runtime.rs
+5
View File
@@ -227,6 +227,11 @@ impl RpcHandler {
.and_then(|v| v.as_bool())
.unwrap_or(false);
// Disable + remove Quadlet companion units BEFORE the rm loop.
// Otherwise systemd's Restart=always will respawn each companion
// within ~10s of `podman rm`, leaving them orphaned post-uninstall.
crate::container::companion::remove_for(package_id).await;
let containers_to_remove = get_containers_for_app(package_id).await?;
if containers_to_remove.is_empty() {
tracing::warn!("Uninstall {}: no containers found", package_id);