lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

fix(recovery): quiet the fleet logs — skip running containers, back off failed companion repairs
Demo images / Build & push demo images (push) Failing after 32s
Details

Browse Source 
Fleet log sweep (2026-07-22): (1) recovery paths podman-start'ed
containers that were already running, producing hundreds of benign but
scary conmon 'Failed to create container' + cgroup Permission-denied
pairs per node per day — both paths now check state first; (2) the 30s
companion-reconcile loop retried registry pulls/builds forever against
unreachable registries (~174×/image/day on an offline node) — failing
rounds now back off exponentially to a 1h cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-22 19:44:32 -04:00
co-authored by Claude Fable 5
parent b193e64ffb
commit 6347d16a2f
2 changed files with 37 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/container/boot_reconciler.rs
+19 -3
View File
@@ -108,17 +108,33 @@ impl BootReconciler {
let orchestrator = self.orchestrator.clone();
let interval = self.interval;
Some(tokio::spawn(async move {
let mut failure_rounds: u32 = 0;
loop {
let installed = orchestrator.manifest_ids().await;
for (companion, err) in crate::container::companion::reconcile(&installed).await
{
let failures =
crate::container::companion::reconcile(&installed).await;
for (companion, err) in &failures {
tracing::warn!(
companion = %companion,
error = %err,
"companion reconcile failed"
);
}
time::sleep(interval).await;
// A failed repair can involve registry pulls and full
// image builds; retrying every 30s hammered unreachable
// registries ~174×/image/day on an offline node
// (archy-x250-dev log sweep, 2026-07-22). Back off
// exponentially while rounds keep failing — 30s doubling
// to a 1h cap — and reset the moment a round is clean.
failure_rounds = if failures.is_empty() {
0
} else {
failure_rounds.saturating_add(1)
};
let backoff = interval
.saturating_mul(2u32.saturating_pow(failure_rounds.min(7)))
.min(Duration::from_secs(3600));
time::sleep(backoff).await;
}
}))
} else {