lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 1 Actions Packages Projects Releases 64 Wiki Activity

test(lifecycle): tolerate slow-but-healthy heavy-app recovery under 5x churn

Browse Source 
The 5x destructive gate on heavy nodes false-failed on transient windows
during stack recovery, not real regressions:

- immich.bats: lan_address port-publish probe 30s -> 90s. The postgres->redis
  ->server (DB migrations on boot) stack can take >30s to republish :2283 after
  a churn-induced recreate; destructive-tier immich tests already allow 180-240s.
- mempool.bats: orphan-container check now polls to steady state (<=30s) instead
  of a single-shot count, which caught a recreated member briefly visible
  alongside its replacement mid-reconcile.
- run-gate.sh: settle cap 180s -> 300s and also gate on immich's :2283 when
  installed, so the next iteration's read-only probe doesn't race a still-
  recovering stack. Settle returns the instant every probe is green.

A genuinely unexposed/orphaned/unhealthy app still fails these checks; they only
absorb the transient recreate window under sustained churn.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-06-25 09:18:34 -04:00
co-authored by Claude Opus 4.8
parent a721532f55
commit 41e7f500f8
3 changed files with 37 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/lifecycle/bats/mempool.bats
+18 -6
View File
@@ -75,12 +75,24 @@ mempool_skip_if_absent() {
}
@test "no orphan mempool-related containers beyond the known set" {
local total known
total=$(podman ps -a --format '{{.Names}}' \
| grep -Ec '^(mempool|archy-mempool)' || true)
known=$(podman ps -a --format '{{.Names}}' \
| grep -Ec '^(mempool|mempool-api|archy-mempool-db|archy-mempool-web)$' || true)
[ "$total" -eq "$known" ]
# Poll for steady state (don't single-shot): a stack restart in a prior tier
# briefly leaves a recreated member visible alongside its replacement, so a
# one-shot count can momentarily see total>known even though the reconciler
# converges within seconds. A genuine orphan never clears, so this still
# catches it — it just tolerates the transient recreate window.
local total known deadline=$(( $(date +%s) + 30 ))
while (( $(date +%s) < deadline )); do
total=$(podman ps -a --format '{{.Names}}' \
| grep -Ec '^(mempool|archy-mempool)' || true)
known=$(podman ps -a --format '{{.Names}}' \
| grep -Ec '^(mempool|mempool-api|archy-mempool-db|archy-mempool-web)$' || true)
[ "$total" -eq "$known" ] && return 0
sleep 3
done
echo "orphan mempool container persisted >30s (total=$total known=$known):" >&2
podman ps -a --format '{{.Names}}' | grep -E '^(mempool|archy-mempool)' \
| grep -vE '^(mempool|mempool-api|archy-mempool-db|archy-mempool-web)$' >&2 || true
return 1
}
# ────────────────────────────────────────────────────────────────────