lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

fix(orchestrator): order only live containers on package start/restart

Browse Source 
package.restart resolved its container list via
ordered_containers_for_start, which injected every name from the
union startup_order list that wasn't already present — including
variant names not live on a given node (mysql-mempool,
archy-mempool-api, archy-mempool-web). The phantom mysql-mempool is
2nd in the mempool start order, so do_orchestrator_package_start hit
its unknown-app-id fallback, do_package_start failed the inspect
("no such object"), and the `?` aborted the whole start sequence —
leaving mempool-api + the frontend down until the health monitor
recovered them minutes later. That was the source of the 5× gate
flakes #73 (frontend not running in 180s) and #74 (api not queryable
in 300s); root-caused from the .228 journal
("Start failed: mysql-mempool").

Replace the inject-then-sort logic with a pure helper
order_present_containers that orders only the actually-present
containers and never adds phantom entries. startup_order remains a
union of name variants across install generations — it's now used
purely to order what's live, not to inject what isn't. +3 unit tests.

Also harden bitcoin-knots.bats "valid state" probe: poll ≤30s for a
settled state instead of a single-shot read, so a container caught
mid-reconcile (transient restarting/configured) can't flake a 20-min
iteration. A genuinely-stuck container never settles, so real
breakage is still caught.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-06-23 02:22:50 -04:00
co-authored by Claude Opus 4.8
parent 57a013bc66
commit 92d7f52dd6
2 changed files with 89 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/lifecycle/bats/bitcoin-knots.bats
+15 -5
View File
@@ -36,11 +36,21 @@ teardown_file() {
}
@test "container-list reports a valid state for bitcoin-knots" {
run rpc_result container-list
[ "$status" -eq 0 ]
local state
state=$(echo "$output" | jq -r '.[] | select(.name == "bitcoin-knots") | .state')
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]]
# Poll briefly: a container caught mid-reconcile can momentarily report a
# transient state ("restarting"/"configured"/"removing") or no state at all.
# A genuinely-stuck container never settles, so this still catches real
# breakage; it only absorbs churn (e.g. another container bouncing right
# before the read-only tier runs).
local state="" deadline=$(( $(date +%s) + 30 ))
while (( $(date +%s) < deadline )); do
run rpc_result container-list
[ "$status" -eq 0 ]
state=$(echo "$output" | jq -r '.[] | select(.name == "bitcoin-knots") | .state')
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]] && return 0
sleep 3
done
echo "bitcoin-knots never reported a settled valid state within 30s (last: '$state')" >&2
return 1
}
@test "container-status returns a valid status object for bitcoin-knots" {