fix: wallet receive reliability, bitcoin install self-heal, ElectrumX app tile
Fixes three Bitcoin/wallet failures observed across the fleet on v1.7.90-alpha
(all nodes were already on the latest build — these were live bugs, not stale
builds), plus the missing ElectrumX tile, and adds automated coverage so each
can't regress silently.
Receive address (".116 receive fails", ".228 false 'wallet is locked'"):
- LND publishes its REST API on a host port that can drift from the manifest
(a container created when the mapping was 8080 kept publishing 8080 after the
manifest moved to 18080). The in-process client connects to the manifest port,
gets connection-refused, and wallet init fails forever while the container
looks "Up". Add published-port drift detection to the reconciler
(container_ports_drifted / host_port_bindings_drifted) that recreates a
drifted backend even for restart-sensitive apps — a drifted container is
already broken, so leaving it "untouched" only perpetuates the failure.
- Receive errors now carry a stable [CODE] token (REST_UNREACHABLE, WALLET_LOCKED,
WALLET_UNINITIALIZED, SYNCING) and always start with "Bitcoin address" so they
survive the RPC error sanitizer instead of collapsing to the generic
"Operation failed". The UI maps the code instead of guessing wallet state from
substrings — so an unreachable REST endpoint is no longer mislabelled "locked".
Bitcoin install (".198 bitcoin gone / reinstall just stops"):
- bitcoin-knots requires the secret bitcoin-rpc-txrelay-rpcauth, which was only
generated by the tx-relay flow. Nodes that never used tx-relay lacked it, so
secret resolution hard-failed and the whole Bitcoin stack cascaded. Generate
it idempotently before bitcoin starts (ensure_app_secrets, reusing
ensure_txrelay_credentials), and name the missing secret in the error so a
genuine gap is actionable instead of a bare "IO error".
ElectrumX app tile missing on every node with it installed:
- The catalog generator dropped electrumx because the manifest had no
interfaces.main block, so the tile had no launch URL and was hidden. Declare
the companion UI port (50002) in the manifest, regenerate the catalog, and let
an app with a known launch URL stay launchable while its backend is still
"starting" (ElectrumX indexes for 10m+).
Test harness:
- New lifecycle bats suites: bitcoin-receive, port-drift, secret-completeness
(validated live; port-drift catches the real .116 drift).
- Rust unit tests for drift detection, the receive reason-code classifier, and
the named-missing-secret error; vitest for the UI code mapping.
- create-release.sh now runs tests/release/run.sh and aborts the release on
failure — previously it ran no tests at all.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
bb808df89a
commit
0ed892a412
@@ -58,10 +58,29 @@ v1.7.52 tags.
|
||||
| L1 RPC | 70 | bitcoin-knots, lnd, electrumx, btcpay, mempool, fedimint, required-stack, package-update-smoke | ● for the 6 core apps |
|
||||
| L2 UI | 9 | ui-coverage | ● for dashboard + 7 proxy paths + bitcoin-ui:8334 |
|
||||
| L3 lifecycle survival | 14 | companion-survives-archipelago-restart, backend-survives-archipelago-restart, required-stack-destructive, use-quadlet-backends-install | ◐ companions ● ; backends ◐ regression-gate (will fail until Phase 3 Quadlet ships); quadlet post-condition gate ✅ skip-clean today, hard gate when flag flipped |
|
||||
| L1 wallet-receive / drift / secrets | 5 | bitcoin-receive, port-drift, secret-completeness | ● guards the v1.7.9x wallet fleet failures |
|
||||
| L4 browser journey | 0 | none | ○ not started |
|
||||
| L5 chaos | 0 | none | ○ not started |
|
||||
| L6 performance | 0 | none | ○ not started |
|
||||
|
||||
### Wallet / Bitcoin fleet-failure regression suites (added after v1.7.90-alpha)
|
||||
|
||||
Three production failures shipped on v1.7.90-alpha despite the existing harness,
|
||||
because nothing exercised the receive path, port-mapping drift, or secret
|
||||
completeness on a live node. New suites close those gaps (all run on the archy
|
||||
host, read-only, so they join `run.sh`/`run-20x.sh` automatically):
|
||||
|
||||
| Suite | Failure it guards | Asserts |
|
||||
|---|---|---|
|
||||
| `bitcoin-receive.bats` | .116 ("Operation failed" on receive) and .228 (false "wallet is locked") | LND REST reachable on the **manifest** host port; `lnd.newaddress` returns a `bc1…` address on a running node; receive errors are specific, never the generic catch-all |
|
||||
| `port-drift.bats` | .116 (lnd REST stuck on host 8080 vs manifest 18080) | every installed backend's live `podman inspect` PortBindings match its manifest `ports:` (the external mirror of the orchestrator's `host_port_bindings_drifted`) |
|
||||
| `secret-completeness.bats` | .198 (bitcoin-knots needs `bitcoin-rpc-txrelay-rpcauth`, never generated → stack cascade) | every `secret_file` referenced by an installed backend manifest exists in the secrets dir |
|
||||
|
||||
Backed by L0 unit tests (`cargo test … drift missing_secret lnd`) and a vitest
|
||||
for the frontend reason-code mapping (`bitcoinReceive.test.ts`). The release
|
||||
gate `scripts/create-release.sh` now runs `tests/release/run.sh` (which includes
|
||||
these) and **aborts the release on failure** — previously it ran no tests at all.
|
||||
|
||||
## Run commands
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/bitcoin-receive.bats
|
||||
#
|
||||
# Regression coverage for the Bitcoin "Receive" flow. Receive addresses come
|
||||
# from LND's hot wallet via the `lnd.newaddress` RPC, so this exercises the
|
||||
# exact path that broke on the fleet:
|
||||
# - .116: LND REST published on the wrong host port (8080 vs the manifest's
|
||||
# 18080) -> connection refused -> receive failed with the generic
|
||||
# "Operation failed. Check server logs." message.
|
||||
# - .228: the same family surfaced to the UI as a *false* "wallet is locked".
|
||||
#
|
||||
# These tests run on the archy host (they shell into podman / curl localhost).
|
||||
#
|
||||
# Tiers: read-only only — generating a receive address is non-destructive.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# Resolve the LND REST host port from the manifest (single source of truth) so
|
||||
# this test follows the manifest rather than hard-coding 18080.
|
||||
_lnd_rest_host_port() {
|
||||
local mf
|
||||
for mf in \
|
||||
"${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yml" \
|
||||
"${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yaml" \
|
||||
"$BATS_TEST_DIRNAME/../../../apps/lnd/manifest.yml"; do
|
||||
[[ -r "$mf" ]] || continue
|
||||
# The REST mapping is the `- host: <N>` whose following `container:` is 8080.
|
||||
awk '
|
||||
/- host:/ { host=$3 }
|
||||
/container:/ { if ($2 == 8080 && host != "") { print host; exit } }
|
||||
' "$mf"
|
||||
return 0
|
||||
done
|
||||
}
|
||||
|
||||
_lnd_running() {
|
||||
rpc_result container-list 2>/dev/null \
|
||||
| jq -e '.[] | select(.name == "lnd" and .state == "running")' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "LND REST is reachable on the manifest host port (catches port drift)" {
|
||||
_lnd_running || skip "lnd not running"
|
||||
local port
|
||||
port=$(_lnd_rest_host_port)
|
||||
[[ -n "$port" ]] || skip "could not resolve LND REST host port from manifest"
|
||||
|
||||
# A TCP connect is enough: drift (container published on a different host
|
||||
# port) shows up as connection-refused here, exactly as on .116.
|
||||
run curl -sk -o /dev/null --max-time 8 "https://127.0.0.1:${port}/v1/getinfo"
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo "LND REST not reachable on host port ${port} (curl exit $status) — likely published-port drift" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "lnd.newaddress returns a bech32 address when lnd is running" {
|
||||
_lnd_running || skip "lnd not running"
|
||||
|
||||
run rpc_call lnd.newaddress
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
local err addr
|
||||
err=$(echo "$output" | jq -r '.error.message // .error // empty')
|
||||
addr=$(echo "$output" | jq -r '.result.address // empty')
|
||||
|
||||
# The whole point of the fix: a running lnd must hand back a real address.
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "lnd.newaddress errored on a running node: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$addr" != bc1* ]]; then
|
||||
echo "expected a bech32 (bc1…) address, got: '$addr'" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "receive errors are specific, never the generic catch-all" {
|
||||
# Even when receive legitimately can't produce an address, the message must be
|
||||
# actionable (start with 'Bitcoin address' and/or carry a [CODE] token) — the
|
||||
# generic 'Operation failed' is what hid the real cause on .116.
|
||||
run rpc_call lnd.newaddress
|
||||
[ "$status" -eq 0 ]
|
||||
local err
|
||||
err=$(echo "$output" | jq -r '.error.message // .error // empty')
|
||||
if [[ "$err" == "Operation failed. Check server logs for details." ]]; then
|
||||
echo "receive returned the generic catch-all instead of a specific reason" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/port-drift.bats
|
||||
#
|
||||
# Regression guard for the .116 failure class: a backend container that is
|
||||
# "Up" but publishes its ports to the WRONG host ports because the manifest
|
||||
# changed after the container was created (e.g. lnd REST stuck on host 8080
|
||||
# while the manifest — and every in-process client — expects 18080).
|
||||
#
|
||||
# This mirrors the orchestrator's `host_port_bindings_drifted` check, but from
|
||||
# the outside: it compares the live `podman inspect` PortBindings against the
|
||||
# manifest `ports:` for each installed backend. Runs on the archy host.
|
||||
#
|
||||
# Tiers: read-only.
|
||||
|
||||
_apps_dir() {
|
||||
local d
|
||||
for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
|
||||
"$BATS_TEST_DIRNAME/../../../apps"; do
|
||||
[[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_manifest_for() {
|
||||
local app="$1" dir
|
||||
dir=$(_apps_dir) || return 1
|
||||
local mf
|
||||
for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
|
||||
[[ -r "$mf" ]] && { echo "$mf"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Emit "host container" pairs from a manifest's ports: block.
|
||||
_manifest_ports() {
|
||||
awk '
|
||||
/^[[:space:]]*ports:/ { inports=1; next }
|
||||
inports && /^[[:space:]]*[a-z_]+:[[:space:]]*$/ && !/protocol:|host:|container:/ { inports=0 }
|
||||
inports && /- host:/ { host=$3 }
|
||||
inports && /container:/ { print host, $2 }
|
||||
' "$1"
|
||||
}
|
||||
|
||||
# For a given container + (host,container) port, emit a "DRIFT: …" line on
|
||||
# mismatch (and nothing otherwise). Stays silent for unpublished / host-net
|
||||
# ports — those are handled elsewhere and must never be treated as drift.
|
||||
_drift_line() {
|
||||
local cname="$1" want_host="$2" cport="$3"
|
||||
local bindings actual
|
||||
bindings=$(podman inspect "$cname" --format '{{json .HostConfig.PortBindings}}' 2>/dev/null) || return 0
|
||||
actual=$(echo "$bindings" | jq -r --arg k "${cport}/tcp" '.[$k][]?.HostPort // empty' 2>/dev/null)
|
||||
[[ -n "$actual" ]] || return 0
|
||||
echo "$actual" | grep -qx "$want_host" && return 0
|
||||
echo "DRIFT: $cname container-port $cport published on host [$actual] but manifest wants $want_host"
|
||||
}
|
||||
|
||||
@test "backend containers publish ports that match their manifest" {
|
||||
command -v podman >/dev/null 2>&1 || skip "podman not available"
|
||||
local checked=0 violations="" app cname mf line
|
||||
# container-name : manifest-app-id
|
||||
for pair in "lnd:lnd" "bitcoin-knots:bitcoin-knots" "electrumx:electrumx"; do
|
||||
cname="${pair%%:*}"; app="${pair##*:}"
|
||||
podman container exists "$cname" 2>/dev/null || continue
|
||||
mf=$(_manifest_for "$app") || continue
|
||||
while read -r host cport; do
|
||||
[[ -n "$host" && -n "$cport" ]] || continue
|
||||
checked=$((checked + 1))
|
||||
line=$(_drift_line "$cname" "$host" "$cport")
|
||||
[[ -n "$line" ]] && violations+="${line}"$'\n'
|
||||
done < <(_manifest_ports "$mf")
|
||||
done
|
||||
[[ "$checked" -gt 0 ]] || skip "no installed backend containers with published ports to check"
|
||||
if [[ -n "$violations" ]]; then
|
||||
echo "published-port drift detected:" >&2
|
||||
echo "$violations" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/secret-completeness.bats
|
||||
#
|
||||
# Regression guard for the .198 failure class: a manifest references a
|
||||
# `secret_env.secret_file` that was never generated on the node, so secret
|
||||
# resolution hard-fails and the container won't start — cascading the whole
|
||||
# Bitcoin stack. (bitcoin-knots gained `bitcoin-rpc-txrelay-rpcauth`, which old
|
||||
# nodes lacked, so bitcoind never came up and reinstall "just stopped".)
|
||||
#
|
||||
# For every installed backend, assert every secret_file it references exists in
|
||||
# the secrets dir. Runs on the archy host.
|
||||
#
|
||||
# Tiers: read-only.
|
||||
|
||||
SECRETS_DIR="${ARCHY_SECRETS_DIR:-/var/lib/archipelago/secrets}"
|
||||
|
||||
_apps_dir() {
|
||||
local d
|
||||
for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
|
||||
"$BATS_TEST_DIRNAME/../../../apps"; do
|
||||
[[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_manifest_for() {
|
||||
local app="$1" dir mf
|
||||
dir=$(_apps_dir) || return 1
|
||||
for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
|
||||
[[ -r "$mf" ]] && { echo "$mf"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_secret_files_in() {
|
||||
# Emit each `secret_file:` value referenced by the manifest.
|
||||
grep -E '^[[:space:]]*secret_file:' "$1" 2>/dev/null | awk '{print $2}'
|
||||
}
|
||||
|
||||
_secret_exists() {
|
||||
local f="$SECRETS_DIR/$1"
|
||||
[[ -e "$f" ]] && return 0
|
||||
sudo -n test -f "$f" 2>/dev/null
|
||||
}
|
||||
|
||||
@test "every installed backend's referenced secrets exist on disk" {
|
||||
command -v podman >/dev/null 2>&1 || skip "podman not available"
|
||||
[[ -d "$SECRETS_DIR" ]] || sudo -n test -d "$SECRETS_DIR" 2>/dev/null || skip "secrets dir not present"
|
||||
|
||||
local checked=0 missing="" app cname mf sf
|
||||
# container-name : manifest-app-id (the bitcoin stack that cascades)
|
||||
for pair in \
|
||||
"bitcoin-knots:bitcoin-knots" "lnd:lnd" "electrumx:electrumx" \
|
||||
"mempool-api:mempool-api" "btcpay-server:btcpay-server" \
|
||||
"archy-nbxplorer:archy-nbxplorer" "fedimint:fedimint" \
|
||||
"fedimint-gateway:fedimint-gateway"; do
|
||||
cname="${pair%%:*}"; app="${pair##*:}"
|
||||
podman container exists "$cname" 2>/dev/null || continue
|
||||
mf=$(_manifest_for "$app") || continue
|
||||
while read -r sf; do
|
||||
[[ -n "$sf" ]] || continue
|
||||
checked=$((checked + 1))
|
||||
_secret_exists "$sf" || missing+="${app} -> ${sf}\n"
|
||||
done < <(_secret_files_in "$mf")
|
||||
done
|
||||
|
||||
[[ "$checked" -gt 0 ]] || skip "no installed backends with secret references to check"
|
||||
if [[ -n "$missing" ]]; then
|
||||
echo "installed apps reference missing secrets:" >&2
|
||||
echo -e "$missing" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
Reference in New Issue
Block a user