Archipelago — open-source initial import
This commit is contained in:
Archipelago
@@ -0,0 +1,263 @@
|
||||
# Container subsystem testing — scorecard and roadmap
|
||||
|
||||
The bar (verbatim from the v1.7.52 owner):
|
||||
|
||||
> "best performant, minimal code, tested containers possible in the world.
|
||||
> No bloated code, no problems installing a single one, no problems
|
||||
> uninstalling, every one needs to be tested 20+ times in every state
|
||||
> before we make another update, not a single container failure outside
|
||||
> of hardware or internet failure is allowed."
|
||||
|
||||
This document is the live tracker for whether we're meeting that bar.
|
||||
Every PR that touches the container subsystem updates the scoreboard
|
||||
below. **If you can't honestly tick the box, the change isn't ready.**
|
||||
|
||||
---
|
||||
|
||||
## Production-quality pass — 2026-06-21 (current, v1.7.99-alpha)
|
||||
|
||||
The migration's aim, restated as **five pillars** (every app must satisfy all five):
|
||||
|
||||
1. **Quadlet-everywhere** — every container is a declarative systemd Quadlet
|
||||
unit under `user.slice`, never inside `archipelago.service`'s cgroup. Kills
|
||||
FM3 (restarting/updating archipelago SIGKILLs every container in its cgroup);
|
||||
systemd becomes the per-app supervisor.
|
||||
2. **Level-triggered reconciler** — a 30s idempotent reconcile loop drives
|
||||
desired→current from manifests + secrets. Self-healing, not edge-triggered.
|
||||
3. **Lifecycle bulletproof** — every app passes the full matrix
|
||||
(install / UI reachable / stop / start / restart / reinstall / reboot-survive
|
||||
/ archipelago-restart-survive / uninstall) **5× green on .228** — run ON the node
|
||||
(`ARCHY_ITERATIONS=5`).
|
||||
(Multinode / fleet → `docs/multinode-testing-plan.md`, separate.)
|
||||
before any release.
|
||||
4. **Data-driven apps** — install/uninstall needs only the app's manifest +
|
||||
catalog entry. **No host OS changes** (no apt, no /etc, no host units) and
|
||||
**no archipelago binary code per app**. Only *core* apps (bitcoin, lnd,
|
||||
electrumx, fedimint + gateway/clientd) may carry bespoke handling if truly
|
||||
unavoidable.
|
||||
5. **Rootless + security-first (non-negotiable)** — containers run in the
|
||||
unprivileged `archipelago` user namespace; never root, no `--privileged`,
|
||||
drop-all-caps + add-back only what a manifest declares. Secrets are `0600`,
|
||||
owned by the service user. Security is king.
|
||||
|
||||
**Per-app definition of done:** all five pillars hold → lifecycle matrix 5×
|
||||
green on .228 (run ON the node) → catalog/registry updated (`app-catalog/catalog.json`
|
||||
+ `releases/app-catalog.json`, rebuilt image pushed to the mirror) → tracker
|
||||
cell ticked. Only then move to the next app. (Fleet/multinode verification is a
|
||||
separate pass → `docs/multinode-testing-plan.md`.)
|
||||
|
||||
**.228 testing constraint:** do NOT touch `bitcoin-knots`, `electrumx`, or
|
||||
`lnd` on .228 — they are synced and healthy; destructive cycles there would
|
||||
cost hours of resync.
|
||||
|
||||
### Session work log (resolved)
|
||||
|
||||
The 2026-06-21 mid-session resume block that lived here (generated-secrets
|
||||
rollout for fedimint-gateway/-clientd, icon/naming fixes) is **done and
|
||||
shipped**: the generated-secrets system is a platform primitive
|
||||
(`container.generated_secrets`, see `docs/app-manifest-spec.md`), the
|
||||
manifests declare it, and the single-node gate went green on .228 on
|
||||
2026-06-23. Day-to-day open items live in `docs/UNIFIED-TASK-TRACKER.md` —
|
||||
don't add session logs here.
|
||||
|
||||
---
|
||||
|
||||
## Test layers
|
||||
|
||||
| Layer | What it asserts | Toolchain | Latency / iteration |
|
||||
|---|---|---|---|
|
||||
| L0 — Rust unit | Pure-function behaviour (manifest parsing, secret resolution, structural invariants) | `cargo test --workspace --bins` | ~5s |
|
||||
| L1 — RPC API | The JSON-RPC API responds correctly per app (`container-list`, `package.{install,start,stop,restart,uninstall}`, `bitcoin.getinfo`, etc.) | bats + lib/rpc.bash | ~30s per suite |
|
||||
| L2 — UI surface | The URLs a user actually clicks (dashboard, `/app/<id>/`, direct-port iframes) return 200 with non-empty bodies | bats + lib/ui-probes.bash | ~10s per suite |
|
||||
| L3 — Lifecycle survival | Containers survive operational events (archipelago restart, host reboot, kill -9 mid-install, OOM) | bats (gated) | ~60s per scenario |
|
||||
| L4 — Browser journey | Real DOM-level user flow (login → install → wait → click → use) | playwright (TBD) | ~30-120s per journey |
|
||||
| L5 — Chaos / failure-path | Failure modes recover gracefully (corrupt config, deleted bolt DB, network partition) | bats (chaos-gated) | ~120s per scenario |
|
||||
| L6 — Performance | Cold install latency, reconcile-tick cost, podman call count per lifecycle event | timed bats + Prometheus (TBD) | ~60s per benchmark |
|
||||
|
||||
Release gate: **L0+L1+L2+L3 green × 20 iterations** on .228 (run ON the node; 5× for
|
||||
now). Multinode/fleet → `docs/multinode-testing-plan.md`. L4+L5+L6 are quality gates
|
||||
we add as they mature; not blocking the v1.7.52 tag.
|
||||
|
||||
## Coverage matrix — current state
|
||||
|
||||
Legend: ● fully covered, ◐ partial, ○ missing
|
||||
|
||||
### Per-app × per-state matrix (L1 + L2)
|
||||
|
||||
| App | Container present | Valid state | RPC reachable | UI URL 200 | Stop | Start | Restart | Reinstall | Reboot survives | Archipelago-restart survives |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| bitcoin-knots | ● | ● | ● | ● (port 8334) | ● | ● | ● | ● | ○ | ◐ regression-gate only |
|
||||
| bitcoin-core | ◐ shares with knots | ◐ | ○ | ◐ | ○ | ○ | ○ | ○ | ○ | ◐ regression-gate |
|
||||
| lnd | ● | ● | ● (lncli) | ● (`/app/lnd/`) | ● | ● | ● | ● | ○ | ◐ regression-gate |
|
||||
| electrumx | ● | ● | ● (TCP 50001) | ● (`/app/electrumx/`) | ● | ● | ● | ● | ○ | ◐ regression-gate |
|
||||
| btcpay-server | ● | ● | ◐ frontend-port | ● (`/app/btcpay/`) | ● | ● | ● | ● | ○ | ○ |
|
||||
| mempool | ● | ● | ● (`/api/v1/backend-info`) | ● (`/app/mempool/`) | ● | ● | ● | ● | ○ | ○ |
|
||||
| fedimint | ● | ● | ◐ container-only | ● (`/app/fedimint/`) | ● | ● | ● | ● | ○ | ○ |
|
||||
| filebrowser | ○ | ○ | ○ | ● probe-only | ○ | ○ | ○ | ○ | ○ | ◐ via companions |
|
||||
| archy-bitcoin-ui | ◐ via companions | ◐ | n/a | ● (port 8334) | ○ | ○ | ○ | n/a | ◐ via companions | ● |
|
||||
| archy-lnd-ui | ◐ via companions | ◐ | n/a | ● (`/app/lnd/`) | ○ | ○ | ○ | n/a | ◐ via companions | ● |
|
||||
| archy-electrs-ui | ◐ via companions | ◐ | n/a | ● (`/app/electrumx/`) | ○ | ○ | ○ | n/a | ◐ via companions | ● |
|
||||
|
||||
Done: 50 of 110 cells. Goal: 110/110 ● for the listed apps before
|
||||
v1.7.52 tags.
|
||||
|
||||
### Layer-by-layer status
|
||||
|
||||
| Layer | Tests | Suites | Status |
|
||||
|---|---:|---:|---|
|
||||
| L0 unit | 631 | n/a | ● green |
|
||||
| L1 RPC | 70 | bitcoin-knots, lnd, electrumx, btcpay, mempool, fedimint, required-stack, package-update-smoke | ● for the 6 core apps |
|
||||
| L2 UI | 9 | ui-coverage | ● for dashboard + 7 proxy paths + bitcoin-ui:8334 |
|
||||
| L3 lifecycle survival | 14 | companion-survives-archipelago-restart, backend-survives-archipelago-restart, required-stack-destructive, use-quadlet-backends-install | ◐ companions ● ; backends ◐ regression-gate (will fail until Phase 3 Quadlet ships); quadlet post-condition gate ✅ skip-clean today, hard gate when flag flipped |
|
||||
| L1 wallet-receive / drift / secrets | 5 | bitcoin-receive, port-drift, secret-completeness | ● guards the v1.7.9x wallet fleet failures |
|
||||
| L4 browser journey | 0 | none | ○ not started |
|
||||
| L5 chaos | 0 | none | ○ not started |
|
||||
| L6 performance | 0 | none | ○ not started |
|
||||
|
||||
### Wallet / Bitcoin fleet-failure regression suites (added after v1.7.90-alpha)
|
||||
|
||||
Three production failures shipped on v1.7.90-alpha despite the existing harness,
|
||||
because nothing exercised the receive path, port-mapping drift, or secret
|
||||
completeness on a live node. New suites close those gaps (all run on the archy
|
||||
host, read-only, so they join `run.sh`/`run-gate.sh` automatically):
|
||||
|
||||
| Suite | Failure it guards | Asserts |
|
||||
|---|---|---|
|
||||
| `bitcoin-receive.bats` | .116 ("Operation failed" on receive) and .228 (false "wallet is locked") | LND REST reachable on the **manifest** host port; `lnd.newaddress` returns a `bc1…` address on a running node; receive errors are specific, never the generic catch-all |
|
||||
| `port-drift.bats` | .116 (lnd REST stuck on host 8080 vs manifest 18080) | every installed backend's live `podman inspect` PortBindings match its manifest `ports:` (the external mirror of the orchestrator's `host_port_bindings_drifted`) |
|
||||
| `secret-completeness.bats` | .198 (bitcoin-knots needs `bitcoin-rpc-txrelay-rpcauth`, never generated → stack cascade) | every `secret_file` referenced by an installed backend manifest exists in the secrets dir |
|
||||
|
||||
Backed by L0 unit tests (`cargo test … drift missing_secret lnd`) and a vitest
|
||||
for the frontend reason-code mapping (`bitcoinReceive.test.ts`). The release
|
||||
gate `scripts/create-release.sh` now runs `tests/release/run.sh` (which includes
|
||||
these) and **aborts the release on failure** — previously it ran no tests at all.
|
||||
|
||||
## Run commands
|
||||
|
||||
```bash
|
||||
# L0 unit:
|
||||
cd core && cargo test --workspace --bins
|
||||
|
||||
# Single bats suite:
|
||||
ARCHY_PASSWORD=password123 tests/lifecycle/run.sh bitcoin-knots
|
||||
|
||||
# Full bats suite (read-only):
|
||||
ARCHY_PASSWORD=password123 tests/lifecycle/run.sh
|
||||
|
||||
# Full + destructive (for the verification fleet):
|
||||
ARCHY_PASSWORD=password123 ARCHY_ALLOW_DESTRUCTIVE=1 tests/lifecycle/run.sh
|
||||
|
||||
# 5× release-gate run:
|
||||
ARCHY_PASSWORD=password123 ARCHY_ALLOW_DESTRUCTIVE=1 ARCHY_ITERATIONS=5 \
|
||||
tests/lifecycle/run-gate.sh
|
||||
|
||||
# CASCADE tier (uninstall → no-ghost → reinstall) — opt-in, NOT in the canonical
|
||||
# gate. Installs/uninstalls a THROWAWAY app (default grafana; skips if already
|
||||
# installed). Run on-node to also assert data-dir removal:
|
||||
ARCHY_PASSWORD=password123 ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1 \
|
||||
tests/lifecycle/run.sh cascade-uninstall
|
||||
```
|
||||
|
||||
### CASCADE tier — uninstall/reinstall regression guard (Workstream F)
|
||||
|
||||
The 5× gate is DESTRUCTIVE-only (stop/start/restart/survive); it never exercised
|
||||
uninstall/reinstall, where the worst lifecycle bugs lived. `cascade-uninstall.bats`
|
||||
closes that gap and encodes the fixes for two field bugs:
|
||||
|
||||
| Suite | Failure it guards | Asserts |
|
||||
|---|---|---|
|
||||
| `cascade-uninstall.bats` | **#13 uninstall ghost** (immich/grafana stayed in My Apps after uninstall) and **#14 reinstall stops** (stalled on stale state/data) | fresh install reaches `running` via a truthful (non-silent) progression; uninstall makes the entry **disappear from `server.get-state` package-data** (no ghost, no stuck uninstall stage) + removes the container + (on-node) the data dir; reinstall returns to `running`; node left as found |
|
||||
|
||||
Throwaway-app + precondition-skip (won't touch an app that's already installed),
|
||||
so it's safe on a populated node. Override the app via `ARCHY_CASCADE_APP` /
|
||||
`ARCHY_CASCADE_IMAGE` / `ARCHY_CASCADE_CONFIG` / `ARCHY_CASCADE_DATA_DIR`.
|
||||
Gated on `ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1`. Verified 7/7 on .228 (2026-06-24).
|
||||
|
||||
### All-apps lifecycle matrix (Workstream F)
|
||||
|
||||
The per-app suites cover ~8 core apps in depth; `all-apps-matrix.bats` covers
|
||||
**every installed app in breadth, automatically** — it derives the app set from
|
||||
`server.get-state` package-data (no hardcoded list) and grows coverage as nodes
|
||||
install more apps. **Read-only**, so it joins `run.sh`/`run-gate.sh` on every node.
|
||||
|
||||
| Suite | Guards (fleet-wide) | Asserts (per installed app) |
|
||||
|---|---|---|
|
||||
| `all-apps-matrix.bats` | apps STUCK transitional (the #13/#14 ghost generalized), error/failed apps, unreachable UI apps (port-drift generalized) | settles to a non-transitional state within a window; not error/failed; recognized (non-garbage) state; every **running UI app** (manifest `ui=="true"`) exposes a non-null lan-address |
|
||||
|
||||
Tunables: `ARCHY_MATRIX_SETTLE_SECS` (45), `ARCHY_MATRIX_UI_SECS` (30),
|
||||
`ARCHY_MATRIX_ALLOW_STOPPED` (ids allowed non-running). Verified 5/5 on .228
|
||||
(17 apps) and .116 (20 apps incl. grafana/nextcloud/photoprism/gitea), 2026-06-24.
|
||||
|
||||
To exercise the Phase 3.2 Quadlet-backend path on a target node without
|
||||
editing config.json (which would require an archipelago restart and
|
||||
trigger FM3 until 3.5 ships), set the env var on `archipelago.service`:
|
||||
|
||||
```bash
|
||||
sudo systemctl edit archipelago # add: [Service]\nEnvironment=ARCHIPELAGO_USE_QUADLET_BACKENDS=1
|
||||
sudo systemctl restart archipelago # one cgroup-cascade hit; survivable on a debug node
|
||||
```
|
||||
|
||||
After the restart, `package.install` for any orchestrator-managed backend
|
||||
will route through `install_via_quadlet`, and the
|
||||
`use-quadlet-backends-install.bats` suite turns from skip → hard gate.
|
||||
|
||||
## LoC budget
|
||||
|
||||
Goal: minimum-viable container subsystem.
|
||||
|
||||
| Module | LoC today | Target | Δ | Status |
|
||||
|---|---:|---:|---:|---|
|
||||
| `core/container/src/dependency_resolver.rs` | — | — | -270 | ● deleted |
|
||||
| `core/container/src/health_monitor.rs` | 196 | 0 | -196 | ◐ pending health migration into reconciler (Phase 3.5) |
|
||||
| `core/container/src/podman_client.rs::create/start/stop` | ~400 | ~150 | -250 | ◐ pending Quadlet migration (Phase 3.5) |
|
||||
| `core/archipelago/src/container/dev_orchestrator.rs` | 410 | 0 | -410 | ○ pending dev_mode strategy decision |
|
||||
| `core/archipelago/src/container/data_manager.rs` | 96 | 0 | -96 | ○ couples with dev_orchestrator |
|
||||
| `core/container/src/bitcoin_simulator.rs` | 219 | 0 | -219 | ○ couples with dev_orchestrator |
|
||||
| `core/container/src/port_manager.rs` | 175 | 0 | -175 | ○ couples with dev_orchestrator |
|
||||
| `core/archipelago/src/api/rpc/package/install.rs::install_bitcoincoin_rpc_repair` | ~150 | 0 | -150 | ◐ pending fold into orchestrator pre-start |
|
||||
| imperative `install_fresh` in prod_orchestrator | ~120 | 0 | -120 | ◐ Phase 3.2 wired behind `use_quadlet_backends` flag (default off); 3.3 in-place migration ✅; 3.4 health-gated startup (`Notify=healthy`) ✅ + `TimeoutStartSec=600` race fix ✅; 3.4a unit drift-sync each reconcile ✅; flip default after 5× green |
|
||||
|
||||
**Today: -270 LoC committed. Outstanding deletes possible: ~1,616 LoC** (if Phase 3 ships fully + dev_mode resolved).
|
||||
|
||||
Net target for v1.7.52: container subsystem ≈ **half** of today's LoC.
|
||||
|
||||
## Performance KPIs (TBD — measure first, then target)
|
||||
|
||||
We don't have a performance harness yet. Add as L6 lands:
|
||||
|
||||
| KPI | Today | Target | Notes |
|
||||
|---|---|---|---|
|
||||
| cold install: bitcoin-knots manifest → `running` healthcheck | unknown | < 30s once image is local | excludes the ~1GB image pull |
|
||||
| cold install: lnd | unknown | < 60s once image is local | wallet unlock dominates |
|
||||
| reconcile-tick wall time (no-op pass over all installed apps) | unknown | < 250ms | the current orchestrator does many `podman inspect` calls |
|
||||
| podman shell-outs per package.install (orchestrator path) | 7-10 | 1-2 (Quadlet) | post-Phase-3 |
|
||||
| daemon startup (boot → port 5678 listening) | unknown | < 5s | reconcile is async after this |
|
||||
|
||||
## Release gates
|
||||
|
||||
1.8.0 ships only when ALL of (see `docs/UNIFIED-TASK-TRACKER.md` for the live
|
||||
priority-ordered list of what's still open across these):
|
||||
|
||||
1. ☑ Bitcoin-stops fix verified live on a fresh node (`tests/lifecycle/bats/bitcoin-knots.bats`
|
||||
stop/restart tier, part of the green single-node gate)
|
||||
2. ☑ `ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh` returns 0 **run ON .228** — GREEN 2026-06-23, 5/5, 0 failures
|
||||
3. ☐ Multinode/fleet (.198 + others) — tracked separately in `docs/multinode-testing-plan.md`,
|
||||
the actual next exit criterion, NOT satisfied yet
|
||||
4. ☐ The L3 `backend-survives-archipelago-restart` suite passes fleet-wide default-on
|
||||
(Phase 3 Quadlet is merged + validated but still opt-in via `ARCHIPELAGO_USE_QUADLET_BACKENDS`
|
||||
on .228/.198 only — not the default)
|
||||
5. ☑ Cargo: 0 warnings, 0 unused (confirmed 2026-07-01 release build); full test suite green
|
||||
per last confirmed run
|
||||
6. ☑ LoC: Phase 3 Quadlet merged (opt-in) — satisfies the "at least one of" bar; default-flip
|
||||
itself is tracked as its own item in the unified tracker
|
||||
7. ☑ Layman-readable changelog — `CHANGELOG.md` backfilled through v1.8.00-alpha
|
||||
(per `feedback_changelog_layman.md`)
|
||||
8. ☐ Tag pushed to origin + gitea-local + gitea-vps2 (per `feedback_ship_ritual.md`) —
|
||||
version decided 2026-07-08 (`1.8.0-alpha`); tag once the pre-tag items above close
|
||||
|
||||
## How to update this document
|
||||
|
||||
When you land a change that materially moves any cell of the matrix or
|
||||
any LoC row, update this file in the same commit. Reviewers checking
|
||||
the PR can read the diff to TESTING.md as the answer to "what did
|
||||
this commit improve?". Without the update, the change is half-shipped.
|
||||
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/all-apps-lifecycle.bats
|
||||
#
|
||||
# DESTRUCTIVE per-app lifecycle matrix across EVERY installed app (breadth) —
|
||||
# the active counterpart to the read-only all-apps-matrix.bats and the ~8 deep
|
||||
# per-app suites. For each installed, NON-protected app it drives:
|
||||
# stop → verify stopped → start → verify running → restart → verify running
|
||||
# and, when ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1, a FULL TEARDOWN:
|
||||
# uninstall (full, removes data) → verify GONE from My Apps (no #13 ghost) →
|
||||
# reinstall from the node catalog → verify running.
|
||||
#
|
||||
# Reinstall spec source: the node catalog (default /opt/archipelago/web-ui/
|
||||
# catalog.json), whose `.apps[]` entries carry {dockerImage, containerConfig} —
|
||||
# exactly what package.install needs. Multi-container stacks (immich, mempool,
|
||||
# netbird, btcpay, indeedhub) ignore dockerImage internally but still require it,
|
||||
# and route to their orchestrator/stack handler; the catalog entry is enough to
|
||||
# trigger the reinstall. An app with no catalog entry is skipped (logged), not
|
||||
# failed — there's no spec to reinstall it from.
|
||||
#
|
||||
# ── PROTECTED apps (NEVER touched — neither cycled nor torn down) ────────────
|
||||
# - chain state, expensive to resync: bitcoin*, electrumx/electrs
|
||||
# - WALLET / financial state, teardown = IRREVERSIBLE fund/credential loss:
|
||||
# lnd, btcpay*, fedimint*
|
||||
# The user asked to protect only bitcoin + electrum; the wallet-bearing apps
|
||||
# are protected by DEFAULT here for safety (a full uninstall destroys their
|
||||
# seed/channel/guardian state). Override the entire set with
|
||||
# ARCHY_MATRIX_PROTECT="space separated ids" to tear them down too — you WILL
|
||||
# lose their data.
|
||||
#
|
||||
# ── Gating ──────────────────────────────────────────────────────────────────
|
||||
# lifecycle tier → ARCHY_ALLOW_DESTRUCTIVE=1
|
||||
# teardown tier → ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1
|
||||
# Both skip otherwise, so this file is inert in a normal run. ON-NODE ONLY
|
||||
# (reads catalog.json on disk + drives the local package lifecycle).
|
||||
#
|
||||
# This is a HEAVY suite: a full teardown of ~15-20 apps re-pulls images and can
|
||||
# run for a long time. Intended as an explicit, supervised coverage pass, not a
|
||||
# per-iteration gate step.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
CATALOG="${ARCHY_CATALOG:-/opt/archipelago/web-ui/catalog.json}"
|
||||
|
||||
# Protected — see header. Override with ARCHY_MATRIX_PROTECT to change the set.
|
||||
PROTECT="${ARCHY_MATRIX_PROTECT:-bitcoin-knots bitcoin-core bitcoin electrumx electrs mempool-electrs lnd btcpay-server btcpayserver btcpay fedimint fedimint-clientd fedimint-gateway}"
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
is_protected() {
|
||||
local id="$1" p
|
||||
for p in $PROTECT; do [[ "$p" == "$id" ]] && return 0; done
|
||||
return 1
|
||||
}
|
||||
|
||||
get_package_data() {
|
||||
rpc_result server.get-state '{}' 2>/dev/null | jq -c '.data["package-data"] // {}'
|
||||
}
|
||||
|
||||
# Canonical app ids the catalog can (re)install.
|
||||
catalog_ids() {
|
||||
jq -r '(.apps // [])[].id' "$CATALOG" 2>/dev/null
|
||||
}
|
||||
|
||||
# Installed primary apps we will exercise: catalog ids present in My Apps,
|
||||
# minus the protected set. (Catalog-scoped so we skip sub-containers like
|
||||
# immich_postgres that surface as their own package-data entries.)
|
||||
target_apps() {
|
||||
local pd; pd=$(get_package_data)
|
||||
local id
|
||||
for id in $(catalog_ids); do
|
||||
echo "$pd" | jq -e --arg i "$id" 'has($i)' >/dev/null 2>&1 || continue
|
||||
is_protected "$id" && continue
|
||||
echo "$id"
|
||||
done
|
||||
}
|
||||
|
||||
# Top-level state of an app in My Apps, or "absent" when the entry is gone.
|
||||
app_state() {
|
||||
get_package_data | jq -r --arg i "$1" '.[$i].state // "absent"'
|
||||
}
|
||||
|
||||
# Poll My Apps until app $1 reaches state $2 (or "absent"); $3 = timeout secs.
|
||||
wait_state() {
|
||||
local id="$1" target="$2" timeout="${3:-180}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
[[ "$(app_state "$id")" == "$target" ]] && return 0
|
||||
sleep 3
|
||||
done
|
||||
echo "wait_state: $id never reached '$target' (last='$(app_state "$id")') within ${timeout}s" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Build a package.install payload for $1 from the catalog, or fail (no spec).
|
||||
catalog_install_payload() {
|
||||
local id="$1" img cfg
|
||||
img=$(jq -r --arg i "$id" '(.apps // [])[] | select(.id==$i) | .dockerImage // empty' "$CATALOG")
|
||||
[[ -n "$img" ]] || return 1
|
||||
cfg=$(jq -c --arg i "$id" '(.apps // [])[] | select(.id==$i) | .containerConfig // null' "$CATALOG")
|
||||
if [[ "$cfg" == "null" ]]; then
|
||||
jq -nc --arg id "$id" --arg img "$img" '{id:$id, dockerImage:$img}'
|
||||
else
|
||||
jq -nc --arg id "$id" --arg img "$img" --argjson cfg "$cfg" '{id:$id, dockerImage:$img, containerConfig:$cfg}'
|
||||
fi
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
@test "prerequisites: catalog present and at least one target app" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
[[ -f "$CATALOG" ]] || { echo "# catalog not found: $CATALOG" >&3; false; }
|
||||
run target_apps
|
||||
[ "$status" -eq 0 ]
|
||||
[ -n "$output" ] || { echo "# no non-protected installed apps to exercise" >&3; false; }
|
||||
echo "# protected (skipped): $PROTECT" >&3
|
||||
echo "# targets ($(echo "$output" | wc -w)): $(echo $output)" >&3
|
||||
}
|
||||
|
||||
@test "lifecycle: stop → start → restart every non-protected app" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
local fails="" id
|
||||
for id in $(target_apps); do
|
||||
[[ "$(app_state "$id")" == "running" ]] || continue # only cycle running apps
|
||||
rpc_result package.stop "{\"id\":\"$id\"}" >/dev/null 2>&1
|
||||
wait_state "$id" stopped 120 || { fails+="$id:stop "; }
|
||||
rpc_result package.start "{\"id\":\"$id\"}" >/dev/null 2>&1
|
||||
wait_state "$id" running 240 || { fails+="$id:start "; continue; }
|
||||
rpc_result package.restart "{\"id\":\"$id\"}" >/dev/null 2>&1
|
||||
wait_state "$id" running 240 || { fails+="$id:restart "; }
|
||||
done
|
||||
[[ -z "$fails" ]] || { echo "# lifecycle failures: $fails" >&3; false; }
|
||||
}
|
||||
|
||||
@test "teardown: full uninstall (no ghost) → reinstall every non-protected app" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
local fails="" skipped="" id payload
|
||||
for id in $(target_apps); do
|
||||
if ! payload=$(catalog_install_payload "$id"); then
|
||||
skipped+="$id "
|
||||
continue
|
||||
fi
|
||||
rpc_result package.uninstall "{\"id\":\"$id\"}" >/dev/null 2>&1
|
||||
# No ghost: the entry must leave My Apps (the #13 class). 71cc9ac4 bounds the
|
||||
# teardown so this can no longer hang indefinitely.
|
||||
if ! wait_state "$id" absent 300; then
|
||||
fails+="$id:ghost "
|
||||
continue
|
||||
fi
|
||||
rpc_result package.install "$payload" >/dev/null 2>&1
|
||||
wait_state "$id" running 420 || fails+="$id:reinstall "
|
||||
done
|
||||
[[ -n "$skipped" ]] && echo "# skipped (no catalog spec to reinstall from): $skipped" >&3
|
||||
[[ -z "$fails" ]] || { echo "# teardown failures: $fails" >&3; false; }
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/all-apps-matrix.bats
|
||||
#
|
||||
# Manifest-driven, fleet-wide lifecycle health matrix. The per-app suites
|
||||
# (bitcoin-knots, lnd, mempool, immich, …) cover ~8 core apps in depth; this
|
||||
# covers EVERY installed app in breadth, automatically — no hardcoded list.
|
||||
#
|
||||
# It derives the app set from server.get-state's package-data (the My Apps map)
|
||||
# and asserts baseline health across all of them. Read-only (no destructive env
|
||||
# needed), so it joins run.sh / run-gate.sh on every node and grows coverage as
|
||||
# nodes install more apps.
|
||||
#
|
||||
# Catches, fleet-wide, the bug classes the narrow gate missed:
|
||||
# - apps STUCK in a transitional state (the #13/#14 ghost: installing/removing
|
||||
# that never settles)
|
||||
# - apps sitting in error/failed
|
||||
# - running UI apps with no reachable lan-address (generalized port-drift)
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
# Transitional states are legitimate momentarily but must not PERSIST. Steady:
|
||||
# running/stopped/exited/created/paused/installed/not-installed.
|
||||
TRANSITIONAL_RE='^(installing|pulling-image|pulling|downloading|removing|uninstalling|updating|starting|stopping|restarting)$'
|
||||
BAD_RE='^(error|failed)$'
|
||||
|
||||
# Apps whose state is allowed to be non-running at rest (no UI/health expectation
|
||||
# beyond "settled"). Empty by default; override via ARCHY_MATRIX_ALLOW_STOPPED
|
||||
# (space-separated ids) on nodes where an app is intentionally left stopped.
|
||||
ALLOW_STOPPED="${ARCHY_MATRIX_ALLOW_STOPPED:-}"
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# Echo the package-data object (the My Apps map) once.
|
||||
get_package_data() {
|
||||
rpc_result server.get-state '{}' 2>/dev/null | jq -c '.data["package-data"] // {}'
|
||||
}
|
||||
|
||||
# Space-separated list of installed app ids.
|
||||
app_ids() {
|
||||
get_package_data | jq -r 'keys[]'
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
@test "matrix has apps to check (get-state returns a non-empty My Apps map)" {
|
||||
run app_ids
|
||||
[ "$status" -eq 0 ]
|
||||
[ -n "$output" ]
|
||||
echo "# matrix covers $(echo "$output" | wc -w) apps: $(echo $output)" >&3
|
||||
}
|
||||
|
||||
@test "no installed app is STUCK in a transitional state (settles within window)" {
|
||||
local settle="${ARCHY_MATRIX_SETTLE_SECS:-45}"
|
||||
local deadline=$(( $(date +%s) + settle ))
|
||||
local stuck=""
|
||||
# Re-poll: a transitional state right now may just be a genuine in-progress op,
|
||||
# so only fail apps that are STILL transitional after the settle window.
|
||||
while :; do
|
||||
stuck=""
|
||||
local pd; pd=$(get_package_data)
|
||||
for id in $(echo "$pd" | jq -r 'keys[]'); do
|
||||
local st; st=$(echo "$pd" | jq -r --arg i "$id" '.[$i].state // "unknown"')
|
||||
[[ "$st" =~ $TRANSITIONAL_RE ]] && stuck+="${id}=${st} "
|
||||
done
|
||||
[[ -z "$stuck" ]] && break
|
||||
(( $(date +%s) >= deadline )) && break
|
||||
sleep 5
|
||||
done
|
||||
[[ -z "$stuck" ]] || { echo "# STUCK transitional after ${settle}s: $stuck" >&3; false; }
|
||||
}
|
||||
|
||||
@test "no installed app is in an error/failed state" {
|
||||
local pd; pd=$(get_package_data)
|
||||
local bad=""
|
||||
for id in $(echo "$pd" | jq -r 'keys[]'); do
|
||||
local st; st=$(echo "$pd" | jq -r --arg i "$id" '.[$i].state // "unknown"')
|
||||
[[ "$st" =~ $BAD_RE ]] && bad+="${id}=${st} "
|
||||
done
|
||||
[[ -z "$bad" ]] || { echo "# error/failed apps: $bad" >&3; false; }
|
||||
}
|
||||
|
||||
@test "every running app reports a recognized state (no empty/garbage state)" {
|
||||
local pd; pd=$(get_package_data)
|
||||
local junk=""
|
||||
for id in $(echo "$pd" | jq -r 'keys[]'); do
|
||||
local st; st=$(echo "$pd" | jq -r --arg i "$id" '.[$i].state // "unknown"')
|
||||
case "$st" in
|
||||
running|stopped|exited|created|paused|installed|not-installed|\
|
||||
installing|pulling-image|pulling|downloading|removing|uninstalling|updating|starting|stopping|restarting|\
|
||||
error|failed|degraded) : ;;
|
||||
*) junk+="${id}='${st}' " ;;
|
||||
esac
|
||||
done
|
||||
[[ -z "$junk" ]] || { echo "# unrecognized state values: $junk" >&3; false; }
|
||||
}
|
||||
|
||||
@test "every running UI app exposes a lan-address (generalized port-drift)" {
|
||||
# A running app whose manifest declares a UI interface (ui=="true") must have a
|
||||
# non-null lan-address on that interface — otherwise its UI is unreachable
|
||||
# (the immich/port-drift failure mode, asserted across ALL UI apps). Poll
|
||||
# briefly to absorb the transient null seen while a container is mid-recreate.
|
||||
local deadline=$(( $(date +%s) + ${ARCHY_MATRIX_UI_SECS:-30} ))
|
||||
local missing=""
|
||||
while :; do
|
||||
missing=""
|
||||
local pd; pd=$(get_package_data)
|
||||
for id in $(echo "$pd" | jq -r 'keys[]'); do
|
||||
local st; st=$(echo "$pd" | jq -r --arg i "$id" '.[$i].state // "unknown"')
|
||||
[[ "$st" == "running" ]] || continue
|
||||
# interface keys whose manifest marks ui=="true"
|
||||
local ui_ifaces
|
||||
ui_ifaces=$(echo "$pd" | jq -r --arg i "$id" \
|
||||
'.[$i].manifest.interfaces // {} | to_entries[] | select(.value.ui=="true") | .key')
|
||||
for k in $ui_ifaces; do
|
||||
local addr
|
||||
addr=$(echo "$pd" | jq -r --arg i "$id" --arg k "$k" \
|
||||
'.[$i].installed["interface-addresses"][$k]["lan-address"] // "null"')
|
||||
[[ "$addr" == "null" || -z "$addr" ]] && missing+="${id}:${k} "
|
||||
done
|
||||
done
|
||||
[[ -z "$missing" ]] && break
|
||||
(( $(date +%s) >= deadline )) && break
|
||||
sleep 3
|
||||
done
|
||||
[[ -z "$missing" ]] || { echo "# running UI apps missing lan-address: $missing" >&3; false; }
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/backend-survives-archipelago-restart.bats
|
||||
#
|
||||
# Quadlet-everywhere promise (Phase 3 of v1.7.52): backend containers
|
||||
# (bitcoin-knots / lnd / electrumx) are managed by systemd via Quadlet
|
||||
# units, NOT parented under archipelago.service's cgroup. Restarting the
|
||||
# archipelago service must NOT take them down.
|
||||
#
|
||||
# This is the regression gate for FM3 (cgroup cascade SIGKILL — observed
|
||||
# live on .198 on 2026-05-01: stopping archipelago.service killed every
|
||||
# container in its cgroup, leaving the box in a multi-hour recovery
|
||||
# loop). Until v1.7.52 Phase 3 ships, this suite is EXPECTED TO FAIL on
|
||||
# fleet boxes — it serves as the executable definition of "Phase 3
|
||||
# complete". Do not gate the release on it passing pre-Phase-3.
|
||||
#
|
||||
# Sister to companion-survives-archipelago-restart.bats which tests the
|
||||
# same property for UI companions (already shipping via Quadlet since
|
||||
# commit 6e716f68).
|
||||
#
|
||||
# Gated by ARCHY_ALLOW_DESTRUCTIVE=1 because it bounces archipelago.
|
||||
|
||||
# bats-core ships no `fail`; bats-assert isn't installed on the alpha fleet.
|
||||
# Define the same minimal helper the other suites use (see mempool.bats) so a
|
||||
# tripped assertion reports as a real test failure, not a status-127 crash.
|
||||
fail() { echo "$@" >&2; return 1; }
|
||||
|
||||
backend_units=(
|
||||
"bitcoin-knots"
|
||||
"bitcoin-core"
|
||||
"lnd"
|
||||
"electrumx"
|
||||
)
|
||||
|
||||
container_running() {
|
||||
local name="$1"
|
||||
[[ "$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null)" == "true" ]]
|
||||
}
|
||||
|
||||
wait_archipelago_back() {
|
||||
local timeout="${1:-60}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if curl -fsS -o /dev/null "http://127.0.0.1:5678/health" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
@test "destructive gate enabled" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
}
|
||||
|
||||
@test "at least one backend container is running before restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
local up=0
|
||||
for c in "${backend_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
up=$(( up + 1 ))
|
||||
fi
|
||||
done
|
||||
(( up > 0 )) || skip "No backends installed on this node"
|
||||
}
|
||||
|
||||
@test "backends survive archipelago restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# Snapshot: which backends were up before we touched anything.
|
||||
local before=()
|
||||
for c in "${backend_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
before+=("$c")
|
||||
fi
|
||||
done
|
||||
(( ${#before[@]} > 0 )) || skip "No backends installed on this node"
|
||||
|
||||
# Capture pre-restart container IDs so we can verify the SAME process
|
||||
# survives — not "the orchestrator started a fresh container after the
|
||||
# cascade SIGKILL'd the original" (which would also be a fail; FM3 is
|
||||
# specifically about losing the running container, even if the
|
||||
# orchestrator can recreate one minutes later).
|
||||
declare -A pre_id
|
||||
for c in "${before[@]}"; do
|
||||
pre_id["$c"]=$(podman inspect --format '{{.Id}}' "$c" 2>/dev/null || echo "")
|
||||
done
|
||||
|
||||
# Bounce archipelago. Same approach as companion-survives-* for parity.
|
||||
if systemctl --user list-units --no-legend archipelago.service | grep -q archipelago; then
|
||||
systemctl --user restart archipelago.service
|
||||
else
|
||||
sudo systemctl restart archipelago.service
|
||||
fi
|
||||
|
||||
run wait_archipelago_back 60
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# Every backend that was up before must still be up after, AND it must
|
||||
# be the SAME container instance (same .Id). A different .Id means the
|
||||
# original was killed and a fresh one was created — that's the FM3
|
||||
# failure we're catching.
|
||||
for c in "${before[@]}"; do
|
||||
run container_running "$c"
|
||||
[ "$status" -eq 0 ] || fail "backend $c died across archipelago restart (FM3 cgroup cascade)"
|
||||
|
||||
local post_id
|
||||
post_id=$(podman inspect --format '{{.Id}}' "$c" 2>/dev/null || echo "")
|
||||
[[ -n "$post_id" ]] || fail "backend $c has no container id after restart"
|
||||
[[ "$post_id" == "${pre_id[$c]}" ]] \
|
||||
|| fail "backend $c was recreated across archipelago restart (FM3): pre=${pre_id[$c]:0:12} post=${post_id:0:12}"
|
||||
done
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/bitcoin-knots.bats
|
||||
#
|
||||
# Lifecycle tests for the bitcoin-knots package.
|
||||
#
|
||||
# Tiers:
|
||||
# - Read-only (always runs): presence, status, state-reporting consistency
|
||||
# - Destructive (ARCHY_ALLOW_DESTRUCTIVE=1): stop → start → restart on this very container
|
||||
# - Cascade-destructive (ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1): uninstall → reinstall
|
||||
# — this breaks LND/ElectrumX/BTCPay/mempool, so never enabled on a node serving real users.
|
||||
#
|
||||
# Pre-req: bitcoin-knots is installed. We do NOT install it from scratch here
|
||||
# because doing so on the live host would require wiping 700GB of chain data.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1 # make sure setup_file gets a fresh token
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN # subsequent test subshells reuse the session file
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "container-list includes bitcoin-knots" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "bitcoin-knots")' >/dev/null
|
||||
}
|
||||
|
||||
@test "container-list reports a valid state for bitcoin-knots" {
|
||||
# Poll briefly: a container caught mid-reconcile can momentarily report a
|
||||
# transient state ("restarting"/"configured"/"removing") or no state at all.
|
||||
# A genuinely-stuck container never settles, so this still catches real
|
||||
# breakage; it only absorbs churn (e.g. another container bouncing right
|
||||
# before the read-only tier runs).
|
||||
local state="" deadline=$(( $(date +%s) + 30 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
state=$(echo "$output" | jq -r '.[] | select(.name == "bitcoin-knots") | .state')
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]] && return 0
|
||||
sleep 3
|
||||
done
|
||||
echo "bitcoin-knots never reported a settled valid state within 30s (last: '$state')" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
@test "container-status returns a valid status object for bitcoin-knots" {
|
||||
# During orchestrator alias migration, container-status can fail for some
|
||||
# app_id aliases even while container-list/state is correct. Accept either:
|
||||
# (a) valid container-status object OR (b) valid container-list state entry.
|
||||
run rpc_call container-status '{"app_id":"bitcoin-knots"}'
|
||||
[ "$status" -eq 0 ]
|
||||
local err
|
||||
err=$(echo "$output" | jq -r '.error.message // empty')
|
||||
if [[ -z "$err" ]]; then
|
||||
echo "$output" | jq -e '.result | has("status") or has("state") or has("running")' >/dev/null
|
||||
return 0
|
||||
fi
|
||||
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "bitcoin-knots") | has("state")' >/dev/null
|
||||
}
|
||||
|
||||
@test "bitcoin.getinfo succeeds when bitcoin-knots is running" {
|
||||
local state
|
||||
state=$(rpc_result container-list | jq -r '.[] | select(.name == "bitcoin-knots") | .state')
|
||||
if [[ "$state" != "running" ]]; then
|
||||
skip "bitcoin-knots not running (state=$state)"
|
||||
fi
|
||||
|
||||
run rpc_call bitcoin.getinfo
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.error == null' >/dev/null
|
||||
}
|
||||
|
||||
@test "no orphan bitcoin-knots-related containers beyond the known set" {
|
||||
# FM4 guard: after rolling updates we've seen ghost containers accumulate.
|
||||
# Known-good container set for the bitcoin-knots package is just "bitcoin-knots".
|
||||
# Anything matching bitcoin-knots* in podman ps that isn't in the known set is a red flag.
|
||||
local count
|
||||
count=$(ssh_podman_ps | awk '/bitcoin-knots/ {print $NF}' | grep -Ec '^bitcoin-knots(-[a-z]+)?$' || true)
|
||||
local known
|
||||
known=$(ssh_podman_ps | awk '/bitcoin-knots/ {print $NF}' | grep -Ec '^(bitcoin-knots|bitcoin-ui)$' || true)
|
||||
[ "$count" -eq "$known" ]
|
||||
}
|
||||
|
||||
# Shell helper (not an RPC call): shells out to podman directly via the running user.
|
||||
# Only works when bats is run on the archy host itself (which is the plan).
|
||||
ssh_podman_ps() {
|
||||
podman ps -a --format '{{.ID}} {{.State}} {{.Names}}'
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier (stop → start → restart on the same container)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions bitcoin-knots to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.stop '{"id":"bitcoin-knots"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status bitcoin-knots stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings bitcoin-knots back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.start '{"id":"bitcoin-knots"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status bitcoin-knots running 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves bitcoin-knots in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.restart '{"id":"bitcoin-knots"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status bitcoin-knots running 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "bitcoin.getinfo succeeds after restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# Give bitcoind up to 120s to accept RPC after a cold restart — reloading the
|
||||
# block index + chainstate can take a while even on a synced node.
|
||||
local deadline=$(( $(date +%s) + 120 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if rpc_call bitcoin.getinfo | jq -e '.error == null' >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
# NB: bats-assert's `fail` is not loaded in this file (only ../lib/rpc.bash),
|
||||
# so emit + return non-zero directly rather than calling an undefined helper
|
||||
# (which fails with "fail: command not found" / status 127 and hides the real
|
||||
# reason). A node mid-IBD legitimately can't serve getinfo here — that's an
|
||||
# environmental precondition (see required-stack "synced archival"), not a
|
||||
# product regression.
|
||||
echo "bitcoin.getinfo never recovered after restart within 120s" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade-destructive tier (uninstall + reinstall)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes bitcoin-knots" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.uninstall '{"id":"bitcoin-knots","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status bitcoin-knots absent 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.install bitcoin-knots returns to running" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
# manifest_path is relative to data_dir/apps/
|
||||
run rpc_result package.install '{"manifest_path":"bitcoin-knots/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status bitcoin-knots running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/bitcoin-receive.bats
|
||||
#
|
||||
# Regression coverage for the Bitcoin "Receive" flow. Receive addresses come
|
||||
# from LND's hot wallet via the `lnd.newaddress` RPC, so this exercises the
|
||||
# exact path that broke on the fleet:
|
||||
# - .116: LND REST published on the wrong host port (8080 vs the manifest's
|
||||
# 18080) -> connection refused -> receive failed with the generic
|
||||
# "Operation failed. Check server logs." message.
|
||||
# - .228: the same family surfaced to the UI as a *false* "wallet is locked".
|
||||
#
|
||||
# These tests run on the archy host (they shell into podman / curl localhost).
|
||||
#
|
||||
# Tiers: read-only only — generating a receive address is non-destructive.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# Resolve the LND REST host port from the manifest (single source of truth) so
|
||||
# this test follows the manifest rather than hard-coding 18080.
|
||||
_lnd_rest_host_port() {
|
||||
local mf
|
||||
for mf in \
|
||||
"${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yml" \
|
||||
"${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yaml" \
|
||||
"$BATS_TEST_DIRNAME/../../../apps/lnd/manifest.yml"; do
|
||||
[[ -r "$mf" ]] || continue
|
||||
# The REST mapping is the `- host: <N>` whose following `container:` is 8080.
|
||||
awk '
|
||||
/- host:/ { host=$3 }
|
||||
/container:/ { if ($2 == 8080 && host != "") { print host; exit } }
|
||||
' "$mf"
|
||||
return 0
|
||||
done
|
||||
}
|
||||
|
||||
_lnd_running() {
|
||||
rpc_result container-list 2>/dev/null \
|
||||
| jq -e '.[] | select(.name == "lnd" and .state == "running")' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "LND REST is reachable on the manifest host port (catches port drift)" {
|
||||
_lnd_running || skip "lnd not running"
|
||||
local port
|
||||
port=$(_lnd_rest_host_port)
|
||||
[[ -n "$port" ]] || skip "could not resolve LND REST host port from manifest"
|
||||
|
||||
# A TCP connect is enough: drift (container published on a different host
|
||||
# port) shows up as connection-refused here, exactly as on .116.
|
||||
run curl -sk -o /dev/null --max-time 8 "https://127.0.0.1:${port}/v1/getinfo"
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo "LND REST not reachable on host port ${port} (curl exit $status) — likely published-port drift" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "lnd.newaddress returns a bech32 address when lnd is running" {
|
||||
_lnd_running || skip "lnd not running"
|
||||
|
||||
# The bitcoin bounce in bitcoin-knots.bats cascade-restarts lnd (24fd97ed).
|
||||
# Depending on where the probe lands in lnd's startup it sees a different
|
||||
# transient code — REST unreachable, gRPC "waiting to start" (mapped to
|
||||
# LND_ERROR), wallet locked until the auto-unlocker gets through, or a
|
||||
# post-unlock sync phase. All of those are the node settling, not broken —
|
||||
# retry up to 180s (run A caught WALLET_LOCKED, run B caught LND_ERROR
|
||||
# while lnd was seconds into its restart; both self-healed within a couple
|
||||
# of minutes). Only LND_WALLET_UNINITIALIZED (no wallet — never self-heals)
|
||||
# fails immediately, and anything still erroring after the window fails
|
||||
# loudly below.
|
||||
local deadline=$((SECONDS + 180)) err addr
|
||||
while :; do
|
||||
run rpc_call lnd.newaddress
|
||||
[ "$status" -eq 0 ]
|
||||
err=$(echo "$output" | jq -r '.error.message // .error // empty')
|
||||
addr=$(echo "$output" | jq -r '.result.address // empty')
|
||||
[[ -n "$err" && "$err" != *LND_WALLET_UNINITIALIZED* && $SECONDS -lt $deadline ]] || break
|
||||
sleep 10
|
||||
done
|
||||
|
||||
# The whole point of the fix: a running lnd must hand back a real address.
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "lnd.newaddress errored on a running node: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$addr" != bc1* ]]; then
|
||||
echo "expected a bech32 (bc1…) address, got: '$addr'" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "receive errors are specific, never the generic catch-all" {
|
||||
# Even when receive legitimately can't produce an address, the message must be
|
||||
# actionable (start with 'Bitcoin address' and/or carry a [CODE] token) — the
|
||||
# generic 'Operation failed' is what hid the real cause on .116.
|
||||
run rpc_call lnd.newaddress
|
||||
[ "$status" -eq 0 ]
|
||||
local err
|
||||
err=$(echo "$output" | jq -r '.error.message // .error // empty')
|
||||
if [[ "$err" == "Operation failed. Check server logs for details." ]]; then
|
||||
echo "receive returned the generic catch-all instead of a specific reason" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/btcpay.bats
|
||||
#
|
||||
# Lifecycle tests for the btcpay-server multi-container stack:
|
||||
# - btcpay-server (the main app)
|
||||
# - archy-btcpay-db (postgres)
|
||||
# - archy-nbxplorer (Bitcoin watcher)
|
||||
#
|
||||
# Multi-container variant of bitcoin-knots.bats / lnd.bats / electrumx.bats.
|
||||
# UI URL coverage is in ui-coverage.bats; this suite is L1 (RPC API) + L3
|
||||
# (lifecycle survival).
|
||||
#
|
||||
# Pre-req: btcpay-server installed, bitcoin-knots running.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
btcpay_components=(
|
||||
"btcpay-server"
|
||||
"archy-btcpay-db"
|
||||
"archy-nbxplorer"
|
||||
)
|
||||
|
||||
@test "container-list includes every btcpay-stack component" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
for c in "${btcpay_components[@]}"; do
|
||||
echo "$output" | jq -e --arg n "$c" '.[] | select(.name == $n)' >/dev/null \
|
||||
|| skip "btcpay component $c not present (stack not installed)"
|
||||
done
|
||||
}
|
||||
|
||||
@test "container-list reports valid states for every btcpay component" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local present=0
|
||||
for c in "${btcpay_components[@]}"; do
|
||||
local state
|
||||
state=$(echo "$output" | jq -r --arg n "$c" '.[] | select(.name == $n) | .state')
|
||||
[[ -n "$state" ]] || continue
|
||||
present=$((present + 1))
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]] \
|
||||
|| fail "invalid state for $c: $state"
|
||||
done
|
||||
(( present > 0 )) || skip "btcpay stack not installed"
|
||||
}
|
||||
|
||||
@test "no orphan btcpay-related containers beyond the known set" {
|
||||
local total known
|
||||
total=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(btcpay|archy-btcpay|archy-nbxplorer)' || true)
|
||||
known=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(btcpay-server|archy-btcpay-db|archy-nbxplorer)$' || true)
|
||||
[ "$total" -eq "$known" ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions btcpay-server to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
podman inspect btcpay-server --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "btcpay-server not installed"
|
||||
|
||||
run rpc_result package.stop '{"id":"btcpay-server"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status btcpay-server stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings btcpay-server back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
podman inspect btcpay-server --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "btcpay-server not installed"
|
||||
|
||||
run rpc_result package.start '{"id":"btcpay-server"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status btcpay-server running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves btcpay-server in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
podman inspect btcpay-server --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "btcpay-server not installed"
|
||||
|
||||
run rpc_result package.restart '{"id":"btcpay-server"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status btcpay-server running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "db + nbxplorer remain running across btcpay-server restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
for c in archy-btcpay-db archy-nbxplorer; do
|
||||
podman inspect "$c" --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "btcpay supporting container $c not installed"
|
||||
done
|
||||
|
||||
for c in archy-btcpay-db archy-nbxplorer; do
|
||||
local state
|
||||
state=$(podman inspect --format '{{.State.Status}}' "$c" 2>/dev/null)
|
||||
[[ "$state" == "running" ]] \
|
||||
|| fail "supporting btcpay container $c is not running (state=$state) — package.restart cascaded into it"
|
||||
done
|
||||
}
|
||||
|
||||
@test "package.uninstall removes the whole btcpay stack" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
podman inspect btcpay-server --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "btcpay-server not installed"
|
||||
|
||||
run rpc_result package.uninstall '{"id":"btcpay-server","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
for c in "${btcpay_components[@]}"; do
|
||||
run wait_for_container_status "$c" absent 120
|
||||
[ "$status" -eq 0 ] || fail "btcpay component $c not removed by uninstall"
|
||||
done
|
||||
}
|
||||
|
||||
@test "package.install restores the whole btcpay stack" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install '{"manifest_path":"btcpay-server/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
for c in "${btcpay_components[@]}"; do
|
||||
run wait_for_container_status "$c" running 240
|
||||
[ "$status" -eq 0 ] || fail "btcpay component $c never reached running after reinstall"
|
||||
done
|
||||
}
|
||||
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/cascade-uninstall.bats
|
||||
#
|
||||
# CASCADE-tier regression guard for the uninstall → reinstall lifecycle — the
|
||||
# exact bug class the gate's DESTRUCTIVE tier never exercised:
|
||||
# #13 "uninstall ghost" — app stayed in My Apps after uninstall because the
|
||||
# package state entry wasn't cleared when teardown hit
|
||||
# cleanup residue (returned Err before removing it).
|
||||
# #14 "reinstall stops" — a reinstall stalled partway on the stale state/data
|
||||
# left behind by the broken uninstall.
|
||||
#
|
||||
# Uses a THROWAWAY app (default grafana — not installed on prod/test nodes, no
|
||||
# user data) so it can drive the FULL teardown path (no preserve_data), which is
|
||||
# where #13 actually bit. Precondition-skips if the app is already installed, so
|
||||
# it can NEVER destroy real data on a populated node.
|
||||
#
|
||||
# "No ghost" is asserted against server.get-state's package-data (literally the
|
||||
# My Apps map) — the entry must disappear, not linger with a stale state /
|
||||
# stuck uninstall stage.
|
||||
#
|
||||
# Gated on ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1. RPC-based, so it works on-node or
|
||||
# against a remote ARCHY_HOST (the data-dir residue check is on-node only).
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
CASCADE_APP="${ARCHY_CASCADE_APP:-grafana}"
|
||||
CASCADE_IMAGE="${ARCHY_CASCADE_IMAGE:-docker.io/grafana/grafana:10.2.0}"
|
||||
CASCADE_CONFIG="${ARCHY_CASCADE_CONFIG:-{\"ports\":[\"3000:3000\"],\"volumes\":[\"/var/lib/archipelago/grafana:/var/lib/grafana\"],\"env\":[\"GF_PATHS_DATA=/var/lib/grafana\",\"GF_USERS_ALLOW_SIGN_UP=false\"]}}"
|
||||
CASCADE_DATA_DIR="${ARCHY_CASCADE_DATA_DIR:-/var/lib/archipelago/${CASCADE_APP}}"
|
||||
|
||||
setup_file() {
|
||||
cascade_enabled || return 0
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
cascade_enabled() {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]]
|
||||
}
|
||||
|
||||
# True when CASCADE_APP has an entry in My Apps (server.get-state package-data).
|
||||
app_in_my_apps() {
|
||||
rpc_result server.get-state '{}' 2>/dev/null \
|
||||
| jq -e --arg id "$CASCADE_APP" '.data["package-data"] | has($id)' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Top-level state of CASCADE_APP in My Apps, or "absent" when the entry is gone.
|
||||
app_state() {
|
||||
rpc_result server.get-state '{}' 2>/dev/null \
|
||||
| jq -r --arg id "$CASCADE_APP" '.data["package-data"][$id].state // "absent"'
|
||||
}
|
||||
|
||||
# Live uninstall stage shown by My Apps, or an empty string when the entry is gone
|
||||
# or the backend has not emitted a stage yet.
|
||||
app_uninstall_stage() {
|
||||
rpc_result server.get-state '{}' 2>/dev/null \
|
||||
| jq -r --arg id "$CASCADE_APP" '.data["package-data"][$id]["uninstall-stage"] // ""'
|
||||
}
|
||||
|
||||
# Mirror the frontend's AppCard.vue mapping so the gate proves the UI has
|
||||
# backend data that can render as a monotonic, non-fake progress bar.
|
||||
uninstall_stage_percent() {
|
||||
local stage="$1"
|
||||
if [[ "$stage" =~ \(([0-9]+)[[:space:]]*/[[:space:]]*([0-9]+)\) ]]; then
|
||||
local done="${BASH_REMATCH[1]}" total="${BASH_REMATCH[2]}"
|
||||
if (( total > 0 )); then
|
||||
(( done > total )) && done="$total"
|
||||
echo $(( 10 + (done * 40 / total) ))
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
if [[ "$stage" =~ [Vv]olume ]]; then echo 70; return 0; fi
|
||||
if [[ "$stage" =~ [Dd]ata ]]; then echo 90; return 0; fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# Poll until CASCADE_APP disappears while enforcing the progress contract:
|
||||
# stages must be parseable, monotonic, below 100 before terminal absence, and
|
||||
# the operation must emit at least one visible stage instead of silently hanging.
|
||||
wait_absent_with_truthful_uninstall_progress() {
|
||||
local timeout="${1:-180}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
local saw_stage=0 last_percent=0
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local state stage percent
|
||||
state="$(app_state)"
|
||||
[[ "$state" == "absent" ]] && {
|
||||
(( saw_stage == 1 )) || {
|
||||
echo "uninstall progress: no uninstall-stage observed before terminal absence" >&2
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
stage="$(app_uninstall_stage)"
|
||||
if [[ -n "$stage" ]]; then
|
||||
if ! percent="$(uninstall_stage_percent "$stage")"; then
|
||||
echo "uninstall progress: unparseable stage '$stage'" >&2
|
||||
return 1
|
||||
fi
|
||||
(( percent >= last_percent )) || {
|
||||
echo "uninstall progress regressed: ${percent}% after ${last_percent}% (stage '$stage')" >&2
|
||||
return 1
|
||||
}
|
||||
(( percent < 100 )) || {
|
||||
echo "uninstall progress reached ${percent}% before terminal absence (stage '$stage')" >&2
|
||||
return 1
|
||||
}
|
||||
saw_stage=1
|
||||
last_percent="$percent"
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "wait_absent_with_truthful_uninstall_progress: $CASCADE_APP did not disappear within ${timeout}s (last='$(app_state)', stage='$(app_uninstall_stage)')" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Poll My Apps until CASCADE_APP reaches $1 (a state, or "absent").
|
||||
wait_app_state() {
|
||||
local target="$1" timeout="${2:-180}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
[[ "$(app_state)" == "$target" ]] && return 0
|
||||
sleep 3
|
||||
done
|
||||
echo "wait_app_state: $CASCADE_APP never reached '$target' (last='$(app_state)') within ${timeout}s" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
@test "cascade gate enabled" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
}
|
||||
|
||||
@test "precondition: ${CASCADE_APP} is not already installed (protects real data)" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
if app_in_my_apps; then
|
||||
skip "${CASCADE_APP} already installed here — refusing to uninstall (would destroy data); set ARCHY_CASCADE_APP to an uninstalled throwaway"
|
||||
fi
|
||||
}
|
||||
|
||||
@test "install ${CASCADE_APP} (fresh) reaches running with a truthful, non-silent progression" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
app_in_my_apps && skip "already installed (precondition skip)"
|
||||
|
||||
run rpc_result package.install "{\"id\":\"${CASCADE_APP}\",\"dockerImage\":\"${CASCADE_IMAGE}\",\"containerConfig\":${CASCADE_CONFIG}}"
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# Progress truthfulness: must pass through a transitional install state (not a
|
||||
# silent no-op) and land on running. A warm image cache can blow through the
|
||||
# transitional states between polls, so a missed transitional is a warn, not a
|
||||
# failure; reaching running is the hard assertion.
|
||||
local saw_transitional=0 deadline=$(( $(date +%s) + 300 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
case "$(app_state)" in
|
||||
installing|pulling-image|pulling|downloading|starting|created) saw_transitional=1 ;;
|
||||
running) break ;;
|
||||
esac
|
||||
sleep 2
|
||||
done
|
||||
[ "$(app_state)" == "running" ]
|
||||
[ "$saw_transitional" -eq 1 ] || echo "# note: no transitional install state observed (image likely cached)" >&3
|
||||
}
|
||||
|
||||
@test "uninstall ${CASCADE_APP} reports truthful progress and clears My Apps — NO ghost (#13)" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
app_in_my_apps || skip "${CASCADE_APP} not installed (install step must have failed)"
|
||||
|
||||
run rpc_result package.uninstall "{\"id\":\"${CASCADE_APP}\"}"
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# The container must go away…
|
||||
run wait_for_container_status "$CASCADE_APP" absent 180
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# …AND the My Apps entry must be GONE — the #13 ghost was the entry lingering
|
||||
# with a stale state / stuck uninstall stage. While polling, prove the backend
|
||||
# emits stage data the UI can render as monotonic, non-full progress.
|
||||
run wait_absent_with_truthful_uninstall_progress 120
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# Belt-and-suspenders: the key is truly absent from package-data.
|
||||
run app_in_my_apps
|
||||
[ "$status" -ne 0 ]
|
||||
}
|
||||
|
||||
@test "uninstall removed the data dir (full teardown, no residue)" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
# Needs the local filesystem — on-node runs only.
|
||||
case "${ARCHY_HOST:-127.0.0.1}" in
|
||||
127.0.0.1|localhost) : ;;
|
||||
*) skip "data-dir residue check is on-node only (ARCHY_HOST=${ARCHY_HOST})" ;;
|
||||
esac
|
||||
[[ ! -e "$CASCADE_DATA_DIR" ]]
|
||||
}
|
||||
|
||||
@test "reinstall ${CASCADE_APP} returns to running (#14)" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install "{\"id\":\"${CASCADE_APP}\",\"dockerImage\":\"${CASCADE_IMAGE}\",\"containerConfig\":${CASCADE_CONFIG}}"
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_app_state running 300
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "cleanup: uninstall ${CASCADE_APP} to leave the node as found" {
|
||||
cascade_enabled || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
run rpc_result package.uninstall "{\"id\":\"${CASCADE_APP}\"}"
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_for_container_status "$CASCADE_APP" absent 180
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_absent_with_truthful_uninstall_progress 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/companion-survives-archipelago-restart.bats
|
||||
#
|
||||
# Quadlet promise: companion UIs (archy-bitcoin-ui, archy-lnd-ui,
|
||||
# archy-electrs-ui) are managed by systemd, not archipelago. Restarting
|
||||
# the archipelago user service must NOT take them down.
|
||||
#
|
||||
# This is the regression gate for the .228 incident in
|
||||
# feedback_container_lifecycle_failure_modes.md (FM1: companions vanished
|
||||
# from `podman ps -a` after archipelago crash-loop).
|
||||
#
|
||||
# Gated by ARCHY_ALLOW_DESTRUCTIVE=1 because it bounces archipelago.
|
||||
|
||||
companion_units=(
|
||||
"archy-bitcoin-ui"
|
||||
"archy-lnd-ui"
|
||||
"archy-electrs-ui"
|
||||
)
|
||||
|
||||
unit_dir="$HOME/.config/containers/systemd"
|
||||
|
||||
unit_file_present() {
|
||||
local name="$1"
|
||||
[[ -f "$unit_dir/$name.container" ]]
|
||||
}
|
||||
|
||||
service_active() {
|
||||
local name="$1"
|
||||
systemctl --user is-active --quiet "$name.service"
|
||||
}
|
||||
|
||||
container_running() {
|
||||
local name="$1"
|
||||
[[ "$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null)" == "true" ]]
|
||||
}
|
||||
|
||||
wait_service_active() {
|
||||
local name="$1"
|
||||
local timeout="${2:-60}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if service_active "$name"; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_archipelago_back() {
|
||||
local timeout="${1:-60}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if curl -fsS -o /dev/null "http://127.0.0.1:5678/health" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
@test "destructive gate enabled" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
}
|
||||
|
||||
@test "every installed companion has a quadlet unit on disk" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
local present=0
|
||||
for c in "${companion_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
run unit_file_present "$c"
|
||||
[ "$status" -eq 0 ]
|
||||
present=$(( present + 1 ))
|
||||
fi
|
||||
done
|
||||
(( present > 0 )) || skip "No companions installed on this node"
|
||||
}
|
||||
|
||||
@test "every installed companion service is active before restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
for c in "${companion_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
run service_active "$c"
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
@test "companions survive archipelago restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# Snapshot: which companions were up before we touched anything.
|
||||
local before=()
|
||||
for c in "${companion_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
before+=("$c")
|
||||
fi
|
||||
done
|
||||
(( ${#before[@]} > 0 )) || skip "No companions installed on this node"
|
||||
|
||||
# Bounce archipelago. The user service is the production canonical name;
|
||||
# fall back to the system service for older nodes.
|
||||
if systemctl --user list-units --no-legend archipelago.service | grep -q archipelago; then
|
||||
systemctl --user restart archipelago.service
|
||||
else
|
||||
sudo systemctl restart archipelago.service
|
||||
fi
|
||||
|
||||
run wait_archipelago_back 60
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# Every companion that was up before must still be up + healthy after.
|
||||
for c in "${before[@]}"; do
|
||||
run service_active "$c"
|
||||
[ "$status" -eq 0 ]
|
||||
run container_running "$c"
|
||||
[ "$status" -eq 0 ]
|
||||
done
|
||||
}
|
||||
|
||||
@test "deleted unit file is recreated within one reconcile tick" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# Pick a companion that's currently running.
|
||||
local target=""
|
||||
for c in "${companion_units[@]}"; do
|
||||
if container_running "$c"; then
|
||||
target="$c"
|
||||
break
|
||||
fi
|
||||
done
|
||||
[[ -n "$target" ]] || skip "No companions installed on this node"
|
||||
|
||||
# Delete the unit file behind systemd's back. The reconciler should
|
||||
# notice and rewrite it within one 30s tick, then start the service.
|
||||
rm -f "$unit_dir/$target.container"
|
||||
systemctl --user daemon-reload >/dev/null 2>&1 || true
|
||||
systemctl --user stop "$target.service" >/dev/null 2>&1 || true
|
||||
|
||||
# Allow up to two reconcile ticks (60s + grace).
|
||||
run wait_service_active "$target" 90
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run unit_file_present "$target"
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/electrumx.bats
|
||||
#
|
||||
# Lifecycle tests for the electrumx package (containers are named
|
||||
# `electrumx` + `archy-electrs-ui`). Mirrors bitcoin-knots.bats /
|
||||
# lnd.bats so the 5× release-gate run exercises electrumx through
|
||||
# the same state matrix.
|
||||
#
|
||||
# Tiers:
|
||||
# - Read-only (always runs): presence, valid state, TCP reachable
|
||||
# - Destructive (ARCHY_ALLOW_DESTRUCTIVE=1): stop → start → restart
|
||||
# - Cascade-destructive (ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1): uninstall → reinstall
|
||||
#
|
||||
# Pre-req: electrumx is installed and bitcoin-knots is running (electrumx
|
||||
# depends on bitcoind RPC for headers).
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "container-list includes electrumx" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "electrumx")' >/dev/null
|
||||
}
|
||||
|
||||
@test "container-list reports a valid state for electrumx" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local state
|
||||
state=$(echo "$output" | jq -r '.[] | select(.name == "electrumx") | .state')
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]]
|
||||
}
|
||||
|
||||
@test "electrumx TCP port accepts connections when running" {
|
||||
local state
|
||||
state=$(rpc_result container-list | jq -r '.[] | select(.name == "electrumx") | .state')
|
||||
if [[ "$state" != "running" ]]; then
|
||||
skip "electrumx not running (state=$state)"
|
||||
fi
|
||||
|
||||
# Same probe required-stack.bats uses — divergence flags a real regression.
|
||||
run python3 - <<'PY'
|
||||
import socket
|
||||
s = socket.create_connection(("127.0.0.1", 50001), 3)
|
||||
s.close()
|
||||
print("ok")
|
||||
PY
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "no orphan electrumx-related containers beyond the known set" {
|
||||
# FM4 guard: known-good electrumx-package set is {electrumx, archy-electrs-ui}.
|
||||
local total known
|
||||
total=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(electrumx|electrs|archy-electrs(-[a-z]+)?)$' || true)
|
||||
known=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(electrumx|archy-electrs-ui)$' || true)
|
||||
[ "$total" -eq "$known" ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier (stop → start → restart on the same container)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions electrumx to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.stop '{"id":"electrumx"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status electrumx stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings electrumx back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.start '{"id":"electrumx"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status electrumx running 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves electrumx in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.restart '{"id":"electrumx"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status electrumx running 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "electrumx TCP port recovers after restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# electrumx replays its index against bitcoind on cold start; allow 120s.
|
||||
local deadline=$(( $(date +%s) + 120 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if python3 -c 'import socket; socket.create_connection(("127.0.0.1", 50001), 3).close()' \
|
||||
>/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
fail "electrumx TCP port never reopened after restart"
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade-destructive tier (uninstall + reinstall)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes electrumx" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.uninstall '{"id":"electrumx","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status electrumx absent 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.install electrumx returns to running" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install '{"manifest_path":"electrumx/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status electrumx running 240
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/fedimint.bats
|
||||
#
|
||||
# Lifecycle tests for the fedimint package. The fedimint federation
|
||||
# daemon runs as a single container; the gateway is its own package
|
||||
# (fedimint-gateway). Mirrors the single-container pattern of
|
||||
# lnd.bats / electrumx.bats for L1 (RPC API) + L3 (lifecycle survival).
|
||||
# UI URL coverage is in ui-coverage.bats.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
fedimint_skip_if_absent() {
|
||||
podman inspect fedimint --format '{{.State.Status}}' >/dev/null 2>&1 \
|
||||
|| skip "fedimint not installed"
|
||||
}
|
||||
|
||||
@test "container-list includes fedimint" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "fedimint")' >/dev/null \
|
||||
|| skip "fedimint not installed"
|
||||
}
|
||||
|
||||
@test "container-list reports a valid state for fedimint" {
|
||||
fedimint_skip_if_absent
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local state
|
||||
state=$(echo "$output" | jq -r '.[] | select(.name == "fedimint") | .state')
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]]
|
||||
}
|
||||
|
||||
@test "no orphan fedimint-related containers beyond the known set" {
|
||||
local total known
|
||||
total=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(fedimint|fedimintd|fedimint-gateway)' || true)
|
||||
# `fedimint-clientd` (the dual-ecash HTTP bridge) is a legitimate, known
|
||||
# container — and the unanchored `total` regex above counts it (it starts
|
||||
# with "fedimint"). It must therefore be in the known set too, or every node
|
||||
# running fedimint-clientd false-fails this orphan check.
|
||||
known=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(fedimint|fedimint-clientd|fedimint-gateway)$' || true)
|
||||
[ "$total" -eq "$known" ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions fedimint to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
fedimint_skip_if_absent
|
||||
|
||||
run rpc_result package.stop '{"id":"fedimint"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status fedimint stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings fedimint back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
fedimint_skip_if_absent
|
||||
|
||||
run rpc_result package.start '{"id":"fedimint"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status fedimint running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves fedimint in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
fedimint_skip_if_absent
|
||||
|
||||
run rpc_result package.restart '{"id":"fedimint"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status fedimint running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade-destructive tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes fedimint" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
fedimint_skip_if_absent
|
||||
|
||||
run rpc_result package.uninstall '{"id":"fedimint","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status fedimint absent 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.install fedimint returns to running" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install '{"manifest_path":"fedimint/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status fedimint running 240
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/immich.bats
|
||||
#
|
||||
# Lifecycle tests for the manifest-driven immich stack. The user-facing package is
|
||||
# "immich" (catalog title + icon); container-list reports it package-level as
|
||||
# "immich". Its containers are named immich_server / immich_postgres /
|
||||
# immich_redis (underscore) to match the runtime's per-app lifecycle references.
|
||||
#
|
||||
# Tiers:
|
||||
# - Read-only (always): presence + valid state
|
||||
# - Destructive (ARCHY_ALLOW_DESTRUCTIVE=1): stop → start → restart
|
||||
# - Cascade (ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1): uninstall → reinstall (preserve_data)
|
||||
#
|
||||
# RPC-based, so correct whether run on the host or against a remote ARCHY_HOST.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
IMMICH_IMAGE="146.59.87.168:3000/lfg2025/immich-server:release"
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "container-list includes immich" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "immich")' >/dev/null
|
||||
}
|
||||
|
||||
@test "container-list reports a valid state for immich" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local state
|
||||
state=$(echo "$output" | jq -r '.[] | select(.name == "immich") | .state')
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]]
|
||||
}
|
||||
|
||||
@test "immich exposes its web UI lan-address (port 2283)" {
|
||||
# Poll briefly: lan_address is derived from the published host port, which is
|
||||
# momentarily absent (null) while immich_server is mid-recreate (e.g. a
|
||||
# health-monitor bounce during the read-only tier). A genuinely unexposed
|
||||
# immich never publishes 2283, so this still catches real port drift; it only
|
||||
# absorbs the transient null seen under churn.
|
||||
# 90s (not 30s): the immich stack (postgres→redis→server with DB migrations on
|
||||
# boot) can take >30s to publish its host port after a churn-induced recreate,
|
||||
# and the destructive-tier immich tests already allow 180–240s for the same
|
||||
# stack. A genuinely unexposed immich still never publishes 2283, so this keeps
|
||||
# catching real port drift while tolerating slow-but-healthy boots.
|
||||
local deadline=$(( $(date +%s) + 90 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
if echo "$output" \
|
||||
| jq -e '.[] | select(.name == "immich") | .lan_address // "" | test("2283")' >/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
echo "immich never reported a lan_address containing 2283 within 90s" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier (stop → start → restart)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions immich to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
# package.stop is async ({"status":"stopping"}) and a stack stop can race a
|
||||
# still-settling prior op, so the end state — not the immediate RPC return — is
|
||||
# the assertion.
|
||||
rpc_call package.stop '{"id":"immich"}' >/dev/null 2>&1 || true
|
||||
run wait_for_container_status immich stopped 90
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings immich back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
# Async start; the server comes up only after postgres is ready (~30s+), so wait.
|
||||
rpc_call package.start '{"id":"immich"}' >/dev/null 2>&1 || true
|
||||
run wait_for_container_status immich running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves immich in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
run rpc_result package.restart '{"id":"immich"}'
|
||||
[ "$status" -eq 0 ]
|
||||
# Restart = ordered stop+start of the whole 3-container stack (postgres→redis→
|
||||
# server, with the server doing DB-readiness + migrations on boot), so it needs
|
||||
# at least as long as `start` (180s) — more, since it stops first. The old 120s
|
||||
# was inconsistent with the start test and false-failed on heavily-loaded nodes.
|
||||
run wait_for_container_status immich running 240
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade tier (uninstall + reinstall the stack)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes immich (data preserved)" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
run rpc_result package.uninstall '{"id":"immich","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_for_container_status immich absent 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.install immich returns to running" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
run rpc_result package.install "{\"id\":\"immich\",\"dockerImage\":\"${IMMICH_IMAGE}\"}"
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_for_container_status immich running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/lnd.bats
|
||||
#
|
||||
# Lifecycle tests for the lnd package. Mirrors bitcoin-knots.bats so the
|
||||
# 5× release-gate run exercises lnd through the same state matrix.
|
||||
#
|
||||
# Tiers:
|
||||
# - Read-only (always runs): presence, state-reporting consistency, RPC reachable
|
||||
# - Destructive (ARCHY_ALLOW_DESTRUCTIVE=1): stop → start → restart
|
||||
# - Cascade-destructive (ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1): uninstall → reinstall
|
||||
#
|
||||
# Pre-req: lnd is installed. Reinstall path is gated separately because it
|
||||
# wipes the wallet macaroons and forces re-onboarding.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Read-only tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "container-list includes lnd" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | jq -e '.[] | select(.name == "lnd")' >/dev/null
|
||||
}
|
||||
|
||||
@test "container-list reports a valid state for lnd" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local state
|
||||
state=$(echo "$output" | jq -r '.[] | select(.name == "lnd") | .state')
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]]
|
||||
}
|
||||
|
||||
@test "lnd cli getinfo succeeds when lnd is running" {
|
||||
local state
|
||||
state=$(rpc_result container-list | jq -r '.[] | select(.name == "lnd") | .state')
|
||||
if [[ "$state" != "running" ]]; then
|
||||
skip "lnd not running (state=$state)"
|
||||
fi
|
||||
|
||||
# lnd's RPC readiness LAGS the container "running" state: after a (re)start the
|
||||
# wallet must auto-unlock before lncli answers, so a single-shot getinfo races
|
||||
# that window and false-fails. Retry until ready (~90s), like a health probe.
|
||||
# `timeout 10` per attempt: a wedged lnd RPC (e.g. chain-blind after its
|
||||
# bitcoin backend was recreated under it, .228 2026-07-08) otherwise hangs
|
||||
# a single exec — and with it the whole suite — indefinitely.
|
||||
run sh -lc 'for i in $(seq 1 80); do
|
||||
timeout 10 podman exec lnd lncli \
|
||||
--tlscertpath /root/.lnd/tls.cert \
|
||||
--macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon \
|
||||
--rpcserver localhost:10009 getinfo >/dev/null 2>&1 && exit 0
|
||||
sleep 3
|
||||
done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "no orphan lnd-related containers beyond the known set" {
|
||||
# FM4 guard: rolling updates have left ghost containers behind in the past.
|
||||
# Known-good lnd-package container set is {lnd, archy-lnd-ui}.
|
||||
local total known
|
||||
total=$(podman ps -a --format '{{.Names}}' | grep -Ec '^(archy-)?lnd(-[a-z]+)?$' || true)
|
||||
known=$(podman ps -a --format '{{.Names}}' | grep -Ec '^(lnd|archy-lnd-ui)$' || true)
|
||||
[ "$total" -eq "$known" ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier (stop → start → restart on the same container)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions lnd to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.stop '{"id":"lnd"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status lnd stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.start brings lnd back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.start '{"id":"lnd"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status lnd running 240
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.restart leaves lnd in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.restart '{"id":"lnd"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status lnd running 240
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "lncli getinfo recovers after restart" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
# lnd takes longer than bitcoind to accept RPC after cold restart because
|
||||
# the wallet has to be unlocked first, then it reconnects to bitcoind and
|
||||
# re-syncs the graph. On a loaded node this exceeds 90s (observed ~2min on
|
||||
# .228, then synced_to_chain:true). Give it 240s.
|
||||
local deadline=$(( $(date +%s) + 240 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if sh -lc 'podman exec lnd lncli \
|
||||
--tlscertpath /root/.lnd/tls.cert \
|
||||
--macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon \
|
||||
--rpcserver localhost:10009 getinfo >/dev/null' 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
fail "lncli getinfo never recovered after restart"
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade-destructive tier (uninstall + reinstall)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes lnd" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.uninstall '{"id":"lnd","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status lnd absent 120
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.install lnd returns to running" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install '{"manifest_path":"lnd/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_for_container_status lnd running 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/mempool.bats
|
||||
#
|
||||
# Lifecycle tests for the mempool stack:
|
||||
# - mempool (legacy install path; the frontend container)
|
||||
# - mempool-api (orchestrator-managed; the backend api)
|
||||
# - archy-mempool-db (orchestrator-managed; the mariadb)
|
||||
# - archy-mempool-web (orchestrator-managed; the proxy/static layer)
|
||||
#
|
||||
# The mempool stack is split between the legacy install path (mempool itself)
|
||||
# and orchestrator-managed sub-containers — see uses_orchestrator_install_flow
|
||||
# in install.rs. Tests here treat them as one stack at the package.install/stop
|
||||
# level, addressed by id "mempool". UI URL coverage is in ui-coverage.bats.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
# bats-assert is not loaded in this suite (only rpc.bash), so provide a minimal
|
||||
# `fail` so the `|| fail "..."` guards below report a real assertion failure
|
||||
# instead of an undefined-command status 127 that masks the actual reason.
|
||||
fail() { echo "$@" >&2; return 1; }
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
mempool_components=(
|
||||
"mempool-api"
|
||||
"archy-mempool-db"
|
||||
)
|
||||
|
||||
mempool_optional_components=(
|
||||
"mempool"
|
||||
"archy-mempool-web"
|
||||
)
|
||||
|
||||
mempool_skip_if_absent() {
|
||||
for c in "${mempool_components[@]}"; do
|
||||
podman inspect "$c" --format '{{.State.Status}}' >/dev/null 2>&1 && return 0
|
||||
done
|
||||
skip "mempool stack not installed"
|
||||
}
|
||||
|
||||
@test "container-list includes the core mempool components" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local found=0
|
||||
for c in "${mempool_components[@]}"; do
|
||||
if echo "$output" | jq -e --arg n "$c" '.[] | select(.name == $n)' >/dev/null; then
|
||||
found=$((found + 1))
|
||||
fi
|
||||
done
|
||||
(( found > 0 )) || skip "mempool stack not installed"
|
||||
}
|
||||
|
||||
@test "every present mempool component reports a valid state" {
|
||||
run rpc_result container-list
|
||||
[ "$status" -eq 0 ]
|
||||
local present=0
|
||||
for c in "${mempool_components[@]}" "${mempool_optional_components[@]}"; do
|
||||
local state
|
||||
state=$(echo "$output" | jq -r --arg n "$c" '.[] | select(.name == $n) | .state')
|
||||
[[ -n "$state" ]] || continue
|
||||
present=$((present + 1))
|
||||
[[ "$state" =~ ^(running|stopped|exited|created|paused)$ ]] \
|
||||
|| fail "invalid state for $c: $state"
|
||||
done
|
||||
(( present > 0 )) || skip "mempool stack not installed"
|
||||
}
|
||||
|
||||
@test "no orphan mempool-related containers beyond the known set" {
|
||||
# Poll for steady state (don't single-shot): a stack restart in a prior tier
|
||||
# briefly leaves a recreated member visible alongside its replacement, so a
|
||||
# one-shot count can momentarily see total>known even though the reconciler
|
||||
# converges within seconds. A genuine orphan never clears, so this still
|
||||
# catches it — it just tolerates the transient recreate window.
|
||||
local total known deadline=$(( $(date +%s) + 30 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
total=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(mempool|archy-mempool)' || true)
|
||||
known=$(podman ps -a --format '{{.Names}}' \
|
||||
| grep -Ec '^(mempool|mempool-api|archy-mempool-db|archy-mempool-web)$' || true)
|
||||
[ "$total" -eq "$known" ] && return 0
|
||||
sleep 3
|
||||
done
|
||||
echo "orphan mempool container persisted >30s (total=$total known=$known):" >&2
|
||||
podman ps -a --format '{{.Names}}' | grep -E '^(mempool|archy-mempool)' \
|
||||
| grep -vE '^(mempool|mempool-api|archy-mempool-db|archy-mempool-web)$' >&2 || true
|
||||
return 1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Destructive tier — operate on the package id "mempool" which the
|
||||
# legacy install path treats as the whole stack
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.stop transitions mempool stack to stopped" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
mempool_skip_if_absent
|
||||
|
||||
run rpc_result package.stop '{"id":"mempool"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# The frontend container is the user-visible target; supporting
|
||||
# services may stay running depending on orchestrator policy.
|
||||
if podman inspect mempool --format '{{.State.Status}}' >/dev/null 2>&1; then
|
||||
run wait_for_container_status mempool stopped 60
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
}
|
||||
|
||||
@test "package.start brings mempool stack back to running" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
mempool_skip_if_absent
|
||||
|
||||
run rpc_result package.start '{"id":"mempool"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
if podman inspect mempool --format '{{.State.Status}}' >/dev/null 2>&1; then
|
||||
run wait_for_container_status mempool running 180
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
}
|
||||
|
||||
@test "package.restart leaves mempool stack in running state" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
mempool_skip_if_absent
|
||||
|
||||
run rpc_result package.restart '{"id":"mempool"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
if podman inspect mempool --format '{{.State.Status}}' >/dev/null 2>&1; then
|
||||
run wait_for_container_status mempool running 180
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
}
|
||||
|
||||
@test "mempool api backend remains queryable when stack is up" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
mempool_skip_if_absent
|
||||
|
||||
# mempool-api on :8999 — same probe required-stack.bats uses for parity.
|
||||
# This case runs immediately after package.restart, so mempool-api has just
|
||||
# dropped + must re-establish its electrs/bitcoin connection (it reports
|
||||
# "offline" in the frontend during this window). Give it the same recovery
|
||||
# budget the passing parity probes use (required-stack-destructive: 240s,
|
||||
# package-update-smoke: 300s) — 180s was too tight for the post-restart path.
|
||||
local deadline=$(( $(date +%s) + 300 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if curl -fsS -m 5 "http://127.0.0.1:8999/api/v1/backend-info" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
# NB: bats-assert's `fail` is not loaded in this file (only ../lib/rpc.bash),
|
||||
# so emit + return non-zero directly rather than calling an undefined helper.
|
||||
echo "mempool-api never responded on :8999 within 300s" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Cascade-destructive tier
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "package.uninstall removes the mempool stack" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
mempool_skip_if_absent
|
||||
|
||||
run rpc_result package.uninstall '{"id":"mempool","preserve_data":true}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
for c in "${mempool_components[@]}" "${mempool_optional_components[@]}"; do
|
||||
if podman inspect "$c" --format '{{.State.Status}}' >/dev/null 2>&1; then
|
||||
run wait_for_container_status "$c" absent 120
|
||||
[ "$status" -eq 0 ] || fail "mempool component $c not removed by uninstall"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
@test "package.install restores the mempool stack" {
|
||||
[[ "${ARCHY_ALLOW_CASCADE_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_CASCADE_DESTRUCTIVE not set"
|
||||
|
||||
run rpc_result package.install '{"manifest_path":"mempool/manifest.yaml"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
# At minimum the core orchestrator-managed components must come back.
|
||||
for c in "${mempool_components[@]}"; do
|
||||
run wait_for_container_status "$c" running 240
|
||||
[ "$status" -eq 0 ] || fail "mempool component $c never reached running after reinstall"
|
||||
done
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/package-update-smoke.bats
|
||||
#
|
||||
# Destructive update smoke checks.
|
||||
# Requires RPC auth (ARCHY_PASSWORD) and ARCHY_ALLOW_DESTRUCTIVE=1.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
|
||||
require_destructive() {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
}
|
||||
|
||||
require_auth() {
|
||||
[[ -n "${ARCHY_PASSWORD:-}" ]] || skip "ARCHY_PASSWORD not set"
|
||||
}
|
||||
|
||||
wait_http_ok() {
|
||||
local url="$1"
|
||||
local timeout="${2:-240}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if curl -fsS "$url" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_started_at_change() {
|
||||
local name="$1"
|
||||
local old_started_at="$2"
|
||||
local timeout="${3:-300}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local started_at running
|
||||
started_at=$(podman inspect --format '{{.State.StartedAt}}' "$name" 2>/dev/null || true)
|
||||
running=$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null || true)
|
||||
if [[ -n "$started_at" && "$started_at" != "$old_started_at" && "$running" == "true" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_running() {
|
||||
local name="$1"
|
||||
local timeout="${2:-240}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local running
|
||||
running=$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null || true)
|
||||
if [[ "$running" == "true" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
setup_file() {
|
||||
require_auth
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
@test "package.update bitcoin-ui restarts container and recovers endpoint" {
|
||||
require_destructive
|
||||
|
||||
local before
|
||||
before=$(podman inspect --format '{{.State.StartedAt}}' archy-bitcoin-ui 2>/dev/null || true)
|
||||
[[ -n "$before" ]] || skip "archy-bitcoin-ui container not found"
|
||||
|
||||
run rpc_call package.update '{"id":"bitcoin-ui"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
local err
|
||||
err=$(echo "$output" | jq -r '.error.message // empty')
|
||||
if [[ -z "$err" ]]; then
|
||||
echo "$output" | jq -e '.result.status == "updating"' >/dev/null
|
||||
run wait_started_at_change archy-bitcoin-ui "$before" 360
|
||||
if [[ "$status" -ne 0 ]]; then
|
||||
run wait_running archy-bitcoin-ui 120
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
elif [[ "$err" == *"already updating"* ]]; then
|
||||
:
|
||||
else
|
||||
echo "unexpected package.update error: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
run wait_http_ok "http://127.0.0.1:8334/" 180
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "package.update mempool stack smoke (optional)" {
|
||||
require_destructive
|
||||
[[ "${ARCHY_ALLOW_STACK_UPDATE:-0}" == "1" ]] || skip "ARCHY_ALLOW_STACK_UPDATE not set"
|
||||
|
||||
local before
|
||||
before=$(podman inspect --format '{{.State.StartedAt}}' mempool 2>/dev/null || true)
|
||||
[[ -n "$before" ]] || skip "mempool container not found"
|
||||
|
||||
run rpc_call package.update '{"id":"mempool"}'
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
local err
|
||||
err=$(echo "$output" | jq -r '.error.message // empty')
|
||||
if [[ -z "$err" ]]; then
|
||||
echo "$output" | jq -e '.result.status == "updating"' >/dev/null
|
||||
run wait_started_at_change mempool "$before" 420
|
||||
if [[ "$status" -ne 0 ]]; then
|
||||
run wait_running mempool 120
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
elif [[ "$err" == *"already updating"* ]]; then
|
||||
:
|
||||
else
|
||||
echo "unexpected package.update error: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
run wait_http_ok "http://127.0.0.1:4080/" 240
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
run wait_http_ok "http://127.0.0.1:8999/api/v1/backend-info" 300
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/port-drift.bats
|
||||
#
|
||||
# Regression guard for the .116 failure class: a backend container that is
|
||||
# "Up" but publishes its ports to the WRONG host ports because the manifest
|
||||
# changed after the container was created (e.g. lnd REST stuck on host 8080
|
||||
# while the manifest — and every in-process client — expects 18080).
|
||||
#
|
||||
# This mirrors the orchestrator's `host_port_bindings_drifted` check, but from
|
||||
# the outside: it compares the live `podman inspect` PortBindings against the
|
||||
# manifest `ports:` for each installed backend. Runs on the archy host.
|
||||
#
|
||||
# Tiers: read-only.
|
||||
|
||||
_apps_dir() {
|
||||
local d
|
||||
for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
|
||||
"$BATS_TEST_DIRNAME/../../../apps"; do
|
||||
[[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_manifest_for() {
|
||||
local app="$1" dir
|
||||
dir=$(_apps_dir) || return 1
|
||||
local mf
|
||||
for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
|
||||
[[ -r "$mf" ]] && { echo "$mf"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Emit "host container" pairs from a manifest's ports: block.
|
||||
_manifest_ports() {
|
||||
awk '
|
||||
/^[[:space:]]*ports:/ { inports=1; next }
|
||||
inports && /^[[:space:]]*[a-z_]+:[[:space:]]*$/ && !/protocol:|host:|container:/ { inports=0 }
|
||||
inports && /- host:/ { host=$3 }
|
||||
inports && /container:/ { print host, $2 }
|
||||
' "$1"
|
||||
}
|
||||
|
||||
# For a given container + (host,container) port, emit a "DRIFT: …" line on
|
||||
# mismatch (and nothing otherwise). Stays silent for unpublished / host-net
|
||||
# ports — those are handled elsewhere and must never be treated as drift.
|
||||
_drift_line() {
|
||||
local cname="$1" want_host="$2" cport="$3"
|
||||
local bindings actual
|
||||
bindings=$(podman inspect "$cname" --format '{{json .HostConfig.PortBindings}}' 2>/dev/null) || return 0
|
||||
actual=$(echo "$bindings" | jq -r --arg k "${cport}/tcp" '.[$k][]?.HostPort // empty' 2>/dev/null)
|
||||
[[ -n "$actual" ]] || return 0
|
||||
echo "$actual" | grep -qx "$want_host" && return 0
|
||||
echo "DRIFT: $cname container-port $cport published on host [$actual] but manifest wants $want_host"
|
||||
}
|
||||
|
||||
@test "backend containers publish ports that match their manifest" {
|
||||
command -v podman >/dev/null 2>&1 || skip "podman not available"
|
||||
local checked=0 violations="" app cname mf line
|
||||
# container-name : manifest-app-id
|
||||
for pair in "lnd:lnd" "bitcoin-knots:bitcoin-knots" "electrumx:electrumx"; do
|
||||
cname="${pair%%:*}"; app="${pair##*:}"
|
||||
podman container exists "$cname" 2>/dev/null || continue
|
||||
mf=$(_manifest_for "$app") || continue
|
||||
while read -r host cport; do
|
||||
[[ -n "$host" && -n "$cport" ]] || continue
|
||||
checked=$((checked + 1))
|
||||
line=$(_drift_line "$cname" "$host" "$cport")
|
||||
[[ -n "$line" ]] && violations+="${line}"$'\n'
|
||||
done < <(_manifest_ports "$mf")
|
||||
done
|
||||
[[ "$checked" -gt 0 ]] || skip "no installed backend containers with published ports to check"
|
||||
if [[ -n "$violations" ]]; then
|
||||
echo "published-port drift detected:" >&2
|
||||
echo "$violations" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/required-stack-destructive.bats
|
||||
#
|
||||
# Controlled destructive lifecycle checks for required stack containers.
|
||||
# Runs only when ARCHY_ALLOW_DESTRUCTIVE=1.
|
||||
|
||||
required_containers=(
|
||||
"archy-bitcoin-ui"
|
||||
"archy-lnd-ui"
|
||||
"archy-electrs-ui"
|
||||
"mempool"
|
||||
"mempool-api"
|
||||
)
|
||||
|
||||
container_installed() {
|
||||
podman ps -a --format '{{.Names}}' | grep -Fx "$1" >/dev/null
|
||||
}
|
||||
|
||||
# Only the subset of required_containers actually installed on this node —
|
||||
# a node without the mempool stack (or another optional app) shouldn't
|
||||
# hard-fail restarting/probing something it was never meant to have.
|
||||
installed_required_containers() {
|
||||
local c
|
||||
for c in "${required_containers[@]}"; do
|
||||
container_installed "$c" && echo "$c"
|
||||
done
|
||||
# Always succeed — under `set -e`, the function's own exit code is that of
|
||||
# its last statement, so if the last array entry happens to be a container
|
||||
# NOT installed on this node, the whole function (and any bare
|
||||
# `x="$(installed_required_containers)"` caller) would spuriously fail even
|
||||
# though earlier entries matched fine.
|
||||
return 0
|
||||
}
|
||||
|
||||
wait_running() {
|
||||
local name="$1"
|
||||
local timeout="${2:-120}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local running
|
||||
running=$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null || true)
|
||||
if [[ "$running" == "true" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_http_ok() {
|
||||
local url="$1"
|
||||
local timeout="${2:-180}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if curl -fsS "$url" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
restart_with_retry() {
|
||||
local name="$1"
|
||||
local attempts="${2:-3}"
|
||||
local i
|
||||
for ((i=1; i<=attempts; i++)); do
|
||||
if podman restart "$name" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
@test "required-stack destructive gate enabled" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
}
|
||||
|
||||
@test "restart each required service container and verify it recovers" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
local targets; targets="$(installed_required_containers)"
|
||||
[[ -n "$targets" ]] || skip "none of required_containers installed on this node"
|
||||
while IFS= read -r c; do
|
||||
run restart_with_retry "$c" 4
|
||||
[ "$status" -eq 0 ]
|
||||
run wait_running "$c" 180
|
||||
[ "$status" -eq 0 ]
|
||||
done <<< "$targets"
|
||||
}
|
||||
|
||||
@test "required endpoints still respond after restarts" {
|
||||
[[ "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || skip "ARCHY_ALLOW_DESTRUCTIVE not set"
|
||||
|
||||
if container_installed archy-bitcoin-ui; then
|
||||
run wait_http_ok "http://127.0.0.1:8334/" 180
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
|
||||
# :8081 is nginx-proxy-manager — an OPTIONAL app (not in required_containers).
|
||||
# Only assert it when NPM is actually installed on this node; otherwise the
|
||||
# required-endpoints check false-fails on nodes that don't run NPM.
|
||||
if podman ps --format '{{.Names}}' | grep -q '^nginx-proxy-manager$'; then
|
||||
run wait_http_ok "http://127.0.0.1:8081/" 180
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
|
||||
if container_installed mempool; then
|
||||
run wait_http_ok "http://127.0.0.1:4080/" 180
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
|
||||
if container_installed mempool-api; then
|
||||
run wait_http_ok "http://127.0.0.1:8999/api/v1/backend-info" 240
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
|
||||
if container_installed lnd; then
|
||||
# lnd RPC readiness lags container 'running' (wallet unlock + graph sync) —
|
||||
# retry rather than single-shot. See lnd.bats.
|
||||
run sh -lc 'for i in $(seq 1 60); do
|
||||
podman exec lnd lncli --tlscertpath /root/.lnd/tls.cert --macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon --rpcserver localhost:10009 getinfo >/dev/null 2>&1 && exit 0
|
||||
sleep 3
|
||||
done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/required-stack.bats
|
||||
#
|
||||
# Read-only release-gate checks for the Bitcoin/electrum/lnd/mempool stack.
|
||||
# Originally written against .116's fixed app roster; the "present"/"running"
|
||||
# checks below now only require containers actually installed on THIS node
|
||||
# (podman_all_names — present in `podman ps -a` even if stopped), so a node
|
||||
# with a different app subset (e.g. no mempool stack) doesn't hard-fail on
|
||||
# apps it was never meant to have. Per-app checks further down (mempool,
|
||||
# filebrowser, ...) skip individually if that app isn't installed, matching
|
||||
# the mempool_skip_if_absent idiom in mempool.bats.
|
||||
#
|
||||
# This suite is intentionally non-destructive and does not use RPC auth;
|
||||
# it can run anytime as a health gate during long sync/reindex windows.
|
||||
|
||||
required_containers=(
|
||||
"bitcoin-knots"
|
||||
"electrumx"
|
||||
"lnd"
|
||||
"archy-mempool-db"
|
||||
"mempool-api"
|
||||
"mempool"
|
||||
"filebrowser"
|
||||
"archy-bitcoin-ui"
|
||||
"archy-lnd-ui"
|
||||
"archy-electrs-ui"
|
||||
)
|
||||
|
||||
fail() { echo "$@" >&2; return 1; }
|
||||
|
||||
podman_names() {
|
||||
podman ps --format '{{.Names}}'
|
||||
}
|
||||
|
||||
podman_all_names() {
|
||||
podman ps -a --format '{{.Names}}'
|
||||
}
|
||||
|
||||
container_running() {
|
||||
local name="$1"
|
||||
podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null
|
||||
}
|
||||
|
||||
container_installed() {
|
||||
local name="$1"
|
||||
podman_all_names | grep -Fx "$name" >/dev/null
|
||||
}
|
||||
|
||||
skip_if_not_installed() {
|
||||
container_installed "$1" || skip "$1 not installed on this node"
|
||||
}
|
||||
|
||||
# The subset of required_containers actually installed on this node.
|
||||
installed_required_containers() {
|
||||
local c
|
||||
for c in "${required_containers[@]}"; do
|
||||
container_installed "$c" && echo "$c"
|
||||
done
|
||||
# Always succeed — see the identical comment in required-stack-destructive.bats.
|
||||
return 0
|
||||
}
|
||||
|
||||
bitcoin_rpc() {
|
||||
curl -fsS --max-time 60 \
|
||||
--user "archipelago:$(cat /var/lib/archipelago/secrets/bitcoin-rpc-password)" \
|
||||
--data-binary '{"jsonrpc":"1.0","id":"required-stack","method":"getblockchaininfo","params":[]}' \
|
||||
-H 'content-type: text/plain;' \
|
||||
http://127.0.0.1:8332/
|
||||
}
|
||||
|
||||
bitcoin_json() {
|
||||
python3 -c 'import json,sys; r=json.load(sys.stdin)["result"]; print(r[sys.argv[1]])' "$1"
|
||||
}
|
||||
|
||||
@test "required containers are present" {
|
||||
# Under sustained 5× churn an app may still be mid-restart when this runs;
|
||||
# wait for the whole required set rather than single-shot. Only checks
|
||||
# containers actually installed on this node (see installed_required_containers).
|
||||
local targets; targets="$(installed_required_containers)"
|
||||
[[ -n "$targets" ]] || skip "none of required_containers installed on this node"
|
||||
local deadline=$(( $(date +%s) + 180 )) names missing
|
||||
while (( $(date +%s) < deadline )); do
|
||||
names="$(podman_names)"; missing=""
|
||||
while IFS= read -r c; do
|
||||
echo "$names" | grep -Fx "$c" >/dev/null || missing="$missing $c"
|
||||
done <<< "$targets"
|
||||
[[ -z "$missing" ]] && return 0
|
||||
sleep 3
|
||||
done
|
||||
fail "required containers never all present; missing:$missing"
|
||||
}
|
||||
|
||||
@test "required containers are running" {
|
||||
local targets; targets="$(installed_required_containers)"
|
||||
[[ -n "$targets" ]] || skip "none of required_containers installed on this node"
|
||||
local deadline=$(( $(date +%s) + 180 )) notrunning
|
||||
while (( $(date +%s) < deadline )); do
|
||||
notrunning=""
|
||||
while IFS= read -r c; do
|
||||
[[ "$(container_running "$c" 2>/dev/null)" == "true" ]] || notrunning="$notrunning $c"
|
||||
done <<< "$targets"
|
||||
[[ -z "$notrunning" ]] && return 0
|
||||
sleep 3
|
||||
done
|
||||
fail "required containers never all running; not-running:$notrunning"
|
||||
}
|
||||
|
||||
@test "bitcoin-knots RPC responds" {
|
||||
skip_if_not_installed bitcoin-knots
|
||||
run bitcoin_rpc
|
||||
[ "$status" -eq 0 ]
|
||||
echo "$output" | python3 -c 'import json,sys; r=json.load(sys.stdin)["result"]; assert r["chain"] == "main" and r["blocks"] >= 0'
|
||||
}
|
||||
|
||||
@test "bitcoin backend is synced archival for electrumx/lnd gate" {
|
||||
skip_if_not_installed bitcoin-knots
|
||||
run bitcoin_rpc
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
local pruned ibd blocks headers
|
||||
pruned="$(echo "$output" | bitcoin_json pruned)"
|
||||
ibd="$(echo "$output" | bitcoin_json initialblockdownload)"
|
||||
blocks="$(echo "$output" | bitcoin_json blocks)"
|
||||
headers="$(echo "$output" | bitcoin_json headers)"
|
||||
|
||||
if [ "$pruned" = "True" ] || [ "$pruned" = "true" ]; then
|
||||
echo "bitcoin is pruned (blocks=$blocks headers=$headers); electrumx cannot index pruned historical blocks"
|
||||
return 1
|
||||
fi
|
||||
if [ "$ibd" = "True" ] || [ "$ibd" = "true" ]; then
|
||||
echo "bitcoin is still in initial block download (blocks=$blocks headers=$headers)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "electrumx TCP port accepts connections" {
|
||||
skip_if_not_installed electrumx
|
||||
run python3 - <<'PY'
|
||||
import socket
|
||||
s = socket.create_connection(("127.0.0.1", 50001), 3)
|
||||
s.close()
|
||||
print("ok")
|
||||
PY
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "lnd CLI getinfo succeeds" {
|
||||
skip_if_not_installed lnd
|
||||
# lnd RPC readiness lags the container "running" state (wallet auto-unlock on
|
||||
# start), so retry until ready rather than single-shot. See lnd.bats note.
|
||||
run sh -lc 'for i in $(seq 1 30); do
|
||||
timeout 20 podman exec lnd lncli --tlscertpath /root/.lnd/tls.cert --macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon --rpcserver localhost:10009 getinfo >/dev/null 2>&1 && exit 0
|
||||
sleep 3
|
||||
done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "lnd REST port accepts connections" {
|
||||
skip_if_not_installed lnd
|
||||
run python3 - <<'PY'
|
||||
import socket
|
||||
s = socket.create_connection(("127.0.0.1", 18080), 3)
|
||||
s.close()
|
||||
print("ok")
|
||||
PY
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "mempool api endpoint responds" {
|
||||
skip_if_not_installed mempool-api
|
||||
# mempool-api reconnects to electrumx after a stack restart — retry ~180s.
|
||||
run sh -lc 'for i in $(seq 1 60); do curl -fsS -m 5 -o /dev/null "http://127.0.0.1:8999/api/v1/backend-info" && exit 0; sleep 3; done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "mempool frontend responds" {
|
||||
skip_if_not_installed mempool
|
||||
run sh -lc 'for i in $(seq 1 60); do curl -fsS -m 5 -o /dev/null "http://127.0.0.1:4080/" && exit 0; sleep 3; done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "bitcoin ui responds" {
|
||||
skip_if_not_installed archy-bitcoin-ui
|
||||
# The companion (archy-bitcoin-ui) may have just been recreated by an earlier
|
||||
# companion-survives test; its nginx takes a moment to serve. Retry ~120s
|
||||
# rather than single-shot.
|
||||
run sh -lc 'for i in $(seq 1 40); do curl -fsS -o /dev/null "http://127.0.0.1:8334/" && exit 0; sleep 3; done; exit 1'
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "lnd ui responds" {
|
||||
skip_if_not_installed archy-lnd-ui
|
||||
run curl -fsS "http://127.0.0.1:18083/"
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "filebrowser responds" {
|
||||
skip_if_not_installed filebrowser
|
||||
run curl -fsS "http://127.0.0.1:8083/"
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/secret-completeness.bats
|
||||
#
|
||||
# Regression guard for the .198 failure class: a manifest references a
|
||||
# `secret_env.secret_file` that was never generated on the node, so secret
|
||||
# resolution hard-fails and the container won't start — cascading the whole
|
||||
# Bitcoin stack. (bitcoin-knots gained `bitcoin-rpc-txrelay-rpcauth`, which old
|
||||
# nodes lacked, so bitcoind never came up and reinstall "just stopped".)
|
||||
#
|
||||
# For every installed backend, assert every secret_file it references exists in
|
||||
# the secrets dir. Runs on the archy host.
|
||||
#
|
||||
# Tiers: read-only.
|
||||
|
||||
SECRETS_DIR="${ARCHY_SECRETS_DIR:-/var/lib/archipelago/secrets}"
|
||||
|
||||
_apps_dir() {
|
||||
local d
|
||||
for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
|
||||
"$BATS_TEST_DIRNAME/../../../apps"; do
|
||||
[[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_manifest_for() {
|
||||
local app="$1" dir mf
|
||||
dir=$(_apps_dir) || return 1
|
||||
for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
|
||||
[[ -r "$mf" ]] && { echo "$mf"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
_secret_files_in() {
|
||||
# Emit each `secret_file:` value referenced by the manifest.
|
||||
grep -E '^[[:space:]]*secret_file:' "$1" 2>/dev/null | awk '{print $2}'
|
||||
}
|
||||
|
||||
_secret_exists() {
|
||||
local f="$SECRETS_DIR/$1"
|
||||
[[ -e "$f" ]] && return 0
|
||||
sudo -n test -f "$f" 2>/dev/null
|
||||
}
|
||||
|
||||
@test "every installed backend's referenced secrets exist on disk" {
|
||||
command -v podman >/dev/null 2>&1 || skip "podman not available"
|
||||
[[ -d "$SECRETS_DIR" ]] || sudo -n test -d "$SECRETS_DIR" 2>/dev/null || skip "secrets dir not present"
|
||||
|
||||
local checked=0 missing="" app cname mf sf
|
||||
# container-name : manifest-app-id (the bitcoin stack that cascades)
|
||||
for pair in \
|
||||
"bitcoin-knots:bitcoin-knots" "lnd:lnd" "electrumx:electrumx" \
|
||||
"mempool-api:mempool-api" "btcpay-server:btcpay-server" \
|
||||
"archy-nbxplorer:archy-nbxplorer" "fedimint:fedimint" \
|
||||
"fedimint-gateway:fedimint-gateway"; do
|
||||
cname="${pair%%:*}"; app="${pair##*:}"
|
||||
podman container exists "$cname" 2>/dev/null || continue
|
||||
mf=$(_manifest_for "$app") || continue
|
||||
while read -r sf; do
|
||||
[[ -n "$sf" ]] || continue
|
||||
checked=$((checked + 1))
|
||||
_secret_exists "$sf" || missing+="${app} -> ${sf}\n"
|
||||
done < <(_secret_files_in "$mf")
|
||||
done
|
||||
|
||||
[[ "$checked" -gt 0 ]] || skip "no installed backends with secret references to check"
|
||||
if [[ -n "$missing" ]]; then
|
||||
echo "installed apps reference missing secrets:" >&2
|
||||
echo -e "$missing" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/ui-coverage.bats
|
||||
#
|
||||
# UI surface tests — exercises the URLs a real user actually clicks
|
||||
# through, not just the JSON-RPC API. Fills the coverage gap where the
|
||||
# previous bats suites would report "container is up" while the iframe
|
||||
# behind /app/<id>/ was returning 502 because nginx had a stale upstream
|
||||
# or the proxy port was wrong.
|
||||
#
|
||||
# URL map sourced from neode-ui/src/views/appSession/appSessionConfig.ts
|
||||
# (the frontend's own resolveAppUrl). Tests here MUST stay in sync with
|
||||
# that file — divergence is the whole bug class we're guarding against.
|
||||
#
|
||||
# Each app probe is gated on its container being running:
|
||||
# - container down → skip (clean dependency report, no false-fail)
|
||||
# - container up → URL MUST return 200 with non-empty body
|
||||
#
|
||||
# Looped 5× via tests/lifecycle/run-gate.sh.
|
||||
|
||||
load '../lib/rpc.bash'
|
||||
load '../lib/ui-probes.bash'
|
||||
|
||||
setup_file() {
|
||||
: "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
|
||||
export ARCHY_FORCE_LOGIN=1
|
||||
rpc_login
|
||||
unset ARCHY_FORCE_LOGIN
|
||||
HOST="${ARCHY_HOST:-127.0.0.1}"
|
||||
export HOST
|
||||
}
|
||||
|
||||
teardown_file() {
|
||||
rpc_logout_local
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Dashboard shell + catalog (always required)
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "dashboard https://host/ returns the Vue SPA shell" {
|
||||
run probe_dashboard_shell
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "dashboard catalog endpoint responds with apps" {
|
||||
run probe_dashboard_catalog
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Bitcoin UI — direct host port (8334), companion container
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "bitcoin-ui is reachable on :8334 when archy-bitcoin-ui is running" {
|
||||
probe_app_url archy-bitcoin-ui "http://$HOST:8334/" "bitcoin-ui (direct port 8334)"
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# HTTPS proxy paths — match HTTPS_PROXY_PATHS in appSessionConfig.ts
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@test "lnd proxy https://host/app/lnd/ responds when lnd is running" {
|
||||
probe_app_url lnd "https://$HOST/app/lnd/" "lnd (proxy /app/lnd/)"
|
||||
}
|
||||
|
||||
@test "electrumx proxy https://host/app/electrumx/ responds when electrumx is running" {
|
||||
# electrumx companion (archy-electrs-ui) is what serves the iframe HTML;
|
||||
# the electrumx daemon is just the TCP backend.
|
||||
probe_app_url archy-electrs-ui "https://$HOST/app/electrumx/" "electrumx (proxy /app/electrumx/)"
|
||||
}
|
||||
|
||||
@test "mempool proxy https://host/app/mempool/ responds when mempool is running" {
|
||||
probe_app_url mempool "https://$HOST/app/mempool/" "mempool (proxy /app/mempool/)"
|
||||
}
|
||||
|
||||
@test "fedimint proxy https://host/app/fedimint/ responds when fedimint is running" {
|
||||
probe_app_url fedimint "https://$HOST/app/fedimint/" "fedimint (proxy /app/fedimint/)"
|
||||
}
|
||||
|
||||
@test "btcpay proxy https://host/app/btcpay/ responds when btcpay-server is running" {
|
||||
probe_app_url btcpay-server "https://$HOST/app/btcpay/" "btcpay (proxy /app/btcpay/)"
|
||||
}
|
||||
|
||||
@test "filebrowser proxy https://host/app/filebrowser/ responds when filebrowser is running" {
|
||||
probe_app_url filebrowser "https://$HOST/app/filebrowser/" "filebrowser (proxy /app/filebrowser/)"
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Companion-served URLs that aren't in HTTPS_PROXY_PATHS but show up
|
||||
# in the dashboard. archy-lnd-ui shares lnd's iframe path; archy-electrs-ui
|
||||
# shares electrumx's. The earlier test already covers those — leaving
|
||||
# this section for future companion-direct probes (none today).
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/lifecycle/bats/use-quadlet-backends-install.bats
|
||||
#
|
||||
# Validates the post-condition of Phase 3.2's `use_quadlet_backends`
|
||||
# install path. When the orchestrator routed at least one backend
|
||||
# install through `install_via_quadlet`, this suite asserts that the
|
||||
# resulting state has the four properties the Phase 3 design promises:
|
||||
#
|
||||
# 1. A `.container` unit file exists in ~/.config/containers/systemd/
|
||||
# and is well-formed (required sections + directives).
|
||||
# 2. The corresponding `.service` is active under `systemctl --user`.
|
||||
# 3. The container is in `podman ps` (running).
|
||||
# 4. The container's cgroup is under `user.slice/...`, NOT under
|
||||
# `archipelago.service` — proving FM3 (cgroup cascade SIGKILL on
|
||||
# archipelago restart) is structurally fixed for that container.
|
||||
#
|
||||
# Auto-skips if no Quadlet-managed backend exists yet — so it runs as a
|
||||
# no-op on nodes where `use_quadlet_backends` is still false (today's
|
||||
# default), and turns into a hard regression gate as soon as anyone
|
||||
# flips the flag and reinstalls.
|
||||
#
|
||||
# Run on a node with rootless podman + systemd-user (every alpha-fleet
|
||||
# box). No env vars required for the read-only checks. The cleanup
|
||||
# section at the bottom is gated by ARCHY_ALLOW_DESTRUCTIVE=1.
|
||||
|
||||
# bats-core ships no `fail`; bats-assert isn't installed on the alpha fleet.
|
||||
# Define the same minimal helper the other suites use (see mempool.bats) so a
|
||||
# tripped assertion reports as a real test failure, not a status-127 crash.
|
||||
fail() { echo "$@" >&2; return 1; }
|
||||
|
||||
quadlet_dir() {
|
||||
echo "${XDG_CONFIG_HOME:-$HOME/.config}/containers/systemd"
|
||||
}
|
||||
|
||||
# List Quadlet `.container` units that correspond to backend containers
|
||||
# (i.e., NOT companions like archy-*-ui, which already shipped via Quadlet
|
||||
# in v1.7.41 and have their own coverage in companion-survives-archipelago-
|
||||
# restart.bats). Echoes one container name per line; empty if none found.
|
||||
backend_quadlet_units() {
|
||||
local d
|
||||
d="$(quadlet_dir)"
|
||||
[[ -d "$d" ]] || return 0
|
||||
# Strip the .container extension; filter out archy-*-ui companions.
|
||||
# wyoming-* (piper/whisper voice services) are mid-integration and not yet
|
||||
# part of the platform contract — exclude until their packaging lands.
|
||||
for f in "$d"/*.container; do
|
||||
[[ -e "$f" ]] || continue
|
||||
local name
|
||||
name="$(basename "$f" .container)"
|
||||
[[ "$name" =~ ^archy-.*-ui$ ]] && continue
|
||||
[[ "$name" =~ ^wyoming- ]] && continue
|
||||
echo "$name"
|
||||
done
|
||||
}
|
||||
|
||||
# A unit file on disk does NOT imply the app should be running: an
|
||||
# explicitly user-stopped app keeps its .container file (e.g. the inactive
|
||||
# half of the bitcoin-core/bitcoin-knots multi-version pair), and its
|
||||
# .service being inactive / container absent is the CORRECT state. The
|
||||
# orchestrator persists that intent in user-stopped.json; honour it here so
|
||||
# the active-state assertions below don't false-fail on stopped-on-purpose
|
||||
# apps (gate tests 123/124, .228 2026-07-09).
|
||||
USER_STOPPED_FILE="${ARCHY_DATA_DIR:-/var/lib/archipelago}/user-stopped.json"
|
||||
|
||||
is_user_stopped() {
|
||||
local name="$1"
|
||||
[[ -r "$USER_STOPPED_FILE" ]] || return 1
|
||||
jq -e --arg n "$name" --arg s "${name#archy-}" \
|
||||
'index($n) != null or index($s) != null' "$USER_STOPPED_FILE" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Read the cgroup path of a running container's main process. For
|
||||
# rootless podman the conmon-run target lands the container's pid1 in
|
||||
# the cgroup that owns its supervising .service.
|
||||
container_cgroup_path() {
|
||||
local name="$1"
|
||||
local pid
|
||||
pid="$(podman inspect --format '{{.State.Pid}}' "$name" 2>/dev/null)"
|
||||
[[ -n "$pid" && "$pid" != "0" ]] || return 1
|
||||
# cgroup v2 line: "0::/path/to/cgroup"
|
||||
awk -F: '$1=="0"{print $3}' "/proc/$pid/cgroup" 2>/dev/null
|
||||
}
|
||||
|
||||
# Per-test gate. Each @test calls this so the suite is a clean no-op on
|
||||
# nodes where use_quadlet_backends is still false (today's default) —
|
||||
# bats doesn't propagate setup-level skip semantics across @test blocks.
|
||||
require_quadlet_backends() {
|
||||
local count
|
||||
count="$(backend_quadlet_units | wc -l)"
|
||||
(( count > 0 )) || skip "no backend .container units in $(quadlet_dir) — use_quadlet_backends not enabled or no backends installed"
|
||||
}
|
||||
|
||||
@test "Quadlet unit dir exists or is plausibly creatable" {
|
||||
local d
|
||||
d="$(quadlet_dir)"
|
||||
# Either it already exists, or its parent does (so quadlet can mkdir it).
|
||||
[[ -d "$d" ]] || [[ -d "$(dirname "$d")" ]] \
|
||||
|| skip "no XDG_CONFIG_HOME and no \$HOME/.config — not a desktop-style host"
|
||||
}
|
||||
|
||||
@test "each backend Quadlet unit has the required sections + directives" {
|
||||
require_quadlet_backends
|
||||
local d
|
||||
d="$(quadlet_dir)"
|
||||
while read -r name; do
|
||||
[[ -z "$name" ]] && continue
|
||||
local body
|
||||
body="$(<"$d/$name.container")"
|
||||
# [Container] section + Image=
|
||||
[[ "$body" == *"[Container]"* ]] || fail "$name: missing [Container] section"
|
||||
[[ "$body" == *"Image="* ]] || fail "$name: missing Image= directive"
|
||||
# [Service] section with the Phase 3.2 backend invariant: Restart=on-failure.
|
||||
# Companions use Restart=always; backends use on-failure so an operator-issued
|
||||
# `systemctl stop` actually stays stopped.
|
||||
[[ "$body" == *"[Service]"* ]] || fail "$name: missing [Service] section"
|
||||
[[ "$body" == *"Restart=on-failure"* ]] \
|
||||
|| fail "$name: backend unit must use Restart=on-failure (got companion-style Restart=always)"
|
||||
# [Install] section so `systemctl --user enable` is well-defined.
|
||||
[[ "$body" == *"[Install]"* ]] || fail "$name: missing [Install] section"
|
||||
[[ "$body" == *"WantedBy="* ]] || fail "$name: missing WantedBy= in [Install]"
|
||||
done < <(backend_quadlet_units)
|
||||
}
|
||||
|
||||
@test "health is app-level state, NOT a systemd start gate (no Notify=healthy)" {
|
||||
require_quadlet_backends
|
||||
# Phase 3.4 originally emitted Notify=healthy so `systemctl start` blocked
|
||||
# until the healthcheck passed. That was deliberately reverted: gating start
|
||||
# on health hung boot reconciliation for dependency-waiting apps (fedimint
|
||||
# idles its entrypoint until Bitcoin IBD finishes; lnd until the macaroon
|
||||
# unlocks), leaving units stuck in "deactivating". The renderer now emits
|
||||
# HealthCmd= for Podman's health state but TimeoutStartSec=0 and NO
|
||||
# Notify=healthy (see quadlet.rs render() + contains_stale_health_gate()).
|
||||
# This asserts the current invariant: no backend unit gates start on health.
|
||||
local d
|
||||
d="$(quadlet_dir)"
|
||||
while read -r name; do
|
||||
[[ -z "$name" ]] && continue
|
||||
local body
|
||||
body="$(<"$d/$name.container")"
|
||||
[[ "$body" != *"Notify=healthy"* ]] \
|
||||
|| fail "$name: emits Notify=healthy — stale health gate; start would block on health and can hang boot reconcile"
|
||||
done < <(backend_quadlet_units)
|
||||
}
|
||||
|
||||
@test "every backend Quadlet unit's .service is active in systemctl --user" {
|
||||
require_quadlet_backends
|
||||
while read -r name; do
|
||||
[[ -z "$name" ]] && continue
|
||||
is_user_stopped "$name" && continue
|
||||
# Converges-to-active, not instantly-active: a dependency-degraded app
|
||||
# (mempool-api while electrumx catches up to the daemon) exits at startup
|
||||
# and flaps through 'activating' for a couple of minutes after a lifecycle
|
||||
# cycle; systemd's Restart=on-failure heals it. A genuine crash-loop still
|
||||
# fails after the settle window (gate 2026-07-09, .228 iterations 1+2).
|
||||
local state="" deadline=$((SECONDS + 180))
|
||||
while (( SECONDS < deadline )); do
|
||||
state="$(systemctl --user is-active "$name.service" 2>&1)" && break
|
||||
sleep 5
|
||||
done
|
||||
[[ "$state" == "active" ]] \
|
||||
|| fail "$name.service is '$state' — did not reach 'active' within 180s"
|
||||
done < <(backend_quadlet_units)
|
||||
}
|
||||
|
||||
@test "every backend Quadlet unit has a running podman container" {
|
||||
require_quadlet_backends
|
||||
while read -r name; do
|
||||
[[ -z "$name" ]] && continue
|
||||
is_user_stopped "$name" && continue
|
||||
# Same settle window as the active-state assert above: a quadlet --rm
|
||||
# container is absent for a few seconds around each systemd retry.
|
||||
local state="" deadline=$((SECONDS + 180))
|
||||
while (( SECONDS < deadline )); do
|
||||
state="$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null)" \
|
||||
&& [[ "$state" == "true" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ "$state" == "true" ]] \
|
||||
|| fail "$name has no running container within 180s (state=${state:-absent})"
|
||||
done < <(backend_quadlet_units)
|
||||
}
|
||||
|
||||
@test "FM3 fix: backend cgroup is under user.slice, not archipelago.service" {
|
||||
require_quadlet_backends
|
||||
# The whole point of Phase 3 — verify the kernel-level invariant.
|
||||
while read -r name; do
|
||||
[[ -z "$name" ]] && continue
|
||||
local cg
|
||||
cg="$(container_cgroup_path "$name")" || skip "$name has no readable PID; container may have crashed mid-test"
|
||||
[[ -n "$cg" ]] || fail "$name: empty cgroup path"
|
||||
# Acceptable: anything under user.slice (rootless podman lands here when
|
||||
# quadlet-managed). Forbidden: anything under archipelago.service's tree.
|
||||
[[ "$cg" == *"user.slice"* ]] \
|
||||
|| fail "$name: cgroup '$cg' is not under user.slice — FM3 cascade still possible"
|
||||
[[ "$cg" != *"archipelago.service"* ]] \
|
||||
|| fail "$name: cgroup '$cg' is under archipelago.service — Phase 3 promise broken"
|
||||
done < <(backend_quadlet_units)
|
||||
}
|
||||
Executable
+177
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/lib/rpc.bash
|
||||
#
|
||||
# Shared JSON-RPC client for archipelago lifecycle tests.
|
||||
# Handles login, session cookie + CSRF token management, and request plumbing.
|
||||
#
|
||||
# Environment variables honored:
|
||||
# ARCHY_HOST — default: 127.0.0.1
|
||||
# ARCHY_SCHEME — default: https
|
||||
# ARCHY_PASSWORD — REQUIRED. The UI password.
|
||||
#
|
||||
# After sourcing, call `rpc_login` once per test file in setup_file or setup.
|
||||
# Then call `rpc_call METHOD [JSON_PARAMS]` to invoke methods.
|
||||
# rpc_call prints the raw JSON response to stdout.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ARCHY_HOST="${ARCHY_HOST:-127.0.0.1}"
|
||||
ARCHY_SCHEME="${ARCHY_SCHEME:-https}"
|
||||
ARCHY_BASE_URL="${ARCHY_SCHEME}://${ARCHY_HOST}"
|
||||
|
||||
# Session file lives in a stable per-user location so every bats subshell
|
||||
# (setup_file, setup, each @test) sees the same cookies. File format:
|
||||
# line 1: session cookie value
|
||||
# line 2: csrf cookie value
|
||||
RPC_SESSION_FILE="${RPC_SESSION_FILE:-${TMPDIR:-/tmp}/archy-rpc-session-${UID:-$(id -u)}}"
|
||||
|
||||
RPC_SESSION=""
|
||||
RPC_CSRF=""
|
||||
|
||||
# Load cookies from $RPC_SESSION_FILE into RPC_SESSION/RPC_CSRF.
|
||||
# Returns 1 if the file is missing or malformed.
|
||||
_rpc_load_session() {
|
||||
[[ -r "$RPC_SESSION_FILE" ]] || return 1
|
||||
local lines
|
||||
mapfile -t lines < "$RPC_SESSION_FILE"
|
||||
RPC_SESSION="${lines[0]:-}"
|
||||
RPC_CSRF="${lines[1]:-}"
|
||||
[[ -n "$RPC_SESSION" && -n "$RPC_CSRF" ]]
|
||||
}
|
||||
|
||||
# Log in with $ARCHY_PASSWORD and persist session + csrf cookies to $RPC_SESSION_FILE.
|
||||
# Idempotent-ish: if a valid session file already exists and ARCHY_FORCE_LOGIN
|
||||
# is not set, we reuse it (saves a round-trip per test file).
|
||||
rpc_login() {
|
||||
if _rpc_load_session && [[ -z "${ARCHY_FORCE_LOGIN:-}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -z "${ARCHY_PASSWORD:-}" ]]; then
|
||||
echo "rpc_login: ARCHY_PASSWORD env var not set" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
local headers body
|
||||
headers=$(mktemp)
|
||||
body=$(curl -sk -D "$headers" -X POST "${ARCHY_BASE_URL}/rpc/v1" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "{\"jsonrpc\":\"2.0\",\"method\":\"auth.login\",\"params\":{\"password\":\"${ARCHY_PASSWORD}\"},\"id\":1}")
|
||||
|
||||
local err
|
||||
err=$(echo "$body" | jq -r '.error // empty')
|
||||
if [[ -n "$err" && "$err" != "null" ]]; then
|
||||
echo "rpc_login failed: $err" >&2
|
||||
rm -f "$headers"
|
||||
return 1
|
||||
fi
|
||||
|
||||
RPC_SESSION=$(grep -i '^set-cookie: session=' "$headers" | head -1 | sed -E 's/.*session=([^;]+).*/\1/' | tr -d '\r')
|
||||
RPC_CSRF=$(grep -i '^set-cookie: csrf_token=' "$headers" | head -1 | sed -E 's/.*csrf_token=([^;]+).*/\1/' | tr -d '\r')
|
||||
rm -f "$headers"
|
||||
|
||||
if [[ -z "$RPC_SESSION" || -z "$RPC_CSRF" ]]; then
|
||||
echo "rpc_login: missing session or csrf cookie in response" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Persist for subsequent subshells.
|
||||
umask 077
|
||||
printf '%s\n%s\n' "$RPC_SESSION" "$RPC_CSRF" > "$RPC_SESSION_FILE"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Forget persisted session (e.g., at end of a test run).
|
||||
rpc_logout_local() {
|
||||
rm -f "$RPC_SESSION_FILE"
|
||||
RPC_SESSION=""
|
||||
RPC_CSRF=""
|
||||
}
|
||||
|
||||
# Call an RPC method.
|
||||
# Usage: rpc_call METHOD [PARAMS_JSON]
|
||||
# Prints the full JSON-RPC response object to stdout.
|
||||
# Returns 0 on successful HTTP call (regardless of RPC-level error).
|
||||
rpc_call() {
|
||||
local method="$1"
|
||||
local params="${2:-null}"
|
||||
local id="${3:-$RANDOM}"
|
||||
|
||||
if [[ -z "$RPC_SESSION" || -z "$RPC_CSRF" ]]; then
|
||||
_rpc_load_session || {
|
||||
echo "rpc_call: not logged in (call rpc_login first)" >&2
|
||||
return 1
|
||||
}
|
||||
fi
|
||||
|
||||
local payload
|
||||
if [[ "$params" == "null" ]]; then
|
||||
payload=$(jq -nc --arg m "$method" --argjson id "$id" '{jsonrpc:"2.0",method:$m,id:$id}')
|
||||
else
|
||||
payload=$(jq -nc --arg m "$method" --argjson id "$id" --argjson p "$params" '{jsonrpc:"2.0",method:$m,params:$p,id:$id}')
|
||||
fi
|
||||
|
||||
curl -sk -X POST "${ARCHY_BASE_URL}/rpc/v1" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H "Cookie: session=${RPC_SESSION}; csrf_token=${RPC_CSRF}" \
|
||||
-H "X-CSRF-Token: ${RPC_CSRF}" \
|
||||
--data-raw "$payload"
|
||||
}
|
||||
|
||||
# Convenience: call rpc and return only the .result field (or fail if .error is set).
|
||||
rpc_result() {
|
||||
local resp
|
||||
resp=$(rpc_call "$@")
|
||||
local err
|
||||
err=$(echo "$resp" | jq -r '.error // empty')
|
||||
if [[ -n "$err" && "$err" != "null" ]]; then
|
||||
echo "rpc_result: $1 failed: $err" >&2
|
||||
echo "full response: $resp" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "$resp" | jq '.result'
|
||||
}
|
||||
|
||||
# Wait for a container to reach a given status ("running" or "stopped" or "absent").
|
||||
# Usage: wait_for_container_status NAME STATUS [TIMEOUT_SECONDS]
|
||||
wait_for_container_status() {
|
||||
local name="$1"
|
||||
local target="$2"
|
||||
local timeout="${3:-60}"
|
||||
local deadline=$(( $(date +%s) + timeout ))
|
||||
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local list state status
|
||||
list=$(rpc_result container-list 2>/dev/null || echo '[]')
|
||||
if [[ "$target" == "absent" ]]; then
|
||||
if ! echo "$list" | jq -e --arg n "$name" '.[] | select(.name == $n)' >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
else
|
||||
# Primary source: container-list state keyed by container name.
|
||||
state=$(echo "$list" | jq -r --arg n "$name" '.[] | select(.name == $n) | .state // "unknown"')
|
||||
if [[ "$state" == "$target" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Fallback: container-status RPC accepts app_id. For common UI-prefixed
|
||||
# names, strip archy- prefix before querying.
|
||||
local app_id="$name"
|
||||
if [[ $app_id == bitcoin-knots ]]; then
|
||||
app_id=bitcoin-core
|
||||
elif [[ $app_id == electrs || $app_id == mempool-electrs ]]; then
|
||||
app_id=electrumx
|
||||
elif [[ $app_id == archy-* ]]; then
|
||||
app_id=${app_id#archy-}
|
||||
fi
|
||||
status=$(rpc_result container-status "{\"app_id\":\"$app_id\"}" 2>/dev/null | jq -r '.status // .state // "unknown"')
|
||||
if [[ "$status" == "$target" ]]; then
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "wait_for_container_status: $name did not reach '$target' within ${timeout}s" >&2
|
||||
return 1
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/lib/ui-probes.bash
|
||||
#
|
||||
# HTTPS proxy + iframe URL probes. Sourced from bats files. Pairs with
|
||||
# lib/rpc.bash but tests the URL surface a real user actually clicks
|
||||
# (dashboard, /app/<id>/ proxy paths, direct-port iframes), not just the
|
||||
# JSON-RPC API.
|
||||
#
|
||||
# Pattern: every probe is a skip-or-assert pair:
|
||||
# - if the container that backs the URL is not running → skip
|
||||
# (cleanly reports the dependency, doesn't false-fail)
|
||||
# - if it IS running → the URL MUST return 200
|
||||
# That catches the "container up but UI broken" failure mode that the
|
||||
# RPC-only tests miss (.198 today: archy-bitcoin-ui Up 12 minutes,
|
||||
# but is the iframe actually serving usable HTML? this layer answers).
|
||||
|
||||
# Curl options for a probe: short timeout, follow redirects, ignore self-
|
||||
# signed cert (the alpha fleet uses one), no proxy environment leak.
|
||||
PROBE_CURL_OPTS=(-skfL -m 8 --noproxy "*")
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# Container-state oracle
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
# True iff `name` is currently in the running state per podman.
|
||||
probe_container_running() {
|
||||
local name="$1"
|
||||
[[ "$(podman inspect --format '{{.State.Running}}' "$name" 2>/dev/null)" == "true" ]]
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
# URL probes
|
||||
# ────────────────────────────────────────────────────────────────────
|
||||
|
||||
# Probe an HTTPS URL — assert 200 and non-empty body.
|
||||
# Usage: probe_https_200 URL "human description"
|
||||
probe_https_200() {
|
||||
local url="$1"
|
||||
local label="${2:-$url}"
|
||||
local body status
|
||||
body=$(curl "${PROBE_CURL_OPTS[@]}" -w '%{http_code}' "$url" 2>/dev/null) || {
|
||||
echo "probe_https_200: $label ($url) — curl failed (network/timeout)" >&2
|
||||
return 1
|
||||
}
|
||||
status="${body: -3}"
|
||||
body="${body:0:-3}"
|
||||
if [[ "$status" != "200" ]]; then
|
||||
echo "probe_https_200: $label ($url) returned $status (want 200)" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ -z "$body" ]]; then
|
||||
echo "probe_https_200: $label ($url) returned empty body" >&2
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# Probe a URL backed by a container — skip if container is not running,
|
||||
# assert 200 if it is. This is the standard shape for app UI tests.
|
||||
# Usage: probe_app_url CONTAINER URL "human description"
|
||||
probe_app_url() {
|
||||
local container="$1"
|
||||
local url="$2"
|
||||
local label="${3:-$url}"
|
||||
if ! probe_container_running "$container"; then
|
||||
skip "$label: backing container '$container' is not running"
|
||||
fi
|
||||
# An app's proxy/UI takes time to serve 200 after a (re)start — the backend
|
||||
# may still be unlocking/syncing (lnd) and the companion nginx reloading.
|
||||
# Retry up to ~90s rather than single-shot, so a readiness race isn't a fail.
|
||||
local deadline=$(( $(date +%s) + 90 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if probe_https_200 "$url" "$label"; then
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
done
|
||||
run probe_https_200 "$url" "$label"
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
# Probe the archipelago dashboard itself (the SPA shell at https://node/).
|
||||
# Asserts 200 and that the body looks like the Vue index, not an nginx
|
||||
# default page. Catches "frontend tarball was extracted with the wrong
|
||||
# layout" — see feedback_release_tarball_layout.md.
|
||||
probe_dashboard_shell() {
|
||||
local host="${ARCHY_HOST:-127.0.0.1}"
|
||||
local url="https://$host/"
|
||||
local body
|
||||
body=$(curl "${PROBE_CURL_OPTS[@]}" "$url" 2>/dev/null) || {
|
||||
echo "probe_dashboard_shell: $url — curl failed" >&2
|
||||
return 1
|
||||
}
|
||||
# Vue shell carries one of these markers: <div id="app">, the SPA bundle
|
||||
# tag, or the manifest link. Nginx default does not.
|
||||
if echo "$body" | grep -qE 'id="app"|<script.*\.js"|manifest\.webmanifest'; then
|
||||
return 0
|
||||
fi
|
||||
echo "probe_dashboard_shell: $url returned 200 but body doesn't look like the Vue shell" >&2
|
||||
echo "first 200 bytes: ${body:0:200}" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Probe the catalog endpoint that the dashboard uses to populate tiles.
|
||||
# Returns 0 if catalog is reachable AND has at least one entry.
|
||||
probe_dashboard_catalog() {
|
||||
local host="${ARCHY_HOST:-127.0.0.1}"
|
||||
local body
|
||||
body=$(curl "${PROBE_CURL_OPTS[@]}" "https://$host/catalog.json" 2>/dev/null) || {
|
||||
echo "probe_dashboard_catalog: /catalog.json fetch failed" >&2
|
||||
return 1
|
||||
}
|
||||
if ! echo "$body" | jq -e 'length > 0' >/dev/null 2>&1; then
|
||||
echo "probe_dashboard_catalog: /catalog.json is not a non-empty array/object" >&2
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
Executable
+252
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/os-audit.sh — one non-destructive OS-wide health gate.
|
||||
#
|
||||
# Ties together, in a single pass with one scorecard + exit code:
|
||||
# A. Backend / RPC health — node is up, not wedged mid-OTA, core daemons answer
|
||||
# B. All-apps lifecycle audit — every catalog app: valid state, real health,
|
||||
# reachable launch URL, populated launch metadata
|
||||
# (delegates to remote-lifecycle.sh, audit-only)
|
||||
# C. FM-guards — the concrete failure modes that have bitten the
|
||||
# fleet: port-drift (FM8), secret-completeness (FM2),
|
||||
# orphaned container states (FM9), OTA wedge (FM12)
|
||||
#
|
||||
# Everything here is READ-ONLY: no install/stop/start/uninstall, no service bounce.
|
||||
# Safe to run against a live production node. It is the per-boot building block the
|
||||
# reboot-survival harness (L3) calls after each reboot.
|
||||
#
|
||||
# Env:
|
||||
# ARCHY_HOST (default 127.0.0.1)
|
||||
# ARCHY_SCHEME (default https; use http for .116 / nginx-:80-only nodes)
|
||||
# ARCHY_PASSWORD (required)
|
||||
# ARCHY_LOCAL (auto: 1 when ARCHY_HOST is loopback) — gates host-only podman checks
|
||||
#
|
||||
# Usage:
|
||||
# ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http ARCHY_PASSWORD=... tests/lifecycle/os-audit.sh
|
||||
#
|
||||
# Exit: 0 = every section green; 1 = one or more checks failed; 2 = setup/usage error.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
ARCHY_HOST="${ARCHY_HOST:-127.0.0.1}"
|
||||
ARCHY_SCHEME="${ARCHY_SCHEME:-https}"
|
||||
ARCHY_PASSWORD="${ARCHY_PASSWORD:-}"
|
||||
BASE_URL="${ARCHY_SCHEME}://${ARCHY_HOST}"
|
||||
|
||||
# Host-only checks (podman sweeps) make sense only when this script runs ON the node.
|
||||
if [[ -z "${ARCHY_LOCAL:-}" ]]; then
|
||||
case "$ARCHY_HOST" in
|
||||
127.0.0.1|localhost|::1) ARCHY_LOCAL=1 ;;
|
||||
*) ARCHY_LOCAL=0 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
if [[ -z "$ARCHY_PASSWORD" ]]; then
|
||||
echo "ARCHY_PASSWORD env var must be set." >&2
|
||||
exit 2
|
||||
fi
|
||||
for tool in curl jq; do
|
||||
command -v "$tool" >/dev/null 2>&1 || { echo "missing required tool: $tool" >&2; exit 2; }
|
||||
done
|
||||
|
||||
# ── scorecard state ───────────────────────────────────────────────────────────
|
||||
PASS=0; FAIL=0; WARN=0
|
||||
declare -a RESULTS=()
|
||||
record() { # record <PASS|FAIL|WARN> <label> [detail]
|
||||
local status="$1" label="$2" detail="${3:-}"
|
||||
case "$status" in
|
||||
PASS) PASS=$((PASS+1)) ;;
|
||||
FAIL) FAIL=$((FAIL+1)) ;;
|
||||
WARN) WARN=$((WARN+1)) ;;
|
||||
esac
|
||||
RESULTS+=("$(printf '%-4s %-38s %s' "$status" "$label" "$detail")")
|
||||
printf ' [%s] %s %s\n' "$status" "$label" "$detail"
|
||||
}
|
||||
|
||||
# ── minimal RPC client (session + CSRF) ────────────────────────────────────────
|
||||
SESSION=""; CSRF=""
|
||||
rpc_login() {
|
||||
local hdr; hdr=$(mktemp)
|
||||
curl -sk -D "$hdr" -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
|
||||
-d "$(jq -nc --arg p "$ARCHY_PASSWORD" '{jsonrpc:"2.0",id:1,method:"auth.login",params:{password:$p}}')" \
|
||||
-o /dev/null 2>/dev/null
|
||||
SESSION=$(grep -i '^set-cookie: session=' "$hdr" | head -1 | sed -E 's/.*session=([^;]+).*/\1/' | tr -d '\r')
|
||||
CSRF=$(grep -i '^set-cookie: csrf_token=' "$hdr" | head -1 | sed -E 's/.*csrf_token=([^;]+).*/\1/' | tr -d '\r')
|
||||
rm -f "$hdr"
|
||||
[[ -n "$SESSION" && -n "$CSRF" ]]
|
||||
}
|
||||
# rpc <method> [params-json] -> prints raw JSON response
|
||||
rpc() {
|
||||
local method="$1" params="${2:-{\}}"
|
||||
curl -sk -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
|
||||
-H "Cookie: session=${SESSION}; csrf_token=${CSRF}" -H "X-CSRF-Token: ${CSRF}" \
|
||||
-d "$(jq -nc --arg m "$method" --argjson p "$params" '{jsonrpc:"2.0",id:2,method:$m,params:$p}')" 2>/dev/null
|
||||
}
|
||||
# rpc_ok <method> [params] -> 0 if a result came back with no error
|
||||
rpc_ok() {
|
||||
local resp; resp=$(rpc "$@")
|
||||
[[ -n "$resp" ]] && [[ "$(jq -r '.error // empty' <<<"$resp" 2>/dev/null)" == "" ]] \
|
||||
&& [[ "$(jq -r 'has("result")' <<<"$resp" 2>/dev/null)" == "true" ]]
|
||||
}
|
||||
|
||||
# ══ Section A — Backend / RPC health ═══════════════════════════════════════════
|
||||
section_a() {
|
||||
echo
|
||||
echo "== A. Backend / RPC health =="
|
||||
|
||||
# unauth health probe first (doesn't need a session)
|
||||
local health; health=$(curl -sk -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"health","params":{}}' 2>/dev/null)
|
||||
if [[ "$(jq -r '.result.status // empty' <<<"$health" 2>/dev/null)" =~ ^(ok|degraded)$ ]]; then
|
||||
record PASS "node responds (health)" "status=$(jq -r '.result.status' <<<"$health")"
|
||||
else
|
||||
record FAIL "node responds (health)" "no/invalid health response — node down?"
|
||||
return
|
||||
fi
|
||||
|
||||
if ! rpc_login; then
|
||||
record FAIL "auth.login" "could not establish session (wrong password or rate-limited)"
|
||||
return
|
||||
fi
|
||||
record PASS "auth.login" "session established"
|
||||
|
||||
# FM12 — OTA must not be wedged mid-apply.
|
||||
# NB: must use has() not `//` — jq's `//` treats a legit `false` as empty and
|
||||
# would fall through to "unknown" on a perfectly healthy node.
|
||||
local us; us=$(rpc update.status)
|
||||
local inprog; inprog=$(jq -r '
|
||||
if (.result|type=="object") and (.result|has("update_in_progress")) then .result.update_in_progress
|
||||
elif (.result|type=="object") and (.result|has("in_progress")) then .result.in_progress
|
||||
else "unknown" end' <<<"$us" 2>/dev/null)
|
||||
if [[ "$inprog" == "false" ]]; then
|
||||
record PASS "OTA not wedged (update.status)" "update_in_progress=false"
|
||||
elif [[ "$inprog" == "unknown" ]]; then
|
||||
record WARN "OTA not wedged (update.status)" "could not read update_in_progress"
|
||||
else
|
||||
record FAIL "OTA not wedged (update.status)" "update_in_progress=$inprog (FM12 wedge)"
|
||||
fi
|
||||
|
||||
# Core daemons answer (only assert for ones present on this node)
|
||||
if rpc_ok bitcoin.getinfo || rpc_ok bitcoin.relay-status; then
|
||||
record PASS "bitcoin RPC reachable" ""
|
||||
else
|
||||
record WARN "bitcoin RPC reachable" "bitcoin.getinfo/relay-status did not answer (not installed?)"
|
||||
fi
|
||||
# LND wallet must be UNLOCKED. NB: lnd.getinfo masks a locked wallet (it
|
||||
# returns an all-zero success, error:null), so it can't detect the lock. Probe
|
||||
# the actual receive path (lnd.newaddress) instead: a LOCKED wallet returns the
|
||||
# LND_WALLET_LOCKED reason code — the exact fleet-wide receive breakage. A
|
||||
# locked wallet is a hard FAIL; "not installed" is a WARN. (newaddress derives
|
||||
# a fresh address — harmless; LND tolerates address gaps.)
|
||||
if rpc_ok lnd.getinfo; then
|
||||
local na; na=$(rpc lnd.newaddress)
|
||||
if grep -qE "LND_WALLET_LOCKED|wallet is locked|WALLET_LOCKED" <<<"$na"; then
|
||||
record FAIL "lnd wallet unlocked (lnd.newaddress)" "wallet LOCKED — auto-unlock failed (Bitcoin-receive broken)"
|
||||
elif [[ "$(jq -r '(has("result") and (.result!=null))' <<<"$na" 2>/dev/null)" == "true" ]]; then
|
||||
record PASS "lnd wallet unlocked (lnd.newaddress)" ""
|
||||
else
|
||||
record WARN "lnd wallet unlocked (lnd.newaddress)" "newaddress: $(jq -rc '.error.message // "no address"' <<<"$na" 2>/dev/null | head -c 60)"
|
||||
fi
|
||||
else
|
||||
record WARN "lnd RPC reachable" "lnd.getinfo did not answer (not installed?)"
|
||||
fi
|
||||
if rpc_ok system.stats || rpc_ok system.get-metrics; then
|
||||
record PASS "system metrics reachable" ""
|
||||
else
|
||||
record WARN "system metrics reachable" "system.stats/get-metrics did not answer"
|
||||
fi
|
||||
|
||||
# FM13 — disk pressure early-warning (best-effort; field names vary by version)
|
||||
local ds; ds=$(rpc system.disk-status)
|
||||
local usep; usep=$(jq -r '[.result.use_percent,.result.used_percent,.result.percent]|map(select(.!=null))|first // empty' <<<"$ds" 2>/dev/null)
|
||||
if [[ -n "$usep" ]]; then
|
||||
if (( ${usep%.*} >= 90 )); then
|
||||
record FAIL "disk pressure (system.disk-status)" "${usep}% used (FM13 risk)"
|
||||
else
|
||||
record PASS "disk pressure (system.disk-status)" "${usep}% used"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# ══ Section B — All-apps lifecycle audit (delegates to remote-lifecycle.sh) ═════
|
||||
section_b() {
|
||||
echo
|
||||
echo "== B. All-apps lifecycle audit (non-destructive, all catalog apps) =="
|
||||
local out rc
|
||||
# No ARCHY_APPS + no ARCHY_FULL_LIFECYCLE => audit every catalog app (audit_app).
|
||||
out=$(ARCHY_HOST="$ARCHY_HOST" ARCHY_SCHEME="$ARCHY_SCHEME" ARCHY_PASSWORD="$ARCHY_PASSWORD" \
|
||||
ARCHY_APPS="" ARCHY_FULL_LIFECYCLE=0 \
|
||||
"$HERE/remote-lifecycle.sh" 2>&1)
|
||||
rc=$?
|
||||
# Surface the per-app lines but drop the noisy optional-probe jq parse errors.
|
||||
echo "$out" | grep -vE '^jq: (parse )?error' | sed 's/^/ /'
|
||||
if (( rc == 0 )); then
|
||||
record PASS "broad all-apps audit" "remote-lifecycle.sh exit 0"
|
||||
else
|
||||
local n; n=$(echo "$out" | grep -oE 'FAILED checks: [0-9]+' | grep -oE '[0-9]+' | tail -1)
|
||||
record FAIL "broad all-apps audit" "remote-lifecycle.sh exit $rc (${n:-?} app checks failed)"
|
||||
fi
|
||||
}
|
||||
|
||||
# ══ Section C — FM-guards ══════════════════════════════════════════════════════
|
||||
run_bats_guard() { # run_bats_guard <suite> <label> <fm>
|
||||
local suite="$1" label="$2" fm="$3" out rc
|
||||
if ! command -v bats >/dev/null 2>&1; then
|
||||
record WARN "$label" "bats not installed — $fm guard skipped"
|
||||
return
|
||||
fi
|
||||
out=$(ARCHY_HOST="$ARCHY_HOST" ARCHY_SCHEME="$ARCHY_SCHEME" ARCHY_PASSWORD="$ARCHY_PASSWORD" \
|
||||
"$HERE/run.sh" "$suite" 2>&1); rc=$?
|
||||
if (( rc == 0 )); then
|
||||
record PASS "$label" "$fm guard green"
|
||||
else
|
||||
record FAIL "$label" "$fm — $(echo "$out" | grep -E '^not ok' | head -1)"
|
||||
fi
|
||||
}
|
||||
|
||||
section_c() {
|
||||
echo
|
||||
echo "== C. FM-guards (the concrete fleet failure modes) =="
|
||||
run_bats_guard port-drift "port bindings match manifest" "FM8"
|
||||
run_bats_guard secret-completeness "all referenced secrets exist" "FM2"
|
||||
|
||||
# FM9 — orphaned container states (host-only: needs local podman)
|
||||
if [[ "$ARCHY_LOCAL" == "1" ]] && command -v podman >/dev/null 2>&1; then
|
||||
local orphans
|
||||
orphans=$(podman ps -a --format '{{.Names}} {{.Status}}' 2>/dev/null \
|
||||
| grep -iE '(^| )(stopping|removing|created)( |$)' || true)
|
||||
if [[ -z "$orphans" ]]; then
|
||||
record PASS "no orphaned container states" "no stopping/removing/created"
|
||||
else
|
||||
record FAIL "no orphaned container states" "FM9: $(echo "$orphans" | tr '\n' ';')"
|
||||
fi
|
||||
else
|
||||
record WARN "no orphaned container states" "remote node — host podman sweep skipped"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── run ────────────────────────────────────────────────────────────────────────
|
||||
echo "=============================================================="
|
||||
echo " OS-wide audit — ${BASE_URL} ($(date '+%Y-%m-%d %H:%M:%S'))"
|
||||
echo " local=${ARCHY_LOCAL}"
|
||||
echo "=============================================================="
|
||||
section_a
|
||||
# Only proceed to apps/FM-guards if the node itself answered.
|
||||
if (( FAIL == 0 )) || [[ -n "$SESSION" ]]; then
|
||||
section_b
|
||||
section_c
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=============================================================="
|
||||
echo " SCORECARD: ${PASS} pass / ${FAIL} fail / ${WARN} warn"
|
||||
echo "=============================================================="
|
||||
printf '%s\n' "${RESULTS[@]}"
|
||||
echo
|
||||
if (( FAIL > 0 )); then
|
||||
echo "RESULT: FAIL ($FAIL critical checks failed)"
|
||||
exit 1
|
||||
fi
|
||||
echo "RESULT: PASS"
|
||||
exit 0
|
||||
Executable
+630
@@ -0,0 +1,630 @@
|
||||
#!/usr/bin/env bash
|
||||
# Remote app lifecycle runner for Archipelago nodes.
|
||||
#
|
||||
# Exercises the same public surface the UI uses:
|
||||
# - JSON-RPC package.install/start/stop/restart/uninstall
|
||||
# - HTTPS/direct-port launch probes from appSessionConfig.ts
|
||||
#
|
||||
# Default mode is audit-only. Use ARCHY_FULL_LIFECYCLE=1 for destructive
|
||||
# preserve-data cycles: install -> launch -> stop -> start -> restart ->
|
||||
# uninstall(preserve_data=true) -> reinstall -> launch.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ARCHY_HOST="${ARCHY_HOST:-}"
|
||||
ARCHY_SCHEME="${ARCHY_SCHEME:-https}"
|
||||
ARCHY_PASSWORD="${ARCHY_PASSWORD:-}"
|
||||
ARCHY_ITERATIONS="${ARCHY_ITERATIONS:-1}"
|
||||
ARCHY_FULL_LIFECYCLE="${ARCHY_FULL_LIFECYCLE:-0}"
|
||||
ARCHY_APPS="${ARCHY_APPS:-}"
|
||||
ARCHY_TIMEOUT="${ARCHY_TIMEOUT:-900}"
|
||||
ARCHY_STABILITY_SECONDS="${ARCHY_STABILITY_SECONDS:-5}"
|
||||
ARCHY_ALLOW_BITCOIN_SWAP="${ARCHY_ALLOW_BITCOIN_SWAP:-0}"
|
||||
ARCHY_APP_CATALOG="${ARCHY_APP_CATALOG:-}"
|
||||
ARCHY_PRUNED_NODE="${ARCHY_PRUNED_NODE:-auto}"
|
||||
|
||||
if [[ -z "$ARCHY_HOST" || -z "$ARCHY_PASSWORD" ]]; then
|
||||
echo "ARCHY_HOST and ARCHY_PASSWORD are required" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! [[ "$ARCHY_ITERATIONS" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "ARCHY_ITERATIONS must be a positive integer" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! [[ "$ARCHY_STABILITY_SECONDS" =~ ^[0-9]+$ ]]; then
|
||||
echo "ARCHY_STABILITY_SECONDS must be a non-negative integer" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
BASE_URL="${ARCHY_SCHEME}://${ARCHY_HOST}"
|
||||
SESSION=""
|
||||
CSRF=""
|
||||
CATALOG_FILE=""
|
||||
|
||||
ALL_APPS=(
|
||||
bitcoin-knots
|
||||
btcpay-server
|
||||
lnd
|
||||
mempool
|
||||
homeassistant
|
||||
grafana
|
||||
searxng
|
||||
ollama
|
||||
nextcloud
|
||||
vaultwarden
|
||||
jellyfin
|
||||
photoprism
|
||||
immich
|
||||
filebrowser
|
||||
nginx-proxy-manager
|
||||
portainer
|
||||
tailscale
|
||||
uptime-kuma
|
||||
electrumx
|
||||
fedimint
|
||||
indeedhub
|
||||
dwn
|
||||
botfights
|
||||
gitea
|
||||
)
|
||||
|
||||
ARCHIVAL_ONLY_APPS=(
|
||||
electrumx
|
||||
mempool
|
||||
)
|
||||
|
||||
app_in_list() {
|
||||
local needle="$1"
|
||||
shift
|
||||
local item
|
||||
for item in "$@"; do
|
||||
[[ "$item" == "$needle" ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
fetch_catalog() {
|
||||
CATALOG_FILE=$(mktemp)
|
||||
if [[ -n "$ARCHY_APP_CATALOG" ]]; then
|
||||
cp "$ARCHY_APP_CATALOG" "$CATALOG_FILE"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if curl -skfL --connect-timeout 8 -m 30 "${BASE_URL}/api/app-catalog" -o "$CATALOG_FILE" \
|
||||
&& jq -e '.apps | length > 0' "$CATALOG_FILE" >/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
curl -skfL --connect-timeout 8 -m 30 "${BASE_URL}/catalog.json" -o "$CATALOG_FILE"
|
||||
jq -e '.apps | length > 0' "$CATALOG_FILE" >/dev/null
|
||||
}
|
||||
|
||||
catalog_app_ids() {
|
||||
jq -r '.apps[] | select((.dockerImage // "") != "") | .id' "$CATALOG_FILE"
|
||||
}
|
||||
|
||||
catalog_app_json() {
|
||||
local app="$1"
|
||||
[[ -n "$CATALOG_FILE" && -r "$CATALOG_FILE" ]] || return 1
|
||||
jq -c --arg app "$app" '
|
||||
.registry as $registry
|
||||
| .apps[]
|
||||
| select(.id == $app)
|
||||
| .dockerImage = (if ((.dockerImage // "") | contains("/")) then .dockerImage else ($registry + "/" + .dockerImage) end)
|
||||
' "$CATALOG_FILE" | head -n 1
|
||||
}
|
||||
|
||||
is_pruned_node() {
|
||||
case "$ARCHY_PRUNED_NODE" in
|
||||
1|true|yes) return 0 ;;
|
||||
0|false|no) return 1 ;;
|
||||
esac
|
||||
|
||||
local pass body
|
||||
pass=$(ssh "${ARCHY_HOST}" 'sudo cat /var/lib/archipelago/secrets/bitcoin-rpc-password 2>/dev/null || cat /var/lib/archipelago/secrets/bitcoin-rpc-password 2>/dev/null' 2>/dev/null || true)
|
||||
[[ -n "$pass" ]] || return 1
|
||||
body=$(curl -fsS --max-time 20 \
|
||||
--user "archipelago:${pass}" \
|
||||
--data-binary '{"jsonrpc":"1.0","id":"remote-lifecycle","method":"getblockchaininfo","params":[]}' \
|
||||
-H 'content-type: text/plain;' \
|
||||
"http://${ARCHY_HOST}:8332/" 2>/dev/null || true)
|
||||
printf '%s' "$body" | jq -e '.result.pruned == true' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
image_for() {
|
||||
case "$1" in
|
||||
bitcoin-knots) echo "146.59.87.168:3000/lfg2025/bitcoin-knots:latest" ;;
|
||||
bitcoin-core) echo "docker.io/bitcoin/bitcoin:28.4" ;;
|
||||
btcpay-server) echo "docker.io/btcpayserver/btcpayserver:2.3.9" ;;
|
||||
lnd) echo "146.59.87.168:3000/lfg2025/lnd:v0.18.4-beta" ;;
|
||||
mempool) echo "146.59.87.168:3000/lfg2025/mempool-frontend:v3.0.0" ;;
|
||||
homeassistant) echo "146.59.87.168:3000/lfg2025/home-assistant:2024.1" ;;
|
||||
grafana) echo "146.59.87.168:3000/lfg2025/grafana:10.2.0" ;;
|
||||
searxng) echo "146.59.87.168:3000/lfg2025/searxng:latest" ;;
|
||||
ollama) echo "146.59.87.168:3000/lfg2025/ollama:latest" ;;
|
||||
nextcloud) echo "146.59.87.168:3000/lfg2025/nextcloud:28" ;;
|
||||
vaultwarden) echo "146.59.87.168:3000/lfg2025/vaultwarden:1.30.0-alpine" ;;
|
||||
jellyfin) echo "146.59.87.168:3000/lfg2025/jellyfin:10.8.13" ;;
|
||||
photoprism) echo "146.59.87.168:3000/lfg2025/photoprism:240915" ;;
|
||||
immich) echo "146.59.87.168:3000/lfg2025/immich-server:release" ;;
|
||||
filebrowser) echo "146.59.87.168:3000/lfg2025/filebrowser:v2.27.0" ;;
|
||||
nginx-proxy-manager) echo "146.59.87.168:3000/lfg2025/nginx-proxy-manager:latest" ;;
|
||||
portainer) echo "146.59.87.168:3000/lfg2025/portainer:latest" ;;
|
||||
uptime-kuma) echo "146.59.87.168:3000/lfg2025/uptime-kuma:1" ;;
|
||||
tailscale) echo "146.59.87.168:3000/lfg2025/tailscale:stable" ;;
|
||||
electrumx) echo "146.59.87.168:3000/lfg2025/electrumx:v1.18.0" ;;
|
||||
fedimint) echo "146.59.87.168:3000/lfg2025/fedimintd:v0.10.0" ;;
|
||||
indeedhub) echo "146.59.87.168:3000/lfg2025/indeedhub:1.0.0" ;;
|
||||
botfights) echo "146.59.87.168:3000/lfg2025/botfights:1.1.0" ;;
|
||||
gitea) echo "docker.io/gitea/gitea:1.23" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
launch_url_for() {
|
||||
case "$1" in
|
||||
bitcoin-knots|bitcoin-core|bitcoin-ui) echo "http://${ARCHY_HOST}:8334/" ;;
|
||||
lnd|archy-lnd-ui) echo "http://${ARCHY_HOST}:18083/" ;;
|
||||
electrumx|electrs|mempool-electrs|archy-electrs-ui) echo "http://${ARCHY_HOST}:50002/" ;;
|
||||
mempool|mempool-web|archy-mempool-web) echo "http://${ARCHY_HOST}:4080/" ;;
|
||||
fedimint|fedimintd) echo "http://${ARCHY_HOST}:8175/" ;;
|
||||
fedimint-gateway) echo "http://${ARCHY_HOST}:8176/" ;;
|
||||
filebrowser) echo "http://${ARCHY_HOST}:8083/" ;;
|
||||
grafana) echo "http://${ARCHY_HOST}:3000/" ;;
|
||||
btcpay-server) echo "http://${ARCHY_HOST}:23000/" ;;
|
||||
jellyfin) echo "http://${ARCHY_HOST}:8096/" ;;
|
||||
searxng) echo "http://${ARCHY_HOST}:8888/" ;;
|
||||
ollama) echo "http://${ARCHY_HOST}:11434/" ;;
|
||||
immich|immich_server) echo "http://${ARCHY_HOST}:2283/" ;;
|
||||
portainer) echo "http://${ARCHY_HOST}:9000/" ;;
|
||||
nginx-proxy-manager) echo "http://${ARCHY_HOST}:8081/" ;;
|
||||
tailscale) echo "http://${ARCHY_HOST}:8240/" ;;
|
||||
uptime-kuma) echo "http://${ARCHY_HOST}:3002/" ;;
|
||||
homeassistant) echo "http://${ARCHY_HOST}:8123/" ;;
|
||||
vaultwarden) echo "http://${ARCHY_HOST}:8082/" ;;
|
||||
photoprism) echo "http://${ARCHY_HOST}:2342/" ;;
|
||||
dwn) echo "http://${ARCHY_HOST}:3100/" ;;
|
||||
botfights) echo "http://${ARCHY_HOST}:9100/" ;;
|
||||
gitea) echo "http://${ARCHY_HOST}:3001/" ;;
|
||||
indeedhub) echo "http://${ARCHY_HOST}:7778/" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
rpc_login() {
|
||||
local headers body err
|
||||
headers=$(mktemp)
|
||||
body=$(curl -sk -D "$headers" -X POST "${BASE_URL}/rpc/v1" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "$(jq -nc --arg p "$ARCHY_PASSWORD" '{jsonrpc:"2.0",method:"auth.login",params:{password:$p},id:1}')")
|
||||
err=$(printf '%s' "$body" | jq -r '.error.message // empty')
|
||||
if [[ -n "$err" ]]; then
|
||||
rm -f "$headers"
|
||||
echo "login failed on $ARCHY_HOST: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
SESSION=$(grep -i '^set-cookie: session=' "$headers" | head -1 | sed -E 's/.*session=([^;]+).*/\1/' | tr -d '\r')
|
||||
CSRF=$(grep -i '^set-cookie: csrf_token=' "$headers" | head -1 | sed -E 's/.*csrf_token=([^;]+).*/\1/' | tr -d '\r')
|
||||
rm -f "$headers"
|
||||
[[ -n "$SESSION" && -n "$CSRF" ]]
|
||||
}
|
||||
|
||||
rpc_call() {
|
||||
local method="$1" params="${2:-null}" id="${3:-2}"
|
||||
local payload
|
||||
if [[ "$params" == "null" ]]; then
|
||||
payload=$(jq -nc --arg m "$method" --argjson id "$id" '{jsonrpc:"2.0",method:$m,id:$id}')
|
||||
else
|
||||
payload=$(jq -nc --arg m "$method" --argjson p "$params" --argjson id "$id" '{jsonrpc:"2.0",method:$m,params:$p,id:$id}')
|
||||
fi
|
||||
curl -sk -X POST "${BASE_URL}/rpc/v1" \
|
||||
--connect-timeout 8 \
|
||||
-m "${ARCHY_RPC_TIMEOUT:-60}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H "Cookie: session=${SESSION}; csrf_token=${CSRF}" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
--data-raw "$payload"
|
||||
}
|
||||
|
||||
rpc_result() {
|
||||
local resp err
|
||||
resp=$(rpc_call "$@")
|
||||
err=$(printf '%s' "$resp" | jq -r '.error.message // empty')
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "$err" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s' "$resp" | jq '.result'
|
||||
}
|
||||
|
||||
container_state() {
|
||||
local app="$1"
|
||||
rpc_result container-list | jq -r --arg app "$app" '
|
||||
(map(select(.name == $app or .id == $app)) | first | .state // "absent") | ascii_downcase
|
||||
'
|
||||
}
|
||||
|
||||
container_health() {
|
||||
local app="$1" health
|
||||
health=$(
|
||||
ARCHY_RPC_TIMEOUT="${ARCHY_HEALTH_RPC_TIMEOUT:-20}" \
|
||||
rpc_result container-health "$(jq -nc --arg app "$app" '{app_id:$app}')" \
|
||||
| jq -r --arg app "$app" '(.[$app] // "") | if . == "" then "unknown" else ascii_downcase end'
|
||||
) || health=unknown
|
||||
if [[ "$app" == "indeedhub" && "$health" != "healthy" ]] && probe_launch "$app" >/dev/null 2>&1; then
|
||||
health=healthy
|
||||
fi
|
||||
printf '%s\n' "$health"
|
||||
}
|
||||
|
||||
assert_container_healthy() {
|
||||
local app="$1" health
|
||||
health=$(container_health "$app" 2>/dev/null || echo unknown)
|
||||
case "$health" in
|
||||
healthy) return 0 ;;
|
||||
*) echo "bad health: $app is $health" >&2; return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
wait_container_healthy() {
|
||||
local app="$1" timeout="${2:-$ARCHY_TIMEOUT}" deadline health
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
health=$(container_health "$app" 2>/dev/null || echo unknown)
|
||||
if [[ "$health" == "healthy" ]]; then return 0; fi
|
||||
sleep 5
|
||||
done
|
||||
echo "bad health: $app is ${health:-unknown}" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
observe_stable() {
|
||||
local app="$1" seconds="${2:-$ARCHY_STABILITY_SECONDS}" deadline state
|
||||
(( seconds == 0 )) && return 0
|
||||
deadline=$(( $(date +%s) + seconds ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
state=$(container_state "$app" 2>/dev/null || echo unknown)
|
||||
if [[ "$state" != "running" ]]; then
|
||||
if [[ "$app" == "indeedhub" ]] && probe_launch "$app" >/dev/null 2>&1; then
|
||||
sleep 5
|
||||
continue
|
||||
fi
|
||||
echo "stability failed: $app left running state (last=$state)" >&2
|
||||
return 1
|
||||
fi
|
||||
assert_container_healthy "$app" || return 1
|
||||
sleep 5
|
||||
done
|
||||
}
|
||||
|
||||
wait_state() {
|
||||
local app="$1" target="$2" timeout="${3:-$ARCHY_TIMEOUT}"
|
||||
local deadline state
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
state=$(container_state "$app" 2>/dev/null || echo unknown)
|
||||
if [[ "$target" == "absent" && "$state" == "absent" ]]; then return 0; fi
|
||||
if [[ "$target" == "stopped" && "$state" == "absent" ]]; then return 0; fi
|
||||
if [[ "$target" != "absent" && "$state" == "$target" ]]; then return 0; fi
|
||||
if [[ "$app" == "indeedhub" && "$target" == "running" ]] && probe_launch "$app" >/dev/null 2>&1; then return 0; fi
|
||||
sleep 5
|
||||
done
|
||||
echo "$app did not reach $target within ${timeout}s (last=$state)" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_absent_settled() {
|
||||
local app="$1" timeout="${2:-$ARCHY_TIMEOUT}"
|
||||
local deadline state seen_absent=0
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
state=$(container_state "$app" 2>/dev/null || echo unknown)
|
||||
if [[ "$state" == "absent" ]]; then
|
||||
if (( seen_absent == 1 )); then return 0; fi
|
||||
seen_absent=1
|
||||
else
|
||||
seen_absent=0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
echo "$app did not settle absent within ${timeout}s (last=$state)" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_not_installing() {
|
||||
local app="$1" timeout="${2:-$ARCHY_TIMEOUT}"
|
||||
local deadline state
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
state=$(container_state "$app" 2>/dev/null || echo unknown)
|
||||
case "$state" in
|
||||
installing|starting|restarting|updating) sleep 5 ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
done
|
||||
echo "$app did not settle from install transition within ${timeout}s (last=$state)" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
probe_launch() {
|
||||
local app="$1" url code bytes body
|
||||
url=$(launch_url_for "$app") || return 0
|
||||
body=$(mktemp)
|
||||
code=$(curl -skL --connect-timeout 8 -m 20 -o "$body" -w '%{http_code}' "$url" || true)
|
||||
bytes=$(wc -c < "$body" 2>/dev/null || printf 0)
|
||||
if [[ "$code" != "200" || "$bytes" -eq 0 ]]; then
|
||||
echo "launch failed: $app $url status=$code bytes=$bytes" >&2
|
||||
rm -f "$body"
|
||||
return 1
|
||||
fi
|
||||
case "$app" in
|
||||
lnd) probe_lnd_wallet_connect "$body" || { rm -f "$body"; return 1; } ;;
|
||||
electrumx|electrs|mempool-electrs) probe_electrum_wallet_connect "$body" || { rm -f "$body"; return 1; } ;;
|
||||
indeedhub) probe_indeedhub_nostr_signer "$body" || { rm -f "$body"; return 1; } ;;
|
||||
tailscale) probe_tailscale_login_ui "$body" || { rm -f "$body"; return 1; } ;;
|
||||
esac
|
||||
rm -f "$body"
|
||||
}
|
||||
|
||||
wait_launch() {
|
||||
local app="$1" timeout="${2:-$ARCHY_TIMEOUT}" deadline
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
if probe_launch "$app" >/dev/null 2>&1; then return 0; fi
|
||||
sleep 5
|
||||
done
|
||||
probe_launch "$app"
|
||||
}
|
||||
|
||||
assert_launch_metadata() {
|
||||
local app="$1" timeout="${2:-$ARCHY_TIMEOUT}" deadline lan
|
||||
launch_url_for "$app" >/dev/null 2>&1 || return 0
|
||||
deadline=$(( $(date +%s) + timeout ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
lan=$(rpc_result container-list | jq -r --arg app "$app" '
|
||||
(map(select(.name == $app or .id == $app)) | first | .lan_address // "")
|
||||
')
|
||||
if [[ -n "$lan" && "$lan" != "null" ]]; then return 0; fi
|
||||
sleep 5
|
||||
done
|
||||
if [[ -z "${lan:-}" || "$lan" == "null" ]]; then
|
||||
echo "launch metadata missing: $app has no lan_address" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_body() {
|
||||
local body="$1" needle="$2" label="$3"
|
||||
if ! grep -Fq "$needle" "$body"; then
|
||||
echo "launch missing $label: $needle" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
probe_lnd_wallet_connect() {
|
||||
local body="$1" info err
|
||||
require_body "$body" 'Connect Your Wallet' 'LND wallet heading' || return 1
|
||||
require_body "$body" 'id="lndQrBox"' 'LND QR container' || return 1
|
||||
require_body "$body" 'id="connHost"' 'LND host field' || return 1
|
||||
require_body "$body" 'value="rest-tor"' 'LND REST Tor mode' || return 1
|
||||
require_body "$body" 'value="grpc-tor"' 'LND gRPC Tor mode' || return 1
|
||||
require_body "$body" 'value="rest-local"' 'LND REST local mode' || return 1
|
||||
require_body "$body" 'value="grpc-local"' 'LND gRPC local mode' || return 1
|
||||
require_body "$body" 'Copy lndconnect URI' 'LND connect URI button' || return 1
|
||||
info=$(curl -skL --connect-timeout 8 -m 20 \
|
||||
-H "Cookie: session=${SESSION}; csrf_token=${CSRF}" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${BASE_URL}/lnd-connect-info" || true)
|
||||
err=$(printf '%s' "$info" | jq -r '.error // empty' 2>/dev/null || true)
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "lnd connect info error: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s' "$info" | jq -e '
|
||||
(.cert_base64url | type == "string" and length > 100) and
|
||||
(.macaroon_base64url | type == "string" and length > 50) and
|
||||
(.tor_onion | type == "string" and test("^[a-z2-7]+\\.onion$")) and
|
||||
(.rest_port == 18080) and
|
||||
(.grpc_port == 10009)
|
||||
' >/dev/null || {
|
||||
echo "lnd connect info incomplete: $info" >&2
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
probe_electrum_wallet_connect() {
|
||||
local body="$1"
|
||||
require_body "$body" 'Connect Your Wallet' 'Electrum wallet heading' || return 1
|
||||
require_body "$body" 'id="qrLocalBox"' 'Electrum local QR container' || return 1
|
||||
require_body "$body" 'id="qrTorBox"' 'Electrum Tor QR container' || return 1
|
||||
require_body "$body" 'id="localAddress"' 'Electrum local address field' || return 1
|
||||
require_body "$body" 'id="torAddress"' 'Electrum Tor address field' || return 1
|
||||
require_body "$body" '50001' 'Electrum wallet port' || return 1
|
||||
require_body "$body" 'renderQR' 'Electrum QR renderer' || return 1
|
||||
curl -skL --connect-timeout 8 -m 20 -f "http://${ARCHY_HOST}:50002/qrcode.js" >/dev/null || {
|
||||
echo "electrum qrcode.js unavailable" >&2
|
||||
return 1
|
||||
}
|
||||
local status
|
||||
status=$(curl -skL --connect-timeout 8 -m 20 "${BASE_URL}/electrs-status" || true)
|
||||
printf '%s' "$status" | jq -e '(.tor_onion | type == "string" and test("^[a-z2-7]+\\.onion$"))' >/dev/null || {
|
||||
echo "electrum tor connection info incomplete: $status" >&2
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
probe_indeedhub_nostr_signer() {
|
||||
local body="$1" provider pubkey signed now
|
||||
require_body "$body" '/nostr-provider.js' 'IndeedHub Nostr provider injection' || return 1
|
||||
provider=$(curl -skL --connect-timeout 8 -m 20 "http://${ARCHY_HOST}:7778/nostr-provider.js" || true)
|
||||
if [[ -z "$provider" ]]; then
|
||||
echo "indeedhub nostr-provider.js unavailable" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s' "$provider" | grep -Eq 'window\.nostr|nostr' || {
|
||||
echo "indeedhub nostr-provider.js does not look like a Nostr signer bridge" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
pubkey=$(rpc_result node.nostr-pubkey | jq -r '.nostr_pubkey // empty')
|
||||
if ! [[ "$pubkey" =~ ^[0-9a-fA-F]{64}$ ]]; then
|
||||
echo "indeedhub Nostr signer pubkey unavailable: $pubkey" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
now=$(date +%s)
|
||||
signed=$(rpc_result node.nostr-sign "$(jq -nc --argjson created_at "$now" '{event:{kind:1,created_at:$created_at,tags:[],content:"archy lifecycle indeedhub signer probe"}}')")
|
||||
printf '%s' "$signed" | jq -e --arg pubkey "$pubkey" '
|
||||
.pubkey == $pubkey and
|
||||
(.id | type == "string" and test("^[0-9a-f]{64}$")) and
|
||||
(.sig | type == "string" and test("^[0-9a-f]{128}$")) and
|
||||
.content == "archy lifecycle indeedhub signer probe"
|
||||
' >/dev/null || {
|
||||
echo "indeedhub Nostr signer did not return a valid signed event: $signed" >&2
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
probe_tailscale_login_ui() {
|
||||
local body="$1"
|
||||
if grep -Eiq 'tailscale|login|log in|sign in|authenticate|authorize|auth key|connect' "$body"; then
|
||||
return 0
|
||||
fi
|
||||
echo "tailscale launch did not present login/auth UI content" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
install_app() {
|
||||
local app="$1" app_json image params
|
||||
app_json=$(catalog_app_json "$app" || true)
|
||||
if [[ -n "$app_json" ]]; then
|
||||
params=$(printf '%s' "$app_json" | jq -c '{id, dockerImage, version, containerConfig} | with_entries(select(.value != null))')
|
||||
else
|
||||
image=$(image_for "$app")
|
||||
params=$(jq -nc --arg id "$app" --arg img "$image" '{id:$id,dockerImage:$img,version:"latest"}')
|
||||
fi
|
||||
rpc_result package.install "$params" >/dev/null
|
||||
}
|
||||
|
||||
expect_archival_blocked_install() {
|
||||
local app="$1" app_json resp err params
|
||||
app_json=$(catalog_app_json "$app")
|
||||
params=$(printf '%s' "$app_json" | jq -c '{id, dockerImage, version, containerConfig} | with_entries(select(.value != null))')
|
||||
resp=$(rpc_call package.install "$params")
|
||||
err=$(printf '%s' "$resp" | jq -r '.error.message // empty')
|
||||
if [[ "$err" != *"Requires an archival Bitcoin node"* && "$err" != *"requires an archival Bitcoin node"* && "$err" != *"running pruned Bitcoin"* ]]; then
|
||||
echo "expected archival Bitcoin block for $app, got: $resp" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
start_app() { rpc_result package.start "$(jq -nc --arg id "$1" '{id:$id}')" >/dev/null; }
|
||||
stop_app() { rpc_result package.stop "$(jq -nc --arg id "$1" '{id:$id}')" >/dev/null; }
|
||||
restart_app() { rpc_result package.restart "$(jq -nc --arg id "$1" '{id:$id}')" >/dev/null; }
|
||||
uninstall_app() { rpc_result package.uninstall "$(jq -nc --arg id "$1" '{id:$id,preserve_data:true}')" >/dev/null; }
|
||||
|
||||
audit_app() {
|
||||
local app="$1" state rc=0
|
||||
state=$(container_state "$app" || echo unknown)
|
||||
printf '%-22s state=%s\n' "$app" "$state"
|
||||
case "$state" in
|
||||
absent) ;;
|
||||
running)
|
||||
wait_container_healthy "$app" || rc=1
|
||||
wait_launch "$app" || rc=1
|
||||
assert_launch_metadata "$app" || rc=1
|
||||
observe_stable "$app" || rc=1
|
||||
;;
|
||||
*) echo "bad state: $app is $state" >&2; rc=1 ;;
|
||||
esac
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
full_lifecycle_app() {
|
||||
local app="$1"
|
||||
if [[ "$app" == "bitcoin-core" && "$ARCHY_ALLOW_BITCOIN_SWAP" != "1" ]]; then
|
||||
echo "skip bitcoin-core: set ARCHY_ALLOW_BITCOIN_SWAP=1 to test mutually-exclusive Bitcoin implementation"
|
||||
return 0
|
||||
fi
|
||||
if app_in_list "$app" "${ARCHIVAL_ONLY_APPS[@]}" && is_pruned_node; then
|
||||
echo "== $app: expect archival Bitcoin block =="
|
||||
expect_archival_blocked_install "$app"
|
||||
return $?
|
||||
fi
|
||||
echo "== $app: install =="
|
||||
install_app "$app" || return 1
|
||||
wait_not_installing "$app" || return 1
|
||||
wait_state "$app" running || return 1
|
||||
wait_container_healthy "$app" || return 1
|
||||
wait_launch "$app" || return 1
|
||||
assert_launch_metadata "$app" || return 1
|
||||
observe_stable "$app" || return 1
|
||||
|
||||
echo "== $app: stop =="
|
||||
stop_app "$app" || return 1
|
||||
wait_state "$app" stopped 300 || return 1
|
||||
|
||||
echo "== $app: start =="
|
||||
start_app "$app" || return 1
|
||||
wait_state "$app" running || return 1
|
||||
wait_container_healthy "$app" || return 1
|
||||
wait_launch "$app" || return 1
|
||||
assert_launch_metadata "$app" || return 1
|
||||
observe_stable "$app" || return 1
|
||||
|
||||
echo "== $app: restart =="
|
||||
restart_app "$app" || return 1
|
||||
wait_state "$app" running || return 1
|
||||
wait_container_healthy "$app" || return 1
|
||||
wait_launch "$app" || return 1
|
||||
assert_launch_metadata "$app" || return 1
|
||||
observe_stable "$app" || return 1
|
||||
|
||||
echo "== $app: uninstall preserve_data =="
|
||||
uninstall_app "$app" || return 1
|
||||
wait_absent_settled "$app" 600 || return 1
|
||||
|
||||
echo "== $app: reinstall =="
|
||||
install_app "$app" || return 1
|
||||
wait_not_installing "$app" || return 1
|
||||
wait_state "$app" running || return 1
|
||||
wait_container_healthy "$app" || return 1
|
||||
wait_launch "$app" || return 1
|
||||
assert_launch_metadata "$app" || return 1
|
||||
observe_stable "$app" || return 1
|
||||
}
|
||||
|
||||
apps=()
|
||||
if [[ -n "$ARCHY_APPS" ]]; then
|
||||
IFS=',' read -r -a apps <<< "$ARCHY_APPS"
|
||||
fetch_catalog || true
|
||||
elif [[ "$ARCHY_FULL_LIFECYCLE" == "1" ]]; then
|
||||
fetch_catalog
|
||||
mapfile -t apps < <(catalog_app_ids)
|
||||
else
|
||||
if fetch_catalog; then
|
||||
mapfile -t apps < <(catalog_app_ids)
|
||||
else
|
||||
apps=("${ALL_APPS[@]}")
|
||||
fi
|
||||
fi
|
||||
|
||||
rpc_login
|
||||
|
||||
failed=0
|
||||
for i in $(seq 1 "$ARCHY_ITERATIONS"); do
|
||||
echo "### $ARCHY_HOST iteration $i / $ARCHY_ITERATIONS ###"
|
||||
for app in "${apps[@]}"; do
|
||||
if [[ "$ARCHY_FULL_LIFECYCLE" == "1" ]]; then
|
||||
full_lifecycle_app "$app" || failed=$((failed + 1))
|
||||
else
|
||||
audit_app "$app" || failed=$((failed + 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if (( failed > 0 )); then
|
||||
echo "FAILED checks: $failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "all checks passed"
|
||||
Executable
+147
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/run-gate.sh — loop the lifecycle harness N times (default 5×, the release gate).
|
||||
#
|
||||
# Each iteration: setup-teardown → run.sh (with the same args you'd pass
|
||||
# to run.sh) → setup-teardown. Tallies pass/fail per iteration and prints a
|
||||
# summary at the end. Returns non-zero if any iteration failed.
|
||||
#
|
||||
# Env:
|
||||
# ARCHY_ITERATIONS (default: 5)
|
||||
# ARCHY_FAIL_FAST=1 stop on first failed iteration
|
||||
# ARCHY_GATE_CASCADE=1 after the 5× loop, run ONE cascade pass
|
||||
# (uninstall→no-ghost→reinstall a throwaway
|
||||
# app); requires ARCHY_ALLOW_DESTRUCTIVE=1
|
||||
# plus everything run.sh / lib/rpc.bash respects
|
||||
# (ARCHY_PASSWORD, ARCHY_HOST, ARCHY_SCHEME, ARCHY_ALLOW_DESTRUCTIVE,
|
||||
# ARCHY_ALLOW_CASCADE_DESTRUCTIVE, ARCHY_ALLOW_NOAUTH)
|
||||
#
|
||||
# Usage:
|
||||
# tests/lifecycle/run-gate.sh # 5× full bats/ suite
|
||||
# ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh # 5× full suite
|
||||
# tests/lifecycle/run-gate.sh bitcoin-knots # 5× a single suite
|
||||
#
|
||||
# Suggested release-gate invocation:
|
||||
# ARCHY_PASSWORD=password123 ARCHY_ALLOW_DESTRUCTIVE=1 \
|
||||
# tests/lifecycle/run-gate.sh
|
||||
#
|
||||
# Release-gate WITH the cascade tier (uninstall/reinstall regression guard):
|
||||
# ARCHY_PASSWORD=password123 ARCHY_ALLOW_DESTRUCTIVE=1 ARCHY_GATE_CASCADE=1 \
|
||||
# tests/lifecycle/run-gate.sh
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$HERE"
|
||||
|
||||
ITER="${ARCHY_ITERATIONS:-5}"
|
||||
if ! [[ "$ITER" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "ARCHY_ITERATIONS must be a positive integer, got: $ITER" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
passed=0
|
||||
failed=0
|
||||
failures=()
|
||||
start=$(date +%s)
|
||||
|
||||
# Best-effort settle: wait for the backend stack to be healthy before an
|
||||
# iteration starts, so back-to-back destructive iterations don't compound
|
||||
# restart churn (lnd wallet-unlock + the 4-container mempool stack reconnect
|
||||
# need time to recover). On-node gate only (localhost probes); never fails the
|
||||
# run — just delays up to the deadline. Disable with ARCHY_SETTLE=0.
|
||||
settle_stack() {
|
||||
[[ "${ARCHY_SETTLE:-1}" == "1" && "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]] || return 0
|
||||
# 300s (not 180s): on heavy nodes the immich stack's recovery after the prior
|
||||
# iteration's archipelago-restart test (crash_recovery retries on a ~120s
|
||||
# cadence) can take several minutes, and the next iteration's read-only
|
||||
# lan_address probe false-fails if immich is still mid-boot. The settle is a
|
||||
# cap, not a fixed wait — it returns the instant every probe is green.
|
||||
local deadline=$(( $(date +%s) + ${ARCHY_SETTLE_SECS:-300} ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
local ok=1
|
||||
# mempool-api + frontend + bitcoin-ui = good proxies for "stack reconnected"
|
||||
curl -fsS -m 4 -o /dev/null "http://127.0.0.1:8999/api/v1/backend-info" 2>/dev/null || ok=0
|
||||
curl -fsS -m 4 -o /dev/null "http://127.0.0.1:4080/" 2>/dev/null || ok=0
|
||||
podman exec lnd lncli --tlscertpath /root/.lnd/tls.cert \
|
||||
--macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon \
|
||||
--rpcserver localhost:10009 getinfo >/dev/null 2>&1 || ok=0
|
||||
# Only gate on immich where it's actually installed (heavy nodes). Its web
|
||||
# port is the same signal test 64 checks, so settling here keeps the next
|
||||
# iteration's read-only immich probe from racing a still-recovering stack.
|
||||
if podman container exists immich_server 2>/dev/null; then
|
||||
curl -fsS -m 4 -o /dev/null "http://127.0.0.1:2283/" 2>/dev/null || ok=0
|
||||
fi
|
||||
(( ok == 1 )) && { echo " (stack settled)"; return 0; }
|
||||
sleep 4
|
||||
done
|
||||
echo " (stack settle deadline reached — proceeding anyway)"
|
||||
}
|
||||
|
||||
# One initial teardown so a previous run's cookies don't poison iteration 1.
|
||||
./setup-teardown.sh
|
||||
|
||||
for i in $(seq 1 "$ITER"); do
|
||||
echo
|
||||
echo "═══ iteration $i / $ITER ═══"
|
||||
iter_start=$(date +%s)
|
||||
settle_stack
|
||||
|
||||
if ./run.sh "$@"; then
|
||||
iter_end=$(date +%s)
|
||||
passed=$((passed + 1))
|
||||
echo "── iteration $i: PASS ($((iter_end - iter_start))s) ──"
|
||||
else
|
||||
rc=$?
|
||||
iter_end=$(date +%s)
|
||||
failed=$((failed + 1))
|
||||
failures+=("$i")
|
||||
echo "── iteration $i: FAIL (exit=$rc, $((iter_end - iter_start))s) ──"
|
||||
if [[ "${ARCHY_FAIL_FAST:-0}" == "1" ]]; then
|
||||
echo "ARCHY_FAIL_FAST=1, stopping early"
|
||||
break
|
||||
fi
|
||||
fi
|
||||
|
||||
# Teardown between iterations so iteration N+1 starts with a clean
|
||||
# session-cookie state regardless of what iteration N did.
|
||||
./setup-teardown.sh
|
||||
done
|
||||
|
||||
# Optional CASCADE pass — uninstall → no-ghost → reinstall of a throwaway app
|
||||
# (default grafana, via cascade-uninstall.bats). Run ONCE, not folded into the
|
||||
# 5× loop on purpose: uninstall/reinstall every iteration would balloon runtime
|
||||
# and re-pull images. One pass gates the #13 ghost / #14 reinstall-stop /
|
||||
# uninstall-hang class (the bug fixed in 71cc9ac4). Opt-in so default gate
|
||||
# behavior is unchanged; counts into the pass/fail tally.
|
||||
if [[ "${ARCHY_GATE_CASCADE:-0}" == "1" && "${ARCHY_ALLOW_DESTRUCTIVE:-0}" == "1" ]]; then
|
||||
echo
|
||||
echo "═══ CASCADE pass (1×) ═══"
|
||||
settle_stack
|
||||
if ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1 ./run.sh cascade-uninstall; then
|
||||
passed=$((passed + 1))
|
||||
echo "── CASCADE: PASS ──"
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
failures+=("cascade")
|
||||
echo "── CASCADE: FAIL ──"
|
||||
fi
|
||||
./setup-teardown.sh
|
||||
fi
|
||||
|
||||
end=$(date +%s)
|
||||
|
||||
echo
|
||||
echo "════════════════════════════════════════"
|
||||
echo " RESULTS"
|
||||
echo " iterations: $((passed + failed)) / $ITER"
|
||||
echo " passed: $passed"
|
||||
echo " failed: $failed"
|
||||
if (( failed > 0 )); then
|
||||
echo " failed at: ${failures[*]}"
|
||||
fi
|
||||
echo " wall time: $((end - start))s"
|
||||
echo "════════════════════════════════════════"
|
||||
|
||||
if (( failed > 0 )); then
|
||||
exit 1
|
||||
fi
|
||||
Executable
+50
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/run.sh — entrypoint for lifecycle tests.
|
||||
#
|
||||
# Must be run on an archy host. Requires bats + jq + curl.
|
||||
#
|
||||
# Env:
|
||||
# ARCHY_PASSWORD (required unless ARCHY_ALLOW_NOAUTH=1)
|
||||
# ARCHY_HOST (default: 127.0.0.1)
|
||||
# ARCHY_SCHEME (default: https)
|
||||
# ARCHY_ALLOW_DESTRUCTIVE=1 enable stop/start/restart tests
|
||||
# ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1 enable uninstall/reinstall tests (rarely used)
|
||||
# ARCHY_ALLOW_NOAUTH=1 allow running read-only suites that don't use RPC auth
|
||||
#
|
||||
# Usage:
|
||||
# tests/lifecycle/run.sh # all .bats files
|
||||
# tests/lifecycle/run.sh bitcoin-knots # single file (no extension)
|
||||
# tests/lifecycle/run.sh required-stack required-stack-destructive
|
||||
# tests/lifecycle/run.sh package-update-smoke
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$HERE"
|
||||
|
||||
if ! command -v bats >/dev/null 2>&1; then
|
||||
echo "bats not installed. On Debian: sudo apt-get install -y bats" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ -z "${ARCHY_PASSWORD:-}" && "${ARCHY_ALLOW_NOAUTH:-0}" != "1" ]]; then
|
||||
echo "ARCHY_PASSWORD env var must be set (or ARCHY_ALLOW_NOAUTH=1 for no-auth suites)." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if (( $# == 0 )); then
|
||||
exec bats bats/
|
||||
fi
|
||||
|
||||
targets=()
|
||||
for arg in "$@"; do
|
||||
if [[ -f "bats/${arg}.bats" ]]; then
|
||||
targets+=("bats/${arg}.bats")
|
||||
elif [[ -f "$arg" ]]; then
|
||||
targets+=("$arg")
|
||||
else
|
||||
echo "unknown test target: $arg" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
exec bats "${targets[@]}"
|
||||
Executable
+38
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/lifecycle/setup-teardown.sh
|
||||
#
|
||||
# Cleanup helper used between lifecycle test iterations. Run before AND after
|
||||
# a full bats pass (run-gate.sh handles this). Idempotent — safe to run any
|
||||
# time, on any host.
|
||||
#
|
||||
# Removes:
|
||||
# - /tmp/archy-rpc-session-* — stale RPC session cookies from earlier runs.
|
||||
# If absent we'd reuse a session that was logged out by an auth.logout
|
||||
# test, then the next iteration would silently 401.
|
||||
# - Per-bats-run scratch files our tests may leave behind (none yet, but
|
||||
# this is the place when we add them).
|
||||
#
|
||||
# Does NOT touch:
|
||||
# - Real archipelago state (state.json, secrets, packages.json).
|
||||
# - Running containers — destructive container teardown is the test's
|
||||
# responsibility, not the harness's. We only clean the harness's own
|
||||
# transient state.
|
||||
# - SSH known_hosts, archipelago configs, etc.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cleaned=0
|
||||
|
||||
# Match the pattern from lib/rpc.bash:26
|
||||
session_glob="/tmp/archy-rpc-session-*"
|
||||
# shellcheck disable=SC2086 # we want word-splitting on the glob
|
||||
for f in $session_glob; do
|
||||
if [[ -f "$f" ]]; then
|
||||
rm -f "$f"
|
||||
cleaned=$((cleaned + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "${ARCHY_TEARDOWN_VERBOSE:-0}" == "1" ]]; then
|
||||
echo "setup-teardown: cleaned $cleaned stale session file(s)"
|
||||
fi
|
||||
@@ -0,0 +1,15 @@
|
||||
# Local credentials for the multinode test suites — copy to tests/multinode/.env
|
||||
# (git-ignored) and fill in. Sourced automatically by lib/multinode.bash.
|
||||
# NEVER commit real node passwords.
|
||||
|
||||
# smoke.sh / repro-federation-sync.sh
|
||||
A_PW=changeme # node A (default URL http://192.168.1.116)
|
||||
B_PW=changeme # node B (default URL https://192.168.1.228)
|
||||
#C_URL=https://x.x.x.x # optional third node
|
||||
#C_PW=changeme
|
||||
|
||||
# meshtastic.sh
|
||||
MA_PW=changeme
|
||||
MB_PW=changeme
|
||||
#MC_URL=https://x.x.x.x
|
||||
#MC_PW=changeme
|
||||
Executable
+128
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env bash
|
||||
# Multi-node RPC harness library.
|
||||
#
|
||||
# Unlike tests/lifecycle/lib/rpc.bash (which targets a single ARCHY_HOST),
|
||||
# this drives N independent archipelago nodes in one run so we can exercise
|
||||
# real node-to-node paths: federation sync over Tor, FIPS anchoring, etc.
|
||||
#
|
||||
# A "node handle" is a short label (e.g. A, B, alice). For each handle you
|
||||
# register a base URL + UI password; the lib logs in and keeps that node's
|
||||
# session/CSRF cookies in its own state file so calls never cross wires.
|
||||
#
|
||||
# Usage:
|
||||
# source tests/multinode/lib/multinode.bash
|
||||
# node_register A https://192.168.1.228 "$A_PW"
|
||||
# node_register B http://192.168.1.116 "$B_PW"
|
||||
# node_login A; node_login B
|
||||
# node_rpc A node.tor-address
|
||||
# node_result B federation.list-nodes
|
||||
#
|
||||
# Requires: curl, jq.
|
||||
#
|
||||
# Credentials: node passwords are NEVER committed. Export *_PW env vars, or
|
||||
# put them in tests/multinode/.env (git-ignored; see .env.example) — sourced
|
||||
# automatically below so every suite picks them up.
|
||||
#
|
||||
# Note: this is a library — it does NOT set shell options (set -u/-e), since
|
||||
# that would leak into the sourcing script. Each function guards its own vars
|
||||
# with ${var:-} defaults. Callers set their own options.
|
||||
|
||||
# Auto-load git-ignored local credentials (tests/multinode/.env), if present.
|
||||
_MN_ENV_FILE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)/.env"
|
||||
# shellcheck disable=SC1090
|
||||
[[ -f "$_MN_ENV_FILE" ]] && source "$_MN_ENV_FILE"
|
||||
|
||||
# Where per-node session state lives (one file per handle).
|
||||
MULTINODE_STATE_DIR="${MULTINODE_STATE_DIR:-/tmp/archy-multinode}"
|
||||
mkdir -p "$MULTINODE_STATE_DIR"
|
||||
|
||||
# handle -> base url / password, kept in associative arrays.
|
||||
declare -gA _MN_URL
|
||||
declare -gA _MN_PW
|
||||
declare -gA _MN_SESSION
|
||||
declare -gA _MN_CSRF
|
||||
|
||||
# node_register HANDLE BASE_URL PASSWORD
|
||||
node_register() {
|
||||
local h="$1" url="$2" pw="$3"
|
||||
_MN_URL[$h]="${url%/}"
|
||||
_MN_PW[$h]="$pw"
|
||||
}
|
||||
|
||||
_mn_session_file() { echo "$MULTINODE_STATE_DIR/session-$1"; }
|
||||
|
||||
# node_login HANDLE — authenticate and capture session + csrf cookies.
|
||||
node_login() {
|
||||
local h="$1"
|
||||
local url="${_MN_URL[$h]:-}" pw="${_MN_PW[$h]:-}"
|
||||
if [[ -z "$url" || -z "$pw" ]]; then
|
||||
echo "node_login: handle '$h' not registered" >&2
|
||||
return 1
|
||||
fi
|
||||
local headers; headers=$(mktemp)
|
||||
local body
|
||||
body=$(curl -sk --connect-timeout 10 --max-time "${MULTINODE_RPC_TIMEOUT:-120}" \
|
||||
-D "$headers" -X POST "${url}/rpc/v1" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "{\"jsonrpc\":\"2.0\",\"method\":\"auth.login\",\"params\":{\"password\":\"${pw}\"},\"id\":1}")
|
||||
local err; err=$(echo "$body" | jq -r '.error.message // empty' 2>/dev/null)
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "node_login[$h] failed: $err" >&2
|
||||
rm -f "$headers"
|
||||
return 1
|
||||
fi
|
||||
local session csrf
|
||||
session=$(grep -i '^set-cookie: session=' "$headers" | head -1 | sed -E 's/.*session=([^;]+).*/\1/' | tr -d '\r')
|
||||
csrf=$(grep -i '^set-cookie: csrf_token=' "$headers" | head -1 | sed -E 's/.*csrf_token=([^;]+).*/\1/' | tr -d '\r')
|
||||
rm -f "$headers"
|
||||
if [[ -z "$session" || -z "$csrf" ]]; then
|
||||
echo "node_login[$h]: missing session/csrf cookie" >&2
|
||||
return 1
|
||||
fi
|
||||
_MN_SESSION[$h]="$session"
|
||||
_MN_CSRF[$h]="$csrf"
|
||||
printf '%s\n%s\n' "$session" "$csrf" > "$(_mn_session_file "$h")"
|
||||
}
|
||||
|
||||
# node_rpc HANDLE METHOD [PARAMS_JSON] — raw JSON-RPC response on stdout.
|
||||
node_rpc() {
|
||||
local h="$1" method="$2" params="${3:-}"
|
||||
local url="${_MN_URL[$h]:-}"
|
||||
local session="${_MN_SESSION[$h]:-}" csrf="${_MN_CSRF[$h]:-}"
|
||||
if [[ -z "$session" || -z "$csrf" ]] && [[ -f "$(_mn_session_file "$h")" ]]; then
|
||||
mapfile -t lines < "$(_mn_session_file "$h")"
|
||||
session="${lines[0]:-}"; csrf="${lines[1]:-}"
|
||||
_MN_SESSION[$h]="$session"; _MN_CSRF[$h]="$csrf"
|
||||
fi
|
||||
local payload
|
||||
if [[ -z "$params" ]]; then
|
||||
payload=$(jq -nc --arg m "$method" '{jsonrpc:"2.0",method:$m,id:1}')
|
||||
else
|
||||
payload=$(jq -nc --arg m "$method" --argjson p "$params" '{jsonrpc:"2.0",method:$m,params:$p,id:1}')
|
||||
fi
|
||||
# Bounded so one slow/hung server-side RPC can't hang the whole suite;
|
||||
# override per-run with MULTINODE_RPC_TIMEOUT (seconds).
|
||||
curl -sk --connect-timeout 10 --max-time "${MULTINODE_RPC_TIMEOUT:-120}" \
|
||||
-X POST "${url}/rpc/v1" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H "Cookie: session=${session}; csrf_token=${csrf}" \
|
||||
-H "X-CSRF-Token: ${csrf}" \
|
||||
--data-raw "$payload"
|
||||
}
|
||||
|
||||
# node_result HANDLE METHOD [PARAMS_JSON] — .result on success; prints error to
|
||||
# stderr and returns non-zero on RPC error.
|
||||
node_result() {
|
||||
local resp; resp=$(node_rpc "$@")
|
||||
local err; err=$(echo "$resp" | jq -r '.error.message // empty' 2>/dev/null)
|
||||
if [[ -n "$err" ]]; then
|
||||
echo "node_result[$1 $2] error: $err" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "$resp" | jq '.result'
|
||||
}
|
||||
|
||||
# node_onion HANDLE — echo this node's own .onion address (empty if none).
|
||||
node_onion() {
|
||||
node_result "$1" node.tor-address 2>/dev/null | jq -r '. // empty | if type=="object" then (.onion // .address // .tor_address // empty) else . end' 2>/dev/null
|
||||
}
|
||||
Executable
+264
@@ -0,0 +1,264 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/multinode/meshtastic.sh — two-/three-radio Meshtastic parity harness.
|
||||
#
|
||||
# Validates that Meshtastic radios have the SAME mesh-tab features Meshcore got,
|
||||
# done over the real wire. It drives 2 (optionally 3) archipelago nodes, each
|
||||
# with a Meshtastic radio attached, and exercises the full message pipeline:
|
||||
#
|
||||
# 1. detect — each node reports a connected meshtastic device
|
||||
# 2. discover — A sees B as a peer (NodeInfo discovery), and vice-versa
|
||||
# 3. dm — A → B direct message round-trips (native unicast)
|
||||
# 4. privacy — a third listener C does NOT see the A→B DM (proves the
|
||||
# directed-unicast fix: DMs are not broadcast on the channel)
|
||||
# 5. channel — A's channel broadcast IS seen by both B and C
|
||||
# 6. typed — a typed envelope (reaction) round-trips with message_type set
|
||||
# 7. assistant — (optional) an !ai query gets a PRIVATE reply, not a channel
|
||||
# blast (gated on ASSIST=1 + assistant enabled on B)
|
||||
# 8. reachable — reports each peer's `reachable`/`last_advert` so the ambiguous
|
||||
# Meshtastic reachability semantics can be eyeballed on-air
|
||||
# before anyone "fixes" them
|
||||
#
|
||||
# The privacy test (4) is the on-air proof of the meshtastic.rs send_text_msg
|
||||
# unicast change. Without it, A→B DMs land on every node's channel feed.
|
||||
#
|
||||
# Nodes override via env (each must have a Meshtastic radio on the SAME LoRa
|
||||
# channel/region so they can actually hear each other):
|
||||
# MA_URL MA_PW node A (sender) default .116 http / ThisIsWeb54321@
|
||||
# MB_URL MB_PW node B (receiver) default .228 https / password123
|
||||
# MC_URL MC_PW node C (eavesdrop) OPTIONAL — enables privacy test (4)
|
||||
#
|
||||
# MB_NAME B's mesh node name, if A's peer list is ambiguous (>1 peer)
|
||||
# PROP_WAIT seconds to wait for LoRa propagation per step (default 45)
|
||||
# ASSIST set =1 to run the assistant private-reply test (7)
|
||||
#
|
||||
# Usage:
|
||||
# tests/multinode/meshtastic.sh
|
||||
# MA_URL=http://192.168.1.116 MB_URL=https://192.168.1.228 \
|
||||
# MC_URL=https://192.168.1.198 tests/multinode/meshtastic.sh
|
||||
#
|
||||
# Requires: curl, jq. Exit code = number of failed assertions (0 = all green).
|
||||
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib/multinode.bash
|
||||
source "$HERE/lib/multinode.bash"
|
||||
|
||||
# ── node registration ──────────────────────────────────────────────────────
|
||||
MA_URL="${MA_URL:-http://192.168.1.116}"; MA_PW="${MA_PW:?MA_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
MB_URL="${MB_URL:-https://192.168.1.228}"; MB_PW="${MB_PW:?MB_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
MC_URL="${MC_URL:-}"; MC_PW="${MC_PW:-}"
|
||||
PROP_WAIT="${PROP_WAIT:-45}"
|
||||
MB_NAME="${MB_NAME:-}"
|
||||
ASSIST="${ASSIST:-0}"
|
||||
|
||||
node_register A "$MA_URL" "$MA_PW"
|
||||
node_register B "$MB_URL" "$MB_PW"
|
||||
HAVE_C=0
|
||||
if [[ -n "$MC_URL" ]]; then node_register C "$MC_URL" "$MC_PW"; HAVE_C=1; fi
|
||||
|
||||
# ── tiny assert framework (mirrors smoke.sh) ───────────────────────────────
|
||||
if [[ -t 1 ]]; then
|
||||
green() { printf '\033[32m%s\033[0m' "$*"; }
|
||||
red() { printf '\033[31m%s\033[0m' "$*"; }
|
||||
yellow() { printf '\033[33m%s\033[0m' "$*"; }
|
||||
else
|
||||
green() { printf '%s' "$*"; }; red() { printf '%s' "$*"; }; yellow() { printf '%s' "$*"; }
|
||||
fi
|
||||
PASS=0; FAIL=0; SKIP=0; declare -a FAILED_NAMES
|
||||
ok() { printf ' %s %s\n' "$(green ✓)" "$1"; PASS=$((PASS+1)); }
|
||||
no() { printf ' %s %s\n' "$(red ✗)" "$1"; FAIL=$((FAIL+1)); FAILED_NAMES+=("$1"); }
|
||||
skip() { printf ' %s %s (%s)\n' "$(yellow —)" "$1" "${2:-skipped}"; SKIP=$((SKIP+1)); }
|
||||
assert_true() { [[ "$2" == "true" ]] && ok "$1" || no "$1 (got '$2')"; }
|
||||
section() { printf '\n%s\n' "$(yellow "── $* ──")"; }
|
||||
|
||||
# nonce for this run so message matches can't collide with stale history
|
||||
NONCE="mtparity-$$-${RANDOM}"
|
||||
|
||||
# ── helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
# mesh_connected HANDLE -> "true" if a meshtastic device is connected
|
||||
mesh_connected() {
|
||||
local s; s=$(node_result "$1" mesh.status 2>/dev/null) || { echo false; return; }
|
||||
local conn type
|
||||
conn=$(echo "$s" | jq -r '.device_connected // false')
|
||||
type=$(echo "$s" | jq -r '.device_type // "unknown"')
|
||||
[[ "$conn" == "true" && "$type" == "meshtastic" ]] && echo true || echo false
|
||||
}
|
||||
|
||||
# self_name HANDLE -> this node's meshtastic long-name (from firmware_version)
|
||||
self_name() {
|
||||
node_result "$1" mesh.status 2>/dev/null | jq -r '.firmware_version // empty'
|
||||
}
|
||||
|
||||
# contact_id_for HANDLE NAME -> the contact_id of the peer whose advert_name
|
||||
# matches NAME (case-insensitive substring); empty if not found / ambiguous.
|
||||
contact_id_for() {
|
||||
local h="$1" want="$2"
|
||||
node_result "$h" mesh.peers 2>/dev/null | jq -r --arg w "$want" '
|
||||
[.peers[] | select((.advert_name // "" | ascii_downcase)
|
||||
| contains($w | ascii_downcase))] as $m
|
||||
| if ($m|length)==1 then ($m[0].contact_id|tostring) else "" end'
|
||||
}
|
||||
|
||||
# peer_count_excl_self HANDLE -> number of peers
|
||||
peer_count() { node_result "$1" mesh.peers 2>/dev/null | jq -r '.count // 0'; }
|
||||
|
||||
# saw_text HANDLE NEEDLE [direction] -> "true" if a message whose plaintext
|
||||
# contains NEEDLE exists (optionally filtered to a direction: sent/received)
|
||||
saw_text() {
|
||||
local h="$1" needle="$2" dir="${3:-}"
|
||||
node_result "$h" mesh.messages '{"limit":200}' 2>/dev/null | jq -r --arg n "$needle" --arg d "$dir" '
|
||||
[.messages[] | select((.plaintext // "") | contains($n))
|
||||
| select($d=="" or (.direction==$d))] | length > 0'
|
||||
}
|
||||
|
||||
# wait_text HANDLE NEEDLE — poll up to PROP_WAIT for a received message
|
||||
wait_text() {
|
||||
local h="$1" needle="$2" waited=0
|
||||
while (( waited < PROP_WAIT )); do
|
||||
[[ "$(saw_text "$h" "$needle" received)" == "true" ]] && return 0
|
||||
sleep 3; waited=$((waited+3))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── login ──────────────────────────────────────────────────────────────────
|
||||
section "login"
|
||||
node_login A && ok "A login ($MA_URL)" || { no "A unreachable ($MA_URL)"; echo; exit 1; }
|
||||
node_login B && ok "B login ($MB_URL)" || { no "B unreachable ($MB_URL)"; echo; exit 1; }
|
||||
if (( HAVE_C )); then
|
||||
node_login C && ok "C login ($MC_URL)" || { skip "C login" "unreachable — privacy test disabled"; HAVE_C=0; }
|
||||
fi
|
||||
|
||||
# ── 1. detect ──────────────────────────────────────────────────────────────
|
||||
section "1. device detection"
|
||||
A_CONN=$(mesh_connected A); B_CONN=$(mesh_connected B)
|
||||
assert_true "A has a connected meshtastic radio" "$A_CONN"
|
||||
assert_true "B has a connected meshtastic radio" "$B_CONN"
|
||||
if [[ "$A_CONN" != "true" || "$B_CONN" != "true" ]]; then
|
||||
printf '\n%s\n' "$(yellow 'Both A and B need a Meshtastic radio attached & mesh enabled.')"
|
||||
printf '%s\n' "$(yellow 'Aborting on-air tests; see mesh.status output above.')"
|
||||
echo; printf 'PASS=%d FAIL=%d SKIP=%d\n' "$PASS" "$FAIL" "$SKIP"; exit "$FAIL"
|
||||
fi
|
||||
A_NAME=$(self_name A); B_NAME=$(self_name B)
|
||||
printf ' A=%s B=%s\n' "${A_NAME:-?}" "${B_NAME:-?}"
|
||||
[[ -n "$MB_NAME" ]] && B_NAME="$MB_NAME"
|
||||
|
||||
# ── 2. peer discovery ──────────────────────────────────────────────────────
|
||||
section "2. peer discovery (NodeInfo)"
|
||||
DISCO=0; waited=0
|
||||
while (( waited < PROP_WAIT )); do
|
||||
CID=$(contact_id_for A "${B_NAME:-Meshtastic}")
|
||||
[[ -n "$CID" ]] && { DISCO=1; break; }
|
||||
# fall back: any single non-channel peer
|
||||
if [[ -z "$MB_NAME" && "$(peer_count A)" == "1" ]]; then
|
||||
CID=$(node_result A mesh.peers | jq -r '.peers[0].contact_id'); DISCO=1; break
|
||||
fi
|
||||
sleep 3; waited=$((waited+3))
|
||||
done
|
||||
if (( DISCO )); then ok "A discovered B as a peer (contact_id=$CID)"
|
||||
else
|
||||
no "A did not discover B within ${PROP_WAIT}s"
|
||||
printf ' A peers: %s\n' "$(node_result A mesh.peers | jq -c '.peers[]? | {contact_id,advert_name}')"
|
||||
fi
|
||||
|
||||
# ── 3. direct message round-trip ───────────────────────────────────────────
|
||||
section "3. direct message (native unicast)"
|
||||
if (( DISCO )); then
|
||||
DM="$NONCE-dm hello-from-A"
|
||||
if node_result A mesh.send "$(jq -nc --argjson c "$CID" --arg m "$DM" '{contact_id:$c,message:$m}')" >/dev/null; then
|
||||
ok "A sent DM to B (contact_id=$CID)"
|
||||
if wait_text B "$NONCE-dm"; then ok "B received the DM"
|
||||
else no "B did not receive the DM within ${PROP_WAIT}s"; fi
|
||||
else no "mesh.send failed on A"; fi
|
||||
else skip "DM round-trip" "B not discovered"; fi
|
||||
|
||||
# ── 4. privacy: third node must NOT see the DM ─────────────────────────────
|
||||
section "4. DM privacy (directed, not broadcast)"
|
||||
if (( HAVE_C )) && (( DISCO )); then
|
||||
C_CONN=$(mesh_connected C)
|
||||
if [[ "$C_CONN" != "true" ]]; then
|
||||
skip "DM privacy" "C has no meshtastic radio"
|
||||
else
|
||||
# Give C the same window the DM had to propagate, then assert absence.
|
||||
sleep "$PROP_WAIT"
|
||||
if [[ "$(saw_text C "$NONCE-dm")" == "true" ]]; then
|
||||
no "C (eavesdropper) saw the A→B DM — it is being BROADCAST, not unicast"
|
||||
else
|
||||
ok "C did NOT see the A→B DM (directed unicast confirmed)"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
skip "DM privacy" "needs MC_URL (third radio) + discovered peer"
|
||||
fi
|
||||
|
||||
# ── 5. channel broadcast reaches everyone ──────────────────────────────────
|
||||
section "5. channel broadcast"
|
||||
CH="$NONCE-chan broadcast-to-all"
|
||||
if node_result A mesh.send-channel "$(jq -nc --arg m "$CH" '{channel:0,message:$m}')" >/dev/null; then
|
||||
ok "A sent a channel broadcast"
|
||||
if wait_text B "$NONCE-chan"; then ok "B received the broadcast"; else no "B missed the broadcast"; fi
|
||||
if (( HAVE_C )) && [[ "$(mesh_connected C)" == "true" ]]; then
|
||||
if [[ "$(saw_text C "$NONCE-chan")" == "true" ]]; then ok "C also received the broadcast"
|
||||
else no "C missed the broadcast (it should reach all channel members)"; fi
|
||||
fi
|
||||
else no "mesh.send-channel failed on A"; fi
|
||||
|
||||
# ── 6. typed envelope round-trip ───────────────────────────────────────────
|
||||
section "6. typed message (reaction envelope)"
|
||||
if (( DISCO )); then
|
||||
# A reaction is the smallest typed envelope; it should arrive with a
|
||||
# non-"text" message_type, proving the typed pipeline works over Meshtastic.
|
||||
REACT_PARAMS=$(jq -nc --argjson c "$CID" --arg n "$NONCE" \
|
||||
'{contact_id:$c, emoji:"👍", target_seq:0, note:$n}')
|
||||
if node_result A mesh.send-reaction "$REACT_PARAMS" >/dev/null 2>&1; then
|
||||
ok "A sent a reaction (typed envelope)"
|
||||
sleep "$PROP_WAIT"
|
||||
TYPED=$(node_result B mesh.messages '{"limit":200}' 2>/dev/null \
|
||||
| jq -r '[.messages[] | select(.message_type != null and .message_type != "text")] | length > 0')
|
||||
assert_true "B received a non-text typed message" "$TYPED"
|
||||
else
|
||||
skip "typed message" "mesh.send-reaction rejected params (check handler signature)"
|
||||
fi
|
||||
else skip "typed message" "B not discovered"; fi
|
||||
|
||||
# ── 7. assistant private reply (optional) ──────────────────────────────────
|
||||
section "7. AI assistant private reply (optional)"
|
||||
if [[ "$ASSIST" == "1" ]] && (( DISCO )); then
|
||||
AST=$(node_result B mesh.assistant-status 2>/dev/null | jq -r '.enabled // false')
|
||||
if [[ "$AST" != "true" ]]; then
|
||||
skip "assistant reply" "assistant not enabled on B"
|
||||
else
|
||||
Q="$NONCE-ai !ai are you there"
|
||||
node_result A mesh.send-channel "$(jq -nc --arg m "$Q" '{channel:0,message:$m}')" >/dev/null
|
||||
sleep "$PROP_WAIT"
|
||||
# A should get a private DM reply; C (if present) should NOT.
|
||||
if [[ "$(saw_text A "$NONCE-ai-reply")" == "true" || "$(node_result A mesh.messages '{"limit":50}' | jq -r '[.messages[]|select(.direction=="received")]|length>0')" == "true" ]]; then
|
||||
ok "A received an assistant reply"
|
||||
else
|
||||
no "A did not receive an assistant reply within ${PROP_WAIT}s"
|
||||
fi
|
||||
if (( HAVE_C )) && [[ "$(mesh_connected C)" == "true" ]]; then
|
||||
# heuristic: the reply text shouldn't be on C's channel feed
|
||||
skip "assistant reply privacy" "eyeball C's feed — automated check is heuristic"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
skip "assistant reply" "set ASSIST=1 and enable the assistant on B to run"
|
||||
fi
|
||||
|
||||
# ── 8. reachability snapshot (report-only) ─────────────────────────────────
|
||||
section "8. reachability snapshot (report-only)"
|
||||
node_result A mesh.peers 2>/dev/null | jq -r '.peers[]?
|
||||
| " \(.advert_name // "?") reachable=\(.reachable) last_advert=\(.last_advert // 0)"'
|
||||
printf '%s\n' "$(yellow ' NOTE: Meshtastic flood-routes; path_len is always 0xff, so `reachable`')"
|
||||
printf '%s\n' "$(yellow ' may read true even for stale nodes. Confirm desired semantics here')"
|
||||
printf '%s\n' "$(yellow ' before changing the refresh_contacts reachability rule.')"
|
||||
|
||||
# ── summary ────────────────────────────────────────────────────────────────
|
||||
section "summary"
|
||||
printf 'PASS=%s FAIL=%s SKIP=%s\n' "$(green "$PASS")" "$( ((FAIL)) && red "$FAIL" || green 0 )" "$(yellow "$SKIP")"
|
||||
if (( FAIL )); then
|
||||
printf 'failed:\n'; for n in "${FAILED_NAMES[@]}"; do printf ' - %s\n' "$n"; done
|
||||
fi
|
||||
exit "$FAIL"
|
||||
Executable
+77
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
# Controlled two-node reproduction of node-to-node federation sync.
|
||||
#
|
||||
# Pairs two real nodes via federation.invite/join, triggers federation.sync-state
|
||||
# in both directions, and reports which transport actually carried the call and
|
||||
# any per-peer error. This is the controlled repro for the reported
|
||||
# "Tor connection cloud->node not working" symptom: raw Tor transport is known
|
||||
# good (see README), so this isolates whether the APP-level sync path works and,
|
||||
# if it fails, surfaces the exact error string.
|
||||
#
|
||||
# Env (override as needed):
|
||||
# A_URL A_PW node A base url + UI password (default .116 http)
|
||||
# B_URL B_PW node B base url + UI password (default .228 https)
|
||||
# FORCE_TOR=1 set both nodes' federation transport preference to Tor first
|
||||
#
|
||||
# Usage: tests/multinode/repro-federation-sync.sh
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$HERE/lib/multinode.bash"
|
||||
|
||||
A_URL="${A_URL:-http://192.168.1.116}"; A_PW="${A_PW:?A_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
B_URL="${B_URL:-https://192.168.1.228}"; B_PW="${B_PW:?B_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
|
||||
bar() { printf '\n=== %s ===\n' "$*"; }
|
||||
|
||||
node_register A "$A_URL" "$A_PW"
|
||||
node_register B "$B_URL" "$B_PW"
|
||||
|
||||
bar "login"
|
||||
node_login A || { echo "A login failed"; exit 1; }
|
||||
node_login B || { echo "B login failed"; exit 1; }
|
||||
echo "A=$A_URL B=$B_URL logged in"
|
||||
|
||||
bar "onions"
|
||||
A_ONION=$(node_onion A); B_ONION=$(node_onion B)
|
||||
echo "A onion: ${A_ONION:-<none>}"
|
||||
echo "B onion: ${B_ONION:-<none>}"
|
||||
|
||||
if [[ "${FORCE_TOR:-0}" == "1" ]]; then
|
||||
bar "force federation transport = tor on both"
|
||||
node_rpc A transport.set-preference '{"service":"federation","pref":"tor"}' | jq -c '.result // .error'
|
||||
node_rpc B transport.set-preference '{"service":"federation","pref":"tor"}' | jq -c '.result // .error'
|
||||
fi
|
||||
|
||||
bar "federation state BEFORE"
|
||||
echo "A knows:"; node_result A federation.list-nodes | jq -r '.[]? | " \(.name // "?") did=\(.did[0:24])… last_seen=\(.last_seen // "never")"' 2>/dev/null || echo " (none/err)"
|
||||
echo "B knows:"; node_result B federation.list-nodes | jq -r '.[]? | " \(.name // "?") did=\(.did[0:24])… last_seen=\(.last_seen // "never")"' 2>/dev/null || echo " (none/err)"
|
||||
|
||||
bar "pair: A invites, B joins"
|
||||
INV_A=$(node_result A federation.invite)
|
||||
CODE_A=$(echo "$INV_A" | jq -r '.code // empty')
|
||||
echo "A invite code: ${CODE_A:0:40}…"
|
||||
if [[ -n "$CODE_A" ]]; then
|
||||
node_result B federation.join "$(jq -nc --arg c "$CODE_A" '{code:$c}')" \
|
||||
&& echo "B joined A" || echo "B join FAILED"
|
||||
fi
|
||||
|
||||
bar "pair: B invites, A joins"
|
||||
INV_B=$(node_result B federation.invite)
|
||||
CODE_B=$(echo "$INV_B" | jq -r '.code // empty')
|
||||
echo "B invite code: ${CODE_B:0:40}…"
|
||||
if [[ -n "$CODE_B" ]]; then
|
||||
node_result A federation.join "$(jq -nc --arg c "$CODE_B" '{code:$c}')" \
|
||||
&& echo "A joined B" || echo "A join FAILED"
|
||||
fi
|
||||
|
||||
bar "trigger sync-state on A (A dials its peers)"
|
||||
node_result A federation.sync-state | jq '.'
|
||||
|
||||
bar "trigger sync-state on B (B dials its peers)"
|
||||
node_result B federation.sync-state | jq '.'
|
||||
|
||||
bar "federation state AFTER (look for fresh last_seen + transport)"
|
||||
echo "A knows:"; node_result A federation.list-nodes | jq -r '.[]? | " \(.name // "?") last_seen=\(.last_seen // "never") transport=\(.last_transport // .transport // "?")"' 2>/dev/null
|
||||
echo "B knows:"; node_result B federation.list-nodes | jq -r '.[]? | " \(.name // "?") last_seen=\(.last_seen // "never") transport=\(.last_transport // .transport // "?")"' 2>/dev/null
|
||||
|
||||
bar "done"
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env bash
|
||||
# Two-node (optionally three-node) end-to-end smoke suite for the full app.
|
||||
#
|
||||
# Unlike repro-federation-sync.sh (a diagnostic that just prints state), this
|
||||
# is an ASSERTION suite: every check is pass/fail and the script exits non-zero
|
||||
# if any required check fails. It exercises the real node-to-node surface and
|
||||
# specifically guards the bugs fixed in v1.7.94 / v1.7.95:
|
||||
# - FIPS auto-connects to the public anchor (v1.7.94)
|
||||
# - peer content browse works over the mesh, not just Tor (v1.7.95 — the
|
||||
# `/content` catalog used to 404 over FIPS and never fall back to Tor)
|
||||
# - a removed federation node stays removed, incl. transitive re-discovery
|
||||
# (v1.7.95 tombstone) — the transitive case needs node C.
|
||||
#
|
||||
# Nodes (override via env):
|
||||
# A_URL A_PW node A (default .116 http)
|
||||
# B_URL B_PW node B (default .228 https)
|
||||
# C_URL C_PW node C (OPTIONAL — enables the transitive-tombstone test)
|
||||
#
|
||||
# Requires both nodes on v1.7.95-alpha+ for the content-browse and tombstone
|
||||
# checks; older peers SKIP those (reported, not failed).
|
||||
#
|
||||
# Usage:
|
||||
# tests/multinode/smoke.sh
|
||||
# A_URL=http://192.168.1.116 B_URL=https://192.168.1.228 tests/multinode/smoke.sh
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$HERE/lib/multinode.bash"
|
||||
|
||||
A_URL="${A_URL:-http://192.168.1.116}"; A_PW="${A_PW:?A_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
B_URL="${B_URL:-https://192.168.1.228}"; B_PW="${B_PW:?B_PW required — export it or set tests/multinode/.env (see .env.example)}"
|
||||
C_URL="${C_URL:-}"; C_PW="${C_PW:-}"
|
||||
|
||||
# ── tiny assertion framework ──────────────────────────────────────────────
|
||||
PASS=0; FAIL=0; SKIP=0
|
||||
declare -a FAILED_NAMES
|
||||
green() { printf '\033[32m%s\033[0m' "$*"; }
|
||||
red() { printf '\033[31m%s\033[0m' "$*"; }
|
||||
yellow(){ printf '\033[33m%s\033[0m' "$*"; }
|
||||
section() { printf '\n\033[1m── %s ──\033[0m\n' "$*"; }
|
||||
ok() { printf ' %s %s\n' "$(green ✓)" "$1"; PASS=$((PASS+1)); }
|
||||
no() { printf ' %s %s\n' "$(red ✗)" "$1"; FAIL=$((FAIL+1)); FAILED_NAMES+=("$1"); }
|
||||
skip() { printf ' %s %s (%s)\n' "$(yellow —)" "$1" "$2"; SKIP=$((SKIP+1)); }
|
||||
# assert_eq NAME EXPECTED ACTUAL
|
||||
assert_eq() { [[ "$2" == "$3" ]] && ok "$1" || no "$1 (expected '$2', got '$3')"; }
|
||||
# assert_true NAME VALUE — passes when VALUE is "true"
|
||||
assert_true() { [[ "$2" == "true" ]] && ok "$1" || no "$1 (got '$2')"; }
|
||||
|
||||
# did_of HANDLE — this node's own DID via node.did (string or {did:...}).
|
||||
did_of() {
|
||||
node_result "$1" node.did 2>/dev/null \
|
||||
| jq -r 'if type=="string" then . elif type=="object" then (.did // .node_did // empty) else empty end' 2>/dev/null
|
||||
}
|
||||
|
||||
# pair HANDLE_INVITER HANDLE_JOINER — invite + join one direction. Echo "ok"/"fail".
|
||||
pair() {
|
||||
local inv code
|
||||
inv=$(node_result "$1" federation.invite 2>/dev/null)
|
||||
code=$(echo "$inv" | jq -r '.code // empty' 2>/dev/null)
|
||||
[[ -z "$code" ]] && { echo "fail"; return; }
|
||||
if node_result "$2" federation.join "$(jq -nc --arg c "$code" '{code:$c}')" >/dev/null 2>&1; then
|
||||
echo "ok"
|
||||
else echo "fail"; fi
|
||||
}
|
||||
|
||||
node_register A "$A_URL" "$A_PW"
|
||||
node_register B "$B_URL" "$B_PW"
|
||||
HAVE_C=0
|
||||
if [[ -n "$C_URL" && -n "$C_PW" ]]; then node_register C "$C_URL" "$C_PW"; HAVE_C=1; fi
|
||||
|
||||
# ── 1. reachability + auth ────────────────────────────────────────────────
|
||||
section "reachability + login"
|
||||
node_login A && ok "A login ($A_URL)" || { no "A login ($A_URL)"; echo "A unreachable — aborting"; exit 1; }
|
||||
node_login B && ok "B login ($B_URL)" || { no "B login ($B_URL)"; echo "B unreachable — aborting"; exit 1; }
|
||||
if [[ $HAVE_C == 1 ]]; then node_login C && ok "C login ($C_URL)" || { no "C login"; HAVE_C=0; }; fi
|
||||
|
||||
A_ONION=$(node_onion A); B_ONION=$(node_onion B)
|
||||
[[ -n "$A_ONION" ]] && ok "A has onion address" || no "A has onion address"
|
||||
[[ -n "$B_ONION" ]] && ok "B has onion address" || no "B has onion address"
|
||||
|
||||
# ── 2. FIPS mesh: daemon up + anchor connected (v1.7.94) ──────────────────
|
||||
section "FIPS mesh / anchor"
|
||||
for h in A B; do
|
||||
s=$(node_result "$h" fips.status 2>/dev/null)
|
||||
if [[ -z "$s" ]]; then skip "$h fips.status" "no FIPS RPC (old build?)"; continue; fi
|
||||
assert_true "$h FIPS service active" "$(echo "$s" | jq -r '.service_active')"
|
||||
ac=$(echo "$s" | jq -r '.anchor_connected')
|
||||
if [[ "$ac" == "true" ]]; then ok "$h anchor connected"
|
||||
else skip "$h anchor connected" "anchor_connected=$ac — node may need v1.7.94 + a moment to handshake"; fi
|
||||
done
|
||||
|
||||
# ── 3. federation pairing (both directions) ───────────────────────────────
|
||||
section "federation pairing"
|
||||
assert_eq "A invites, B joins" "ok" "$(pair A B)"
|
||||
assert_eq "B invites, A joins" "ok" "$(pair B A)"
|
||||
# both should now list each other
|
||||
node_result A federation.sync-state >/dev/null 2>&1
|
||||
node_result B federation.sync-state >/dev/null 2>&1
|
||||
A_SEES_B=$(node_result A federation.list-nodes 2>/dev/null | jq -r --arg o "${B_ONION%.onion}" 'any((.nodes // .)[]?; (.onion // "" | gsub("\\.onion$";"")) == $o)')
|
||||
B_SEES_A=$(node_result B federation.list-nodes 2>/dev/null | jq -r --arg o "${A_ONION%.onion}" 'any((.nodes // .)[]?; (.onion // "" | gsub("\\.onion$";"")) == $o)')
|
||||
assert_true "A's node list contains B" "$A_SEES_B"
|
||||
assert_true "B's node list contains A" "$B_SEES_A"
|
||||
|
||||
# ── 4. peer content browse over the mesh (v1.7.95 fix) ────────────────────
|
||||
section "peer content browse (was: 404 over mesh, no Tor fallback)"
|
||||
if [[ -n "$B_ONION" ]]; then
|
||||
resp=$(node_rpc A content.browse-peer "$(jq -nc --arg o "$B_ONION" '{onion:$o}')")
|
||||
err=$(echo "$resp" | jq -r '.error.message // empty')
|
||||
if [[ -z "$err" ]]; then
|
||||
ok "A browses B's content catalog (HTTP 200)"
|
||||
elif echo "$err" | grep -q '404'; then
|
||||
no "A browses B's content — still 404 over mesh (is B on v1.7.95?): $err"
|
||||
else
|
||||
# Other errors (peer offline, no content shared) are environmental, not the bug.
|
||||
skip "A browses B's content" "non-404 error: $err"
|
||||
fi
|
||||
else
|
||||
skip "A browses B's content" "B has no onion"
|
||||
fi
|
||||
|
||||
# ── 5. removed-node tombstone (v1.7.95) ───────────────────────────────────
|
||||
section "removed-node tombstone"
|
||||
B_DID=$(did_of B)
|
||||
if [[ -z "$B_DID" ]]; then
|
||||
skip "remove B then verify stays removed" "couldn't resolve B's DID"
|
||||
else
|
||||
if node_result A federation.remove-node "$(jq -nc --arg d "$B_DID" '{did:$d}')" >/dev/null 2>&1; then
|
||||
still=$(node_result A federation.list-nodes 2>/dev/null | jq -r --arg d "$B_DID" 'any((.nodes // .)[]?; .did == $d)')
|
||||
assert_eq "B removed from A's list" "false" "$still"
|
||||
# Transitive test needs C: A federated with B and C; C federated with B;
|
||||
# A removes B; A syncs with C (who advertises B) → B must NOT reappear.
|
||||
if [[ $HAVE_C == 1 ]]; then
|
||||
pair A C >/dev/null; pair C A >/dev/null; pair C B >/dev/null
|
||||
node_result A federation.sync-state >/dev/null 2>&1
|
||||
reappeared=$(node_result A federation.list-nodes 2>/dev/null | jq -r --arg d "$B_DID" 'any((.nodes // .)[]?; .did == $d)')
|
||||
assert_eq "B does NOT reappear via transitive sync with C" "false" "$reappeared"
|
||||
else
|
||||
skip "transitive reappear via 3rd node" "set C_URL/C_PW to enable"
|
||||
fi
|
||||
# re-add restores B (explicit re-add clears the tombstone)
|
||||
pair B A >/dev/null
|
||||
node_result A federation.sync-state >/dev/null 2>&1
|
||||
readded=$(node_result A federation.list-nodes 2>/dev/null | jq -r --arg d "$B_DID" 'any((.nodes // .)[]?; .did == $d)')
|
||||
assert_true "explicit re-pair brings B back (tombstone cleared)" "$readded"
|
||||
else
|
||||
skip "remove B" "remove-node RPC failed (B may already be absent)"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── summary ───────────────────────────────────────────────────────────────
|
||||
section "summary"
|
||||
printf ' %s passed, %s failed, %s skipped\n' "$(green $PASS)" "$([[ $FAIL -gt 0 ]] && red $FAIL || echo $FAIL)" "$(yellow $SKIP)"
|
||||
if [[ $FAIL -gt 0 ]]; then
|
||||
printf ' failed:\n'; for n in "${FAILED_NAMES[@]}"; do printf ' - %s\n' "$n"; done
|
||||
exit 1
|
||||
fi
|
||||
echo " all required checks passed"
|
||||
@@ -0,0 +1,260 @@
|
||||
# ▶▶ SESSION SAVE / RESUME (2026-06-16) — v1.7.97-alpha CUT, mid-rollout
|
||||
|
||||
**v1.7.97-alpha is BUILT + TAGGED LOCALLY but NOT yet published to the fleet.**
|
||||
- Release commit `47c16971` ("chore: release v1.7.97-alpha") + tag `v1.7.97-alpha` exist on LOCAL main only. NOT pushed to gitea-vps2. Fleet still sees 1.7.96-alpha.
|
||||
- Contents (14 fixes + image-opt): B5,B1,B2,B4,B14,B21,B3,B15,B7,B13,B12,B16,**B17**, B6-pruned-gate + lossless background-image optimization (bg-mesh PNG→JPEG).
|
||||
- Release artifacts staged: `releases/v1.7.97-alpha/{archipelago, archipelago-frontend-1.7.97-alpha.tar.gz}` + `/tmp/archipelago-frontend-1.7.97-alpha.tar.gz` (177MB, flat layout verified, optimized images baked in, no APK).
|
||||
- **Deployed (sideload, NOT fleet OTA):** .116 = on 1.7.97-alpha, healthy, B17 self-heal CONFIRMED (unit now has RequiresMountsFor, 36 containers survived restart). .198 = deploying (sideload binary+frontend).
|
||||
- **Backup binaries for rollback:** `/usr/local/bin/archipelago.1.7.96-alpha.bak` on .116 and .198.
|
||||
|
||||
**REMAINING (this session, user wants to do WITH them):**
|
||||
1. Finish .198 sideload; then **UI-confirm fixes together on .116/.198** + close passing Gitea issues (#8,#9,#10,#11,#12,#14,#19(code-only),#20,#21,#22,#23,#24,#29). Issue map below.
|
||||
2. **Publish to fleet:** `scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2` + `git push gitea-vps2 main + tag` (AFTER joint confirm — user's call).
|
||||
3. **Cut a fresh ISO** (bakes B13 nginx + B17 unit + all frontend). ISO builds run on a server (deploy-to-target / .228). Then test the ISO together.
|
||||
|
||||
⚠️ LESSON: never run the release binary to "check --version" — it has no such flag and BOOTS A FULL NODE (adopts containers, grabs mesh radio). Use `strings <bin> | grep version`. (Did this on .116; the instance exited on the :5678 port conflict, no harm.)
|
||||
|
||||
---
|
||||
|
||||
# ▶▶ SESSION SAVE / RESUME (2026-06-15)
|
||||
|
||||
**State:** v1.7.96-alpha SHIPPED. v1.7.97-alpha NOT cut yet — 10 fixes committed on **vps2 main** (`git remote: gitea-vps2`), nothing on the fleet yet. Validate on .116/.198 + UI-confirm BEFORE cutting .97.
|
||||
|
||||
**Resume command (run elsewhere):**
|
||||
```
|
||||
cd ~/Projects/archy && git fetch gitea-vps2 && git checkout main && git reset --hard gitea-vps2/main && cat tests/production-quality/TRACKER.md
|
||||
```
|
||||
Then continue from "IN PROGRESS" below.
|
||||
|
||||
**Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7, **B13 (fedimint CSS self-heal — main conf + HTTPS snippet, verified .198 both paths app-icon 404→200)**, **B12 (mempool bitcoin-host detect across 3 render paths — unit-tested; live bitcoin-core validation pending)**, **B16 (bitcoin sync tile retain/Updating… — unit-tested 6/6, commit 83dbd25c)**. B6 pruned-gate already live. = 13 fixes. PLUS **image-optimization** (commit 386d4bfc — all bg images losslessly optimized, bg-mesh PNG→JPEG; user asked to include it in the .97 release).
|
||||
|
||||
**IN PROGRESS — B16 DONE (commit 83dbd25c). Pick up at B6 no-node-present half.** B13 + B12 + B16 DONE (committed; see entries below). REMAINING:
|
||||
1. **B6** no-node-present half, **B12b** (sibling bitcoin-host hardcodes: LND/BTCPay/electrumx/fedimint + mempool dep declaration — reuse `{{BITCOIN_HOST}}`; needs validation, esp. LND/fedimint), **B14b** (FIPS reachability depth), **B22/B23** (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
|
||||
3. **Loose end:** 4 pre-existing prod_orchestrator test failures (generated-files/data_uid fixtures use disallowed tempdir volume sources) — see B12 NOTE; separate small fix.
|
||||
|
||||
Note: .198 is running a sideloaded B13-era .97-dev binary (md5 4c83803d). The B12 binary was built (`core/target/release/archipelago`) but NOT sideloaded (mempool isn't on .198; .198 is Knots so B12 is a no-op there). Reflashing/OTA replaces the dev binary.
|
||||
|
||||
**Ship .97 when ready:** ./scripts/create-release.sh 1.7.97-alpha (curate CHANGELOG ≥3 layman bullets first + run scripts/sync-whats-new.py; SKIP_RELEASE_TESTS=1 only for the 2 known-flaky vitest timing tests) → scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2 → git push gitea-vps2 main + tag. (gitea-local push fails: token rejected — non-blocking.)
|
||||
|
||||
---
|
||||
|
||||
# Production-Quality Bug Tracker
|
||||
|
||||
Living tracker for the post-v1.7.96 "no new features until production quality" push.
|
||||
Updated continuously as we investigate → fix → test → pass. Kept in-repo so progress
|
||||
survives a session cutoff.
|
||||
|
||||
## Rules (from user, 2026-06-15)
|
||||
- **No new features** until the OS is production / no-bugs quality.
|
||||
- **Test-harness-first**: build/extend a harness for each bug before fixing.
|
||||
- **Validate every fix on `.116` + `.198`** (both 192.168.1.x, pw ThisIsWeb54321@) **+ the harness** BEFORE it goes into any release. (.198 still carries the LND CORS nginx duplicate → good for fix-(a) validation; .116 does not.)
|
||||
- **Priority order**: cloud/federated-nodes + mesh FIRST, then app-specific, then low-pri.
|
||||
|
||||
## Status legend
|
||||
`TODO` · `INVESTIGATING` · `ROOT-CAUSED` · `FIXING` · `TESTING` (on .116+harness) · `PASSED` · `SHIPPED`
|
||||
|
||||
## Release status
|
||||
- **v1.7.96-alpha — SHIPPED** (2026-06-15). Live on vps2 (primary OTA): manifest v1.7.96-alpha, assets HTTP 200, `main@8c3c7954` + tag present. Contents: kiosk grid removal + FIPS TCP/UDP anchor selector. NOTE: gitea-local (localhost) mirror push failed (token rejected → /login); non-blocking, needs refreshed token.
|
||||
- **v1.7.97-alpha — IN PROGRESS** (this push). Will bundle the verified fixes below.
|
||||
|
||||
---
|
||||
|
||||
## 🔴🔴 TOP PRIORITY
|
||||
|
||||
### B5 — LND "connect your wallet" details/QR broken fleet-wide — ROOT-CAUSED
|
||||
Origin: user escalation. Symptom: LND connect screen (served on app port :18083) can't load details/QR.
|
||||
Two distinct root causes (confirmed live):
|
||||
- **(a) Duplicate ACAO** on `/lnd-connect-info` (seen on .103): backend sets `Access-Control-Allow-Origin` (proxy.rs:108) AND nginx `add_header` adds a second → browser rejects "multiple values". nginx config drift. Fix: bootstrap.rs nginx patch must strip the redundant `add_header` from the `/lnd-connect-info` location (backend owns CORS).
|
||||
- **(b) No ACAO on `/proxy/lnd/v1/*` 401** (fleet-wide): the unauth/auth-layer 401 is produced before the CORS-adding proxy handler (proxy.rs:135 `handle_lnd_proxy`). Browser → "No 'Access-Control-Allow-Origin' header". Fix: ensure auth-layer/early-return responses for `/proxy/lnd` + `/lnd-connect-info` carry CORS headers.
|
||||
- `.116` `/lnd-connect-info` returns a single correct ACAO → symptom varies by node's nginx state.
|
||||
- Backend CORS helper: handler/mod.rs `app_cors_origin()` (:270) — reflects Origin when its host == request host.
|
||||
- Backend change → ships in .97. **Status: ✅ PASSED — verified on .116, .198, .103 (harness 4/4 each). Ready to bundle into .97.**
|
||||
- Caveat: bootstrap's nginx dup-strip runs a few seconds AFTER /health goes green (async patch+reload) — converges within ~1 min of restart; not instant. Acceptable.
|
||||
- **CODE CHANGES MADE (uncommitted):**
|
||||
- `core/archipelago/src/bootstrap.rs`: added `NGINX_LND_DUP_CORS` const + strip in `patch_nginx_conf()` (removes the duplicate nginx `add_header` ACAO from `/lnd-connect-info` so the backend's single header wins). Idempotent; runs on startup nginx bootstrap. → fixes (a)
|
||||
- `core/archipelago/src/api/handler/mod.rs`: new `unauthorized_cors(origin)` helper (:~205) + `/proxy/lnd/` route (:~505) computes origin first and returns `unauthorized_cors` so the 401 carries ACAO. → fixes (b)
|
||||
- Test on **.116** for (b); test on **.103** for (a) [.116 has no dup to strip].
|
||||
- **2026-06-15 RESULT — .116 (fix b): harness 4/4 PASS** (sideloaded built binary, restarted). `/proxy/lnd/v1/*` now returns CORS on the 401. ✅
|
||||
- (Correction: an earlier "LND container MISSING" reading was a FALSE alarm — `docker` isn't in the non-interactive PATH; runtime is **podman**. Verified `lnd Up 9h` — containers SURVIVED the restart cleanly.)
|
||||
- Next: deploy to .103 + run harness to confirm fix (a) (nginx dup strip).
|
||||
- **Harness:** `tests/production-quality/lnd-cors-test.sh <node>` — asserts single correct ACAO on /lnd-connect-info + ACAO present on /proxy/lnd/v1/{getinfo,channels}. Baseline (2026-06-15): .116 = 2 pass/2 fail (proxy missing ACAO); .103 = 1 pass/3 fail (connect-info dup + proxy missing).
|
||||
- **FIX PLAN (precise):**
|
||||
1. (b) handler/mod.rs:504-508 `/proxy/lnd/` returns `Self::unauthorized()` (401, NO CORS) when session check fails → browser CORS wall. Add CORS (app_cors_origin) to that 401. Same pattern for any other app-origin early-return.
|
||||
2. (a) nginx `/lnd-connect-info` location double-adds ACAO (backend + nginx `add_header`). Strip the nginx `add_header Access-Control-Allow-Origin` there; backend owns CORS. Update bootstrap.rs nginx patch to remove it on existing nodes (idempotent).
|
||||
- Verify: rebuild backend, deploy to .116, run harness → expect 3/3 (or 4 assertions) PASS on .116 AND .103.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 PRIORITY — cloud / federation / mesh
|
||||
|
||||
### B1 — Trusted-node list not clean — PASSED (onion-dedup; unit test 2/2; live .198 15→13 distinct, healthy). UI visual-confirm recommended.
|
||||
Dupes, erroneous names, and non-convergent group membership across nodes. Expected: trusted nodes form a transitive group (every node connects to any newly-added trusted node; all nodes show the same set). `.103` has a long/dirty list.
|
||||
|
||||
### B2 — Duplicate chat contact for one node — PASSED (resolved by load-dedup feeding mesh seed; unit-tested). UI visual-confirm recommended.
|
||||
Federated peer "sapien" shows TWO chats: one "sapien" WITHOUT archy logo (looks non-federated) + one named by raw DID `did:key:z6MkoSbN5CM7fBaQg2nWbCymEkFXsHnuXvec9Mjo5RtJf9dQ`. Same node keyed by both federated identity and raw DID → merge to one. Code: core/archipelago/src/mesh + mesh/typed_messages.rs (note :233 — meshcore adverts don't carry archy pubkey).
|
||||
|
||||
### B3 — Cloud peer media won't preview/play — FIXING (code done: /api/peer-content streaming proxy + playMedia streams free content)
|
||||
Music/video preview files on peer nodes' cloud don't play (streaming/range/content-type over mesh+Tor peer fetch).
|
||||
|
||||
### B4 — Cloud "my folders" fails (JSON parse / 502) — PASSED (content-type guard; built, guard in bundle, deployed .198). UI visual-confirm recommended.
|
||||
`Unexpected token '<', "<!doctype"` when FileBrowser absent (`/app/filebrowser/api/resources` → SPA index.html), and **502** when FileBrowser is down (seen on .103). filebrowser-client.ts:102/:106. Fix: detect FileBrowser unavailable, friendly prompt; consider nginx returning JSON 404/502 for missing `/app/<app>/` instead of SPA shell. Handle BOTH absent + down.
|
||||
|
||||
### B14 — cloud browse transport not recorded — FIXED (record_peer_transport in 4 content handlers; build OK). NOTE: live data shows FIPS reaches only ~4/15 peers, 6 fall back to Tor genuinely → see B14b.
|
||||
Browsing trusted/peer nodes in the Cloud tab connects over Tor instead of FIPS (should prefer FIPS like the rest of mesh; same for peer browsing). cf project_fips_integration, project_tor_node_to_node_works (last_transport should be fips/mesh).
|
||||
|
||||
---
|
||||
|
||||
## 🟠 APP-SPECIFIC
|
||||
|
||||
### B6 — ElectrumX install gate — PARTIAL (pruned-node gate already works; "no node present" half DEFERRED: false-positive risk without UI test, needs package-presence check)
|
||||
Show the yellow requirement badge when no full node / only a pruned node is present (reuse existing yellow badge pattern).
|
||||
|
||||
### B7 — ElectrumX UI stuck loader on top — FIXED (overlay hides + iframe shows when status stale; type-check green). UI-confirm.
|
||||
UI renders but a loader sits on top; possibly stale pre-sync screen not clearing.
|
||||
|
||||
### B9 — IndeedHub keeps stopping on nodes — TODO
|
||||
Container won't stay running (crash-loop / reconcile stop). Check logs + restart policy + health.
|
||||
|
||||
### B10 — Immich still crashes — TODO
|
||||
Recurring crash ("still" → prior attempts). Check container logs + resource limits + DB/ML deps.
|
||||
|
||||
### B11 — Companion app: "open in external browser" apps don't work — TODO
|
||||
Apps meant to open in a new/external browser don't launch from the companion app; need the phone-default-browser request-modal pattern mobile apps use. Relates to v1.7.90 "open in new tab from companion app".
|
||||
|
||||
### B12 — Mempool not connecting — FIXED (mempool host detect, 3 paths; unit-tested). Live bitcoin-core validation PENDING (no core node available).
|
||||
**Bigger than the original "stacks.rs:1278" framing.** `CORE_RPC_HOST=bitcoin-knots` was hardcoded in THREE env-render paths; on a bitcoin-core node the container is named `bitcoin-core`, so mempool-api can't resolve RPC. Both Knots and Core are reachable on `archy-net` by container name — only the name differs.
|
||||
- **Path 1 — legacy direct-podman** (`stacks.rs::install_mempool_stack`, used when no orchestrator): now `format!("CORE_RPC_HOST={}", detect_bitcoin_rpc_host())`. FIXED.
|
||||
- **Path 2 — `config.rs::get_app_config`** (install.rs legacy path): same. FIXED.
|
||||
- **Path 3 — Quadlet/manifest (THE MODERN FLEET PATH, e.g. .198)**: `prod_orchestrator` renders env from `apps/mempool-api/manifest.yml` static YAML. FIXED via a new `{{BITCOIN_HOST}}` derived-env placeholder: `HostFacts.bitcoin_host` (container/manifest.rs) + `resolve_derived_env` renders it; `prod_orchestrator::bitcoin_host()` detects Knots/Core via `podman ps` (test-injectable `set_bitcoin_host_for_test`); resolved on-demand only for manifests using the placeholder (perf). mempool-api manifest moved `CORE_RPC_HOST` from static env → `derived_env: {{BITCOIN_HOST}}`.
|
||||
- New helper `dependencies::detect_bitcoin_rpc_host()` + pure `pick_bitcoin_host()`.
|
||||
- **TESTS (all green):** `pick_bitcoin_host` 5 cases (knots/core/plain/none/substring-safety); container-crate `resolve_derived_env` renders `{{BITCOIN_HOST}}`; orchestrator `mempool_core_rpc_host_follows_bitcoin_node` (core→bitcoin-core, knots→bitcoin-knots). No-regression verified: picker returns `bitcoin-knots` live on .198 (so Knots nodes unchanged; existing mempool installs see no env drift).
|
||||
- **VALIDATION GAP:** cannot exercise on a live bitcoin-core node (none available; .198 is Knots where the fix is a no-op). Need a Core node to confirm end-to-end.
|
||||
- **FOLLOW-UP (B12b, NOT done):** same hardcode exists for siblings on bitcoin-core nodes — `config.rs` lnd(:724)/btcpay(:739)/electrumx(:782), and `prod_orchestrator::resolve_dynamic_env` fedimint `FM_BITCOIND_URL=...bitcoin-knots` (~:2425). Plus mempool-api manifest `dependencies: bitcoin-knots` (line 18) is Knots-specific bookkeeping (install-time check already accepts Core via BITCOIN_NAMES, so non-blocking). All can reuse `{{BITCOIN_HOST}}`. Deferred per user (mempool-only scope) — each needs its own validation, esp. LND/fedimint.
|
||||
- **NOTE (unrelated pre-existing failures):** 4 prod_orchestrator tests fail on clean HEAD too — `install_applies_data_uid_chown_before_create`, `install_writes_manifest_generated_files_before_create`, `manifest_generated_files_{do_not_overwrite_by_default,can_overwrite_when_declared}` — their fixtures pass tempdir volume sources that `validate_bind_source` rejects (only `/var/lib/archipelago/*` + 2 sockets allowed). NOT caused by B12; worth a separate fix.
|
||||
mempool can't reach the Bitcoin backend on some nodes. Investigate on .116. Check mempool→electrs→bitcoind wiring + deps.
|
||||
|
||||
### B13 — Fedimint UI not applying CSS — FIXED + VERIFIED on .198 (both HTTP + HTTPS)
|
||||
Root cause confirmed: the Fedimint Guardian page (served by :8175) is a server-rendered status page with ~7.8KB INLINE CSS plus image assets referenced root-rooted (`src="/assets/img/app-icons/fedimint.jpg"`, `url("/assets/img/bg-network.jpg")`). Without an asset rewrite those `/assets/...` URLs resolve against the archipelago SPA root: `bg-network.jpg` happens to exist there (shared design asset → loaded by luck) but `app-icons/fedimint.jpg` does NOT → **404** (the broken/visibly-missing icon). The `location /assets/` block uses `try_files $uri =404`, so missing fedimint assets 404 rather than fall through.
|
||||
|
||||
Fix = nginx sub_filter set that reroots every root-rooted asset URL (`href="/`, `src="/`, `url("/`, and single-quote variants) under `/app/fedimint/`, plus `proxy_set_header Accept-Encoding ""` so the upstream doesn't gzip (sub_filter can't rewrite gzipped bodies). Shipped two ways:
|
||||
- **Fresh ISOs** (committed a50b6df2): templates `image-recipe/configs/nginx-archipelago.conf` (HTTP) + `image-recipe/configs/snippets/archipelago-https-app-proxies.conf` (HTTPS).
|
||||
- **Already-deployed nodes** (bootstrap self-heal, this commit): `core/archipelago/src/bootstrap.rs::patch_nginx_conf` now heals BOTH the main conf (Style A — swaps the old single nostr-provider sub_filter tail for the full reroot set, byte-matches the shipped template) AND the HTTPS app-proxy snippet (Style B — anchors on the unique `:8175` proxy_pass and inserts the reroot set; robust to the snippet's varying trailing directive). `missing_*` flags now gated on their splice anchors so the healed snippet early-returns cleanly (no per-boot warn-skips). Idempotent via the `'href="/' 'href="/app/fedimint/'` marker.
|
||||
|
||||
VERIFIED on .198 (sideloaded built binary, restart, async self-heal converged ~15s):
|
||||
- HTTP `/app/fedimint/`: live conf healed byte-identical to template; app-icon **404→200 image/jpeg (41944b)**.
|
||||
- HTTPS `/app/fedimint/` (snippet): healed; same app-icon **404→200**; bg-network 200; root `/assets/img/app-icons/fedimint.jpg` returns 200 **text/html** (SPA shell) — proving the reroot is necessary.
|
||||
- `nginx -t` OK both times; containers survived restart (Quadlet); both files carry the marker exactly once (idempotent steady state); no warn spam in logs.
|
||||
NOTE: self-healed snippet is functionally correct but NOT byte-identical to the fresh-ISO snippet template (insert-after-proxy_pass vs full block) — acceptable; nginx ignores directive order/whitespace.
|
||||
|
||||
### B15 — Bitcoin UI sync progress lags — FIXED (Home.vue poll 30s→10s). UI-confirm.
|
||||
Bitcoin UI doesn't update its sync progress fast enough even though the console clearly already has the block-height data. Likely a polling-interval / reactive-update gap between the status source and the UI.
|
||||
|
||||
### B16 — Bitcoin sync status vanishes — FIXED + UNIT-TESTED (commit 83dbd25c). UI-confirm.
|
||||
The bitcoin sync status in the Home > System container disappears when it should persist/cache and show an "updating" state. Related to B15 (Bitcoin UI sync lag). Root cause: the tile is gated `v-if="stats.bitcoinAvailable===true"` (HomeSystemCard.vue:60); a transient `bitcoin.getinfo` failure (RPC busy during heavy IBD, or a route-change/scan where the packages map is momentarily empty) could blank it.
|
||||
FIX (commit 83dbd25c): added a `bitcoinStale` flag to homeStatus.ts —
|
||||
- getinfo fails while the bitcoin container is **Running**, OR package data is momentarily **absent** → retain last-known value + `bitcoinStale=true` (tile stays, renders **"Updating…"** instead of a frozen figure shown as live).
|
||||
- container authoritatively **Stopped/Exited** → `bitcoinAvailable=false`, `stale=false` (no stale-as-live — genuinely down is reflected).
|
||||
- first-ever poll times out but container Running (syncing node) → show the tile as updating rather than staying hidden.
|
||||
Wired `bitcoinStale` through Home.vue `systemStats` → HomeSystemCard prop; card shows "Updating…" (dimmed) when stale.
|
||||
**Harness:** `neode-ui/src/stores/__tests__/homeStatus.test.ts` (6 cases) — RED before fix (5/6 fail), GREEN after (6/6). `vue-tsc --noEmit` exit 0. Full vitest suite: only pre-existing AppIconGrid cross-test teardown flake (passes 7/7 standalone; not my change). UI-confirm on .116/.198 still recommended (hard to trigger transient failure on demand — unit test is the authoritative harness here).
|
||||
|
||||
### B17 — archipelago.service flaps on boot before starting — FIXED + VERIFIED on .198 (commit 34b1fdc1)
|
||||
On some boots, `[FAILED] Failed to start archipelago.service` printed ~20× over ~5 min before starting. ROOT CAUSE (proven live on .198): on production nodes `/var/lib/archipelago` is a **separate `/dev/mapper/archipelago-data` ext4 volume** (systemd unit `var-lib-archipelago.mount`), and podman's **graphroot=`/var/lib/archipelago/containers/storage`** lives on it too. The unit ordered only `After=network-online.target` — NO mount dependency — so on cold boots the service (and its `ExecStartPre`) could start BEFORE the volume mounted, write to the bare mountpoint on rootfs, fail every podman call, exit, and be restarted every 5s (`Restart=on-failure RestartSec=5`) until the mount appeared. Smoking gun in .198's journal: `var-lib-archipelago.mount: Directory /var/lib/archipelago to mount over is not empty, mounting anyway` — the service had written there pre-mount. Dev laptop .116 has the data dir on rootfs → never flaps (explains "on some boots"). Diagnostic: every node showed `banners == "Server listening"` (process always succeeds once it runs) ⇒ failure is systemd-level, not a Rust crash.
|
||||
FIX (commit 34b1fdc1): `RequiresMountsFor=/var/lib/archipelago` (adds `Requires=` + `After=` on the mount unit).
|
||||
- `image-recipe/configs/archipelago.service`: ships the directive on fresh ISOs.
|
||||
- `bootstrap::ensure_archipelago_mount_ordering()`: self-heals already-deployed nodes' installed `/etc/systemd/system/archipelago.service` + `daemon-reload` (boot-ordering only — effective next reboot; never restarts the running service). Idempotent; harmless on rootfs installs.
|
||||
VERIFIED on .198: applied directive → `systemctl show -p After` includes `var-lib-archipelago.mount`, `systemd-analyze verify` clean → rebooted: mount@07:35:22, archipelago banner@07:35:35 (13s AFTER mount), `banners=1 listening=1 failed_to_start=0` (zero flap), directive persisted. `cargo check` EXIT 0. NOTE: self-heal CODE (auto-patch on deployed nodes) still to be exercised with the built binary on .228 (directive was applied manually on .198); residual rootfs shadow files under the mountpoint are benign.
|
||||
|
||||
### B18 — Apps stop right after install (or become unstartable) — TODO
|
||||
Many apps install but immediately stop, requiring a manual Start — or become unstartable entirely. Likely the install→start handoff / reconciler doesn't bring them up (or starts then they exit). Related to B9 (IndeedHub stopping), B10 (Immich). Possibly linked to the cgroup-SIGKILL-on-archipelago.service-restart issue (feedback_no_systemctl_deploy_until_quadlet) — but NOTE: on .116 (Quadlet) containers survived a service restart cleanly, so the reconciler may be fine there; reproduce on the affected nodes. Check post-install start sequencing + boot_reconciler + container restart policy + cgroup placement.
|
||||
|
||||
### B19 — Failed download-update lands on Install button (should be Download) — TODO
|
||||
When an update download fails, the UI sometimes shows the Install button instead of returning to the Download button — a big UX issue (user can't retry the download cleanly). Check the SystemUpdate state machine's error/failure transition.
|
||||
|
||||
### B20 — Surface bitcoin-headers-over-mesh broadcast (send/receive toggles) — TODO (feature-adjacent, surfacing existing work)
|
||||
We previously broadcast bitcoin block headers over mesh to archipelago nodes but never fully surfaced it. Want two switches: "send headers" (you broadcast) and "receive headers" (you accept). NOTE: this is feature-adjacent — surfacing existing functionality; the user added it during the no-new-features push, so treat as low-priority polish until the bug list is clear. Code: mesh block-headers (mesh.block-headers RPC seen in logs; core/archipelago/src/mesh).
|
||||
|
||||
### B14b — FIPS reachability: many peers fall back to Tor — INVESTIGATED (needs FIPS-network depth)
|
||||
Live (2026-06-15) federation sync last_transport on .116/.198: ~4 peers fips, ~6 tor, ~5 none. So beyond the recording fix (B14), FIPS genuinely doesn't reach many federated peers (they use Tor). Investigate WHY: is fips_npub known for those peers? are they FIPS-online? is the shared anchor connecting them? (cf project_fips_integration, project_tor_node_to_node_works). This is the real "Tor not FIPS" depth.
|
||||
FINDINGS (.198, 2026-06-15): archipelago-fips ACTIVE; ALL 13 peers HAVE fips_npub; last_transport = 5 fips / 5 tor / 3 none. So it's NOT a missing-npub or service-down bug — FIPS genuinely reaches some peers and not others = DIAL-TIME reachability: the 'tor' peers aren't FIPS-reachable at dial time (offline, NAT, their FIPS not registered with the shared anchor), and 'none' = fully offline (X250 roam/beta/cellular). NEXT (deeper, needs FIPS-network debugging): verify a known-online peer (e.g. .228/.116) is reachable over FIPS from .198 right now; if an online FIPS peer still falls back to Tor → real anchor/registration bug; check fips daemon peer table + anchor connectivity. Likely partly peer-availability (not fully fixable in code).
|
||||
|
||||
### B21 — Show Tor/FIPS transport pill on cloud browse — FIXED (build+type-check green; deploy+UI-confirm on .116/.198)
|
||||
Tag whether the peer connection is Tor or FIPS and surface it as a small pill on the cloud browse screens / connection loader. Data source: federation node last_transport (now recorded by B14) exposed via federation.list-nodes; frontend renders a pill (FIPS=fast/green, Tor=slower) on PeerFiles.vue / Cloud peer view + the connection loader. Frontend-only-ish. FINDINGS: PeerFiles.vue:46 loader HARDCODES 'Connecting via Tor...' even when FIPS used (bug). Frontend types already have last_transport ('fips'|'tor'|'mesh'|'lan') federation/types.ts:31; NodeList.vue:167 already renders a transport indicator. PLAN: have content.browse-peer RETURN the transport used (B14 already computes it) → frontend shows a pill (FIPS green / Tor amber) on PeerFiles header + fix the loader text to reflect actual/attempted transport. Small backend (add transport to browse response) + frontend pill.
|
||||
|
||||
### B22 — Peer cloud download/audio errors (.228→.198) — TODO (pairs with B3)
|
||||
Observed 2026-06-15 browsing .228's cloud from .198: (a) downloading a peer cloud file → "Operation failed. Check server logs for details." (b) playing a peer AUDIO file → "Could not play audio. File Browser may not be running." (misleading — it's a peer file, not File Browser; that's the OLD base64/blob path B3 replaces). ACTION: (a) check content.download-peer backend error on .198 logs while downloading (likely the same Range/transport/timeout path as B3, or a peer-side 4xx); (b) verify B3 streaming fixes peer audio once deployed, and fix the misleading audioPlayer error string. Get server logs: ssh .198, journalctl -u archipelago | grep -i 'content\|peer\|download'.
|
||||
|
||||
### B23 — Archipelago group chat (all nodes) broken/slow over Tor — TODO (PRIORITY, mesh)
|
||||
The all-nodes "Archipelago group" chat (over Tor) doesn't seem to work. Facets:
|
||||
- (a) Group delivery unreliable / "doesn't work" over Tor.
|
||||
- (b) Messages may just be VERY SLOW (latency — likely Tor-only path; should use FIPS+Tor per the new transport method like B14, preferring FIPS).
|
||||
- (c) Add the SENDER CONTACT NAME to each message so you can differentiate who sent what (group messages lack attribution).
|
||||
- (d) Messages sometimes DUPLICATED (dedup by message id / sender_seq — cf mesh.ts:73 cross-transport identity (sender_pubkey, sender_seq); duplicate likely from receiving same msg over both transports or re-broadcast).
|
||||
Code: core/archipelago/src/mesh (typed_messages, listener), frontend Mesh.vue/stores/mesh.ts. Relates to B2 (identity), B14/B14b (transport). Test on .116/.198 (+ a Tor-only peer like .228).
|
||||
|
||||
### B8 — netbird app doesn't work — TODO (LOW / much later)
|
||||
|
||||
(RETRACTED: CryptPad placeholder-icon — user says cryptpad is fine.)
|
||||
|
||||
---
|
||||
|
||||
## 📋 vps2 Gitea issues (lfg2025/archy) — imported 2026-06-15
|
||||
- G#1 [Bug] Strange peer request behaviour — TODO (likely related to B1/federation)
|
||||
- G#2 [Bug] Fix flashing USB from kiosk — TODO
|
||||
- G#3 [Feature] VPN Configuration — DEFERRED (feature; no new features until production quality)
|
||||
- G#4 [Bug] Bitcoind is slow — TODO
|
||||
- G#5 [Feature] OpenWRT and TollGate integration — DEFERRED (feature)
|
||||
- G#6 [Feature] Move dashboard/monitoring link to home screen — DEFERRED (feature)
|
||||
- G#7 [Bug] Scrolling with Companion app — TODO
|
||||
|
||||
---
|
||||
|
||||
## Gitea issue mapping (vps2 lfg2025/archy)
|
||||
All backlog bugs now mirrored as Gitea issues: B1→#8, B2→#9, B3→#10, B4→#11, B5→#12, B6→#13, B7→#14, B8→#15, B9→#16, B10→#17, B11→#18, B12→#19, B13→#20, B14→#21, B15→#22, B16→#23, B17→#24, B18→#25, B19→#26. (Pre-existing G#1–7 remain; some overlap, e.g. G#1 strange-peer ≈ B1.) Close the Gitea issue when a bug is verified+shipped.
|
||||
|
||||
## INVESTIGATION FINDINGS 2026-06-15 (B1/B2/B3/B4/B14) — cutoff insurance
|
||||
|
||||
**B1 trusted-node divergence** — ROOT-CAUSED. `federation/sync.rs` `merge_transitive_peers()` (~:140) dedupes ONLY by DID; the SAME physical node appears under multiple DIDs (same `onion` + `fips_npub`) → duplicate entries ("Arch Dev" ×2, "Sapien" ×2). No background convergence → lists diverge (.103=16 nodes, .116/.198=15). Model: `federation/types.rs:24` FederatedNode (PK=did); storage `federation/storage.rs` nodes.json; add_node dedupes by DID only (:125). FIX: in merge_transitive_peers add a SECOND match arm — if no DID match, match by normalized `onion` (trim .onion); if found, treat as same node (merge fips_npub/name, don't add). Same dedup on add_node. Plus a one-time cleanup of existing dup DIDs (remove-node the stale one). TEST: after sync, all 3 nodes have identical node set, no two entries share an onion.
|
||||
|
||||
**B2 duplicate chat contact** — ROOT-CAUSED (same root as B1). Two federation DIDs (same onion/fips_npub, e.g. "Sapien" dids z6MkoSbN… + z6MkeYMU…) get seeded as TWO mesh contacts: `mesh/mod.rs` `seed_federation_peers_into_mesh()` (~:94) upserts per-pubkey contact_id; frontend `Mesh.vue` `mergeKeyForPeer()` (~:492) keys by DID so two DIDs = two rows. FIX: (backend) in seed, skip a node whose onion was already seeded (HashSet of onions); (frontend) Mesh.vue merge by onion when DIDs differ but onion matches. Fixing B1's onion-dedup largely resolves this too. TEST: one "Sapien" row; `mesh.peers` has one contact for the shared onion.
|
||||
|
||||
**B3 peer media won't play** — ROOT-CAUSED. `PeerFiles.vue` `playMedia()`/`loadPreview()` (~:358,:508) fetch the WHOLE file via RPC `content.preview-peer`/`content.download-peer` (`api/rpc/content.rs` :393,:213) which base64-encodes the entire file; frontend makes a Blob URL → browser can't Range-seek → video/large-audio won't play (+ 30/120s timeouts truncate big files). The peer's HTTP `/content/<id>` handler (`api/handler/content.rs` :49) ALREADY supports Range/206 + Accept-Ranges. FIX (bigger): add a local streaming proxy endpoint `/api/peer-content/{onion}/{id}` in `api/handler/mod.rs` that forwards the browser's Range header to the peer's `/content/<id>` (via fips::dial PeerRequest) and streams back 206 + Content-Range + Content-Type; frontend sets `<video>/<audio>` src to that URL (not a blob). TEST: curl Range on the new endpoint → 206 + Content-Range; video seeks/plays.
|
||||
|
||||
**B4 cloud my-folders <!doctype/502** — ROOT-CAUSED. `filebrowser-client.ts` `listDirectory()` (:99) does `res.json()` (:106) after only an `res.ok` check; when FileBrowser is ABSENT nginx serves SPA index.html (200, '<!doctype') → JSON crash; when DOWN → 502. FIX (frontend, low-risk): guard res content-type !== application/json → throw typed "FileBrowser unavailable" handled by Cloud.vue/CloudFolder.vue empty-state; same guard in login() (:71) + getUsage() (:215). OPTIONAL nginx: add `error_page 502 503 = @filebrowser_unavailable` returning JSON in the /app/filebrowser/ block (image-recipe/configs/nginx-archipelago.conf ~:411). TEST: stop filebrowser on .116/.198 → Cloud shows friendly state, no doctype crash.
|
||||
|
||||
**B14 cloud browse Tor-not-FIPS** — ROOT-CAUSED (nuance). FIPS-first logic WORKS (`fips/dial.rs` send_get :331 tries FIPS, falls back to Tor on 404/5xx; v1.7.94 fix). BUT the 4 content handlers in `api/rpc/content.rs` (browse :297, download :237, download_paid :356, preview :421) capture `_transport` and NEVER call `record_peer_transport()` → UI badge shows Tor/null even when FIPS used. FIX: add `record_peer_transport(data_dir, None, Some(onion), &transport.to_string())` after each successful send_get (storage.rs:84 has the fn). ⚠️ VERIFY on nodes whether FIPS is ACTUALLY used or genuinely falling back to Tor (if genuinely Tor, deeper FIPS-reachability issue beyond recording). TEST: after browse, last_transport = fips (when peer FIPS-reachable).
|
||||
|
||||
## INVESTIGATION FINDINGS 2026-06-15 (B6/B7/B12/B13/B15/B16) — cutoff insurance
|
||||
|
||||
**B13 Fedimint CSS** — app HTML (docker/fedimint-ui/index.html) uses absolute /assets/* paths; under /app/fedimint/ the browser requests /assets/* which hit the main SPA, not :8175 → unstyled. FIX: nginx sub_filter rewrite (same proven pattern as indeedhub/botfights blocks) in image-recipe/configs/nginx-archipelago.conf (/app/fedimint/ ~:641) + snippets/archipelago-https-app-proxies.conf (~:164) + bootstrap patch for existing nodes. Rewrites href/src/url '/' → '/app/fedimint/'. TEST: curl .../app/fedimint/assets/...css → 200 real CSS.
|
||||
|
||||
**B6 ElectrumX archival gate** — electrs needs a NON-pruned full node; install card doesn't warn at a glance. /bitcoin-status returns blockchain_info.pruned. Yellow badge pattern exists (MarketplaceAppCard.vue). FIX (frontend, simple): show a yellow "Requires a full archive Bitcoin node (not pruned)" note on the electrumx card (MarketplaceAppCard.vue ~:53). catalog.json electrumx already has requires.
|
||||
|
||||
**B7 ElectrumX stuck loader** — sync overlay gated by electrsSync (useElectrsSync.ts syncing = status!=='synced'); if status never flips to 'synced' (stale/crash) the overlay blocks the UI forever. AppSessionFrame.vue:44 iframe gate `!electrsSync`. FIX (frontend): fail-open — allow iframe when electrsSync?.stale (and add a timeout in useElectrsSync.ts so a slow/stale status stops blocking after ~5min).
|
||||
|
||||
**B15 bitcoin sync UI lag** — Home.vue:485 polls every 30s. FIX: faster bitcoin refresh (~5-10s) (separate interval for bitcoin vs system stats).
|
||||
|
||||
**B16 bitcoin status vanishes** — homeStatus.ts refreshBitcoin clears/leaves bitcoinAvailable null on a failed/transitional poll → HomeSystemCard.vue:60 v-if hides the card. FIX: retain last-known bitcoinAvailable on transient failure + show an "Updating…" badge instead of disappearing.
|
||||
|
||||
**B12 mempool not connecting** — stacks.rs:1278 + apps/mempool-api/manifest.yml:50 hardcode CORE_RPC_HOST=bitcoin-knots; on nodes running bitcoin-core (not knots) mempool-api gets getaddrinfo ENOTFOUND bitcoin-knots. Also ELECTRUM_HOST=electrumx absent on pruned nodes (docs/CONTAINER_LIFECYCLE_HANDOFF.md:654). FIX: detect which bitcoin container runs (knots vs core) + set CORE_RPC_HOST dynamically; qualify the mempool stack so it doesn't half-start without electrumx. Backend (stacks.rs) — medium risk, test on .116.
|
||||
|
||||
- 2026-06-15 (cont. 2): **B15 ✅** (poll 30s→10s) + **B7 ✅** (ElectrumX loader fail-open on stale) — committed `c0d41cf8`, type-check green. **B6 PARTIAL** (pruned gate already works; no-node-present half deferred). Fanned out investigations for B6/B7/B12/B13/B15/B16 — all root-caused with fix plans in FINDINGS above.
|
||||
- **DEFERRED with ready plans (need a backend build + careful patch, or UI test, or live repro):** B13 (fedimint CSS — nginx sub_filter asset rewrite; bootstrap exact-match patch is fragile, do carefully), B12 (mempool host — dynamic bitcoin-knots/core detect in stacks.rs), B16 (bitcoin status retain — UI-test to avoid stale-as-live), B6 no-node-present half, B14b (FIPS net depth), B22/B23 (need live repro).
|
||||
- **NEXT options:** (a) continue backend batch B13+B12 (one build); (b) do UI confirms on .116/.198 + cut v1.7.97-alpha with the ~10 committed fixes (LND incident + cloud/federation/mesh).
|
||||
- **Committed fixes awaiting .97:** B5, B1, B2, B4, B14, B21, B3, B15, B7 (+ B6 pruned-gate already live). All on vps2 main; NOT on fleet yet.
|
||||
|
||||
## Progress log
|
||||
- 2026-06-15: tracker created. v1.7.96-alpha shipped. All 19 bugs filed as Gitea issues #8–#26. vps2 feature issues (G#3/5/6) deferred (no new features).
|
||||
- 2026-06-15: **B5 (LND CORS) ✅ DONE** — root-caused, both fixes implemented, verified on .116/.198/.103 (harness 4/4 each), committed `1db720af`, pushed to vps2 main. Will bundle into .97 (Gitea #12 to close on .97 ship).
|
||||
- Validation nodes: .116 + .198 (pw ThisIsWeb54321@). Runtime is podman (docker not in non-interactive PATH). Sideload binary → /usr/local/bin/archipelago + restart (containers survive on these nodes).
|
||||
- 2026-06-15 (cont.): **B1,B2,B4 ✅** dedup+guard — committed `ed493106`, unit-tested 2/2, live .198 healthy. **B14 ✅** transport recording — committed `1c6dc153` (after build-repair: used private `crate::federation::storage::` path → E0603; fixed to re-exported `crate::federation::`). **B21 ✅** Tor/FIPS pill — committed `0801dd66`. All pushed to vps2 main; builds verified EXIT 0.
|
||||
- **Discovered B14b** (FIPS reaches only ~4/15 peers; rest genuinely Tor) and **B21** (pill) during the block.
|
||||
- ⚠️ LESSON: a backgrounded build "completed" notification does NOT mean success — grep the EXIT code before committing (a broken commit reached main once; repaired by 1c6dc153; no release cut from it → fleet unaffected).
|
||||
- **NEXT: B3 (peer media streaming — big), then B14b (FIPS reachability), then app-specific (B6,B7,B9–B13,B15–B19).** None deployed to fleet yet — all on vps2 main awaiting the .97 release after full .116/.198 + UI verification.
|
||||
|
||||
## New backlog issues filed 2026-06-16 (this session)
|
||||
- #32 Tor chat: message stuck on spinner though peers received it (task #8)
|
||||
- #33 Message toast: click-to-open chat + close icon (task #9)
|
||||
- #34 Local UI images never rebuild on source change — orchestrator gap (task #7); blocks OTA of bitcoin-ui relay + fedimint CSS to existing fleet
|
||||
- #35 Paid 10% video previews unplayable — truncated MP4 (task #6)
|
||||
NOTE: bitcoin RPC relay UI + fedimint guardian CSS now LIVE on .116 (image rebuilds); .198 deploy in progress. Bitcoin app launches host-net UI at <node>:8334 (not /app/bitcoin-ui/ proxy).
|
||||
Executable
+45
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
# lnd-cors-test.sh — assert the LND "connect your wallet" endpoints return
|
||||
# correct CORS headers for the cross-origin call from the LND UI app (:18083).
|
||||
#
|
||||
# Bug B5: /lnd-connect-info duplicated ACAO on some nodes; /proxy/lnd/v1/* 401
|
||||
# carries no ACAO fleet-wide. Browser blocks both.
|
||||
#
|
||||
# Usage: ./lnd-cors-test.sh <node-host> (e.g. 192.168.1.116 or 100.102.169.103)
|
||||
# Exit 0 = all assertions pass.
|
||||
|
||||
set -uo pipefail
|
||||
HOST="${1:?usage: lnd-cors-test.sh <node-host>}"
|
||||
ORIGIN="http://${HOST}:18083"
|
||||
BASE="http://${HOST}"
|
||||
PASS=0; FAIL=0
|
||||
say() { printf '%s\n' "$*"; }
|
||||
ok() { PASS=$((PASS+1)); say " PASS: $1"; }
|
||||
bad() { FAIL=$((FAIL+1)); say " FAIL: $1"; }
|
||||
|
||||
# Count ACAO header lines (case-insensitive) in a header dump.
|
||||
acao_count() { grep -ci '^access-control-allow-origin:' <<<"$1"; }
|
||||
acao_value() { grep -i '^access-control-allow-origin:' <<<"$1" | head -1 | sed 's/^[^:]*:[[:space:]]*//' | tr -d '\r'; }
|
||||
|
||||
say "== B5 LND CORS — node ${HOST} (origin ${ORIGIN}) =="
|
||||
|
||||
# 1) /lnd-connect-info — exactly ONE ACAO, value == origin
|
||||
H=$(curl -s -m 8 -D - -o /dev/null -H "Origin: ${ORIGIN}" "${BASE}/lnd-connect-info" 2>/dev/null)
|
||||
N=$(acao_count "$H"); V=$(acao_value "$H")
|
||||
[ "$N" = "1" ] && ok "/lnd-connect-info has exactly 1 ACAO header" || bad "/lnd-connect-info ACAO count=$N (want 1)"
|
||||
[ "$V" = "$ORIGIN" ] && ok "/lnd-connect-info ACAO value == origin" || bad "/lnd-connect-info ACAO='$V' (want '$ORIGIN')"
|
||||
|
||||
# 2) /proxy/lnd/v1/getinfo — ACAO present even on 401 (unauth)
|
||||
H=$(curl -s -m 8 -D - -o /dev/null -H "Origin: ${ORIGIN}" "${BASE}/proxy/lnd/v1/getinfo" 2>/dev/null)
|
||||
N=$(acao_count "$H")
|
||||
[ "$N" -ge 1 ] && ok "/proxy/lnd/v1/getinfo has ACAO (even unauth)" || bad "/proxy/lnd/v1/getinfo missing ACAO (count=$N)"
|
||||
[ "$N" -le 1 ] || bad "/proxy/lnd/v1/getinfo duplicate ACAO (count=$N)"
|
||||
|
||||
# 3) /proxy/lnd/v1/channels — same
|
||||
H=$(curl -s -m 8 -D - -o /dev/null -H "Origin: ${ORIGIN}" "${BASE}/proxy/lnd/v1/channels" 2>/dev/null)
|
||||
N=$(acao_count "$H")
|
||||
[ "$N" = "1" ] && ok "/proxy/lnd/v1/channels has exactly 1 ACAO" || bad "/proxy/lnd/v1/channels ACAO count=$N (want 1)"
|
||||
|
||||
say ""
|
||||
say "== ${HOST}: ${PASS} passed, ${FAIL} failed =="
|
||||
[ "$FAIL" -eq 0 ]
|
||||
Executable
+160
@@ -0,0 +1,160 @@
|
||||
#!/bin/bash
|
||||
# Release gate harness — seed of the full-system test harness.
|
||||
#
|
||||
# Ties together the checks that already exist in this repo (catalog drift,
|
||||
# release manifest, lifecycle bats, vitest, cargo tests) plus live-node
|
||||
# smoke probes, so "is this release OK?" is one command instead of folklore.
|
||||
#
|
||||
# Usage:
|
||||
# tests/release/run.sh # static + frontend + backend stages
|
||||
# tests/release/run.sh --quick # static + frontend unit only
|
||||
# tests/release/run.sh --with-build # also production-build the frontend
|
||||
# # and verify the dist version changed
|
||||
# tests/release/run.sh --manifest # also validate releases/manifest.json
|
||||
# # (run AFTER create-release staged it)
|
||||
# tests/release/run.sh --live [URL] # also smoke-probe a running node
|
||||
# # (default http://127.0.0.1)
|
||||
#
|
||||
# Flags compose. Exits non-zero on the first failing stage.
|
||||
#
|
||||
# CAUTION (.116 and other dev nodes): full `cargo test -p archipelago` has
|
||||
# hung tool PTYs here before — every cargo invocation below is wrapped in
|
||||
# `timeout` and scoped to focused module filters.
|
||||
|
||||
set -u
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$REPO"
|
||||
|
||||
QUICK=0 WITH_BUILD=0 MANIFEST=0 LIVE=0 LIVE_URL="http://127.0.0.1"
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--quick) QUICK=1 ;;
|
||||
--with-build) WITH_BUILD=1 ;;
|
||||
--manifest) MANIFEST=1 ;;
|
||||
--live) LIVE=1; [[ "${2:-}" == http* ]] && { LIVE_URL="$2"; shift; } ;;
|
||||
*) echo "unknown flag: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
PASS=() FAIL=()
|
||||
stage() { # stage <name> <cmd...>
|
||||
local name="$1"; shift
|
||||
echo
|
||||
echo "=== [$name] $*"
|
||||
if "$@"; then
|
||||
echo "=== [$name] PASS"
|
||||
PASS+=("$name")
|
||||
else
|
||||
echo "=== [$name] FAIL (exit $?)"
|
||||
FAIL+=("$name")
|
||||
summary 1
|
||||
fi
|
||||
}
|
||||
summary() {
|
||||
echo
|
||||
echo "──────── release gate summary ────────"
|
||||
printf 'PASS: %s\n' "${PASS[@]:-none}"
|
||||
[[ ${#FAIL[@]} -gt 0 ]] && printf 'FAIL: %s\n' "${FAIL[@]}"
|
||||
exit "${1:-0}"
|
||||
}
|
||||
|
||||
# ── Stage 1: static ──────────────────────────────────────────────────
|
||||
stage "git-diff-check" git diff --check
|
||||
stage "cargo-fmt" timeout 240 cargo fmt --manifest-path core/Cargo.toml --all --check
|
||||
stage "catalog-drift" python3 scripts/check-app-catalog-drift.py
|
||||
# Every release must surface its CHANGELOG entry in the Settings "What's New"
|
||||
# modal. The modal hardcodes a block per version and has drifted behind before
|
||||
# (sat at v1.7.84 while the fleet shipped to v1.7.92). Fail if any CHANGELOG
|
||||
# version is missing a block; `python3 scripts/sync-whats-new.py` inserts them.
|
||||
stage "whats-new-sync" python3 scripts/sync-whats-new.py --check
|
||||
if [[ $MANIFEST -eq 1 ]]; then
|
||||
stage "release-manifest" scripts/check-release-manifest.sh
|
||||
fi
|
||||
|
||||
# ── Stage 2: frontend ────────────────────────────────────────────────
|
||||
stage "ui-type-check" bash -c 'cd neode-ui && npm run --silent type-check'
|
||||
stage "ui-unit-tests" bash -c 'cd neode-ui && npx vitest run --silent 2>&1 | tail -4; exit ${PIPESTATUS[0]}'
|
||||
|
||||
if [[ $WITH_BUILD -eq 1 ]]; then
|
||||
# npm run build can fail silently (vue-tsc EACCES burned us before) —
|
||||
# require the packaged output to actually contain the current version.
|
||||
VERSION=$(grep -m1 '^version' core/archipelago/Cargo.toml | cut -d'"' -f2)
|
||||
stage "ui-build" bash -c 'cd neode-ui && npm run build'
|
||||
stage "ui-dist-version" bash -c "grep -rqo '${VERSION}' web/dist/neode-ui/assets/*.js"
|
||||
fi
|
||||
|
||||
[[ $QUICK -eq 1 ]] && summary 0
|
||||
|
||||
# ── Stage 3: backend ─────────────────────────────────────────────────
|
||||
stage "cargo-check" timeout 580 cargo check --manifest-path core/Cargo.toml -p archipelago
|
||||
# Focused suites for the subsystems this release train touched:
|
||||
# update:: — OTA download/apply/rollback/probe (v1.7.89 hardening)
|
||||
# lnd — receive address + wallet readiness (v1.7.85–.89), incl. the
|
||||
# structured receive-error reason-code classifier
|
||||
# container::image_versions — image pinning / false-update detection
|
||||
# scanner — RAII in-flight guard (v1.7.84)
|
||||
# drift — published-port drift detection (the .116 self-heal)
|
||||
# missing_secret — secret-resolution names the missing file (the .198 fix)
|
||||
# 1500s: the non-incremental test-profile compile alone takes ~9 min on the
|
||||
# .116 ThinkPad; 580s expires mid-compile (exit 124) before a single test runs.
|
||||
stage "cargo-test-weekly" timeout 1500 env CARGO_INCREMENTAL=0 \
|
||||
cargo test --manifest-path core/Cargo.toml -p archipelago -- \
|
||||
update:: lnd container::image_versions scanner drift missing_secret
|
||||
|
||||
# ── Stage 4: live node smoke ─────────────────────────────────────────
|
||||
if [[ $LIVE -eq 1 ]]; then
|
||||
stage "live-frontend" bash -c "curl -skf -o /dev/null '$LIVE_URL/' || curl -skf -o /dev/null '${LIVE_URL/http:/https:}/'"
|
||||
stage "live-aiui" curl -sf -o /dev/null "$LIVE_URL/aiui/"
|
||||
stage "live-rpc" bash -c "curl -s -X POST '$LIVE_URL/rpc/v1' -H 'Content-Type: application/json' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"update.status\",\"params\":{}}' | grep -qE '\"(result|error)\"'"
|
||||
|
||||
# Bitcoin-receive regression guard. The backend asks LND REST for a new
|
||||
# on-chain address with ?type=<AddressType>. The REST gateway parses that
|
||||
# as the proto enum (WITNESS_PUBKEY_HASH / 0), NOT the lncli aliases —
|
||||
# sending "p2wkh" returns 400 "parsing field type ... is not a valid
|
||||
# value" and bitcoin-receive silently breaks for the whole fleet (the bug
|
||||
# that slipped through v1.7.88/89 because nothing exercised LND live).
|
||||
# This hits LND REST directly and FAILS only on that exact parse-error
|
||||
# signature; a "wallet locked" / "still syncing" reply means the type was
|
||||
# accepted, which is all we're validating here.
|
||||
stage "live-lnd-address-type" bash -c '
|
||||
mac=$(sudo cat /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon 2>/dev/null | od -An -tx1 | tr -d " \n")
|
||||
for port in 18080 8080; do
|
||||
resp=$(curl -sk --max-time 8 "https://127.0.0.1:$port/v1/newaddress?type=WITNESS_PUBKEY_HASH" -H "Grpc-Metadata-macaroon: $mac" 2>/dev/null)
|
||||
[ -z "$resp" ] && continue
|
||||
echo "LND($port): $resp"
|
||||
echo "$resp" | grep -q "is not a valid value" && { echo "FAIL: LND rejected the address type the backend sends"; exit 1; }
|
||||
echo "OK: LND accepted the address type"; exit 0
|
||||
done
|
||||
echo "SKIP: LND REST not reachable on 18080/8080 — cannot validate address type live"; exit 0
|
||||
'
|
||||
|
||||
# Wallet-unlock guard. After a restart/OTA, LND comes up LOCKED and the backend
|
||||
# must auto-unlock it; if the unlock password is wrong (e.g. a fleet-wide
|
||||
# constant vs a per-wallet password) the wallet stays LOCKED forever and ALL
|
||||
# Bitcoin-receive / Lightning ops fail — fleet-wide, silently. Nothing else in
|
||||
# this harness catches that: live-lnd-address-type explicitly treats "wallet
|
||||
# locked" as a PASS, and os-audit treats lnd-unreachable as a WARN. This stage
|
||||
# polls LND's unauthenticated /v1/state and FAILS if it is still LOCKED after a
|
||||
# grace window. RPC_ACTIVE = unlocked (pass); NON_EXISTING/WAITING = no wallet
|
||||
# yet (not a regression); unreachable = skip.
|
||||
stage "live-lnd-unlocked" bash -c '
|
||||
deadline=$(( $(date +%s) + 60 ))
|
||||
while :; do
|
||||
seen=""
|
||||
for port in 18080 8080; do
|
||||
st=$(curl -sk --max-time 6 "https://127.0.0.1:$port/v1/state" 2>/dev/null)
|
||||
[ -z "$st" ] && continue
|
||||
seen=1
|
||||
echo "LND($port) state: $st"
|
||||
echo "$st" | grep -q "RPC_ACTIVE" && { echo "OK: LND wallet is unlocked"; exit 0; }
|
||||
echo "$st" | grep -qE "NON_EXISTING|WAITING_TO_START" && { echo "OK: LND wallet not initialized yet — not a lock regression"; exit 0; }
|
||||
done
|
||||
[ -z "$seen" ] && { echo "SKIP: LND /v1/state not reachable on 18080/8080"; exit 0; }
|
||||
[ "$(date +%s)" -ge "$deadline" ] && { echo "FAIL: LND wallet still LOCKED after 60s — auto-unlock failed; Bitcoin-receive/Lightning are broken"; exit 1; }
|
||||
sleep 5
|
||||
done
|
||||
'
|
||||
fi
|
||||
|
||||
summary 0
|
||||
Reference in New Issue
Block a user