lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 1 Actions Packages Projects Releases 64 Wiki Activity

chore(release): stage v1.7.55-alpha

Browse Source
This commit is contained in:
Dorian
2026-05-13 15:09:22 -04:00
parent 3202b79e41
commit 835c525218
65 changed files with 2322 additions and 566 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CONTAINER_LIFECYCLE_HANDOFF.md
+135 -5
View File
@@ -1,17 +1,56 @@
# Container Lifecycle Handoff
Last updated: 2026-05-06
Last updated: 2026-05-11
## 2026-05-13 `.198` Stopping-State Repair Checkpoint
- User directive confirmed: testing target is `.198` until all containers work and the container layer is bulletproof/perfected.
- `.198` service state after this pass:
- `archipelago.service`: active.
- `archipelago-doctor.timer`: inactive.
- `archipelago-reconcile.timer`: inactive.
- `/usr/local/bin/archipelago` sha256: `5d3777d928ae6ee7627e9401faf932442806020ab7ad7a439eb7384d8eb7b8e6`.
- Live blocker found and repaired:
- `nostr-rs-relay` was stuck in raw Podman state `Stopping (healthy)`; focused lifecycle audit failed with `bad state: nostr-rs-relay is stopping`.
- Removed only the wedged container record with `podman rm -f nostr-rs-relay`; bind-mounted relay data under `/var/lib/archipelago/nostr-relay` was preserved.
- Archipelago/runtime recreated the relay and it returned `Up ... (healthy)`.
- Durable local fix added and deployed:
- `core/archipelago/src/container/prod_orchestrator.rs` now treats `ContainerState::Stopping` as a wedged container record during reconcile and force-recreates it from the manifest instead of trying a normal start.
- Added unit coverage intent: `reconcile_force_recreates_stopping_container`.
- `cargo check -p archipelago --bin archipelago` passed locally.
- `cargo build -p archipelago --bin archipelago --release` passed locally and was deployed to `.198`.
- Rust test binary build for the targeted unit test timed out during compilation in this environment before emitting compiler errors; use `cargo check` plus live `.198` audit as the validated gate for this pass.
- Post-deploy validation on `.198`:
- Focused audit passed: `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_APPS=mempool,searxng,nginx-proxy-manager,nostr-rs-relay,grafana,btcpay-server ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=180 tests/lifecycle/remote-lifecycle.sh`.
- Broad audit passed: `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=240 tests/lifecycle/remote-lifecycle.sh`.
- Raw Podman sweep found no `unhealthy`, `Stopping`, `Removing`, `Exited`, or `Created` containers after post-restart startup settled.
- Direct HTTP probes returned healthy responses (`200` or expected `302`) for dashboard, bitcoin-ui, lnd-ui, btcpay, indeedhub, botfights, gitea, filebrowser, vaultwarden, searxng, fedimint, jellyfin, immich, homeassistant, grafana, tailscale, uptime-kuma, nextcloud, nginx-proxy-manager, and nostr-rs-relay.
- Current `.198` broad audit state:
- Running: `bitcoin-knots`, `lnd`, `btcpay-server`, `indeedhub`, `botfights`, `gitea`, `filebrowser`, `vaultwarden`, `searxng`, `fedimint`, `jellyfin`, `immich`, `homeassistant`, `grafana`, `tailscale`, `uptime-kuma`, `nextcloud`.
- Absent/expected in this audit: `bitcoin-core`, `mempool`, `electrumx`, `photoprism`.
- Important observation:
- Immediately after backend restart, `bitcoin-knots` briefly appeared `Exited (137)` during startup/recovery, then self-recovered and was running by inspection. Final broad audit and raw sweep were clean.
- Next recommended gate:
- Run destructive/full lifecycle on `.198` only when ready to intentionally cycle app containers; non-destructive broad audit and raw health are green after the stopping-state fix.
## 2026-05-13 Resume Correction
- User directive: "we're testing on .198 right, until all containers are working and we achieve our goal of bulletproof containers".
- Active target remains `.198`; do not drift back to older `.116`/`.228` release threads except for cross-node context.
- Continue lifecycle hardening until every intended `.198` container/app is working, recoverable, and aligned with the bulletproof-container goal.
## Resume Prompt
```text
Resume Archipelago lifecycle testing from /home/archipelago/Projects/archy. Read docs/CONTAINER_LIFECYCLE_HANDOFF.md first. Preserve data unless explicitly told otherwise. Do not revert unrelated dirty worktree changes. Keep untracked docs/CONTAINER_LIFECYCLE_HANDOFF.md and docs/CHAT_TRANSCRIPT_2026-05-02.md.
Resume Archipelago lifecycle hardening from /home/archipelago/Projects/archy. Read docs/CONTAINER_LIFECYCLE_HANDOFF.md first. Active mission is node `192.168.1.198`, not the older `.116/.228/.67` release thread. SSH key is `/home/archipelago/.ssh/id_ed25519`; lifecycle password is `password123`. Preserve data unless explicitly told otherwise. Keep `archipelago-doctor.timer` and `archipelago-reconcile.timer` inactive during deterministic testing. Do not revert unrelated dirty worktree changes because another agent/user may be working too.
Current focus: multi-node non-destructive hardening across .228, .116, and .67. .228 was live-repaired and verified for dashboard, Bitcoin UI, LND UI, Immich, and authenticated Bitcoin RPC. .116 was live-repaired for stale Bitcoin Knots command drift, Grafana rootless ownership, nginx /bitcoin-status proxying, and stale LND UI companion image/unit drift; focused non-destructive lifecycle audit now passes for bitcoin-knots,lnd,btcpay-server,mempool,grafana. v1.7.54-alpha release artifacts were regenerated from current source and verified to carry runtime payload fixes. .67 remains unreachable from this workspace despite confirmed credentials archipelago/archipelago.
Mission: make every Archipelago app/container on `.198` lifecycle-safe and power-loss/reboot resilient. Containers should not randomly go down; app state must recover through daemon restarts, reboots, stale Podman/Quadlet state, missing host listeners, stuck installs, stopped/exited state drift, and stale stack/container records. Release is blocked until strict lifecycle plus app-specific reachability/launch probes agree with raw Podman health and actual app behavior.
Durable fixes implemented locally: Bitcoin container entrypoint/cmd drift recreation, Grafana data_uid/rootless ownership repair, Immich Postgres 2g memory, IndeedHub boot/start Nostr provider reapply, Apps loading/launch readiness UI fixes, nginx /bitcoin-status backend proxy repair, and LND UI 18083 companion/spec drift repair.
Latest live `.198` status from 2026-05-11: `archipelago.service` active; `archipelago-doctor.timer` inactive; `archipelago-reconcile.timer` inactive; deployed `/usr/local/bin/archipelago` sha256 `ed4df8e4c3c0a12a481ea41f8246da4b5f9e9ad931d0f3f58084b0057c330af0`. Broad audit passed with `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=180 tests/lifecycle/remote-lifecycle.sh`, but this is not enough for release because raw Podman still reports health/state mismatches.
Latest deployed backend checksum on `.116` after live deploy: `c6c7830f14dc80b0e22d803997ad3df31c9ab3d4b08829b3bddc1b03ce77bd0a`. Latest live verification: nginx `/bitcoin-status` returns JSON, LND UI `http://127.0.0.1:18083/` returns HTTP 200 from `localhost/lnd-ui:local`, runtime payload scripts and promoted `/opt/archipelago/scripts` both carry `18083`, and focused non-destructive lifecycle audit passed for `bitcoin-knots,lnd,btcpay-server,mempool,grafana`. Next action: publish/tag v1.7.54-alpha if approved, then continue to `.228` deploy or `.67` reachability.
Current suspected release blocker: reconcile the broad-audit pass with raw Podman health. On `.198`, `mempool-api` is `Up ... (unhealthy)`, `searxng` is `Up ... (unhealthy)`, `botfights` is `Up ... (unhealthy)`, and `nostr-rs-relay` is `Stopping (unhealthy)`. RPC/package state reports the installed audit set as running, so next work is to diagnose these health/state mismatches, decide whether each is a false-negative healthcheck or real app failure, fix the manifest/runtime/reconcile behavior, then rerun focused full lifecycle and browser/direct launch probes for affected apps.
Known `.198` package state from latest broad audit: running `lnd`, `mempool`, `indeedhub`, `botfights`, `gitea`, `filebrowser`, `vaultwarden`, `searxng`, `fedimint`, `jellyfin`, `immich`, `homeassistant`, `tailscale`, `uptime-kuma`, `nextcloud`; absent `bitcoin-knots`, `bitcoin-core`, `btcpay-server`, `electrumx`, `grafana`, `photoprism`. Some absences are expected/blockers from earlier qualification, but `btcpay-server` and `grafana` had previously passed focused checks, so verify whether their absence is intentional before release.
Regenerated release artifacts:
- `releases/v1.7.54-alpha/archipelago`: `77e3a236a6196a5ab9ec2411b150490e78ffc95ea6ab8eb34ab29b3df53cd632`
@@ -20,6 +59,97 @@ Regenerated release artifacts:
- Unbundled ISO: `image-recipe/results/archipelago-installer-1.7.54-alpha-unbundled-x86_64.iso`, sha256 `9828b244e6ffdd5f1b1d5184c1b22bef7474b32078b1ceb4ec3584d9bdb6775b`, size `2.3G`.
```
## 2026-05-11 `.198` Active Mission Checkpoint
## 2026-05-11 Resume Session Update
- Latest user directive: "please resume our work".
- Reconfirmed active mission is `.198` lifecycle hardening, not the older `.116/.228/.67` thread.
- Live `.198` state at resume:
- `archipelago.service`: active.
- `archipelago-doctor.timer`: inactive.
- `archipelago-reconcile.timer`: inactive.
- `/usr/local/bin/archipelago` sha256: `494cd64f77cbecb95c08552237cb8fd3c11c2b2b76d5d39854e6cf92b5900b68`.
- Raw Podman still showed release blockers:
- `mempool-api`: `Up ... (unhealthy)`.
- `nginx-proxy-manager`: `Up ... (unhealthy)`.
- `nostr-rs-relay`: `Stopping (healthy)`.
- `searxng` was healthy by the time of recheck and served `http://127.0.0.1:8888/` with HTTP 200.
- Diagnosed `mempool-api` as a real app failure, not a false-negative healthcheck: logs repeatedly show `getaddrinfo ENOTFOUND electrumx`, and `.198` has no `electrumx` container present. `mempool-api` is configured with `ELECTRUM_HOST=electrumx`, so the broad audit was masking a broken stack member.
- Found and fixed a local backend masking bug: `ProdContainerOrchestrator::health` returned `healthy` for every running container and ignored Podman's actual health status. It now returns Podman's health value for running containers, maps `Stopping` to unhealthy, and `ContainerState` now parses Podman's `stopping` state explicitly.
- Local verification:
- `cargo fmt` passed.
- `cargo test -p archipelago-container parse_podman_ps_json_handles_cli_output` passed.
- `cargo check -p archipelago --bin archipelago` passed.
- `cargo test -p archipelago health_maps_states_to_strings` did not finish within 3 minutes during crate compilation; no compiler error was emitted before timeout.
- Focused live audit command attempted: `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_APPS=mempool,searxng,nginx-proxy-manager ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=60 tests/lifecycle/remote-lifecycle.sh`. It timed out because the deployed `.198` backend still has the old health behavior and Podman operations on the node are intermittently hanging.
- Next continuation point:
- Decide whether to deploy a freshly built backend to `.198`. Do not deploy the current dirty worktree blindly unless the existing unrelated changes are intended for this release, because the workspace contains many modified files from prior work.
- After deploy, rerun focused audit for `mempool,searxng,nginx-proxy-manager` and verify `container-health` reports `mempool` or stack health as unhealthy while `mempool-api` cannot resolve `electrumx`.
- Fix the mempool stack qualification: on a pruned/under-disk node, `mempool` must not install/start into a half-running state that leaves `mempool-api` unhealthy because `electrumx` is absent.
## 2026-05-12 Lifecycle Hardening Completion Checkpoint
- User directive: continue until the work is done.
- Deployed fixed backend to `.198`; final `/usr/local/bin/archipelago` sha256: `616e50ba8a83654e4a7656f931e5c9d1340a92cfa0ba22906edc0d374560df02`.
- `archipelago.service` active; `archipelago-doctor.timer` inactive; `archipelago-reconcile.timer` inactive.
- Local durable fixes made:
- `ProdContainerOrchestrator::health` now respects Podman's health status instead of mapping all running containers to healthy.
- Podman `stopping` state is parsed explicitly and maps to unhealthy/stopping instead of unknown/running.
- `container-health` aggregates stack health for multi-container apps, so stack apps cannot hide unhealthy members like `mempool-api`.
- Health fallback now uses bounded exact-container Podman checks to avoid broad `podman ps` hangs poisoning unrelated app health.
- `mempool` install now runs dependency and archival-Bitcoin checks before dispatching to the stack installer, preventing half-running mempool stacks on pruned/under-disk nodes.
- Nginx Proxy Manager healthcheck now probes `http://localhost:81/`; `/api/` returns 502 on the deployed image while the UI is healthy.
- Runtime start repair now covers Vaultwarden and Nextcloud missing host listeners.
- Nextcloud runtime repair fixes bind-mounted data ownership before start/restart.
- Stale transitional state timeout lowered from 20 minutes to 2 minutes so dead lifecycle tasks clear promptly.
- Live `.198` repairs performed with data preserved:
- Removed broken `mempool` stack via `package.uninstall preserve_data=true`; `mempool` is now absent and full lifecycle correctly reports archival-blocked install.
- Recreated Nginx Proxy Manager container after stale Podman `Removing` state; data under `/var/lib/archipelago/nginx-proxy-manager` preserved.
- Recreated Vaultwarden container after stale conmon/host-listener failure; `/var/lib/archipelago/vaultwarden` preserved.
- Recreated Home Assistant and Nextcloud container records after stale conmon/host-listener failures; data directories preserved.
- Repaired Nextcloud ownership (`/var/lib/archipelago/nextcloud`) so Apache/PHP can write `config.php` and `data/nextcloud.log`.
- Verification passed:
- `cargo fmt`.
- `cargo check -p archipelago --bin archipelago`.
- `cargo build -p archipelago --bin archipelago --release`.
- `cargo test -p archipelago-container parse_podman_ps_json_handles_cli_output` passed earlier in this session.
- `cargo test -p archipelago health_maps_states_to_strings` still fails during local test binary linking with rust-lld undefined hidden symbols; `cargo check` and release build pass.
- Focused audit passed: `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_APPS=mempool,searxng,nginx-proxy-manager ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=120 tests/lifecycle/remote-lifecycle.sh`.
- Broad audit passed: `ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=180 tests/lifecycle/remote-lifecycle.sh`.
- Final raw Podman sweep found no `unhealthy`, `Stopping`, or `Removing` containers.
- Final direct probes returned HTTP 200 for LND UI, IndeedHub, Botfights, Gitea, File Browser, Vaultwarden, SearXNG, Fedimint, Jellyfin, Immich, Home Assistant, Tailscale, Uptime Kuma, Nextcloud, Nginx Proxy Manager, and Nostr Relay.
- Final broad audit state:
- Running: `lnd`, `indeedhub`, `botfights`, `gitea`, `filebrowser`, `vaultwarden`, `searxng`, `fedimint`, `jellyfin`, `immich`, `homeassistant`, `tailscale`, `uptime-kuma`, `nextcloud`.
- Absent/expected for this node or archival-gated: `bitcoin-knots`, `bitcoin-core`, `btcpay-server`, `mempool`, `electrumx`, `grafana`, `photoprism`.
- Remaining release consideration: `.198` is green for the non-destructive broad audit and raw Podman health. Destructive/full lifecycle should still be run only when you are ready to intentionally cycle app containers.
- User corrected the active mission after disconnect: continue `.198` container lifecycle hardening, not the older `.116/.228/.67` thread.
- Mission: build "perfect containers" that do not go down unexpectedly and recover through daemon restarts, server reboots, power loss, stale Podman/Quadlet state, missing rootless host listeners, stuck installs, stopped/exited state drift, and stale stack/container records.
- Preserve app data unless explicitly told otherwise.
- Keep deterministic-test timers paused: `archipelago-doctor.timer` and `archipelago-reconcile.timer` should remain inactive.
- Latest verified `.198` service state:
- `archipelago.service`: active.
- `archipelago-doctor.timer`: inactive.
- `archipelago-reconcile.timer`: inactive.
- `/usr/local/bin/archipelago` sha256: `ed4df8e4c3c0a12a481ea41f8246da4b5f9e9ad931d0f3f58084b0057c330af0`.
- Latest broad audit command passed:
```bash
ARCHY_HOST=192.168.1.198 ARCHY_PASSWORD=password123 ARCHY_STABILITY_SECONDS=5 ARCHY_TIMEOUT=180 tests/lifecycle/remote-lifecycle.sh
```
- Latest broad audit states:
- Running: `lnd`, `mempool`, `indeedhub`, `botfights`, `gitea`, `filebrowser`, `vaultwarden`, `searxng`, `fedimint`, `jellyfin`, `immich`, `homeassistant`, `tailscale`, `uptime-kuma`, `nextcloud`.
- Absent: `bitcoin-knots`, `bitcoin-core`, `btcpay-server`, `electrumx`, `grafana`, `photoprism`.
- Do not treat the broad audit pass as release-ready yet. Raw Podman still showed these concerning health/state mismatches:
- `mempool-api`: `Up ... (unhealthy)`.
- `searxng`: `Up ... (unhealthy)`.
- `botfights`: `Up ... (unhealthy)`.
- `nostr-rs-relay`: `Stopping (unhealthy)`.
- Current suspected release blocker: Archipelago package state and broad audit say apps are running, but raw Podman health/state still reports unhealthy/stopping containers. Next agent should diagnose whether each mismatch is a false-negative healthcheck, stale Podman state, or a real app failure; then fix manifest/runtime/reconcile behavior and rerun focused full lifecycle plus browser/direct launch probes for affected apps.
- Also verify whether `btcpay-server` and `grafana` being absent is intentional, because both had previously passed focused lifecycle checks on `.198`.
## 2026-05-06 Resume Checkpoint
- Goal: make container lifecycle and health recovery durable for every install and existing Archipelago server, while preserving app data.
docs/SESSION-RESUME-2026-04-24.md
+3
View File
@@ -648,3 +648,6 @@ Go/no-go verdict:
> please do not miss AIUI in the release build or remove it from the nodes whatever you do
- Critical release constraint: AIUI must remain bundled in release artifacts and must never be removed from existing nodes during update/deploy.
> please check the resume files for our latest plan and resume the work.
- Current directive: read the resume/plan files, resume the latest active work, and continue from the recorded release/ISO lane while preserving the AIUI release constraint above.