lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 4 Actions Packages Projects Releases 72 Wiki Activity

feat(update): deepen post-OTA verification beyond a frontend 200

Browse Source 
verify_pending_update previously cleared the rollback marker on any
2xx/3xx from GET / — a release with a dead RPC API or broken podman
access passed and never rolled back. Verification now requires, in the
same attempt: the frontend via nginx, backend RPC liveness (an
unauthenticated POST /rpc/v1 — 401 proves the stack is up, 5xx/404/
refused fails it), and rootless podman reachability. A pre-loop check
also asserts the running binary's version matches what the marker says
was applied, catching a silent or half swap deterministically.

Per-app container assertions are deliberately excluded: the
pre-Quadlet service restart legitimately takes containers down and the
boot reconciler can need minutes for heavy apps — that would
false-rollback healthy updates. Revisit after the Phase-3 flip.

§B of the 1.8.0 hardening plan; update suite 38/38 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-04 13:50:00 -04:00
co-authored by Claude Fable 5
parent 4b4a1f88fb
commit bd7edb4376
2 changed files with 112 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/1.8.0-RELEASE-HARDENING-PLAN.md
+9 -5
View File
@@ -86,11 +86,15 @@ atomic swap, single-depth backup). The gaps are **authenticity** (§A) and
**verification depth** — plus the fact that the upgrade path has never run
end-to-end on real hardware.
- [ ] 🔴 **Deepen the post-OTA health check.** `update.rs:456` (`probe_frontend_once`)
passes on any 2xx/3xx from `GET /`, and `verify_pending_update` (494-593) only rolls
back on that. A release with a broken RPC API, dead containers, or failed LND unlock
passes and never rolls back. Add `/rpc/v1 update.status` + container-list/required-stack
health assertions before clearing the pending-verify marker.
- [x] 🔴 **Deepen the post-OTA health check.** DONE 2026-07-03: `verify_pending_update`
now requires, in the same attempt, (1) frontend 2xx/3xx via nginx, (2) backend RPC
liveness — unauthenticated POST `/rpc/v1`; 401/403 = alive, 5xx/404/refused = dead,
so a 502-behind-static-files release now rolls back, (3) rootless `podman ps`
reachability; plus a pre-loop binary-version==marker assertion that catches a silent
or half swap (new frontend + old binary) deterministically. Per-app container
assertions deliberately EXCLUDED — the pre-Quadlet service restart legitimately kills
containers and the reconciler can need minutes (false-rollback risk); revisit after
the Phase-3 flip. LND-unlock-level checks remain out of scope for the 90s window.
- [ ] 🟠 **Run one real upgrade-from-vN-1 soak on hardware before tagging.**
No test installs the previous version, points it at a staged 1.8.0 manifest, applies,
and asserts health + rollback. This is the top release risk for an OTA release. A