fix(robustness): surface swallowed persistence-write failures + federation tombstone durability

§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.

Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-02 21:02:54 -04:00
co-authored by Claude Fable 5
parent 206d5fe8cf
commit 01cbec27ed
12 changed files with 102 additions and 40 deletions
+15 -11
View File
@@ -107,11 +107,13 @@ end-to-end on real hardware.
Note: the `.unwrap()`/`panic!` worry is a **non-issue** — nearly all are in test
modules; production request/boot paths are essentially panic-free. The real risks:
- [ ] 🟠 **Log swallowed persistence writes.** ~30-40 dangerous `let _ = save_*().await`
sites discard durability failures with zero diagnostics: `server.rs:270` (mesh config),
`bitcoin_relay.rs:865` (relay state), `update.rs:163/1223` (mirrors/update state),
`registry.rs:158`, `mesh/status.rs:286`, `scheduler.rs:179`, `install.rs:34`. Convert to
`if let Err(e) = … { warn!(…) }`; leave genuinely fire-and-forget ones commented.
- [x] 🟠 **Log swallowed persistence writes.** DONE 2026-07-02 (full-workspace re-inventory
found 19 production sites): 16 converted to `if let Err(e) = … { warn!(…) }` mesh
config (`server.rs`), relay tor endpoint (`bitcoin_relay.rs`), update mirrors/state +
staging flush/sync (`update.rs`), registry config, radio-contact blocklist, mesh outbox
sweep (`scheduler.rs`), block-header cache (`mesh/mod.rs`), 7× peer-transport badge
(`sync.rs` + `content.rs`). Federation tombstone/untombstone upgraded to hard errors
(see §I). Install-log line write left fire-and-forget with an explanatory comment.
- [ ] 🟠 **Remove blocking `std::process::Command` from async handlers.**
`install.rs:2222` `published_host_port` (sync podman on the install path),
`dependencies.rs:316` (`df`), `system/handlers.rs:578` (`sudo`), `transport/fips.rs:50`
@@ -271,10 +273,10 @@ media (latest artifact only one minor behind).
- [~] 🟠 **Multinode gate pass** — 5× destructive gate was launched on node `.5`; bring the
rest of the fleet to precondition, then run the existing (undocumented-but-present)
`tests/multinode/{smoke,meshtastic}.sh` cross-node suites.
- [ ] 🟠 **Federation `remove-node` tombstone regression.**
`federation/storage.rs:187` does `let _ = tombstone_did(...)` — swallows the write error,
so a removed peer reappears after the next sync. (This is a specific, confirmed instance
of the §C swallowed-writes class.) Needs a careful fix + `smoke.sh` re-verify.
- [~] 🟠 **Federation `remove-node` tombstone regression.** Code fix DONE 2026-07-02:
`remove_node` now tombstones BEFORE trimming the node list and propagates the write
error (idempotent, so retries are clean); `add_node`'s untombstone likewise propagates
before mutating. **Still open: `tests/multinode/smoke.sh` re-verify on real nodes.**
- [ ] 🟠 **Phase-3 Quadlet default-flip** — validated + opt-in on .228/.198; flip
`config.rs:256` once the .5 gate reports clean.
- [ ] 🟠 **Developer CLI suite** (`archy app validate/render/install/test`) — gates external
@@ -300,5 +302,7 @@ media (latest artifact only one minor behind).
6. **Before public GA only (NOT alpha/beta):** remove + rotate the Anthropic key (§F) —
intentionally left in for frictionless AI during alpha/beta.
*Last updated: 2026-07-02 (initial deep-audit synthesis). Update this line + tick
boxes with commit shas as items land.*
*Last updated: 2026-07-02 PM (hardening session 1: §A anchor+catalog `1977bdef`, §A
manifest signature `51647b21`, §D origin checks `206d5fe8`, §C swallowed writes +
§I tombstone fix — this commit). Update this line + tick boxes with commit shas as
items land.*