lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 4 Actions Packages Projects Releases 72 Wiki Activity

security: untrack operations docs; scrub infra identifiers from public docs

Browse Source 
Operations docs move out of git entirely rather than being sanitized. They
stay on disk for local use and are gitignored, so the Phase 6 export (which
takes HEAD) can never carry them. 15 files: the fleet runbook, hotfix
process, node inventories, internal trackers, session handoffs, the key
rotation/signing-posture records, and the open-source plan itself.

For the docs that remain public, infra identifiers are replaced with things
that are better documentation rather than placeholders: curl examples now
use `archipelago.local`, the product's own mDNS name, so a reader can run
them as-is instead of substituting an address that was never theirs.

Deliberately NOT scrubbed, both verified as functional rather than leaked:
- `tx1138.com` is the shipped default block explorer (DEFAULT_TX_EXPLORER in
  useTxExplorer.ts, surfaced in WalletSettingsModal). Product behavior.
- `git.tx1138.com` in core/container/{image_policy,registry}.rs is a retired-
  registry constant the code matches on to strip stale entries from legacy
  node configs. Removing it would break migration for older nodes.
- `192.168.1.254` in bulletproof-containers.md is the LAN gateway in a podman
  bug description, and `192.168.1.x` in user-walkthrough.md is already generic.

Whether a personal domain should be the shipped explorer default in a public
product is a separate product question, not a security one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-07 10:08:18 -04:00
co-authored by Claude Opus 5
parent 39eb6b0553
commit fda7feda60
24 changed files with 49 additions and 5085 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+21
View File
@@ -92,3 +92,24 @@ scripts/resilience/reports/
# app/docs asset path with a descriptive filename.
Screenshot *.png
uploads/
# Operations docs — fleet runbooks, node inventories, internal trackers and
# session handoffs. Kept on disk for local use, deliberately NOT tracked: they
# describe live infrastructure (node addresses, hostnames, deploy procedures)
# and must never reach the public repo. See the ops-docs section of the
# open-source readiness plan.
/docs/operations-runbook.md
/docs/hotfix-process.md
/docs/PRODUCTION-MASTER-PLAN.md
/docs/UNIFIED-TASK-TRACKER.md
/docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md
/docs/HANDOFF-2026-07-20-fips-peer-files.md
/docs/RETICULUM-TRANSPORT-PROGRESS.md
/docs/combined-test-plan-2026-07-22.md
/docs/pine-voice-release-test-plan.md
/docs/OPEN-SOURCE-READINESS-PLAN.md
/docs/archive/HANDOVER-2026-07-02-iso-feedback.md
/docs/archive/SESSION-1.8.0-OTA-PROGRESS.md
/docs/security/KEY-02-FLEET-ROTATION.md
/docs/security/KEY-03-SIGNING-POSTURE.md
/tests/production-quality/TRACKER.md
CHANGELOG.md
+13 -13
View File
@@ -496,7 +496,7 @@
- NetBird's browser proxy now sends API, OAuth, relay, WebSocket, and management traffic through the stable host-published server port at `169.254.1.2:8086`, avoiding stale rootless Podman DNS/IPs after `netbird-server` restarts.
- Mobile App Store category chips now stay visible above the tab bar, Discover is available on mobile, and category selection updates the page route/query so the selected category is actually shown.
- Apps that require a real browser tab now open directly from the app icon tap instead of first entering an in-shell app-session route, including BTCPay, Grafana, Home Assistant, Vaultwarden, Nextcloud, Portainer, OnlyOffice, Tailscale, Uptime Kuma, Gitea, and Nginx Proxy Manager.
- Validation passed with catalog JSON checks, `npm run type-check`, `cargo fmt --all --check --manifest-path core/Cargo.toml`, and `cargo check -p archipelago --manifest-path core/Cargo.toml`; live checks on `100.70.96.88` confirmed Saleor dashboard `9010`/API `8000` and NetBird API/OAuth routes survive `netbird-server` restart.
- Validation passed with catalog JSON checks, `npm run type-check`, `cargo fmt --all --check --manifest-path core/Cargo.toml`, and `cargo check -p archipelago --manifest-path core/Cargo.toml`; live checks on a fleet node confirmed Saleor dashboard `9010`/API `8000` and NetBird API/OAuth routes survive `netbird-server` restart.
## v1.7.75-alpha (2026-05-19)
@@ -529,7 +529,7 @@
## v1.7.71-alpha (2026-05-19)
- NetBird stack installs now pre-create `/var/lib/archipelago/netbird/data` before binding it into `netbird-server`, fixing the failed install/start path seen on `100.70.96.88` where Podman rejected the missing host directory.
- NetBird stack installs now pre-create `/var/lib/archipelago/netbird/data` before binding it into `netbird-server`, fixing the failed install/start path seen on a fleet node where Podman rejected the missing host directory.
- NetBird start/restart ordering now starts `netbird-server` before the dashboard container so lifecycle actions bring the control plane up before the UI.
- App-session invalid IDs and panel-mode fallbacks now return to `/dashboard/apps`, avoiding the stale `/apps` route that could render a 404.
- Mobile launches for apps that block iframes now stay inside the Archipelago app-session fallback instead of automatically opening an external browser tab.
@@ -547,7 +547,7 @@
## v1.7.69-alpha (2026-05-19)
- App installs now allow up to 10 minutes for the initial `package.install` RPC to return, matching slow container image pulls and preventing apps from disappearing from My Apps while the backend is still pulling or retrying mirrors.
- Live diagnostics on `100.70.96.88` confirmed the Gitea install did not fail; the primary registry pull timed out after 300 seconds, the fallback mirror succeeded, and Gitea came up healthy on `3001` while the frontend had already timed out at 15 seconds.
- Live diagnostics on a fleet node confirmed the Gitea install did not fail; the primary registry pull timed out after 300 seconds, the fallback mirror succeeded, and Gitea came up healthy on `3001` while the frontend had already timed out at 15 seconds.
- Gitea and other Docker-image app installs now stay visible during slow registry pulls instead of being marked as failed by the browser before backend install progress can complete.
- Gitea is now categorized as a known Data app in My Apps, so a running Gitea container appears with installed apps instead of being filtered into the Websites/Services split.
- NetBird `0.71.2` is now available in the app catalog and fallback marketplace data as a recommended networking app using the official `docker.io/netbirdio/netbird:0.71.2` image.
@@ -565,8 +565,8 @@
- App session close buttons now return to the previous dashboard screen when possible and otherwise fall back to My Apps, avoiding the 404 page after closing an app launched from an invalid or stale history entry.
- System Update confirmation and mirror modals now teleport to the document body with a full-screen overlay, so they cover the whole app instead of only the right-hand dashboard panel.
- Mobile app launches stay inside Archipelago's app-session webview and hide desktop-only new-tab launch affordances, including apps such as Home Assistant that previously looked like they would leave the mobile shell.
- Live recovery on `100.70.96.88` upgraded only the `btcpay-server` container to `docker.io/btcpayserver/btcpayserver:2.3.9`, preserved the existing datadir and Postgres database, and confirmed the container is healthy after a pre-upgrade backup.
- Public validation confirmed `spay.tx1138.com`/`www` redirect to BTCPay login over HTTPS and `sapien.tx1138.com`/`www` serve the L484 page over HTTPS using the issued Let's Encrypt certificates.
- Live recovery on a fleet node upgraded only the `btcpay-server` container to `docker.io/btcpayserver/btcpayserver:2.3.9`, preserved the existing datadir and Postgres database, and confirmed the container is healthy after a pre-upgrade backup.
- Public validation confirmed ``the BTCPay host`/`www` redirect to BTCPay login over HTTPS and `the L484 host`/`www` serve the L484 page over HTTPS using the issued Let's Encrypt certificates.
## v1.7.67-alpha (2026-05-18)
@@ -575,18 +575,18 @@
- Settings What's New is filled through `v1.7.67-alpha`, including the missing historical `v1.7.44-alpha` through `v1.7.66-alpha` entries.
- Bitcoin/Knots/Core shell lifecycle specs now match the Rust app config memory policy: 8 GiB on normal hosts, 4 GiB on low-memory hosts, and pruned Knots uses a larger dbcache on hosts with enough RAM to improve IBD throughput.
- ElectrumX/electrs shell lifecycle specs now match the 4 GiB memory policy used by the Rust app config, reducing drift between first boot, reconcile, and app lifecycle paths.
- Live assessment of `100.70.96.88` identified the current IBD bottlenecks as CPU/thermal/I/O pressure rather than RAM exhaustion, with follow-up work planned for existing-node swap repair, kiosk Chromium CPU reduction, and reconcile failure cleanup.
- Live assessment of a fleet node identified the current IBD bottlenecks as CPU/thermal/I/O pressure rather than RAM exhaustion, with follow-up work planned for existing-node swap repair, kiosk Chromium CPU reduction, and reconcile failure cleanup.
## v1.7.66-alpha (2026-05-18)
- Nginx Proxy Manager stale-port repair now detects stopped or `Created` Podman records by inspecting `podman ps -a` port metadata, covering records where `podman port nginx-proxy-manager` returns no mapping until start.
- Live recovery on `100.70.96.88` removed only the stale Nginx Proxy Manager container record and recreated it with `8081:81`, `8084:80`, and `8444:443`, preserving `/var/lib/archipelago/nginx-proxy-manager` data.
- Live recovery on a fleet node removed only the stale Nginx Proxy Manager container record and recreated it with `8081:81`, `8084:80`, and `8444:443`, preserving `/var/lib/archipelago/nginx-proxy-manager` data.
- Validation confirmed Nginx Proxy Manager recovered as healthy and responds through direct admin port `8081`, host compatibility port `81`, and `/app/nginx-proxy-manager/`.
## v1.7.65-alpha (2026-05-18)
- Orchestrator-backed app starts now run the same pre-start repairs as the legacy Podman path, so Nginx Proxy Manager stale `81:81` container metadata is removed and recreated before the orchestrator tries to start it.
- Live diagnostics on `100.70.96.88` confirmed host nginx is healthy while Nginx Proxy Manager has no listeners on `8081`, `8084`, or `8444`, causing host nginx `502` responses for NPM proxy paths.
- Live diagnostics on a fleet node confirmed host nginx is healthy while Nginx Proxy Manager has no listeners on `8081`, `8084`, or `8444`, causing host nginx `502` responses for NPM proxy paths.
## v1.7.64-alpha (2026-05-18)
@@ -611,7 +611,7 @@
- Multi-container stack installs now keep their app card in the `Installing` state for up to 20 minutes while dependency containers are being pulled and prepared.
- BTCPay Server installs no longer appear to vanish or fail after two minutes while Postgres and NBXplorer are still being created before the primary `btcpay-server` container exists.
- The stale-transition escape hatch remains short for start, stop, restart, update, and removal operations, so genuinely wedged lifecycle actions still recover quickly.
- Live validation on `100.70.96.88` confirmed BTCPay Server completed installation and responds on port `23000` with the expected HTTP redirect.
- Live validation on a fleet node confirmed BTCPay Server completed installation and responds on port `23000` with the expected HTTP redirect.
## v1.7.60-alpha (2026-05-18)
@@ -619,7 +619,7 @@
- Mesh radio auto-detection now skips known non-mesh serial devices such as Sierra Wireless LTE modems and Zooz/Z-Wave sticks, avoiding interference with production peripherals.
- Meshtastic config sync now sends `want_config_id` with the correct protobuf wire type, fixing radio-side `ignore malformed toradio` errors and allowing node-info/contact ingestion.
- The stable `/dev/mesh-radio` udev rule no longer claims every `ttyACM*` device; it only matches known mesh USB serial adapters and known USB CDC ACM radio vendors.
- Live validation on `100.70.96.88` confirmed Archipelago selects `/dev/ttyUSB0`, identifies the Meshtastic node, and refreshes 103 mesh contacts.
- Live validation on a fleet node confirmed Archipelago selects `/dev/ttyUSB0`, identifies the Meshtastic node, and refreshes 103 mesh contacts.
## v1.7.59-alpha (2026-05-17)
@@ -643,7 +643,7 @@
- Host nginx now serves `/assets/*` hashed frontend chunks as immutable static files with a hard 404 on misses instead of falling back to `index.html`, preventing strict MIME errors when a browser has a stale pre-update HTML shell.
- The SPA HTML shell and service-worker files now revalidate on every load, reducing stale frontend references after OTA updates.
- OTA runtime promotion now installs the bundled `nginx-archipelago.conf` into `/etc/nginx/sites-available/archipelago` and reloads nginx after a successful config test, so frontend cache/fallback fixes reach existing nodes without a manual deploy.
- Local validation passed with `cargo check -p archipelago`; live SSH testing against `100.70.96.88` was not completed because temporary public-key authentication was rejected on the target.
- Local validation passed with `cargo check -p archipelago`; live SSH testing against a fleet node was not completed because temporary public-key authentication was rejected on the target.
## v1.7.57-alpha (2026-05-17)
@@ -724,7 +724,7 @@
- Health monitor no longer pages "Auto-restart failed" for orphaned containers. After a variant switch (bitcoin-core ↔ bitcoin-knots) the previous variant's container could survive uninstall and the health monitor would try restarting it forever. Now skipped silently with a debug log.
- Apps no longer disappear from My Apps when an install fails. The card stays visible with state=Stopped so the user can retry or uninstall, with the failure reason surfaced via the new install_progress.message field.
- "Downloading…" progress now actually advances during multi-image stack pulls. Was sticking at 20% until all pulls finished; now interpolates 20%→70% based on which image of N has landed.
- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on OVH and tx1138. Removes a docker.io dependency from first-boot installs.
- Pulled four docker.io images (bitcoin, gitea, nextcloud, valkey) into the lfg2025 registries on the registry mirrors. Removes a docker.io dependency from first-boot installs.
- Resilience harness improvements: install-fail entries no longer vanish, install/uninstall/probe cells are timing-tolerant (60s retry on ui_probe and auth_probe), dep snapshots no longer leak companion containers into the dependent app's "new containers" set.
## v1.7.45-alpha (2026-04-29)
@@ -770,7 +770,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Infrastructure
- CI pipeline added (.github/workflows/ci.yml) — cargo fmt, clippy, tests + frontend type-check, build
- Update system now fetches from git.tx1138.com Gitea instance (configurable via ARCHIPELAGO_UPDATE_URL)
- Update system now fetches from the release Gitea instance (configurable via ARCHIPELAGO_UPDATE_URL)
- Cleaned up stale git branches (app-store, overnight/2026-03-12, overnight/2026-03-13)
## [1.3.0] - 2026-03-19
docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md
-300
View File
@@ -1,300 +0,0 @@
# FIPS near-100% uptime + optimistic UI state — implementation plan
**Date:** 2026-07-27. **Status:** researched + root-caused live on the fleet; ready to
implement for the next release. Two workstreams: (A) make node↔node FIPS transport
succeed whenever a FIPS path physically exists, (B) stop the UI reloading everything
on every navigation (optimistic/cached cards, stale-while-revalidate) while keeping
data fresh.
**Honesty note on "100%":** if a node's network blackholes every anchor (the .116
WiFi case, `docs/HANDOFF-2026-07-20-fips-peer-files.md:117-133`), Tor fallback is
*correct*. The achievable target is: **FIPS wins whenever a FIPS path exists, and
fallback frequency is measured in-product so regressions are visible.** Today several
paths are 0% FIPS *by construction* regardless of network health — that's the bug.
---
## Part A — why Cloud/FIPS "commonly falls back to Tor": ranked root causes
All verified live on 2026-07-27 (.116 local, .198, .228, Framework PT, x250s) plus a
full code audit of `core/archipelago/src/{fips,transport,federation,server.rs}`.
### RC0 — 🔥 The hardening firewall drops the peer-API port on every hardened node (PROVEN)
The fips0 default-deny baseline (`/etc/fips/fips.nft`) is opened by archipelago's
drop-in `80-web-ui.nft` (`fips/config.rs:236-255`) for **80 + 8443 + app ports only**.
The peer-API listener — which carries *all* federation sync, cloud browse/download,
mesh envelopes, DWN, invoices — is **`PEER_PORT = 5679`** (`fips/dial.rs:35`).
**5679 is not in the allowlist.** The drop-in's own comment claims "web UI + peer
API" but the peer API port was never added.
Live proof (2026-07-27):
- .116 nft chain: 5,965 dropped packets; .198: **28,670 dropped packets** — that's
peers' FIPS dials dying at the firewall.
- .198 → .116 `GET :5679/health`: **timeout (6s)** before; **HTTP 200 in 0.35s**
after `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`.
Same result in reverse direction (200 in 0.64s).
- Explains the exact fleet split in `federation/nodes.json`: hardened-baseline nodes
(Framework PT, .198, .228, x250-dev, x250-mad2) = `last_transport: tor`;
non-hardened nodes (Austin Sapien, X250-Beta, X250-PA) answer :5679 (404 from the
path allowlist = listener reachable) = `last_transport: fips`.
- Every dial to a hardened peer pays the 8s FIPS connect timeout
(`dial.rs:114`) ×2 (retry, `dial.rs:128-140`) → then Tor. That's the "Cloud takes
forever / shows Tor" experience.
**Fix (one line + reload):** add `tcp dport 5679 accept` to the drop-in in
`fips/config.rs` (use a constant shared with `dial.rs::PEER_PORT`, not a literal).
The drop-in reinstalls on every daemon config install, so it heals fleet-wide on OTA.
⚠️ Transient manual rules were inserted on .116 and .198 during diagnosis (2026-07-27)
— they vanish on the next `nft -f /etc/fips/fips.nft` reload or reboot; the code fix
makes them permanent.
### RC1 — .228 (Shorty's) runs fips 0.3.0-dev; the 0.4.1 fleet can't reach it
.228's daemon: `0.3.0-dev (rev 34e00b9f6e)`, both anchor links "connected", but its
ULA is 100% unreachable from 0.4.1 nodes (ping loss 100%). FIPS wire format is not
stable across revs (`docs/HANDOFF-2026-07-23-companion-apk-deploy.md:78`). Everything
to/from .228 rides Tor no matter what else we fix.
**Fix:** fleet fips-version audit + upgrade to v0.4.1 everywhere (in-product updater
exists: `fips/update.rs`; .deb path per `reference_vps2_fips_anchor`). Add a version
check to `fips.status` and surface a "peer daemon outdated" warning.
### RC2 — Direct LAN/endpoint peering is dead code + wrong port + stale seed anchors
Without direct links, all peer traffic hairpins through the vps2 anchor spanning
tree (observed: .116→.198 cold RTT 1.5–3.5s on the same LAN; also the wedged-anchor
latency-rot incident, `HANDOFF-2026-07-23:141-160`).
- **G1 — `lan_fips_anchors()` has never run.** It needs `PeerRecord.fips_npub`, but
`PeerRegistry::set_fips_npub` (`transport/mod.rs:302`) has **zero callers** — mDNS
TXT records only carry `did`/`pubkey`/`version` (`transport/lan.rs:50-54`). So the
"co-located peers form a direct link" feature (`anchors.rs:294-305`,
`server.rs:761-766`) is a fleet-wide no-op.
- **G2 — wrong UDP port.** `anchors.rs:293` dials `8668`, but the generated
fips.yaml binds UDP **2121** (`fips/config.rs:187`, `fips/mod.rs:130`). Even if G1
ran, it would dial a dead port. `.116`'s live `seed-anchors.json` still carries
`.198@192.168.1.198:8668` — **stale IP (LAN renumbered to 192.168.63.x) AND dead
port**; both manual entries are useless today.
- No Tailscale/alternate endpoint fallback when LAN is unreachable (the .116↔.198
fix of 2026-07-20 was hand-applied per-node config, never productized).
**Fix:** (a) `FIPS_UDP_PORT` → `crate::fips::PUBLISHED_UDP_PORT` + drift-guard test;
(b) hydrate `fips_npub` into the registry from federation storage (did-keyed join) so
`lan_fips_anchors` goes live with no wire change; (c) advertise the npub in the mDNS
TXT + `set_fips_npub` on resolve as the proper fix; (d) teach the LAN-anchor tick to
also try a peer's Tailscale/last-known-good endpoint when LAN fails (reviewed change
— this area got handoffs wrong twice, per memory).
### RC3 — No fast-fail on the hottest call sites; retry silently doubles every budget
- `content.browse-peer` — **the Cloud page** — has NO `fips_timeout`
(`api/rpc/content.rs:363-366`): a cold FIPS path burns up to ~16.6s (8s connect +
600ms + 8s retry) before Tor even starts, against a UI deadline of 30s
(`Cloud.vue:720`) — and the frontend then retries ×3. Users see errors, not
fallback. 12 call sites total lack `fips_timeout` (browse/download/preview-peer,
`/blob`, DWN, node_message, rotation notifies).
- `dial.rs:128-140` runs 2 full-budget attempts, so `fips_timeout(6s)` really means
~12.6s everywhere.
**Fix:** wrap `send_with_retry` in a single `tokio::time::timeout(fips_attempt_timeout())`
(call sites `dial.rs:455`, `dial.rs:488`; halve per-attempt client timeout), then add
`.fips_timeout(...)`: `content.rs:366` (6s), `content.rs:281` (8s), `content.rs:1139`
(6s), `typed_messages.rs:822` (8s), `dwn_sync.rs:188/213/272` (6s),
`node_message.rs:376` (8s), `node_message.rs:412` (4s), `tor/mod.rs:501` (6s),
`federation/handlers.rs:869` (6s). **Skip the three 900s streaming downloads**
(`content.rs:552/870/1061`, `proxy.rs:236`) — `dial.rs:311-319` documents why; the
retry-budget wrap covers their connect phase.
### RC4 — Two features are 100% Tor by construction (allowlist 404)
The peer listener path allowlist (`server.rs:1219-1239`) omits `/blob/<cid>` (mesh
file sharing, `typed_messages.rs:813-822`) and `/dwn/health` (step 1 of DWN sync,
`dwn_sync.rs:186`) → deterministic 404 over FIPS (`dial.rs:44-46` treats 404 as
fall-back) → deterministic Tor, after paying the full FIPS cost. Both endpoints are
already cryptographically gated, so they meet the allowlist's stated criterion.
**Fix:** add `|| path.starts_with("/blob/") || path.starts_with("/dwn/")`; extend the
existing test block at `server.rs:1935-1945` (assert `/blob/abc` + `/dwn/health`
allowed, `/blobber` + `/dwnx` denied).
### RC5 — Inbound listener can't heal; anchor flap = 5-minute Tor window; probe overhead
- `peer_late_bind_loop` returns after first successful bind (`server.rs:1203`) and
`accept_loop` `continue`s on errors forever (`server.rs:1249-1258`): a fips0
teardown/re-key leaves the node inbound-dead until process restart → **every peer**
falls back to Tor against it.
- Nothing reacts to anchor-link drops: anchors re-apply only on the 300s tick
(`server.rs:731`); worst-case 5min Tor-only after a flap (the historic "link dead
timeout 30s" flapping made this chronic).
- `is_service_active()` spawns up to 2 `systemctl` per FIPS attempt *and* per peer
per 25s warm tick (`dial.rs:284-294`); `warm_path` skips peers without
`fips_npub` in federation storage (`fips/mod.rs:88-95`); `anchors::apply` is
serial with unbounded subprocess waits (`anchors.rs:234-283`).
**Fix:** rebindable listener; a ~25s connectivity watcher (reuse
`service::peer_connectivity_summary`, `fips/service.rs:178-207`) that re-applies
anchors immediately on a connected→disconnected edge with bounded backoff; 10s TTL
cache for `is_service_active` (mirror `transport/fips.rs:24-107`); warm the union of
federation+registry peers; make `apply()` concurrent with per-connect timeouts.
### RC6 — Zero observability: fallbacks are invisible, so "uptime" is unfalsifiable
Fallbacks log at `debug!` only (`dial.rs:458,491`); no counters; `last_transport` is
written by only 7 of ~20 call sites and **never read** to influence anything
(`storage.rs:120-147`). The parallel `TransportRouter` system can't even see FIPS
(`FipsTransport` is never constructed — `server.rs:422-442` registers Tor/Mesh/LAN
only).
**Fix:** per-reason fallback counters (F1 no-npub / F2 service-inactive / F3
DNS-fail / F4 connect-fail / F5 404 / F6 5xx) surfaced in `fips.status` + `info!`
logs with a `reason` field; call `record_peer_transport` from all peer-dial sites;
UI: per-peer transport badge on Cloud (the response already carries `transport` —
`content.rs:392-400` — Cloud.vue currently throws it away at `:716-721`).
---
## Part A — execution phases
### Phase A0 — fleet triage (no release needed; do first, validates everything)
1. Fleet audit: `fipsctl --version` + `nft list table inet fips` + `ss -tlnp | grep 5679`
on every node (roster: `reference_test_deploy_roster`).
2. Transient `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`
on hardened nodes (already done on .116 + .198, 2026-07-27) — instant fleet-wide
FIPS recovery while the code fix rides the OTA.
3. Upgrade .228 (and any other 0.3.x) fips daemon to v0.4.1.
4. Regenerate/clean stale `seed-anchors.json` on .116 (dead 192.168.1.x + :8668 entries).
5. Baseline measurement: for each node pair, `content.browse-peer` time + transport.
### Phase A1 — P0 code (one commit, mechanical, offline-testable)
1. **nft drop-in: open 5679** — `fips/config.rs` (share the constant with
`dial.rs::PEER_PORT`). ← RC0
2. **Allowlist `/blob/`, `/dwn/`** — `server.rs:1219-1239` + tests. ← RC4
3. **`FIPS_UDP_PORT` = `PUBLISHED_UDP_PORT` (2121)** — `anchors.rs:293` + drift-guard
test against `render_config_yaml()`. ← RC2-G2
4. **Un-deaden `lan_fips_anchors`** — hydrate `fips_npub` from federation storage in
`server.rs:761-766`; then mDNS TXT `fips` key + `set_fips_npub`
(`transport/lan.rs:50-54`, `lan.rs:96-108`, `LanTransport::new` 4th arg via
`crate::identity::fips_npub(&data_dir.join("identity"))`). ← RC2-G1
5. **Retry-budget wrap + `fips_timeout` on 12 call sites** (list in RC3). ← RC3
Verify: `cd core && cargo test -p archipelago` — watch `test_rendered_yaml_exact_snapshot`
(`config.rs:419`) + `test_render_is_deterministic` (`config.rs:476`); item 3 must
not change rendered output.
### Phase A2 — telemetry BEFORE tuning (second commit)
6. Fallback counters by reason + `fips.status` exposure + `info!` reason logs;
`record_peer_transport` from all sites. ← RC6 (gives the baseline that makes A3
measurable and "100%" falsifiable)
### Phase A3 — resilience (third commit, measured against A2 baseline)
7. `is_service_active` 10s TTL cache; warm-path union + `warm_path_unchecked`.
8. Link-state watcher → immediate anchor re-apply on drop (replaces waiting for the
300s tick); concurrent `apply()` with subprocess timeouts.
9. Rebindable peer listener (`server.rs:1203`, `1249-1258`).
10. (Reviewed, separate PR) endpoint-fallback for direct peering: LAN → Tailscale →
last-known-good, npub-keyed. Mesh-routing area — needs careful review per memory.
### Phase A4 — verification gate (on nodes, before tag)
- On .116/.198/framework-pt/.228: `content.browse-peer` to every peer must return
`transport: "fips"` with sub-second latency (LAN pairs) / <3s (WAN), 20/20 calls.
- Kill the fips daemon on one node → calls fall back to Tor gracefully within the
fast-fail budget (<8s), UI shows partial results, no errors.
- Restart daemon → FIPS recovers within one watcher tick (~25s), verified in
`fips.status` counters.
- Flap the anchor link (drop vps2 route) → direct LAN pairs keep FIPS via their
direct link (G1 fix proof).
- Add these as `tests/multinode/` cases per `docs/multinode-testing-plan.md`; also
fix the known `node_rpc()` missing `--max-time` (tracker item).
---
## Part B — optimistic loading + state management (frontend)
Full audit: Pinia exists but pages fetch-on-mount with `loading=true` spinners;
`Dashboard.vue:89` keys the router-view by `route.path`, so **every navigation
unmounts and refetches everything**; no KeepAlive/onActivated anywhere; no dedup,
no abort, no SWR layer. Four hand-rolled cache implementations already exist and
prove the pattern (`useFleetData.ts:198-231` sessionStorage hydrate;
`homeStatus.ts` sticky-ready loadState; `Home.vue:591-621` wallet localStorage
snapshot; `curatedApps.ts:21-77` TTL cache). `SkeletonCard.vue` exists, imported by
zero files.
### B1 — one shared primitive: `useCachedResource` composable + `resources` Pinia store
Semantics (generalize `homeStatus.ts` + `useFleetData.ts`):
- Keyed resource: `{ data, loadState: idle|loading|ready|error|refreshing, fetchedAt, error }`.
- **Hydrate synchronously** from memory (Pinia, survives navigation) → sessionStorage
snapshot (survives reload) → then revalidate in background.
- Sticky-ready: once `ready`, never regress to `loading`
(`loadState = loadState==='ready' ? 'ready' : 'loading'` — the `homeStatus.ts:80` idiom);
keep-last-known-value on error with a stale badge (age from `fetchedAt`).
- TTL per resource; `revalidateOnFocus` + on WS push (debounced, the
`Home.vue:539-542` pattern); explicit `invalidate(key)` for mutations.
- Optimistic mutation helper: apply → RPC → rollback on error (generalize
`TransportPrefsCard.vue:112-127`).
### B2 — rpc-client upgrades (`src/api/rpc-client.ts`)
- `AbortSignal` in `RPCOptions` (today the AbortController at `:87` is timeout-only)
→ abort-on-unmount for fan-outs.
- In-flight dedup keyed `method+JSON(params)` — collapses duplicate concurrent calls.
- Per-call `maxRetries` override; set `maxRetries: 1` for `content.browse-peer` /
`preview-peer` (retry×3 on a 30s timeout is why one slow peer = 90s spinner).
### B3 — Cloud page conversion (worst offender, the marquee win)
- Move `sectionCounts`, `peerNodes`, `myFiles`, `peerFiles`, `paidItems` out of
`Cloud.vue` component state (`:403,:476,:582,:689,:427`) into the cached store —
instant render on revisit, background refresh.
- **Incremental per-peer fan-in**: render each peer's card as its
`content.browse-peer` resolves (today `Promise.allSettled` at `:708-747` blocks on
the slowest peer). Per-peer states: cached/fresh/loading/unreachable.
- **Surface `transport` per peer** (already in the response, discarded at `:716-721`):
FIPS/Tor badge + latency — this is also the fleet-wide FIPS-uptime dashboard the
user asked for, for free.
- Skeleton cards (revive `SkeletonCard.vue`, copy `FileGrid.vue:3-19` shimmer) instead
of spinners for counts/folders/peer grids.
- Stop `CloudFolder.vue:307-319` calling `cloudStore.reset()` on every folder entry —
cache per-path listings, navigate renders cache + revalidates.
- `PeerFiles.vue`: persist catalog + preview cache in the store; cap the
`preview-peer` fan-out (`:832-841`, currently unbounded) with a small concurrency
queue + abort-on-unmount.
### B4 — roll out to remaining offenders (in audit order)
PeerFiles → Web5 wallet/ecash/LND slices → Monitoring → Lightning channels
(`LightningChannelsPanel.vue:650`) → Federation (already has `{showLoader:false}` —
just adopt the store) → Server → Credentials/OpenWrtGateway/ContainerApps.
`Apps.vue`/`Marketplace.vue`/`Fleet.vue` are already good; don't touch.
### B5 — freshness via the existing push channel
`/ws/db` firehose + `sync.ts` JSON-patch already exist. Wire `useCachedResource`
revalidation to relevant WS pushes (debounced 800ms), keep the 30s staleness
reconciliation as backstop. No new backend needed for v1; a per-topic subscribe can
come later.
### Part B verification (on nodes)
- Navigate Cloud → Apps → Cloud: peer files render instantly from cache (0 spinner),
refresh indicator while revalidating, updated data lands without layout jump.
- One unreachable peer: its card shows stale/unreachable state; other peers render
immediately (no 30s all-or-nothing).
- Kill backend mid-view: stale data stays visible with age badge; recovery
revalidates automatically.
- Hard reload: sessionStorage hydrate paints before first RPC completes.
---
## Sequencing for the next release
1. **A0 now** (fleet triage + transient nft rules + .228 daemon upgrade + baseline).
2. **A1 + A2** land together (P0 fixes + telemetry) → deploy to .116/.198 →
Phase A4 checks on the pair → framework-pt → full fleet.
3. **B1 + B2 + B3** (composable + rpc-client + Cloud) in parallel with A-testing —
frontend-only, verifiable against .116 dev (`reference_neode_ui_dev_testing`).
4. **A3** after telemetry baseline exists; **B4/B5** ride the same or next OTA.
5. Gate: Phase A4 checklist green + Part B verification on-device + existing
single-node gate stays green → tag/OTA per ship ritual.
## Success criteria
- `content.browse-peer` transport = fips for ≥99% of calls between healthy 0.4.1
nodes over 24h (measured by the new counters), Tor reserved for genuinely
FIPS-unreachable peers (.116-WiFi-class networks).
- Cloud revisit paints in <100ms from cache; fresh data within one revalidate.
- Fallback counters visible in `fips.status` so regressions are caught on the
dashboard, not by users.
docs/HANDOFF-2026-07-20-fips-peer-files.md
-238
View File
@@ -1,238 +0,0 @@
# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill
Written for a fresh session that will **cut the OTA release and build the ISO**.
Everything below is already committed and pushed to `gitea-ai/main`. Last release
was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**.
---
## 1. What this release carries (3 commits on top of v1.7.105-alpha)
| Commit | What | User-visible? |
|---|---|---|
| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes |
| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) |
| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly |
### `9e3ac9ba` — mobile transport pill
`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header
carries the name on mobile), and the transport pill was nested inside it — so it
vanished below 768px. Added a separate `md:hidden` pill next to the peer icon.
Frontend was rebuilt and the class verified present in the emitted bundle.
Caveats worth knowing (pre-existing, not introduced here):
- On this code path the backend only ever emits `fips` or `tor`, so the `mesh`
and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead.
- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a
hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie.
The peer-files pill does not.
### `3ab7fb52` — full error chain in logs
`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every
peer-files failure read exactly `RPC error on content.browse-peer: Failed to
connect to peer` with the real cause discarded. Now `{:#}`. The client-facing
message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so
no internal detail leaks. **This fix applies to every RPC method, not just
browse-peer.**
### `5fd0d6c3` — typed FIPS config + mDNS
`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal.
Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key
does not degrade — **the daemon refuses to start and the node leaves the mesh**.
Now a typed serde struct tree, verified field-by-field against jmcorgan/fips
**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and
the pre-existing schema test. All pass.
Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0)
so co-located nodes peer directly instead of depending on the public anchor.
> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup
> drift check at `server.rs:864` compares the freshly rendered config against
> what's on disk. The render differs now, so it reinstalls the config and
> restarts the FIPS daemon **once**. This is the intended self-healing path and
> settles immediately. Do not mistake it for a regression.
Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig`
has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it
harmlessly (verified against the v0.3.0 source). It self-activates on upgrade.
---
## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled
Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1**
(2026-07-19). Verified before touching anything:
- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day.
- **Config forward-compatible** — every key we emit exists in 0.4.1.
- **Asset names match** what `fips/update.rs` expects (`fips_<ver>_<arch>.deb` +
`checksums-linux.txt`), so the in-product updater should work.
### Upgraded so far (2 of N)
| Node | Before | After | Result |
|---|---|---|---|
| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` |
| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) |
The OptiPlex was still running the **old string-rendered config** and 0.4.1
accepted it — empirical confirmation of the compat analysis, not just desk work.
### Upgrade recipe (nodes cannot reach GitHub — sideload)
```bash
# 1. On a host with GitHub access:
curl -sL -o fips_0.4.1_amd64.deb \
https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb
curl -sL -o checksums-linux.txt \
https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt
sha256sum fips_0.4.1_amd64.deb # must match checksums-linux.txt
# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded
# 2. Sideload:
scp fips_0.4.1_amd64.deb archipelago@<node>:/tmp/
# 3. On the node — the same command update.rs uses:
sudo -n systemd-run --collect --wait --quiet --pipe -- \
env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \
/tmp/fips_0.4.1_amd64.deb
# 4. Restart the ACTIVE unit — it is archipelago-fips.service,
# NOT fips.service (which is inactive on these nodes):
sudo -n systemctl restart archipelago-fips.service
# 5. Verify:
fipsctl --version
sudo -n fipsctl show links # expect anchor 185.18.221.160:8443 connected
sudo -n fipsctl show tree # expect is_root: false, depth > 0
```
### ISO implication (important)
`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from
**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64
only). So a freshly built ISO will pick up whatever main is that day — probably
≥0.4.1, but it is not deterministic. Pinning is an open item in
`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1
before building the release ISO** so the shipped version is knowable.
---
## 3. The original bug — peer cloud files not loading
**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit
because it would be easy to read this as closed.
What is established:
- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links`
→ `[]`, `show tree` → `is_root: true, depth 0`. An island.
- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the
public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on
ICMP, 443/8443/8668 all time out. `show transports` showed
`packets_sent: 760, packets_recv: 0` on both UDP and TCP.
- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock
Tailscale anti-spoof DROPs).
- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's
WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`).
- With no FIPS tree, everything falls back to Tor. Every peer in
`federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`.
- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×,
HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`.
What is **not** established: why three specific `content.browse-peer` calls
failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never
reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic
is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and
the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is
cold-circuit timeouts on first fetch after idle — **a guess, not a finding.**
`3ab7fb52` means the next occurrence will log the actual cause.
### Corrections to earlier claims in this session
- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the
`ip:port` in `fipsctl connect` is only an underlay endpoint hint.
- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy
(`delivery_ratio` 1.0 both directions, bloom filter syncing). The
`bytes_recv: 0` link counters are simply uninstrumented in 0.3.0.
---
## 4. Open items — decisions NOT taken
1. **Second FIPS anchor (user asked for this; not built).** Needs a host running
FIPS that is reachable from the restricted WiFi. Candidate found: OVH
**`146.59.87.168`** — pings fine from the thinkpad and general egress works
(github 200), while the upstream anchor fails even ICMP there. But it does not
run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** —
a production change, deliberately not made unprompted. Code side is easy after:
`fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list
(`default_public_anchor()` → `default_public_anchors() -> Vec<SeedAnchor>`).
2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228`
(`100.64.204.114`) has been **offline ~20h** and could not be included.
3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet,
so mDNS is not actually live anywhere. That is what this OTA is for.
4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so
mDNS may not rescue that particular node even after the OTA. It will help
co-located nodes on sane networks.
5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO.
---
## 5. Release ritual (from prior sessions — follow exactly)
Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`,
`release-manifest.json`, `releases/manifest.json` modified, and an untracked
`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another
agent may share this tree; never `git add -A`.
```bash
V=1.7.106-alpha
# Frontend build — MUST verify dist actually changed (build can silently no-op)
cd neode-ui && npm run build # → web/dist/neode-ui/
grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js # sanity
# Backend
cd core && cargo build --release -p archipelago
# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache
# corruption — rebuild with CARGO_INCREMENTAL=0
# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s
tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui .
tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3 # ./ then ./index.html
# Exclude the ~17MB companion APK from tarballs.
# Ship
scripts/create-release.sh $V
scripts/publish-release-assets.sh $V gitea-vps2
git push origin main && git push origin --tags # tag or the Releases page stays empty
git push gitea-ai main # main is protected; use the `ai` account
# Verify the live manifest
curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json
```
Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is
done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first
(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with
user benefit.
### ISO
```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh
```
ISO builds are **always unbundled** — the default env silently builds the wrong
full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output
filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is
`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO.
---
## 6. Node access quick reference
- **thinkpad (`.116`) is the local machine** — do not SSH to it; read
`journalctl -u archipelago` and `/var/lib/archipelago/**` directly.
- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user
`archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale.
- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and
it is in real use; don't touch uninvited.
- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex.
- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores.
- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660).
- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74).
docs/OPEN-SOURCE-READINESS-PLAN.md
-322
View File
@@ -1,322 +0,0 @@
# Open-Source Readiness Plan — Archipelago public launch
> Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup.
> A second agent is working the same goal concurrently — before executing any phase,
> diff against `git log` since `7e8d3314` and skip/merge what's already done.
> (Session plan file: `~/.claude/plans/resilient-moseying-reef.md`.)
## Context
The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep
exploration passes (docs/structure, code health, secrets sweep) found the repo is
fundamentally strong — README, `apps/` manifest examples, ADRs, the bats lifecycle gate,
1,104 Rust tests — but has hard blockers: **two live Anthropic API keys committed in
tracked files**, node passwords in 7 tracked files, no LICENSE (README links a 404),
5.5 GB `.git` (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea
registry `146.59.87.168:3000` that make every app image unpullable for outsiders, and
~28 internal AI-session/tracker docs mixed into `docs/`.
**Decisions made by the user:**
1. **Fresh-history publish** — new public repo with a clean initial commit; private repo keeps full history.
2. **Registry: domain + parameterize** — real domain in front of the existing registry; host configurable everywhere.
3. **Deep code cleanup** — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly).
4. **Internal docs: sanitize and keep public** — scrub creds/IPs/hostnames but publish plans/trackers for transparency.
**Invariant throughout:** the single-node production gate (`tests/lifecycle/run-gate.sh`)
is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E
especially). All cargo verification uses `--all-features` to match CI. Stage by explicit
path, never `git add -A` (shared tree).
## Current local pass status
This branch is replayed on top of `origin/main` as `public-prelaunch`.
Completed locally in this pass:
- Redacted the two tracked Anthropic API key literals from
`scripts/setup-aiui-server.sh` and
`image-recipe/_archived/build-auto-installer-iso.sh`.
- Removed `Android/app/debug.keystore` and `core/.env.production` from the
source tree; copies were preserved in
`~/Desktop/archipelago-sensitive-backup-2026-07-27/`.
- Reworked `scripts/audit-secrets.sh` to scan tracked source more aggressively
and to catch non-example env files and credential file patterns.
- Reworked `scripts/validate-app-manifest.sh` so the current `app:` manifest
schema can be audited without a Python `PyYAML` dependency.
- Updated root/community docs, CI, PR template, app developer notes, and
container/deployment docs toward public contributor expectations.
- Fixed native FIPS activation fallback: nodes that have the packaged
`fips.service` but not `archipelago-fips.service` now start the available
unit instead of repeatedly failing activation against a missing unit. This
now covers startup, supervisor self-heal, manual dashboard start/reconnect,
and post-onboarding activation. The UI now labels the action as `Start`
instead of making native FIPS look like an installable app.
- Fixed the FIPS app-port relay design so it binds relays to the node's FIPS
ULA instead of wildcard `[::]`, avoiding collisions with Podman-published app
ports such as FileBrowser `8083` and Botfights `9100`.
- Added `docs/nostr-git-source-hosting.md`, a NIP-34/ngit/GRASP source hosting
plan using a Bitcoin Core-style maintainer model: public review and easy
forks, with canonical merge rights held by a small signed maintainer set.
Verified locally:
- `./scripts/audit-secrets.sh` passes.
- Full `apps/*/manifest.yml` repository audit passes with warnings only.
- `bash -n` passes for the edited shell scripts.
- Targeted FIPS dashboard vitest passes.
- Targeted Rust tests for FIPS service unit detection and FIPS app relay
address selection pass.
Verified on a Linux Archipelago verification node:
- Native FIPS was restored by starting the already-installed packaged
`fips.service`; the daemon became active and joined the FIPS tree.
- Correct local lifecycle API endpoint is HTTP, not HTTPS
(`ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http`).
- Read-only lifecycle run progressed past login and confirmed required
containers, Bitcoin RPC, ElectrumX TCP, and manifest port-drift checks, but
did not complete cleanly: `botfights` and `filebrowser` remained in
`restarting` longer than the matrix window, and the LND `lncli getinfo`
probe hung. Do not run the destructive gate until those live-node issues are
understood.
- After the node updated to `1.7.116-alpha`, `botfights`, `filebrowser`, and
`lnd` were active/running and ports `8083`/`9100` were held by Podman's
`rootlessport` as expected. The packaged `fips.service` remained installed
and enabled but inactive, so the native FIPS service fallback should still
ship before the public launch.
Still required before public publish:
- Rotate/revoke compromised credentials listed in Phase 0.
- Finish Phase 1 password/node/token sanitization beyond the two API keys.
- Publish from fresh history after the sanitized tree is final.
- Run full Rust, frontend, Android, and lifecycle gate verification.
- Resolve the live-node lifecycle blockers above, then rerun the read-only
suite followed by the destructive gate only on an approved verification node.
- Decide the canonical Archipelago maintainer npub and merge-maintainer npub
list before publishing the Nostr Git source-hosting workflow.
---
## Phase 0 — Credential rotation (DEFERRED to the pre-publish gate, 2026-08-07)
> **Sequencing decision (user, 2026-08-07):** rotation/revocation moved from first to
> last. This is safe *only* because the publish is fresh-history — the scrub commits
> never become public, so scrubbing before rotating leaks nothing to outsiders.
>
> **Hard gate: Phase 6 MUST NOT run until every item below is done.** The export is the
> point where a missed literal becomes public and a live key becomes an incident.
> Everything here is still live as of this writing. Phase 6 step 3 now includes an
> explicit rotation sign-off.
Treat all of these as already compromised; rotate even though we're doing fresh-history:
- **Anthropic API key #1**: `image-recipe/_archived/build-auto-installer-iso.sh:2837` (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (`ISO_ANTHROPIC_API_KEY`), keep the alpha-baking behavior if desired but never the literal in git.
- **Anthropic API key #2**: `scripts/setup-aiui-server.sh:28` — a *different* live key, not covered by the documented alpha exception. Revoke; parameterize the script.
- **The shared node SSH/sudo/UI password** (two variants) — was in **8** tracked files (see
Phase 1 status) + 24+ commits. Now scrubbed from the tree; still live on the fleet.
Rotate fleet-wide (user task).
- **Gitea `ai` account password + 2 Gitea tokens** — embedded in `.git/config` remote URLs
(not tracked, but leaks in any directory copy/tarball). **Verified 2026-08-07: both tokens
are already dead** — `localhost:3000` and `146.59.87.168:3000` both return 401. Only the
`ai` password on `source.archipelago-foundation.org` is live. Rotate it; switch remotes to
credential-helper storage instead of URL-embedded creds.
- **Framework node SSH** — its password was rotated out-of-band and is not recorded anywhere;
key auth is also rejected. Whoever holds it should capture it before the fleet rotation, or
that node becomes unreachable for the rotation itself.
## Phase 1 — Secrets & sanitization of tracked files
**Status 2026-08-07: items 1, 2 and 4 DONE** (`e3b98ed1`, `19082a44`). The password was in
**8 files, not 7** — the reworked audit found three in `.planning/` that this list missed.
`scripts/audit-secrets.sh` is 5/5 green and canary-tested. Items 3 (infra identifiers) and
5 (`.gitignore`) remain.
1. Strip the password/credential lines from the 7 files:
`docs/PRODUCTION-MASTER-PLAN.md` (lines ~428–429, 454–457, 483, 521–528, 886 — the fleet cred table),
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`, `docs/archive/HANDOVER-2026-07-02-iso-feedback.md`,
`docs/bitcoin-version-bulletproof-rollout.md`, `tests/production-quality/TRACKER.md`,
`tests/multinode/meshtastic.sh:26`, `neode-ui/test-openwrt.mjs:4` (→ env var).
2. `.gitea/workflows/post-install-tests.yml` — remove `sshpass -p '…'` + default target IP; use secrets/vars.
3. Sanitize infra identifiers repo-wide (in the *sanitize-and-keep* docs and scripts):
replace Tailscale IPs (17 unique, 14 files), LAN IPs (`192.168.1.x`, 93 files), hostnames
(`tx1138`, `shorty-s`, `archy-x250`, `archy-dev-pa`) with placeholders like `<node-a>` /
`NODE_IP`. Key script targets: `scripts/deploy-config-defaults.sh`, `scripts/deploy-tailscale.sh`,
`docs/operations-runbook.md` (opens with real node IPs), `docs/developer-guide.md`, `docs/api-reference.md`, `docs/hotfix-process.md`.
4. Fix the audit tool that let this happen: `scripts/audit-secrets.sh:28` — remove `\.md$` and
bare `test` from ALLOW_PATTERNS; add `sk-ant-` and password-table patterns; scan all
tracked files not just `*.env`. Run it clean as a Phase-1 exit check.
5. `.gitignore` additions: `.claude/`, `*.key`, `*.pem`, `id_rsa*`, `*.sqlite`, `*.db`
(`.claude/settings.local.json` with creds is currently only ignored by a machine-global rule).
6. Product-security note to raise (not fix now): `password123` is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced.
## Phase 2 — Repo restructure: deletions, binaries, layout
Delete (each its own commit):
- **`.planning/` — 199 tracked files, not in the original plan.** GSD phase/session material
(RESUME notes, phase SUMMARYs, `.continue-here.md`); three of them held the fleet password.
Same class as `loop/` and `.agents/`: internal agent working state, not product. Decide
delete-vs-`docs/history/` explicitly — it is the largest un-triaged block of internal
material still tracked.
- `loop/` (AI overnight harness w/ node SSH lines), `.agents/`, `.codex`, `.githooks/pre-push`
(the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history).
- `indeedhub/` submodule + `.gitmodules` entry (points at private HTTP Gitea, breaks `--recursive`
clones); `indeedhub-demo/` (single Dockerfile — merge or drop).
- `RELEASE-NOTES-v1.0.0.md` (superseded by CHANGELOG), `neode-ui/docs/GAMEPAD-NAV-MAP.md` (duplicate of `docs/GAMEPAD-NAV.md`).
- Stray generated HTML: `docs/container-architecture.html` (311 KB), `docs/archive/architecture-review.html`, `docs/archive/lora-functionality.html`.
- `Android/local.properties` from tracking (local absolute path); remove `Android/app/debug.keystore` (standard practice).
Move out of git (→ release assets on the Releases page, referenced by URL):
- `neode-ui/public/packages/archipelago-companion.apk` (27 MB), `wireguard.apk` (17 MB), `atob.s9pk` (23 MB).
- `Android/archipelago-0.3.0-debug.apk.zip` (16 MB, stale).
- `demo/content/music/*` + heavy `demo/aiui/assets` (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size).
- `neode-ui/dev-dist/` (generated Workbox output) → gitignore.
Rename/fix the naming lie: `image-recipe/_archived/` contains the *production* ISO builder
(`build-auto-installer-iso.sh`, referenced by `.gitea/workflows/build-iso.yml`). Move live
files up into `image-recipe/`, delete the genuinely archived rest.
## Phase 3 — Registry domain + parameterization (functional blocker)
Infra (user assists: DNS + TLS):
- Put a domain (e.g. `registry.archipelago-os.org` / `git.archipelago-os.org`) with HTTPS in
front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS.
Repo changes:
- Introduce a single source of truth for the registry host (e.g. `REGISTRY_HOST` in
`scripts/lib/` + a default in the orchestrator config). Replace `146.59.87.168:3000` in:
all 56 `apps/*/manifest.yml`, `app-catalog/catalog.json`, `releases/manifest.json`,
`release-manifest.json`, the 11 scripts (`self-update.sh`, `create-release.sh`,
`generate-app-catalog.sh`, `validate-app-manifest.sh`, `first-boot-containers.sh`, …),
both `demo-images.yml` workflows, `demo-deploy/.env.example`, and the Android sources
(`FipsPreferences.kt`, `PartyScreen.kt`).
- Because the catalog is signed: regenerate + re-sign + republish the app catalog after the
manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise).
Signing needs the user's mnemonic → schedule one ceremony after manifests are final.
- Verify: fresh machine with no LAN/tailnet access can `podman pull` one app image via the
domain and the gate node still installs apps after the re-signed catalog lands.
## Phase 4 — Documentation overhaul
### 4a. Community/legal files (missing today)
- `LICENSE` — MIT (matches existing README badge). Add `[workspace.package] license` +
`license.workspace = true` in the 5 member Cargo.tomls (also see Phase A4).
- `SECURITY.md` — disclosure address, PGP key, supported-versions; cite the March 2026 audit (`docs/archive/security-code-audit-2026-03.md`).
- `CODE_OF_CONDUCT.md` — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today).
- `CONTRIBUTING.md` edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb
the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model,
non-destructive migrations), versioning policy note for the `-alpha` scheme.
- `CLAUDE.md` — rewrite: keep invariants/build-verify (public-worthy), remove status banner,
node numbers, `gitea-ai` push mechanics, MEMORY references (those move to private notes).
### 4b. New developer docs (the three real gaps for app developers)
1. **`docs/quadlet-compilation.md`** — how a manifest becomes a Quadlet/systemd unit: naming,
`systemctl --user` lifecycle, where units land, how to inspect/debug one. (Source:
`core/archipelago/src/container/quadlet*.rs`, prod_orchestrator.)
2. **`docs/container-lifecycle.md`** — the 30 s level-triggered reconciler, install/adopt/
restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped
`docs/bulletproof-containers.md` as the current description; salvage its content.)
3. **`docs/secrets.md`** — `generated_secrets` declaration → materialisation by
`container::secrets` (0600, rootless) → injection; what developers must never do.
- Also: make every example in `docs/app-developer-guide.md` + `apps/*/manifest.yml` copy-paste
work against the new public registry host; add an end-to-end "write your first app" walkthrough
that a stranger can follow with only the public repo + an Archipelago node.
### 4c. Sanitize-and-keep internal docs (user's transparency choice)
- Keep, after Phase-1 scrubbing: `docs/PRODUCTION-MASTER-PLAN.md`, `docs/UNIFIED-TASK-TRACKER.md`,
`docs/1.8.0-RELEASE-HARDENING-PLAN.md`, `docs/RETICULUM-TRANSPORT-PROGRESS.md`, HANDOFF-*, test
plans, `docs/archive/*` — but **move all session/handoff/tracker material under
`docs/history/`** (extending the existing honest `docs/archive/README.md` pattern) so the
top-level `docs/` reads as current reference only. Add a banner to each: "historical working
document, sanitized; not maintained."
- Remove dangling agent-memory references in tracked docs (`docs/bulletproof-containers.md`,
`docs/RETICULUM-TRANSPORT-PROGRESS.md`, `docs/registry-manifest-design.md`,
`docs/bitcoin-multi-version-design.md` progress block).
- De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status
field; e.g. `docs/APP-PACKAGING-MIGRATION-PLAN.md` → public app-platform design doc).
- Extract North-Star narrative from PRODUCTION-MASTER-PLAN into `docs/ROADMAP.md`; extract
the "run the gate ON the node" philosophy from `docs/multinode-testing-plan.md` into
`tests/lifecycle/TESTING.md`.
- Add `docs/README.md` index (bitcoin/bitcoin `doc/` style): Getting started / Architecture /
App development / Operations / Design docs (ADRs) / History.
- README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized
docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at
CONTRIBUTING.md only.
## Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit)
### A. Zero-risk deletions & metadata (S each, own commits)
- **A1** Delete orphan non-compiling StartOS crates: `core/models`, `core/helpers`,
`core/js-engine` (incl. 2 committed `JS_SNAPSHOT.*.bin`), `core/container-init` (~4,100 LOC,
zero references). Verify: `cargo build --workspace && cargo test --all-features`.
- **A2** Delete unreferenced Vue components: `neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue`. Verify: `npm run type-check && npm run build`.
- **A3** Fix `.gitignore` lockfile lines (7: `Cargo.lock`, 15: `package-lock.json`) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles.
- **A4** LICENSE + Cargo license fields (see 4a). Verify with `cargo metadata`.
- **A5** `core/rust-toolchain.toml` pinning `1.95.0`; align `.github/workflows/ci.yml` (remove explicit `toolchain: stable` input so the file wins). Upgrades become deliberate PRs.
- **A6** `core/rustfmt.toml` codifying **defaults only** (`edition = "2021"` + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify `cargo fmt --all -- --check` yields no diff.
### B. CI guards (zero runtime risk)
- **B1** Enable vitest in CI: run `cd neode-ui && npm run test` locally; fix trivial failures, `.skip`+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration).
- **B2** Raw podman/systemctl **ratchet, not migration**: the 132 raw `Command::new("podman"/"systemctl")` sites use subcommands the `core/container/src/podman_client.rs` wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical `install.rs`, and the prod path intentionally uses Quadlet+systemctl. Add `scripts/ci/raw-podman-ratchet.sh` (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design.
### C. Clippy suppression trim (`core/archipelago/src/main.rs:8-18`, per-lint commits)
- Remove cheaply: `assertions_on_constants`, `drop_non_drop`, `wildcard_in_or_patterns`, `doc_lazy_continuation`, `enum_variant_names` (targeted allows on serde enums — never rename wire variants).
- Own careful commit: `unused_io_amount` — a **correctness** lint; fix sites with `read_exact`/`write_all` or documented targeted allows (`mesh/serial.rs:456,496` has raw partial reads; serial framing may be intentional). Full test suite + gate after.
- Keep crate-wide with justifying comment: `too_many_arguments`, `type_complexity`; attempt `ptr_arg` (`&Vec<T>`→`&[T]`, mechanical) if time allows — first to cut.
- Verify each: `cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features`.
### D. dead_code lift — Tiers 1–2 pre-launch, Tier 3 → commented allows + issues
Per-module procedure (one file per commit): remove `#![allow(dead_code)]` → `cargo check
--all-targets --all-features` → triage each warning: (a) genuinely dead → delete;
(b) future-feature/protocol-mandated → targeted `#[allow(dead_code)] // TODO(#NNN): …`;
(c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) →
clippy `-D warnings` + tests → commit.
- **Tier 1 (small/leaf, S each):** `swarm/seed_advert.rs`, `transport/{mesh_transport,lan,chunking,delta}.rs`, `mesh/{crypto,alerts,types,outbox}.rs`, `streaming/mod.rs`, `wallet/mod.rs`.
- **Tier 2 (M each):** `fips/{mod,iface,dial}.rs` (41 external refs → little residual deadness), `mesh/{x3dh,ratchet,steganography,message_types}.rs` — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why).
- **Tier 3 (defer, riskiest):** `mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs`, `transport/mod.rs` — change each blanket allow to `#![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN`.
- Optional S/M win: move `prod_orchestrator.rs`'s 5,034-line `#[cfg(test)]` module to a sibling file via `#[path]` (pure move, halves the 6,291-line file).
### E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated)
Legacy installers for immich/btcpay/mempool/indeedhub (`core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498`, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging `INSTALL ORCH SKIP` (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920).
1. Run the full gate on the node; grep install logs for `INSTALL ORCH SKIP`.
2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + `adopt_stack_if_exists`; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue.
3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers".
### Explicitly deferred → public tracked issues at launch
PodmanClient API extension + call-site migration; god-module splits (`install.rs`, `update.rs`, `mesh/mod.rs`); Playwright in CI; Tier-3 dead_code triage; `password123` default hardening.
## Phase 6 — Fresh-history publish
1. Freeze: all phases merged on internal `main`, gate green, catalog re-signed.
2. Build the public tree: `git archive`-style export of HEAD (never copy `.git/` — it holds
credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"),
optionally preserving CHANGELOG.md as the human-readable history.
3. **Rotation sign-off (blocking):** confirm every Phase 0 item is rotated/revoked — both
Anthropic keys dead, fleet SSH password changed fleet-wide, Gitea `ai` password rotated,
remotes moved off URL-embedded creds. Do not proceed on "it's scrubbed" — scrubbed and
rotated are different things, and only rotation covers the private history.
4. Pre-publish gate on the export: `scripts/audit-secrets.sh` (fixed version) clean; grep-zero for
`sk-ant-`, rotated-password strings, `146.59.87.168`, tailnet `100.` IPs, `192.168.1.`,
internal hostnames; `du -sh .git` sanity (< ~100 MB); fresh `git clone` + `cd core && cargo build`
+ `cd neode-ui && npm ci && npm run build` on a clean machine/container; one app image pull
from the public domain.
5. Publish to GitHub; enable issue templates (already present in `.github/`); file the deferred-work
issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points.
6. Internal repo remains the private full-history remote; decide sync direction post-launch
(recommend: public repo becomes canonical, private keeps only ops/infra notes).
## Verification (end-to-end)
- `tests/lifecycle/run-gate.sh` green on the node after Phases 3 + 5E (and after any lifecycle-touching commit).
- CI green on every phase commit: `cargo fmt --check`, `clippy -D warnings`, `cargo test --all-features`, frontend type-check + build + (new) vitest.
- Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer.
- Docs acceptance: a reader following `docs/app-developer-guide.md` + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra.
## Sequencing / cut-line
Order (revised 2026-08-07): 1 → 2 → (3 ∥ 4) → 5 (A→E) → **0** → 6. Phase 0 rotation now sits
immediately before publish as a blocking gate rather than first; see the Phase 0 banner for why
that is safe under fresh-history. Phases 1–2 are non-negotiable security; Phase 3 is the
functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any
commit (minimum viable: A1–A6, B1–B2, unused_io_amount fix); Phase 6 last. If the timeline
compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds.
docs/PRODUCTION-MASTER-PLAN.md
-1265
View File
File diff suppressed because it is too large Load Diff
docs/RETICULUM-TRANSPORT-PROGRESS.md
-399
View File
@@ -1,399 +0,0 @@
# Reticulum mesh transport — progress tracker
Living status doc for the Reticulum (RNS+LXMF) third-transport work. **Update this after every
meaningful step.** If a session is cut off mid-work, read this file first, then the plan, then
resume at "Next up."
Full plan: `.claude/plans/enchanted-strolling-rocket.md`. Memory pointer:
`project_reticulum_transport_plan.md` (auto-memory index).
**Coordination note (2026-06-30):** a separate agent owns concurrent Meshtastic work, scoped to
`mesh/meshtastic.rs` + `mesh/protocol.rs` (see `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`) and explicitly
avoiding `mesh/listener/session.rs` transport plumbing + `mesh/mod.rs` routing, which this work
owns. Stay out of `meshtastic.rs`/`protocol.rs` to avoid collisions.
## Checkpoint 2026-07-28 — RNode connect + names FIXED, live-verified E2E (read this first)
The fleet reflash back to RNode firmware exposed a stack of bugs that made Reticulum
unusable on CP2102-bridged boards (Heltec V3 etc.) and left every archy node nameless on
RNS. All fixed in `a8c4694c` (backend) + `3f76b496` (UI), live-verified on archi-dev-box
and archy-x250-dev with a real RNode-to-RNode LXMF message (`transport: "reticulum"` in
mesh-messages) plus a cross-transport reply:
1. **probe_rnode boot race** — serial open pulses DTR/RTS via the USB-UART bridge → ESP32
power-cycles → KISS DETECT written 300ms later is eaten during ~2.5-3s of boot. Fix:
immediate probe (fast path) + drain-until-quiet boot settle + second DETECT window.
2. **configure() was a no-op on a running listener** (only enable/disable restarted it) —
the setup modal's apply/keep-as-is and every rename did nothing until process restart.
3. **Name propagation** — `config.advert_name` had no reader; `server.set-name` never
reached mesh; daemon display name fixed at spawn to the "Archy" default; the ARCHY:2
announce blob REPLACED the LXMF name. Now: announces carry msgpack
`[name, stamp_cost, sf, ARCHY-blob]` (Sideband-compatible, blob invisible to stock
clients), daemon has a `set_name` verb, renames bounce the session live.
4. **Daemon-death detection** (was invisible up to the 30-min RX-stall watchdog),
**modal re-trigger loop** (plugged_at used tty mtime → bumps on every open; now
btime/ctime), **ARCHY:2 federation-name clobber**, **mesh.refresh RPC** (Refresh button
now actually re-queries the radio), **Meshtastic mesh.broadcast now sends NodeInfo**.
Still open here: legacy-format peers (old fleet builds) show as `Reticulum <hex4>` until
they OTA; RNode RF params still daemon-hardcoded (EU-868 869.525/125k/SF8/CR5); Phase 4
multi-radio; duty-cycle guard.
## Status at a glance
| Phase | What | Status |
|---|---|---|
| 0 | Gate #1 — deterministic identity from Archy keys | ✅ **DONE**, verified in venv AND in the PyInstaller binary (same dest hash) |
| 0 | Gate #2 — two-node LXMF-over-LoRa on real hardware | ✅ **PASSED 2026-06-30** — real RF announce + encrypted DM exchanged between .116's Heltec V3 RNode and a phone-flashed second RNode running Sideband |
| 0 | Gate #3 — external Sideband/MeshChat interop | ✅ **PASSED 2026-06-30** — same session as gate #2; Sideband is the stock external client this gate calls for |
| 1 | `reticulum-daemon/` (Python rns+lxmf, Unix-socket RPC) | ✅ scaffolded + tested (no radio); signed-identity announce **also done** (see below) |
| 1 | Packaging — PyInstaller single binary | ✅ **DONE + verified** — `reticulum-daemon/build.sh`, 16M standalone binary, selftest passes run from `/tmp` with no venv on PATH |
| 2 | Rust wiring (`DeviceType`, `MeshRadioDevice`, `ReticulumLink`, stamp sites) | ✅ **`cargo check`/`cargo test -p archipelago` GREEN** (99 mesh tests pass) — still untested on real hardware |
| 2c | `MeshConfig.device_kind` reflashable-board pin | ✅ **DONE** this session (was the one open Phase-2 item) |
| 3 | Frontend (~8 label/CSS spots) | ✅ DONE (scoped down — see note below) |
| 4 | Multi-device (run all 3 radios at once) + per-network channels | ⏳ not started (follow-on, after 0–3) |
| 5 | Aurora interop — optional plain-TCP Reticulum interface (radio-less) | ✅ **DONE + verified 2026-07-03** — see checkpoint below. Real Aurora GUI test still open (manual follow-up). |
## Checkpoint 2026-06-30 (late session — read this first if cut off)
This session picked up after Phase 2/3 were already green, and closed out everything that didn't
need real RNode hardware:
1. **Corrected two stale tracker entries** (both were already done, just not reflected here):
- The `_announce_app_data` "TODO" was actually already implemented:
`reticulum_daemon.py`'s `_announce_app_data()` embeds `ARCHY:2:{ed}:{x25519}` when
`--archy-ed-pubkey-hex`/`--archy-x25519-pubkey-hex` are passed, and `reticulum.rs`'s
`daemon_command()`/`open()` already forward `our_ed_pubkey_hex`/`our_x25519_pubkey_hex` from
`session.rs` (`run_mesh_session` → `auto_detect_and_open`/`open_preferred_path` →
`ReticulumLink::open`). Confirmed end-to-end by reading the call chain, not just grepping.
- Phase 3 frontend was already done (see prior entry below) — tracker table above said
"not started", now corrected.
2. **Added `MeshConfig.device_kind: Option<DeviceType>`** (plan §2c, the one explicitly-listed
open Phase-2 item) — `mesh/mod.rs` (field + Default + threaded into `start()`'s
`spawn_mesh_listener` call), `listener/mod.rs` (`spawn_mesh_listener` param → `run_mesh_session`
arg), `listener/session.rs` (`run_mesh_session` param; `auto_detect_and_open` skips
non-matching probes per-path via `device_kind.is_none_or(|k| k == ...)`;
`open_preferred_path` restructured to a `match kind { ... }` that tries **only** the pinned
driver and surfaces its real error, instead of silently falling through to another firmware's
handshake on the same port). `None` (default) preserves today's strict
Meshcore→Meshtastic→Reticulum auto-detect — fully backward compatible, no config migration
needed. `cargo check` + `cargo test -p archipelago` both green after (99 mesh tests, 0 failed).
3. **Built and verified the PyInstaller packaging** (plan's Phase 1 "Packaging" + the file list's
"Ops: release packaging to include the daemon binary" item — previously undone):
- `reticulum-daemon/build.sh` (new) — reproducible build, installs `requirements-build.txt`
(new, `pyinstaller==6.21.0`, build-only/not shipped) into the existing `.venv`, runs
PyInstaller with flags discovered by trial: `--collect-submodules RNS --collect-submodules
LXMF --collect-data RNS -d noarchive`.
- **Non-obvious gotcha, written up in `build.sh`'s comments so it isn't re-discovered:**
`RNS.Interfaces/__init__.py` builds its `__all__` via `glob.glob(os.path.dirname(__file__) +
"/*.py")` at import time (`Reticulum.py` does `from RNS.Interfaces import *`). PyInstaller's
default `--onefile` zips pure-Python modules into an in-binary PYZ archive, so `__file__`
doesn't point at a real directory and the glob comes back empty → `NameError: name
'Interface' is not defined` the moment `RNS.Reticulum(...)` is constructed. `-d noarchive`
(keep modules as loose `.pyc` files on disk inside the onefile bundle's runtime-extraction
dir) fixes it — confirmed by reproducing the failure first, then fixing it.
- **Verified, not just built:** ran the resulting `dist/archy-reticulum-daemon` binary's
`--check` (dest hash matches the venv-derived `06bb31e16f4f8d46a8ae8eac23a4fd21` for the
test seed) and `--selftest` (full RNS+LXMF bring-up, no radio) **both from `/tmp` with the
binary copied away from the repo and the `.venv` not on `PATH`** — confirms it's genuinely
self-contained, not accidentally still depending on the dev venv.
- `dist/`/`build/`/`*.spec` are already gitignored (`reticulum-daemon/.gitignore`); only
`build.sh` + `requirements-build.txt` are new tracked files.
**NOT done this session (still genuinely open):**
- Everything hardware-dependent (Phase 0 gates #2/#3, real RNode probe/spawn). The .116 Heltec V3
reflash mentioned in the prior session's memory was **not** done in this session — no physical
hardware access was exercised, only software.
- `/dev/reticulum-radio` udev symlink (plan §2c) — **deliberately not added**: the existing
`99-mesh-radio.rules` keys on USB vendor/product ID (e.g. CP2102 0x10c4/0xea60), but the whole
point of `device_kind` is that the *same* chip can run any of the three firmwares — a
vendor/product udev rule can't disambiguate them, and a fabricated rule would just be
misleading. Real fix needs either a per-device `ATTRS{serial}==...` rule the operator fills in
once they know their specific board's serial (no such board exists in-repo to template from
yet), or rely on `device_kind` alone (already done, works regardless of `/dev` path naming).
Revisit once a real RNode-flashed board's serial is known.
- PyInstaller binary not yet wired into the release tarball / `scripts/deploy-to-target.sh` (the
daemon binary path is currently resolved via `ARCHY_RETICULUM_DAEMON_BIN` env or the dev venv
fallback in `reticulum.rs`'s `daemon_command()` — production default
`/usr/local/bin/archy-reticulum-daemon` is a real path convention now that `build.sh` produces
exactly that filename, but nothing copies it there yet). Left undone deliberately — wiring
release-tarball plumbing for a binary that's never been run against real RNS network traffic
felt premature; do this once Phase 0 gates #2/#3 pass.
## Phase 2 — Rust wiring detail (what's done vs left)
**Done — `cargo check -p archipelago` is GREEN:**
- `core/archipelago/src/mesh/types.rs` — `DeviceType::Reticulum` (+ `Display` arm) + a
`radio_transport_label(DeviceType) -> &'static str` helper (`"reticulum"` vs `"lora"`).
- `core/archipelago/src/mesh/mod.rs` — all 4 outbound stamp sites use
`radio_transport_label(...)`; `use_typed_envelope` (~1571) extended to
`matches!(device_type, Meshcore | Reticulum)`; `data_dir` threaded into
`spawn_mesh_listener(...)` call (was: `MeshService::start()` → `spawn_mesh_listener`).
- `core/archipelago/src/mesh/listener/mod.rs` — `spawn_mesh_listener` takes `data_dir:
PathBuf`, passes `&data_dir` into `run_mesh_session`.
- `core/archipelago/src/mesh/listener/decode.rs:406,639` and `dispatch.rs:79` — all 3 inbound
stamp sites now use `radio_transport_label(state.status.read().await.device_type)`.
- `core/archipelago/src/mesh/listener/session.rs`:
- `MeshRadioDevice` enum has `Reticulum(ReticulumLink)`; all 18 method arms wired (no-ops:
`ensure_lora_region`, `ensure_channel`, `send_keepalive`, `send_nodeinfo_advert`, `reboot`,
`reset_contact_path`; everything else forwards to `ReticulumLink`).
- `auto_detect_and_open(data_dir: &Path)` and `open_preferred_path(path, data_dir: &Path)`
both now try `ReticulumLink::open(path, data_dir)` **last**, after Meshcore/Meshtastic —
cheap raw-serial KISS-detect probe runs first; the daemon only spawns on a confirmed match.
- `reticulum_contact_id()` helper added (delegates to the canonical
`reticulum::reticulum_contact_id_from_hash`, masked `& 0x7FFF_FFFF`, avoids 0).
- `refresh_contacts()` has an `is_reticulum` branch parallel to `is_meshtastic`; `reachable`
flows through `contact.path_len != 0` unchanged (`ReticulumLink::get_contacts()` already
encodes daemon-reported reachability into `path_len`).
- `data_dir: &Path` threaded through `run_mesh_session` → both probe functions.
- `core/archipelago/src/mesh/reticulum.rs` — **created**. `ReticulumLink`: spawns/supervises the
daemon as a child process, Unix-socket RPC client (matches the tested daemon contract),
`prefix_to_hash: HashMap<[u8;6],[u8;16]>` (mandatory per the plan), synthetic
`InboundFrame` builder byte-matching `meshtastic.rs`'s layout, `Drop` impl that kills the
daemon + cleans up the socket. Has unit tests (KISS-detect byte matching, contact-id masking,
synthetic-frame layout) — **passing, see below**.
**Concurrent-edit note:** a separate in-flight change (not mine) added `MeshPeer.pkc_capable`
and `ParsedContact.pkc_capable` (Meshtastic PKI-capability tracking) while this work was in
progress. Accounted for: `reticulum.rs`'s `ParsedContact` literal sets `pkc_capable: false`
(Reticulum/LXMF is unconditionally E2E via `take_rx_encrypted()`, this field has no analogue);
two incomplete `MeshPeer` literals in `decode.rs` (lines ~330, ~548) were completed with
`pkc_capable: false` to unblock the build for everyone — not reverted, not worked around.
**Self-review fix applied:** the RPC Unix socket originally lived in the shared system temp
dir; moved to `{data_dir}/reticulum/` (0700) instead — archipelago-owned, not shared `/tmp`,
matching the security posture. Re-confirmed `cargo check -p archipelago` GREEN after the move.
**NOT yet done:**
- `MeshConfig.device_kind: Option<DeviceType>` hint (optional reflashable-board disambiguator,
plan §2c) — not added. Auto-detect ordering (Meshcore→Meshtastic→Reticulum, strict probes)
is the only disambiguator right now.
- Phase 3 frontend — **DONE**, but **smaller scope than originally inventoried**: only
`Mesh.vue`'s `transportLabel()` (per-message field) + `mesh-styles.css` `.transport-reticulum`
+ the `mesh.ts` doc comment needed the addition. `transport.ts` `TransportKind`,
`federation/types.ts` `last_transport`, `NodeList.vue` `transportBadge`, and `PeerFiles.vue`
`transportPill` are a COARSER routing-layer category (`mesh`/`lan`/`fips`/`tor`) where
`'mesh'` already covers any radio (meshcore/meshtastic/reticulum) — adding a separate
`'reticulum'` there would be inconsistent with how meshcore/meshtastic are handled. Confirmed
via `vue-tsc --noEmit` (exit 0, zero errors).
- Everything hardware-dependent: real daemon spawn/probe against an actual RNode (the .116
Heltec V3, once reflashed), two-node LXMF-over-LoRa, the `_announce_app_data` signed-identity
TODO in the daemon (currently carries only the plaintext display name, not a verified Archy
DID/pubkey — needed for `bind_federation_twins`-style auto-binding across protocols).
## Verified facts to reuse (don't re-derive)
**RNode KISS-detect handshake** (confirmed against the canonical Reticulum source, not guessed):
```
constants: FEND=0xC0 FESC=0xDB TFEND=0xDC TFESC=0xDD CMD_DETECT=0x08 DETECT_REQ=0x73 DETECT_RESP=0x46
probe tx: C0 08 73 C0 50 00 C0 48 00 C0 49 00 C0 (detect + fw_version + platform + mcu queries)
success: response contains byte sequence ... C0 08 46 ... (FEND, CMD_DETECT, DETECT_RESP)
```
Source: `RNS/Interfaces/RNodeInterface.py` (Liberated Systems mirror), `detect()`/`readLoop()`.
**Synthetic `InboundFrame` layout** for a 1:1 DM, copied exactly from
`meshtastic.rs:1031-1047` (`ReticulumLink` must build the same shape so `frames::handle_frame`
needs zero changes):
```
data = [snr(1)=0][reserved(2)=00,00][sender_prefix(6)][path(1)=0xff][type(1)=0][rx_time(4 LE)][payload…]
code = RESP_CONTACT_MSG_V3_E2E if encrypted else RESP_CONTACT_MSG_V3 (RNS/LXMF is always E2E, so always _E2E)
```
Channel/broadcast equivalent (`RESP_MESHTASTIC_CHANNEL_TEXT`, meshtastic.rs:1019-1028) — N/A for
Reticulum in single-device Phase 2 (LXMF has no shared-channel concept); revisit in Phase 4.
**`resolve_peer`** (decode.rs:316) matches inbound `sender_prefix` against
`peer.pubkey_hex.starts_with(prefix)` — so as long as `refresh_contacts`/announce-handling
populates `pubkey_hex` = full 16-byte RNS hash hex BEFORE a message arrives (same precondition
meshtastic relies on via its `peer_pubkeys` map), no Reticulum-specific fallback is needed there.
**`ParsedContact.public_key_hex`** for Reticulum = hex of the 16-byte RNS dest hash (32 hex
chars, NOT 32 bytes) — the `hex::decode(...).len()==32` checks elsewhere (e.g. the auto-heal
`reset_contact_path` loop in `refresh_contacts`) will naturally skip Reticulum contacts since
their key decodes to 16 bytes, not 32. That's fine — no special-casing needed, just don't "fix"
it to be 32 bytes.
**`data_dir.join("identity").join("node_key")`** is the 32-byte raw Ed25519 seed file — this is
exactly what `reticulum_daemon.py --identity-key <path>` expects (confirmed against
`identity.rs` `NODE_KEY_FILE`/`load_or_create`). The daemon reads the file itself — Rust should
pass the **path**, not pipe the raw key bytes through more hops than already exist.
## Hardware update (2026-06-30)
**.116 has a Heltec V3 available to reflash with RNode firmware.** This unblocks Phase 0 gates
#2/#3 (previously marked blocked — `.198`'s radio is dead, but .116's Heltec V3 is a real path
forward without needing new hardware). Next concrete step once reflashed: run
`reticulum-daemon/reticulum_daemon.py` pointed at the RNode's serial path, confirm `--check`
hash matches `--selftest`, then bring up two instances (.116 + .228, after .228 also gets an
RNode-capable board) for the real two-node LXMF-over-LoRa gate.
## Daemon contract (already built + tested — Phase 2 codes against this, no changes needed)
`reticulum-daemon/reticulum_daemon.py`, RPC over Unix socket (0600), one JSON object per line:
- in: `{"cmd":"send","dest_hash":hex16,"content":...}` / `{"cmd":"announce"}` /
`{"cmd":"status"}` / `{"cmd":"shutdown"}`
- out: `{"event":"ready",...}` / `{"event":"recv",...}` / `{"event":"announce",...}` /
`{"event":"delivered",...}` / `{"event":"status",...}`
Verified: `--check` (hash only), `--selftest` (boots real RNS+LXMF, no radio), and a live
socket round-trip (`ready`→`status`→`shutdown`, clean exit) — see `reticulum-daemon/README.md`.
## Checkpoint 2026-06-30 (hardware session — gates #2/#3 PASSED)
Picked up after a session pipe-break; the live system (archipelago.service + the spawned
`archy-reticulum-daemon`) had kept running uninterrupted the whole time, so nothing was lost.
**What happened, in order:**
1. .116's Heltec V3 (CP2102, USB vendor/product `10c4:ea60`, serial `0001`) was reflashed with
RNode firmware and plugged into `/dev/mesh-radio` (generic udev symlink → `ttyUSB0`, not a
per-serial rule). `mesh-config.json` has `device_path: null` — pure auto-detect, no
`device_kind` pin needed.
2. Auto-detect correctly tried Meshcore → Meshtastic → Reticulum and found it: journal shows
`Found Reticulum (RNode) device via auto-detect path=/dev/mesh-radio` — but only **after**
~4 min of `Failed to spawn reticulum-daemon — is it installed/packaged?` retries, because
`/usr/local/bin/archy-reticulum-daemon` hadn't been copied into place yet from
`reticulum-daemon/dist/` (built via `./build.sh`). Once copied (sha256-verified match to the
`dist/` build), auto-detect succeeded on the very next retry.
3. `mesh.status` RPC confirmed live: `device_type: "reticulum"`, `device_connected: true`,
`dest_hash: 5d146f6e1c9707f89468b5016ed6dfad`. Periodic self-advert (`send_self_advert` →
`{"cmd":"announce"}` → real RNS `Identity.announce()`) firing every ~30s — confirmed this is
**not** the `send_nodeinfo_advert` no-op arm (that one's still legitimately a no-op for
Reticulum; the real announce path is `send_self_advert`, wired correctly).
4. Second RNode flashed onto a phone running **Sideband**. First attempt showed RF energy
(`interference_last_dbm` climbing) but `rxb: 0` — a parameter mismatch, **not** a frequency
problem (energy was detected, just not demodulated). Root cause: Spreading Factor mismatch
in Sideband's manual RNode interface config (frequency display rounds to one decimal so
"869.5" silently passed at first glance — bandwidth/SF/CR are separate fields and SF was
wrong). Once SF was corrected to match (freq `869525000`, BW `125000`, **SF `8`**, CR `5`),
`rxb` went non-zero immediately and a real `{"event":"announce","dest_hash":"1870744d...",
"app_data":"7a617a61"}` (hex for "zaza") arrived over the air.
5. **Gate #2 + gate #3 both passed in the same exchange**: `zaza` shows up as a real, reachable
`mesh.peers` contact; an inbound encrypted LXMF message ("Yoooo") arrived and was correctly
stamped `encrypted: true, transport: "reticulum"`; a reply was sent back and round-tripped.
Sideband is exactly the stock external client gate #3 calls for, so one real RNode-to-RNode
LoRa link covered both gates — no need for a second dedicated archy node.
6. **Two real bugs found from this, both fixed:**
- `record_sent_typed`'s `encrypted` flag was hardcoded `false`/`archy || pkc_capable` on the
Reticulum send path (both the native-text path in `send_message` and the typed-envelope
path in `send_typed_wire`) — correct for Meshcore/Meshtastic (where E2E really is
conditional on PKI/session state not yet threaded through), **wrong** for Reticulum: LXMF
encrypts every send to the destination identity key unconditionally, archy peer or not.
Fixed: both call sites now OR in `device_type == DeviceType::Reticulum`.
- `radio_transport_label()` collapsed Meshcore **and** Meshtastic into one generic `"lora"`
string, so the per-message pill couldn't distinguish them. User asked for 3 distinct pill
colors (Meshtastic mint, Meshcore orange, Reticulum blue) — extended the label fn to
return `"meshtastic"`/`"meshcore"`/`"reticulum"` distinctly, updated `Mesh.vue`'s
`transportLabel()` switch and `mesh-styles.css` (`.transport-meshtastic` `#3eb489`,
`.transport-meshcore` `#fb923c`, `.transport-reticulum` `#60a5fa`; kept `.transport-lora`
`#f59e0b` as a fallback for any already-stored legacy-labelled messages). `cargo check` +
`vue-tsc --noEmit` both green after.
**NOT yet done:**
- The Rust-side fix above (`encrypted` flag, transport-label split) is built but **not yet
deployed to .116's running binary** — the live daemon/auto-detect verification above was all
against the binary already running before this session's edits. Rebuild + redeploy to see the
fix live.
- `tests/lifecycle/run-gate.sh` not re-run after these mesh changes yet (project convention:
run after backend changes land).
- Multi-device (3 radios at once, Phase 4) and the release-tarball/udev-rule wiring (originally
"Next up" #6 below) are both still untouched.
## Next up (resume here)
Phase 0 gates #1–#3 are now **all passed**. What's left:
1. Rebuild the backend + frontend and redeploy to .116 so the `encrypted`-flag fix and the
3-way transport-pill color split actually take effect on the live node (currently only
checked in with `cargo check`/`vue-tsc`, not deployed).
2. Re-verify on-device after redeploy: send another Sideband↔archy DM, confirm the Sent bubble
now shows E2E + a blue "Reticulum" pill, and confirm Meshtastic/Meshcore pills (if any
messages exist) render mint/orange instead of the old generic amber "LoRa".
3. Exercise the rest of the plan's "Verification (definition of done)" items: hot-swap
detection (unplug the RNode mid-session, confirm fallback to FIPS/Tor on the same contact;
replug, confirm it picks Reticulum back up), and `device_kind: Some(Reticulum)` pin path
(currently only auto-detect has been exercised on real hardware).
4. Run `tests/lifecycle/run-gate.sh` to confirm no regression from the mesh changes landing.
5. Only after the above: wire `dist/archy-reticulum-daemon` into the release tarball /
`scripts/deploy-to-target.sh` (target path `/usr/local/bin/archy-reticulum-daemon`, matching
`reticulum.rs`'s default) and add a per-serial-number `/dev/reticulum-radio` udev rule now
that a real board's serial number (`0001` on the CP2102, .116's board) is known — though a
second board will likely report the same `0001` stock serial since CP2102 modules commonly
ship with an unprogrammed default, so this may still need a different disambiguator.
6. Phase 4 (run all 3 radios at once) — still not started, follow-on after the above.
## Checkpoint 2026-07-03 — Phase 5: Aurora interop via plain-TCP Reticulum (radio-less)
**Why:** `~/aurora` (a separate Flutter off-grid messenger) already runs real RNS + LXMF
(`LxmfRouter`, comment "interop with Sideband/NomadNet/MeshChat" in `rns_service.dart`), and its
**default** connectivity mode is plain TCP (`RnsTcpInterface`/`RnsTcpServerInterface`), not radio —
it ships a static bootstrap list of public RNS hubs on port 4242. Archy's daemon could previously
only bring up a serial-RNode interface, so it was unreachable by Aurora (or any TCP-based RNS/LXMF
client) at all, and every interop proof was bottlenecked on scarce LoRa hardware. This phase adds
an **optional, additive, loopback-only plain-TCP interface**, proves interop with a scripted
RNS/LXMF stand-in (the same class of proof the Sideband gate already established), and leaves the
serial/RNode path completely unchanged.
**Done, all verified:**
1. `reticulum-daemon/reticulum_daemon.py` — `_write_rns_config()` gained a third branch
(`--tcp-listen HOST:PORT` → `TCPServerInterface`, `--tcp-connect HOST:PORT` repeatable →
`TCPClientInterface`), mutually exclusive with `--serial-port`. `--tcp-listen` is hard-gated to
loopback (`_require_loopback`) — archy is otherwise Tor-first for inter-node traffic, so a
WAN/LAN-exposed Reticulum port is a deliberate future decision, not something this phase does
silently. Verified: `--selftest` regression still passes; two daemon processes (server +
client, throwaway identities) reached `connected: true` on both sides via `mesh.status`-daemon
RPC, live `TCPServerInterface`/`TCPClientInterface` visible in `get_interface_stats()`.
2. **Bidirectional LXMF DM gate against a scripted Aurora stand-in** (Python RNS+LXMF client
dialing as a `TCPClientInterface` + running its own `LXMRouter` — a legitimate protocol-level
proxy for Aurora's Dart stack, same wire format): forward (stand-in → archy daemon) and reverse
(archy daemon → stand-in) both delivered with matching content and correct source/dest hashes,
confirmed via the daemon's own `recv`/`delivered` RPC events. Direct TCP analogue of the
already-passed Sideband gate (RF → TCP, Sideband → scripted stand-in).
3. **Rust wiring**, fully additive — the serial/RNode path is byte-for-byte unchanged:
- `mesh/reticulum.rs`: new `ReticulumInterface` enum (`Serial`/`TcpServer`/`TcpClient`) threads
through `daemon_command()`/`spawn()`; `open()` (serial) now just wraps
`ReticulumInterface::Serial` — same `probe_rnode` gate as before. New
`open_tcp_server()`/`open_tcp_client()` associated fns skip `probe_rnode` entirely (the
"spawn without a physical RNode" path); `open_tcp_server` hard-enforces
`is_loopback_host()` (mirrors the Python-side guard).
- `mesh/types.rs`: new `ReticulumTcpConfig` enum (`Server { bind }` / `Client { connect }`).
- `mesh/mod.rs`: `MeshConfig.reticulum_tcp: Option<ReticulumTcpConfig>` (`#[serde(default)]`,
`None` by default — no migration, zero behavior change when unset); threaded into
`start()` → `spawn_mesh_listener`.
- `listener/mod.rs` / `listener/session.rs`: `reticulum_tcp` param threaded through
`spawn_mesh_listener`/`run_mesh_session`; new leading branch — if set, a new
`open_reticulum_tcp()` helper dispatches to `open_tcp_server`/`open_tcp_client`; otherwise
falls through to the **untouched** existing `preferred_path`/`auto_detect_and_open` logic.
- Deliberately **not** wired into `mesh.configure`/the frontend — dev/verification-only surface
for now (hand-edit `mesh-config.json`), consistent with how narrowly scoped this phase is.
- `cargo check -p archipelago` + `cargo test -p archipelago` (mesh module): **108 passed, 0
failed, 1 ignored** (the pre-existing hardware-gated `probe_rnode_detects_real_hardware`) —
zero regression to the serial/RNode path, provable without any hardware.
4. **End-to-end Rust integration test** (`mesh::tests::mesh_service_connects_over_reticulum_tcp_client`,
`#[ignore]`d — spawns real subprocesses, skipped in the default `cargo test` run the same way
the rest of the mesh suite skips hardware-gated tests): a real `MeshService::start()` spawns the
daemon in TCP **client** mode (no serial probe at all), dials a second stand-alone daemon
instance in TCP **server** mode (the Aurora-side role), and reaches `device_connected: true` /
`device_type: Reticulum` via the exact `MeshService::status()` call the `mesh.status` RPC uses.
Passed in ~2.6s. Run manually: `cargo test -p archipelago -- --ignored
mesh_service_connects_over_reticulum_tcp` (needs `reticulum-daemon/.venv`, see below).
**Environment note:** this session's Rust toolchain drift — system `rustc` (apt, 1.85.0) is too
old for code already on `main` (`u32::is_multiple_of` in `health_monitor.rs`, stabilized upstream
after 1.85); a pre-installed rustup toolchain at
`~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu` (1.96.0) builds clean. Not something this
phase's changes caused — pre-existing, just newly hit. Put that toolchain's `bin/` first on `PATH`
if `cargo check`/`test` reports `E0658 unsigned_is_multiple_of`.
**Explicitly NOT done (out of scope for this phase, see plan non-goals):**
- Real Aurora Flutter GUI verification — this dev sandbox has no `flutter`, no `$DISPLAY`, and no
`reticulum-dart` sibling checked out (Aurora's actual RNS implementation lives in that separate
repo; Aurora's CI clones it fresh at build time). The scripted-stand-in gate above is the
protocol-level substitute. **Manual follow-up**: point a real Aurora build's TCP hub list (or an
ad hoc connect) at an archy node's `--tcp-listen` address and confirm an LXMF DM in the actual
app UI.
- Any non-loopback (LAN/WAN) TCP bind — hard-gated off on purpose; a real "Aurora hub" deployment
needs its own security review given archy's Tor-first posture for inter-node traffic.
- LXMF propagation-node / always-on-hub role for archy (bridging Aurora's offline BLE peers) —
bigger architectural + storage commitment.
- Identity unification between archy's and Aurora's independent Nostr/secp256k1 keys — both
already have separate Nostr identities with no derivation link; out of scope here.
- `mesh.configure` RPC / frontend exposure of `reticulum_tcp` — stays hand-edit-only until/unless
it becomes user-facing.
docs/SEED-VERIFICATION.md
+2 -2
View File
@@ -371,12 +371,12 @@ RPC alternative (from any machine on the LAN):
```bash
# Node identity
curl -s http://192.168.1.228/api/rpc \
curl -s http://archipelago.local/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.get-node"}' | jq .
# All identities
curl -s http://192.168.1.228/api/rpc \
curl -s http://archipelago.local/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.list"}' | jq .
```
docs/UNIFIED-TASK-TRACKER.md
-435
View File
@@ -1,435 +0,0 @@
# Unified Task Tracker — OTA 1.8.0 + Master Plan
Single working list for everything left before 1.8.0 ships and the next master-plan
exit criteria (multinode + workstreams B/C/D) are met. Supersedes the open-task
sections of `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md`
as the day-to-day tracker — those docs remain the historical record / detailed
narrative and are still linked from here where useful. **Ordered fastest/simplest
first** so we work top-down instead of hunting across docs.
Verified against actual code state on 2026-07-01 (not just doc text — several
items the source docs still listed as "open" turned out to already be shipped;
those are marked ✅ below with the commit that did it, so we stop re-litigating them).
---
## Tier 0 — Quick / mechanical, no blockers
- [ ] **Ship the lightning payment false-failure fix in the next release** (fixed
on main 2026-07-27, needs OTA). Slow multi-hop payments (>15s) surfaced as
"Payment failed" while LND settled them in the background — the shared LND
REST client's 15s timeout aborted the synchronous `/v1/channels/transactions`
wait. Now: payinvoice decodes the invoice first for its payment hash, waits
up to 120s on a dedicated client, returns `status: "pending"` (never a
failure) on timeout, and the new `lnd.paymentstatus` RPC + frontend
`payLightningInvoice()` helper poll to a real terminal state (all 5 UI call
sites migrated). Verify on Framework PT with a real multi-hop payment.
- [ ] **Show the app version on the companion mobile-app banner in the app store
and on its install/pairing modal** (user request 2026-07-27) — so it's
obvious at a glance whether the node is serving the latest APK build.
- [ ] **Optimise the companion QR scan — quicker + better** (user request
2026-07-27; deferred to a later session on purpose). The pairing/scan QR
flow works (user-verified on-device 2026-07-27) but should get faster and
smoother: quicker camera start + decode (scan resolution/framerate,
continuous autofocus), more forgiving in low light / at an angle, and
snappier feedback once the code locks. Touch the native-scan path from
PR #104 and the in-app scan modal together so both benefit.
- [ ] **Update `tests/lifecycle/TESTING.md`'s stale Release Gates checklist** (lines
289–296) — several boxes are unchecked but actually true now:
- #1 bitcoin-stops: covered by `tests/lifecycle/bats/bitcoin-knots.bats` stop/restart
tier, included in the 5/5 green gate run.
- #2 `ARCHY_ITERATIONS=5` on .228: **GREEN 2026-06-23 per CLAUDE.md** — check the box.
- #5 cargo 0 warnings: confirmed 0 warnings on `cargo build --release` (2026-07-01).
- #7 layman changelog: `CHANGELOG.md` is backfilled with layman-readable entries
through v1.8.00-alpha — check the box.
- Leave #3 (multinode), #4 (backend-survives-restart / Phase-3 default-on), #6
(LoC decision), #8 (tag pushed) unchecked — genuinely still open, see Tier 2/3.
- [x] ~~Finish the archival/full-node manifest generalization~~ — investigated 2026-07-01:
the hardcoded fallback names in `dependencies.rs:48-52` (`electrs`, `mempool-electrs`,
`mempool-web`) are legacy **alias** ids for `electrumx`/`mempool`, resolved via
id-mapping in a dozen other places (`install.rs`, `runtime.rs`, `config.rs`, etc.),
not separate un-migrated apps with their own manifests. `electrumx` and `mempool`
themselves already declare `bitcoin:archival`. The fallback is correct as-is —
not tech debt, closing this item rather than risk breaking alias resolution.
- [x] ~~Confirm/close the Portainer image-pin item~~ — confirmed 2026-07-01:
`146.59.87.168:3000/lfg2025/portainer:2.19.4` is present in `podman images` on
all 3 LAN nodes (.116/.198/.228), i.e. actually resolvable/pulled from the mirror.
Not a live bug.
- [x] ~~grafana Quadlet "stuck activating"~~ — checked live on .116 (2026-07-01):
`grafana.service` is `active (running)`, container `Up 2 hours (healthy)`. The
2026-06-21 report is stale for grafana. **strfry still unconfirmed** — not
installed on any of .116/.198/.228 to check directly; low priority until someone
actually needs it installed.
- [ ] **Add `cargo audit` / `cargo deny` to CI, failing on duplicate `rand` majors**
(entropy audit R-05, finding F-07 —
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). `cargo-audit` is not installed
anywhere, so no RustSec check has ever run against this tree. Separately,
`cargo tree` shows **both** `rand 0.8.5` (direct, all first-party key generation)
and `rand 0.9.2` (transitive via `totp-rs` and `tungstenite 0.26.2`) resolved into
one binary. `rand 0.9.0` removed `ThreadRng` fork protection and the orchestrator
forks constantly, so a future bump must be visible rather than silent — add a
`bans` rule so the duplicate majors show up in CI, not in an incident.
- [ ] **Harden the release signing ceremony's mnemonic input** (entropy audit R-08,
finding F-06). `ceremony gen` prints the release master mnemonic to **stdout**
(`core/archipelago/src/ceremony.rs:71-77`) and `load_release_root_key` prefers the
`RELEASE_MASTER_MNEMONIC` **environment variable** over stdin (`:157-160`) — both
leak into shell history, `/proc/<pid>/environ`, tmux scrollback and terminal
recordings. This is the seed that derives the fleet release-root signing key, so a
leak means forged signed manifests fleet-wide. Make stdin/TTY the only supported
input for `sign`/`pubkey`; write `gen`'s output to a `0600` file rather than the
terminal. Small change, but schedule it deliberately — it is the signing ceremony.
- [ ] **Small entropy-audit hygiene batch** (entropy audit R-09 – R-12, R-14 —
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). Five independent one-liners,
each closing a Low/Informational finding:
- Persist the CSPRNG-readiness verdict (`seed.rs:85-91`) as a durable structured
event, so any node can answer post-hoc "was the entropy pool ready when this seed
was born?" — the question Coldcard owners cannot answer today.
- Add a test asserting the `getrandom` crate uses the **blocking** syscall, making
`seed.rs:52-57`'s invariant mechanical instead of a comment.
- Clear `_seed_words` from `sessionStorage` on route-leave from onboarding, not only
on successful verify (`OnboardingSeedVerify.vue:251`), plus a wall-clock expiry
mirroring the server's 10-minute `MNEMONIC_TTL`.
- Replace `% charset.len()` in `totp.rs:305` with `SliceRandom::choose(&mut OsRng)`.
(No bias today — 32 divides 256 — but any future charset edit introduces one
silently. The audit refutes the research's claim that this is currently biased.)
- Comment `pickRandomIndices` (`OnboardingSeedVerify.vue:157`) to record that its
`Math.random()` picks a UX challenge, not key material, so the next auditor does
not re-derive that it is benign.
- [ ] ~~**Swap container `generated_secrets` to explicit `OsRng`** (entropy audit R-13,
finding F-10) — two-line change in `container/secrets.rs:90-102`~~
**SUPERSEDED 2026-08-02 by R-16 / KEY-05.** The audit scoped this at 2 call sites; the
real surface is **41 across 15 files** — see the audit's new §F-10a. `secrets.rs` is 2
of them, and a two-line fix there while 39 other sites inherit the same dependency
default is not a fix.
- [ ] **Crate-wide CSPRNG enforcement — a defaulted RNG cannot be inherited anywhere**
(entropy audit **R-16 / F-10a**, Medium) — tracked as **KEY-05 in Phase 10**, so plan
and execute it there rather than as a standalone item. `session.rs` (16 sites),
`pine_ha.rs` (6), `wallet/bdhke.rs` (2 prod — **Cashu proof secret + blinding factor,
genuine key material**), `storage_crypto.rs` (1 — **AEAD nonce**), `mesh/x3dh.rs` (2 —
prekey *identifiers*, **not** key material — corrected 2026-08-02), +10 more files.
Nothing is broken today (`rand::random()`/`thread_rng()` are ChaCha12 from
`getrandom(2)`), but it is the T1 shape that produced the COLDCARD defect, now with key
material in the blast radius. Five layers: sealed allowlist trait at key-gen seams;
`clippy.toml` `disallowed-methods` ban (compile-time, CI-enforced — no `clippy.toml`
exists yet); `cargo-deny` on duplicate `rand` majors (absorbs R-05); degenerate-entropy
runtime check; persist the CSPRNG-readiness verdict (absorbs R-09). Also retires the
`impl rand::CryptoRng for CountingRng` false promise at `seed.rs:656`.
**Gated: do not start until the concurrent Phase 1 agent is done and synced.**
## Tier 1 — Medium effort, unblocked
- [ ] **Fix the fail-open first-boot secret regeneration in the ISO** (entropy audit
R-02 + R-03, finding F-03 — `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`).
The installed rootfs is a **cached container export shared by every node**
(`image-recipe/_archived/build-auto-installer-iso.sh:717-726`, extracted at
`:2303`), and it bakes SSH host keys (via the `openssh-server` install at `:345`)
and a TLS keypair (`:463-469`). `archipelago-first-boot-secrets.service` correctly
regenerates both per device — but both branches are **fail-open** (`:1647`,
`:1659`) and `touch "$MARKER"` at `:1663` runs **unconditionally**, so a single
transient failure permanently leaves that node on the image-wide shared SSH host
key and TLS private key, with the failure visible only in a log file. Fix:
(a) set the marker only when both regenerations succeeded, so it retries next
boot; (b) surface the failure in the UI/doctor, not just the log; (c) strip the
baked keys from the rootfs tar so a failure degrades to "no key" rather than
"shared key". Needs an ISO rebuild and two fresh flashes to verify.
- [ ] **Reconcile `Argon2::default()` with ADR-005** (entropy audit R-06, finding F-05).
ADR-005 states 64 MB / 3 iterations
(`docs/adr/005-chacha20-backup-encryption.md:31`); `Argon2::default()` in
argon2 0.5.3 is Argon2id at **19 MiB / t=2 / p=1**. Used at
`core/archipelago/src/seed.rs:249` and `:285`, `backup/identity.rs:38`/`:93`,
`backup/full.rs:618`/`:650`. Either raise the parameters behind a versioned
envelope **with a migration** (an existing `master_seed.enc` was encrypted under
the old parameters and will not decrypt under new ones) or amend the ADR to state
the real numbers. Do not change them silently.
- [ ] **Run the on-node entropy verification checklist** (entropy audit R-15, §6 of
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). Everything in that section is
explicitly **UNVERIFIED** — it needs real hardware this environment cannot reach.
Highest value first: **C-3** (are SSH host-key and TLS fingerprints actually
different across two nodes flashed from the same ISO?) and **C-5** (the cross-node
same-ISO seed collision test — the empirical check that would have caught the
Coldcard defect). Also C-1 (`crng init done` vs seed-generation timestamp), C-2
(`machine-id` uniqueness), C-4 (what the rootfs tar actually contains, run on the
build host), C-6 (is `/rpc/v1` reachable unauthenticated from the LAN). Use a
disposable node — C-5 overwrites node identity.
- [x] ~~immich → Quadlet migration~~ — investigated 2026-07-01, turned out already done:
immich uses the same `install_stack_via_orchestrator` primitive as netbird/btcpay
(`immich_stack_app_ids()` in `stacks.rs:690`), and is confirmed running as real
Quadlet units live on .228 (`immich_server.container`, `immich_postgres.container`,
`immich_redis.container`, all active). Not a legacy in-cgroup app — the only
remaining piece is the fleet-wide Phase-3 default-flip, already tracked in Tier 2.
- [x] ~~Netbird reinstall adoption path~~ — investigated 2026-07-01, **not a bug, by
design.** `adopt_stack_if_exists()` (`stacks.rs:140-198`) is only used as a
fallback when the orchestrator has no manifest for the app — there's nothing to
render certs/config from in that case, so skipping rendering is correct. When
the orchestrator *does* have the manifest (the normal path), the reconcile loop
already re-renders certs even for adopted-running containers, fixed in
`4519dbf0` (`prod_orchestrator.rs:1707-1708`).
- [x] ~~TanStack Query (or equivalent) investigation~~ — spike complete 2026-07-01,
**recommendation: don't adopt / close as not needed.** Only 3 stores actually fetch
data, WebSocket push already handles hot data (server-info/package-data), no
cache-invalidation or stale-data bugs found, migration would touch 62 RPC call
sites for no concrete payoff. If boilerplate ever bothers us, extract a
`usePolling()` composable instead — much cheaper than a query-cache migration.
## Tier 2 — High effort, mostly unblocked (the actual next exit criteria)
- [ ] **🔴 Gate the unauthenticated seed RPCs** (entropy audit R-01, finding **F-01,
Critical** — `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). `seed.generate`,
`seed.verify`, `seed.restore` and `seed.save-encrypted` are in
`UNAUTHENTICATED_METHODS` (`core/archipelago/src/api/rpc/middleware.rs:24-28`),
which skips session, RBAC **and** CSRF (`api/rpc/mod.rs:263`, `:295`, `:326`).
Neither handler checks whether onboarding is already complete
(`api/rpc/seed_rpc.rs:93-159`, `:226-305`), and `NodeIdentity::from_seed`
overwrites `node_key`, `nostr_secret` and the FIPS mesh key **unconditionally**
(`identity.rs:79-114`). There is no rate limit (`rate_limit.rs:60-97` has no
`seed.*` entry). The endpoint is proxied to the LAN over plaintext HTTP
(`image-recipe/configs/nginx-archipelago.conf:11`, `:165`, `:192`) and mesh peers
can reach it too (`server.rs:2080` asserts `/rpc/v1` passes the peer path filter).
Net: **one unauthenticated POST can take over or destroy a live node's identity**,
and `seed.restore` lets the attacker choose the mnemonic. The guard already exists
and is simply never called — `NodeIdentity::key_exists` (`identity.rs:117`).
Fix: bail when a node key exists and no onboarding mnemonic is pending; prefer
also gating on `auth_manager.is_onboarding_complete()`; add rate limits at
`auth.changePassword` strictness; narrow the peer path filter. Changes an
authentication boundary on a live fleet — **needs its own `/gsd-plan-phase` with a
federation re-verify**, not an opportunistic patch.
- [x] **PSBT-first signing: Phase 1 — move the Bitcoin private key out of Core** — **DONE
2026-08-02 by deletion, not conversion** (entropy audit R-04, finding **F-13**;
Phase 10 plan 10-05, decision **D-07b**). The handler that imported the BIP-84
account **private** key into Core's `wallet.dat` had no caller anywhere, LND is the
wallet the UI drives, and the endpoint was authenticated *and* password-gated — so
it was deleted outright rather than rewritten watch-only. `bitcoin.rs`'s wallet-init
handler and its `dispatcher.rs` arm are gone; **no daemon code path writes the
BIP-84 private key into Bitcoin Core.** No migration was performed or is needed —
a 4-node fleet census found no wallet the handler created. D-09's key-origin
requirement moved to the PSBT itself: `lnd.create-psbt` now reports
`key_origin` (`psbt_key_origin_report`, `api/rpc/lnd/wallet.rs`).
**Read `docs/security/KEY-03-SIGNING-POSTURE.md` for the current state** — it also
records the verdict that **no fleet node is provisioned watch-only**, so what ships
today is PSBT *transport*, not air-gapped custody.
- [ ] **Finish the Core-wallet fleet census — 6 nodes unchecked** (Phase 10 plan 10-05,
Task 3; standing item). The 2026-08-02 census examined 4 nodes (archi-dev-box,
shorty-s/.228, archy-x250-beta, archy-x250-pa) and found **no** wallet created by
the deleted handler and no wallet holding keys or funds. Six were not examined:
framework-pt, archipelago-1, archipelago, archy-dev-pa and archipelago-5
(SSH auth/connectivity) and archy-x250-dev (offline). Re-run the **read-only**
procedure in `docs/security/KEY-03-SIGNING-POSTURE.md` § *Fleet census* when
credentials or connectivity allow — a natural fold-in for KEY-04's on-node work.
**Never run `listdescriptors true`** (it returns private keys). If any node reports
a wallet named `archipelago`, or any descriptor wallet with
`private_keys_enabled: true` that is not blank/empty, **stop and escalate — do not
migrate or modify it** (D-07b).
- [ ] **PSBT-first signing: Phases 2-7 rollout**
(`docs/security/PSBT-SIGNING-ARCHITECTURE.md` §8) — the spec is written to be
consumed directly by `/gsd-plan-phase`, with per-phase goals, dependencies,
candidate requirements and hardware gating. Sequence: PSBT construct/export →
external-signer import + finalize → air-gap transport (BC-UR v2 primary, BBQr for
Coldcard, file fallback always) → `wsh(sortedmulti)` multisig on BIP-48 → LND
remote signing → hot-wallet spend limits and cold/warm/hot tiering. Two hard rules
the spec fixes in place: a channel-funding PSBT must **never** be self-broadcast
(funds can be lost), and no UI copy may imply a routing node's Lightning channel
keys are cold — they are necessarily hot. Phases 3-6 need real hardware.
- [ ] **Confine the seed-bearing RPCs to loopback/TLS** (entropy audit R-07, finding
F-04 / [ARCHY-4]). The 24-word master mnemonic is returned to the browser over
JSON-RPC (`core/archipelago/src/api/rpc/seed_rpc.rs:147`, `:156-158`), held in
process memory under a 10-minute TTL (`:27`) and deliberately **not** cleared at
verify time (`:205-211`, with a documented and defensible rationale about client
retries) — over a transport that is plaintext HTTP on LAN by design
(`api/rpc/mod.rs:227-241`). Anyone with LAN traffic visibility during onboarding
reads the phrase that unlocks the wallet and the node identity. Fix: force TLS or
loopback for seed methods, shrink the TTL, and clear on an acknowledged verify
with a short grace window. Touches the onboarding transport — needs a phase.
- [~] **Multinode test pass** (`docs/multinode-testing-plan.md`) — worked the
preconditions on .198 2026-07-01:
- ✅ cleared 2 stale failed-unit records (`archy-mempool-db.service`,
`meshtastic.service` — both `not-found`/dead since 6 and 5 days ago, harmless
bookkeeping, `systemctl --user reset-failed`).
- ✅ nginx `/app/lnd/` proxy target confirmed correct (→ `18083`, matches the
running `archy-lnd-ui` port) — the plan's "stale proxy target" concern doesn't
apply here.
- ⛔ .198 disk (448GB) is below the 1TB archival threshold + was only 21%
through IBD — user chose to **swap in a different node** rather than wait/add
storage. **.116 ruled out** (no bitcoin container installed at all, just the
UI companion). **.120 ruled out** (reserved for another developer). **.5**
(archy-x250-beta, Tailscale `100.72.136.5`) chosen: also sub-1TB (472GB, so
still pruned — that ceiling is shared by every non-.228 node), but **fully
synced** (`ibd:false`, blocks==headers 956,240). Bootstrapped bats 1.11.1 +
jq 1.7.1 onto it 2026-07-01 and **launched the 5× destructive gate
(`ARCHY_ITERATIONS=5 ARCHY_ALLOW_DESTRUCTIVE=1`) — running now**, log at
`/tmp/gate.log` on .5, background poller watching for the `RESULTS` banner.
- Once .5's gate reports: bring the rest of the fleet to precondition, then the
cross-node federation/mesh/transport suites. This is the literal
"next exit criterion" called out in `CLAUDE.md`.
- [ ] **Phase-3 Quadlet default-flip** — code is validated + opt-in via
`ARCHIPELAGO_USE_QUADLET_BACKENDS=true` on .228/.198 already (confirmed live
2026-07-01). Ready to flip (`config.rs:256` + its test) the moment the .5 gate
reports clean — deliberately NOT staged uncommitted in the tree (a prior attempt
left an uncommitted flip sitting around and that caused confusion; it's a 2-line
change, faster to just do it fresh once confirmed).
- [x] ~~Per-app test coverage for the ~30 apps with zero automated coverage~~ —
**reframed 2026-07-01, mostly a non-issue.** `all-apps-matrix.bats` +
`all-apps-lifecycle.bats` already give EVERY installed app generic baseline
coverage (no stuck state, no error state, stop/start/restart survives, UI
reachable). The real gap is narrower: **34 apps lack app-specific assertions**
(health endpoints, API queryability, data integrity) beyond that baseline —
aiui, bitcoin-core, botfights, core-lightning, did-wallet, fedimint-clientd,
fedimint-gateway, fips-ui, gitea, grafana, home-assistant, indeedhub (+5
sub-containers), jellyfin, lightning-stack, lnd-ui, morphos-server, netbird
(+2 sub-containers), nextcloud, nostr-rs-relay, photoprism, portainer, router,
searxng, strfry, uptime-kuma, vaultwarden. Not urgent — baseline coverage is
real safety net; treat as a backlog "nice to harden further," not a gate item.
- [x] ~~Convert remaining multi-container legacy stacks to the manifest-owned model~~ —
**investigated 2026-07-01, DONE, nothing left.** All 5 real multi-container
stacks (btcpay, mempool, immich, netbird, indeedhub) are on the
`install_stack_via_orchestrator` pattern (`stacks.rs`). saleor was removed from
the codebase; portainer/home-assistant/grafana are single-container
manifest-driven apps, never stacks; fedimint/fedimint-gateway/fedimint-clientd
are 3 separate single-container apps with manifest dependency edges, not a
coordinated stack. Workstream A's stack-migration tail is fully closed.
- [ ] **Container thrashing/flapping + reconciler churn** (added 2026-07-04 — was
implicit across other tracks, now an explicit pre-tag concern). The root cause
of restart-storm flapping is pre-Quadlet architecture: restarting
`archipelago.service` SIGKILLs every container in its cgroup, then the
reconciler rebuilds the world over several minutes (the post-OTA health check
deliberately skips per-app container assertions because of exactly this).
Consolidated lever list, in order of impact:
- **Phase-3 Quadlet default-flip** (tracked above) — removes the SIGKILL-the-world
behavior entirely; the single biggest fix.
- **Workstream F lifecycle items** — immich/grafana uninstall hangs + ghost
containers, grafana reinstall stops, fedimint guardian sync
(`docs/PRODUCTION-MASTER-PLAN.md` workstream F).
- **Reconciler churn observability** — no metric/log today distinguishes "settling
after restart" from "flapping"; add a per-app restart counter + log line when an
app restarts >N times in M minutes so thrash is visible instead of anecdotal.
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
never revived it (it repairs missing/drifted containers but doesn't
`reset-failed`+start failed .services). Same for the indeedhub trio after the
gate run. The health monitor also can't help (container is gone when the unit
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
not user-stopped → reset-failed + start, with backoff.
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
removes executor stalls that made the API janky under reconcile load),
quadlet entrypoint-split false-drift fix (2026-07-08 — `container_command_drifted`
compared entrypoint/cmd halves separately, but quadlet folds `sh -lc` into
`Entrypoint=sh` + `Exec=-lc …`, so every quadlet-created app with a
multi-element entrypoint read as permanently drifted; electrumx on .228
recreated 114×/6h until the comparator was switched to concatenated argv).
- Perf polish riding along: 93 MB frontend dist shrink (hardening plan §D 🟡).
- [ ] **Developer tooling CLI suite** (validate/render/local-install/lifecycle-test) —
APP-PACKAGING-MIGRATION-PLAN.md step 5, needed before external devs can publish.
- [x] ~~**Consolidated deploy 2026-07-01**: merged PR #67 (reticulum daemon
process-group fix, `469b0203`), the UI/UX work (`8256fde1` — mesh/web5/apps
layout, modal, search UX), and `archy-openwrt` (TollGate/OpenWrt gateway
integration — new `core/openwrt` crate, RPC surface, `OpenWrtGateway.vue`)
into `main`, alongside the indeedhub self-heal fix~~ — all merged clean, no
conflicts. **Found + fixed 2 real build-breaking issues during
verification, not caught by whoever authored them**: a vestigial unused
`ref` in `Web5ConnectedNodes.vue` that broke `vue-tsc`, and a stale
`MeshMap.test.ts` mock missing `federatedPositions` (predated this
session's Mesh Map feature) that crashed on mount. Full test suite green
(667 passed) after fixes. **Deployed fleet-wide 2026-07-01, all 5 nodes
sha256-verified**: .116, .198, .228, .5 (recovered cleanly from one
truncated-transfer hiccup, caught via checksum before it hit the live
service), 100.82.34.38 (non-Quadlet node — all containers survived the
restart intact, unlike the worst-case risk flagged beforehand). Also
built an unbundled installer ISO from this same merged source
(`archipelago-installer-1.7.99-alpha-unbundled-x86_64.iso`, 2.4GB) —
the ISO pipeline was archived from the release process at v1.7.43-alpha
(OTA tarballs are now primary) but the wrapper script still works.
- [ ] **⚠️ NOT YET DEPLOYED — start here next session.** After the fleet deploy
above, found that PR #67 ("kill whole daemon process group on drop",
branch `fix/reticulum-daemon-process-group`, head `be50c886`) is a
**different, separate** reticulum-daemon fix from the one already
deployed (`469b0203` on `fix/reticulum-daemon-pdeathsig`) — I'd
conflated the two by topic similarity and only merged/deployed the
Python-level `pdeathsig` fix, missing PR #67's Rust-level
kill-whole-process-group-on-`Drop` fix entirely. Merged PR #67 into
`main` (`7a7fec21`, clean, `cargo check` green, complementary not
conflicting with the already-deployed fix) and separately fixed a real
bug found live: `OpenWrtGateway.vue`'s back button had no `@click`
handler at all (`7d7ba573`, `vue-tsc` clean). **Both committed + pushed
to `main` but genuinely NOT deployed to any node** — user asked to hold
off deploying to restart their computer. Also spot-checked
`openwrt.scan` live on .116: RPC plumbing works, but no physical
OpenWrt router was available to confirm true-positive detection, and
`detect::scan_subnet` does blocking TCP/SSH calls inside an `async fn`
with no `.await` — untested at scale, worth hardening. **Next steps**:
build release binary + frontend from current `main`, deploy to all 5
fleet nodes (.116/.198/.228/.5/100.82.34.38) the same way as the
earlier consolidated deploy, then verify the back button + (if a real
OpenWrt router is available) router detection live.
- [~] **Cross-node federation/mesh/transport suites** — **big find 2026-07-01: these
already exist**, just aren't wired into the gate or documented as existing:
`tests/multinode/smoke.sh` (federation pairing/sync, FIPS anchor, peer content
browse, tombstone-removal regression tests), `tests/multinode/meshtastic.sh`
(8-stage on-air mesh test), harness in `tests/multinode/lib/multinode.bash`.
**Actually ran `smoke.sh` live against .116↔.228 2026-07-01: 14 passed, 1
failed, 1 skipped.** Confirms federation pairing (both directions), FIPS
anchor connectivity (both nodes), and peer-content-browse-over-mesh (the
v1.7.95 fix) all genuinely work node-to-node right now.
- ⚠️ **Real robustness gap found**: `node_rpc()` in `tests/multinode/lib/multinode.bash`
has no `--max-time` on its curl calls — a slow server-side RPC hangs the whole
suite with zero feedback (this is what looked like a hang before it eventually
completed on its own). Cheap fix, not yet applied.
- 🐛 **Real regression found and root-caused**: removing a federation node
(`federation.remove-node`) doesn't reliably stick — B reappeared in A's peer
list after removal in the live test. Root cause: `remove_node()`
(`core/archipelago/src/federation/storage.rs:187`) does
`let _ = tombstone_did(data_dir, did).await` — **silently swallows the
tombstone write's errors.** If that write fails (disk I/O, permission,
transient issue), the peer is removed from `nodes.json` but never actually
tombstoned, so the next background sync/notify-join re-adds it — the
tombstone check at `handlers.rs:592-599` passes because the DID was never
recorded as removed. Diagnosed as a **pre-existing logic gap**, not a fresh
regression from the v1.7.95 fix. **Not fixed yet** — this is federation/trust
code, deliberately not touching it blind; needs a careful fix (surface the
tombstone-write failure instead of swallowing it, and/or retry) plus
re-verification with `smoke.sh` before considering it closed.
## Tier 3 — Blocked on a decision or resource only you can supply
- [x] ~~Version naming decision~~ — **decided 2026-07-08: `1.8.0-alpha`.** Remaining
work is the mechanical bump + tag + push once the pre-tag items above close.
- [x] ~~Workstream B signing ceremony~~ — **done 2026-07-02.** `anchor.rs` pins
`RELEASE_ROOT_PUBKEY_HEX = 5d15cbee…9951` (signer
`did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur`); mnemonic held
offline per `docs/workstream-b-signing-runbook.md`.
- [ ] **Bitcoin multi-version fleet-wide OTA** — `.228` fully working on branch,
per your prior gating this rollout is explicitly held for your decision on
timing (`docs/bitcoin-version-bulletproof-rollout.md`).
- [ ] **3ccc stock-Meshtastic RF validation** — needs a live send/receive test with
physical radios in your hands; code fix is in place, just unverified live.
## Backlog — deferred, no scope decided, low priority
- [ ] **Marketplace protocol (workstream C)** — design-only (`docs/marketplace-protocol.md`),
no tooling/trust UX built. Future work, not urgent.
- [ ] **DHT distribution (workstream D)** — confirmed design-only, no code
(`docs/dht-distribution-design.md` explicitly says "Status: Design (no code yet)");
an experimental iroh provider skeleton exists behind a feature flag for future
PoC measurement, nothing fleet-facing.
- [ ] **Custom live voice-call protocol** — deprioritized 2026-07-01 per user request;
scope not yet decided. Revisit after the tiers above are worked down.
---
*Historical narrative and detailed per-session logs remain in
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md` §6/§8b —
this doc is the live "what's left, in priority order" list. Update it (don't just
append to the old docs) as items close or new ones surface.*
docs/api-reference.md
+3 -3
View File
@@ -382,17 +382,17 @@ All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`.
```bash
# Login
curl -c cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -c cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"auth.login","params":{"password":"password123"}}'
# Get system stats (authenticated)
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"system.stats"}'
# Get DID
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"node.did"}'
```
docs/app-developer-guide.md
+3 -3
View File
@@ -363,15 +363,15 @@ podman logs my-app
1. Install via the marketplace UI or RPC:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-d '{"method":"package.install","params":{"id":"my-app","dockerImage":"docker.io/myorg/my-app:1.0.0"}}'
```
2. Verify the container is running:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-d '{"method":"container-list"}'
```
3. Check the UI at `http://192.168.1.228/app/my-app/`
3. Check the UI at `http://archipelago.local/app/my-app/`
### Validate Manifest
docs/archive/HANDOVER-2026-07-02-iso-feedback.md
-151
View File
@@ -1,151 +0,0 @@
# Handover — fresh-ISO feedback bug-bash (2026-07-02)
**For: the agent building the next ISO + fleet deploy.** All fixes below are
**merged and pushed: gitea-ai main = `f5d24796`** (merge of `c375ecc4`,
65 files; branch `iso-feedback-fixes-2026-07-02` also pushed). Source
feedback: user's fresh ISO install on a Framework (11th-gen Tiger Lake)
machine, node `192.168.1.81` (SSH `archipelago` / `archipelago`).
Diagnostic bundle: `/home/archipelago/incoming-logs/node-logs-192.168.1.81/`.
**⚠️ Known-red tests on main (NOT from this work):** `trust::anchor::
unset_constant_is_none` + 2 `trust::signed_doc` tests fail because a prior
commit pinned `RELEASE_ROOT_PUBKEY_HEX` without updating them. The signing/
audit agent's uncommitted changes in the shared tree fix exactly these —
coordinate with them; don't "fix" it independently or you'll collide. This
bug-bash branch alone was 898/898 green; merged with main it's 894/898 with
only those three.
## ⚠️ Outstanding user request for the deploy
- **Change .81's web-UI password to `<FLEET_PW>`** — the user forgot the
current one. Node was unreachable from .116 during this session (flaky WiFi
AP, IP flapped .68↔.81). Do this during deploy (SSH works from the user's
machine; `archipelago`/`archipelago`).
## What changed (by file)
### Backend (core/archipelago/src) — builds clean, targeted tests pass
- `api/handler/websocket.rs` — **subscribe BEFORE initial snapshot** (the
"everything needs ctrl-r" root cause: broadcasts in the snapshot→subscribe
gap were silently lost; a stale client never learned containers-scanned).
- `main.rs` — crash check now runs BEFORE writing the PID marker (**crash
recovery had never run on any node** — it always saw its own PID and
skipped); tracing default demoted debug→info (journal volume).
- `crash_recovery.rs` — PID-reuse guard (`process_is_archipelago`); new
**pending-boot-starts registry** (names queued for recovery/reconcile) with
writers in `recover_containers` + stack recovery.
- `server.rs` — scanner overlays Stopped/Exited → **Restarting** for
pending-boot-start ids (user ask: "status should be restarting if they are
being restarted"); `SCANNER_RESTARTING` ownership set so scanner-authored
Restarting resolves immediately instead of wedging in the 20-min
transitional-preserve.
- `container/prod_orchestrator.rs` — reconcile pass + `adopt_existing`
register/deregister pending boot-starts; LND pre-start hook passes detected
`bitcoin_host()` (Knots vs Core) into `lnd::ensure_config`; new
`fedimint-clientd` pre-start hook (mkdir + chown 1000:1000 of
`/var/lib/archipelago/fmcd` — self-heals the crash-loop).
- `container/lnd.rs` — `ensure_config(paths, rpc_pass, bitcoin_host)`;
bitcoind.rpchost no longer hardcoded `bitcoin-knots`; drift check rewrites
host changes; +unit test `ensure_config_repairs_bitcoin_host_drift`.
- `api/rpc/package/dependencies.rs` — bounded **dependency wait**
(`wait_for_install_deps`, 36×5s): installed-but-starting deps wait with
"Waiting for Bitcoin to start…" on the card; not-installed deps fail fast
with `DependencyGateError` marker; +5 unit tests.
- `api/rpc/package/install.rs`, `stacks.rs` — call sites wired to
`gate_install_deps` (lnd/electrumx/mempool/btcpay).
- `api/rpc/package/async_lifecycle.rs` — `DependencyGateError` removes the
optimistic entry (**no more phantom "Stopped" LND tile**) + pushes an Error
notification with the reason.
- `api/rpc/package/progress.rs` — `set_install_message` helper.
- `api/rpc/seed_rpc.rs` — `save_pending_seed_encrypted`; seed.restore also
stashes the mnemonic; `auth.rs` — **auth.setup persists the encrypted seed
backup** (recovery-phrase reveal previously failed on EVERY node because
nothing ever wrote `master_seed.enc`).
- `api/rpc/middleware.rs` — sanitizer allowlist extended (seed/2FA/auth
errors reach the user instead of "Check server logs"); +2 tests.
- `bitcoin_status.rs` — friendly status for "connection reset" (bitcoind
starting); raw URL/os-error chains no longer shown; +3 tests.
- `bootstrap.rs` — journald drop-in self-heal (OTA nodes get log caps);
bitcoin.conf printtoconsole heal. (Log-spam agent's work; verified.)
- `api/rpc/package/config.rs` — bitcoin args `-printtoconsole=0`.
### Manifests / scripts / configs
- `apps/lnd/manifest.yml` — BITCOIND_HOST now `derived_env {{BITCOIN_HOST}}`.
- `apps/bitcoin-knots/manifest.yml`, `apps/bitcoin-core/manifest.yml` —
`-printtoconsole=0` (90.6% of the journal was IBD UpdateTip spam;
debug.log in the datadir keeps full logs).
- `scripts/first-boot-containers.sh` — chown 1000:1000 of
`/var/lib/archipelago/fmcd` in BOTH fmcd blocks (root-owned dir was the
fedimint-clientd "Permission denied os error 13" crash-loop);
printtoconsole=0.
- `scripts/container-doctor.sh`, `scripts/reconcile-containers.sh` —
printtoconsole=0.
- `image-recipe/configs/journald-archipelago.conf` (NEW) — SystemMaxUse=500M,
rate limits; baked by ISO builder + bootstrap self-heal.
- `image-recipe/configs/nginx-archipelago.conf` — `/assets/` 404s no longer
cacheable (the `always` immutable header could pin a missing background for
a YEAR); HTTPS block gained the missing `/assets/` location (was silently
serving index.html as images).
- `image-recipe/configs/archipelago-kiosk.service` — MemoryMax 1500→2800M,
MemoryHigh 1200→2200M (kiosk was riding reclaim-throttle = the lag).
- `image-recipe/_archived/build-auto-installer-iso.sh` — kiosk launcher/service
now spliced from `image-recipe/configs/` at build time (was a stale inline
heredoc that force-disabled GPU); **+ `firmware-intel-graphics` +
`firmware-amd-graphics`** (Debian trixie split the i915 DMC blobs out of
firmware-misc-nonfree; the .81 kernel logged tgl_dmc missing).
### Frontend (neode-ui) — vue-tsc clean, vitest green
- `views/Login.vue` — Enter in field 1 → focus confirm; Enter in confirm →
submit; submit button always clickable (shows inline mismatch/length error
instead of being silently disabled); errors clear on input; **Restart
Onboarding needs a confirming second click** (5s window) — this button is
the likely cause of the "onboarding restarted after mismatch" report.
+`login.restartConfirm` key in en/es locales.
- `stores/sync.ts` — 30s staleness reconciliation (server.get-state) while
connected; already-connected fast path now refetches too.
- `composables/useContainersScanTimeout.ts` (NEW, +tests) — 20s escape hatch;
wired into `Apps.vue` / `Discover.vue` / `Marketplace.vue`; fresh empty node
reaches the real "no apps yet" empty state; "Checking…" can never persist.
- Backgrounds: 10 heaviest bg JPEGs → **WebP q90** (9.4MB→6.6MB; refs updated
in OnboardingWrapper/Dashboard/useRouteTransitions); 7 remaining images
stayed JPEG (WebP came out LARGER on those — noisy sources; deliberate).
- `public/assets/video/video-intro.mp4` — re-encoded CRF20 (SSIM 0.988) with
**+faststart** (moov was at EOF → browser had to download all 15MB before
playing = the intro lag). 12.7MB now, streams immediately.
- LND icon: stale dist artifact; any fresh `npm run build` ships
`app-icons/lnd.png` correctly.
## Verification done here
- `cargo build -p archipelago` + `cargo check` clean; targeted tests
(bitcoin_status, middleware sanitize, dep_wait, lnd, crash_recovery,
boot_reconciler, bitcoin_host, prod_orchestrator lnd hooks): **52 passed,
0 failed**. Full suite: **898 passed, 0 failed, 1 ignored** (22s).
- `npm run build` green; dist verified: 10 bg-*.webp present, `lnd.png`
icon present, `restartConfirm` string in bundle, optimized faststart
video (12,740,782 bytes) in place. Note: main had a latent build breaker
(unused template ref in `Web5ConnectedNodes.vue` from commit 8256fde1,
vue-tsc TS6133) — fixed here by removing the dead ref/binding; without
this fix `npm run build` fails on current main.
- vitest: new composable tests + related suites pass.
- `bash -n` clean on all touched scripts; nginx conf live-verified by agent
(200/404/cache headers on both HTTP+HTTPS blocks).
- ISO kiosk splice byte-verified against configs/ by agent simulation.
## NOT done / left for you
1. **Full test-suite run + gate**: run the complete `cargo test` and (after
deploy) `tests/lifecycle/run-gate.sh` ON .228 per CLAUDE.md before any tag.
2. **Frontend bundle grep before shipping** (per memory/feedback): verify new
strings (e.g. `restartConfirm`, `bg-home.webp`) in the built tarball.
3. **Diagnostics collector** (`data-dir-listing.txt` = 15MB of podman overlay
internals; dmidecode empty) — collector script wasn't found in this repo
(likely lives on-node or in the user's collection script); fix when found.
4. **podman healthcheck cgroup EPERM spam** (1,250 journal errors, healthchecks
unreliable fleet-wide) — real open bug, Quadlet-phase territory, NOT fixed.
5. **DP link-training failures on .81** (display corruption) — likely
cable/dock/port hardware; firmware fix may help; tell user to try another
cable/port if corruption recurs.
6. **LoRa/RNode onboarding surface** — never scoped; user may want it as a
feature (mesh device-found modal exists only on Mesh page post-login).
7. The concurrent audit agent's files (`docs/1.8.0-RELEASE-HARDENING-PLAN.md`,
`core/.../trust/*`, parts of `bootstrap.rs`) are ALSO uncommitted here —
coordinate before committing; don't mix attribution.
docs/archive/INSTALL-SCREENS-DESIGN.md
+2 -2
View File
@@ -92,9 +92,9 @@ Constraints: bash TUI only (no ncurses). ANSI colors available:
After reboot, open the Web UI from any device:
http://192.168.1.198
http://archipelago.local
SSH: ssh archipelago@192.168.1.198
SSH: ssh archipelago@archipelago.local
Password: archipelago
Web Login: password123
docs/archive/SESSION-1.8.0-OTA-PROGRESS.md
-344
View File
@@ -1,344 +0,0 @@
# 1.8.0 OTA Session Progress
Updated: 2026-06-30
> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc is kept
> as the historical session-by-session log; open items were consolidated into the
> unified tracker on 2026-07-01 (several turned out already shipped — see that doc for
> current status instead of re-deriving it from the log below).
---
## ▶️▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (evening) — #17 deployed + verified on .198/.228
**#17 (3ccc / stock-peer E2E pill) is now built, deployed, and live-verified** on `.198` and
`.228` only (`.116` skipped per the hardware notice below — its radio is mid-reflash to RNode).
- Built release binary **sha `b1d695fc626a7382`** from the working tree (`cargo check` +
`cargo test -p archipelago mesh::` both green, 99 passed/0 failed/1 ignored, right before
building — tree was settled, no collision with the Reticulum agent's concurrent edits).
- Deployed via stop/swap/start to `.198` (192.168.1.198) and `.228` (192.168.1.228), sha256
confirmed matching on both, `systemctl is-active` = `active` on both (`.228` took its usual
~couple-minute convergence — heavy resilience node, unrelated bitcoind/fedimint container
startup noise in the logs during that window, no mesh errors).
- **Live-verified the actual fix**, not just deploy: on `.198`, `mesh.peers` shows
`"advert_name":"Meshtastic 3ccc", "pkc_capable":true`, and `mesh.send` to 3ccc
(`contact_id:1128152268`) now returns **`"encrypted":true`** — confirms the
`archy || peer_pkc_capable(contact_id)` TX fix is live, not just compiled.
- `.228`'s RPC password in memory (`password123`) was stale — user confirmed the correct
password is `<FLEET_PW>` (same as `.198`/`.116`, i.e. fully unified now). Re-verified via
RPC: `mesh.peers` shows 3ccc `pkc_capable:true`, and `mesh.send` to 3ccc returns
`"encrypted":true` — #17 confirmed live on `.228` too, not just `.198`.
**NOT yet done:** push commit to gitea-vps2 (still uncommitted in the working tree, by design —
shares the tree with the Reticulum agent's uncommitted work); user on-device confirmation that
the E2E pill actually renders in the Mesh UI for 3ccc.
---
## 🛠️ HARDWARE NOTICE 2026-06-30 (~16:30) — .116's Heltec V3 is being repurposed
**The Reticulum agent is reflashing .116's Heltec V3 (the board on `/dev/ttyUSB0`, currently
.116's live Meshtastic radio) to RNode firmware**, with explicit user approval, to unblock the
Reticulum Phase-0 hardware gates (real RNode needed; see `docs/RETICULUM-TRANSPORT-PROGRESS.md`).
This was user-confirmed specifically because it takes .116 offline as a Meshtastic radio.
**Effect on this workstream: do all on-device Meshtastic testing on .198 and .228 only — .116 no
longer has a Meshtastic-firmware radio attached once this lands.** `cargo check`/`cargo test
-p archipelago` were both confirmed clean (99/99 mesh tests) right before the reflash started, so
the earlier "wait for their edit to settle" blocker above is cleared — software-side it's safe to
build/test/deploy; only .116's *physical radio role* changed.
---
## ▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (later PM, ~15:50) — READ THIS FIRST IF RESUMING
**#17 (3ccc / stock-peer E2E pill) is CODE-COMPLETE in the working tree**, isolated
to `meshtastic.rs`/`protocol.rs`/`types.rs`/`mod.rs` as planned (no `session.rs`
transport-plumbing changes from this side):
- `ParsedContact.pkc_capable` (`protocol.rs`) + `MeshPeer.pkc_capable` (`types.rs`),
both `#[serde(default)]`/defaulted `false` at every construction site.
- `MeshtasticDevice::get_contacts()` now stamps `pkc_capable` per contact from the
existing `peer_is_pkc_capable(node_num)` seam (de-`allow(dead_code)`'d).
- `listener/session.rs::refresh_contacts` ORs the new value into `MeshPeer.pkc_capable`
(capability only grows, never cleared by a transient refresh) — this IS a touch of
session.rs, but additive/non-colliding with the Reticulum device-enum match arms
already there; did not touch transport plumbing/routing.
- `mod.rs::MeshService::send_message` now does `archy || self.peer_pkc_capable(contact_id)`
for the Sent-row `encrypted` flag (was `archy`-only before).
- Verified via `cargo check -p archipelago --bin archipelago` (clean, exit 0) **before**
the other agent's latest edit landed.
**NOT YET DONE:** rebuild release binary → redeploy 5 nodes → push → user on-device test
(same as #16, both still pending live verification).
**⚠️ BLOCKED right now — do not build/deploy/push until this clears:** the Reticulum
agent is actively mid-edit in the *same* working tree. A `cargo test` run right after
the clean `cargo check` above failed with a real (but transient, not mine) signature
mismatch: `session.rs::auto_detect_and_open` / `run_mesh_session` were observed with a
new `device_kind: Option<DeviceType>` param that `listener/mod.rs`'s call site didn't
have yet — a normal in-flight snapshot of their work, not a regression to fix here.
**Action on resume: re-run `cargo check` first; if it's clean, the other agent's edit
has settled and it's safe to proceed to build/test/deploy. If still broken, wait —
do not stash, revert, or patch their in-progress session.rs/listener/mod.rs changes**
(see memory `feedback_concurrent_agent_tree.md`). Also: building/deploying right now
would bundle their not-yet-finished `reticulum.rs` wiring into the binary — confirm
with the user before shipping a combined build, since only the meshtastic `#17` piece
has been asked for/owned by this session.
---
## ▶️▶️ LIVE CHECKPOINT 2026-06-30 (late PM) — READ THIS FIRST
**Fleet state:** all **5 test nodes** on binary **`38c456b0bacec3c4`** + frontend
**`Mesh-CAkPgvLo.js`**, `archipelago` active on each:
`.116`, `.198`, `.228` (LAN, archipelago@ + `~/.ssh/archipelago-deploy`),
`100.72.136.5`, `100.89.209.89` (Tailscale, same key — installed this session;
SSH user `archipelago` / pw `<FLEET_PW>`; NOPASSWD sudo on all 5).
**Shipped this session (commit `12e7990b` on `main`, pushed to gitea-vps2):**
- ✅ **#16 public-channel routing** — inbound Meshtastic text to `BROADCAST_NUM`
now files under the **public channel thread** (contact_id `u32::MAX - idx`),
attributed to its real sender, instead of polluting per-sender DM threads.
Directed text (`to == our node`) still routes to the DM thread (regression test
`packet_to_inbound_frame_directed_dm_stays_a_contact_message`). `send_channel_text`
now sets `MeshPacket.channel` so archy TX's on channel 0 (public).
Code: `meshtastic.rs` (`packet_to_inbound_frame`, `parse_mesh_packet` to/channel,
`send_channel_text`), `protocol.rs` (`RESP_MESHTASTIC_CHANNEL_TEXT = 0x70`),
`listener/frames.rs` (handler + sender attribution), `Mesh.vue` (`senderLabelFor`).
Tests green (95 mesh tests). **Pending: user on-device test with the radios.**
**Push access:** `main` is a PROTECTED branch on gitea-vps2. Direct push uses the
dedicated **`ai`** account via remote **`gitea-ai`** (`git push gitea-ai main`).
See memory `reference_gitea_ai_push_account.md`.
**Coordination:** another agent owns **Reticulum** (`reticulum-daemon/` + Rust
transport wiring). DO NOT touch `mesh/listener/session.rs` transport plumbing or
`mod.rs` routing in ways that collide. Keep #17 work isolated to `meshtastic.rs`
RX/TX + (if needed) the sent-row encrypted flag.
### ✅ CODE-COMPLETE (not yet deployed/tested live) — #17 (3ccc / stock-peer E2E pill)
Goal: DMs **to and from** a PKC-capable stock peer (3ccc, NodeInfo public_key
key_len=32 confirmed) must show the E2E pill.
- **RX side is already correct:** `parse_mesh_packet` reads `public_key` (field 16)
+ `pki_encrypted` (field 17) per the MeshPacket proto; the directed-DM RX path
promotes to `RESP_CONTACT_MSG_V3_E2E` when `pki_encrypted`. (Verify live.)
- **TX bug (root cause) — FIXED:** `mod.rs::send_message` now records the Sent row
with `encrypted = archy || peer_pkc_capable(contact_id)`. `peer_is_pkc_capable`
(meshtastic.rs) is wired out via `get_contacts()` → `ParsedContact.pkc_capable` →
`refresh_contacts` (session.rs) → `MeshPeer.pkc_capable` → `MeshService::peer_pkc_capable`.
See the LIVE CHECKPOINT at the top of this file for the exact touch points.
- NEXT STEP when resuming: confirm `cargo check` is clean (the other agent's
Reticulum work shares this tree and may be mid-edit — see top checkpoint), then
rebuild → redeploy 5 nodes → push → user test (same pending step as #16).
**Remaining open after #17:** #12 (provisioning robustness — HOLD, session.rs churn
risks reticulum collision), #8 (Device-tab settings panel + reboot button — RPC
`mesh.reboot-radio` already exists), #6 (onboarding modal), #7 (.116 re-verify),
#14 (RSSI/SNR per-contact indicator), #15 (peer-location map, POSITION_APP portnum=3).
---
## ▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST
**Goal:** archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill,
identical in off-grid and normal mode. Test bed = `.116` / `.198` / `.228` (all EU_868).
Don't touch the federation/FIPS path.
### ✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)
VERIFIED: `.198→.228` directed DM → `.228` row `RECEIVED enc=True peer="Arch Optiplex"`.
All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary
**`737b16c3235b`** active on all three. Fix source **COMMITTED as `a57ae388`** on `main`
(not yet pushed to gitea-vps2/origin).
**THE fix (receive stream):** archy ignored `FromRadio.rebooted` (field 8). Every config
write reboots the radio → firmware PhoneAPI resets to `STATE_SEND_NOTHING` and stops
streaming received packets until the client re-sends `want_config`. archy never did →
went deaf to inbound (that's why old messages only arrived after a full restart = fresh
want_config). Fix: handle `FROM_RADIO_REBOOTED` → set `pending_reinit` → re-send
want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and
a pinned `modem_preset=LONG_FAST` so all radios share frequency. Combined with the earlier
E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.
**Open follow-ups:** #A surface received msgs under archy identity in all UI views; #6
device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation;
#12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog;
#14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr);
#15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3
lat/lon). See the resume memory `project_session_resume_2026_06_30_lora.md` for the full
task list.
### (historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix
The **archy software is correct and deployed.** The blocker was at the
**radio/RF layer: the three radios are not hearing each other over the air at all.** No
amount of archy code change will fix that until the radios actually RF-link. **Resume by
testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS
FIRST AT HOME" below.** ← this turned out to be the want_config resubscribe bug above.
### What is DONE and deployed (commit pending — see below)
- **E2E send fix** (`core/archipelago/src/mesh/mod.rs` `send_message`, ~L1542): archy↔archy
plain chat text is now sent as a **native `TEXT_MESSAGE_APP` DM** (firmware PKC-encrypts
it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked
`encrypted=true` so the pill shows. Rich typed msgs still use `send_typed_wire`. This was
the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
- **NEW: software radio-reboot** end-to-end, so a wedged/RX-deaf radio can be rebooted
without physical access (and for the Device-tab settings panel the user requested):
- `meshtastic.rs`: `reboot(seconds)` driver method + `ADMIN_REBOOT_SECONDS_FIELD = 97`
(verified vs meshtastic/protobufs admin.proto — `set_owner=32/set_channel=33/set_config=34`
matched our existing constants, confirming the proto read).
- `listener/mod.rs`: `MeshCommand::RebootRadio { seconds }`.
- `listener/session.rs`: device-enum `reboot()` dispatch (Meshtastic only) + handler arm.
- `mesh/mod.rs`: `MeshService::reboot_radio(seconds)`.
- `api/rpc/mesh/messaging.rs`: `handle_mesh_reboot_radio` → RPC **`mesh.reboot-radio`**
`{seconds?}` (default 2); dispatcher arm in `api/rpc/dispatcher.rs`.
- `cargo check` passes. Built release **sha `ba4aed590027690d`** and DEPLOYED + active on
`.116/.198/.228`. The RPC works (`{"reboot":true,"seconds":2}`).
- ⚠️ **Caveat:** when called, archy logged "Sent Meshtastic radio reboot" but the radio did
**not** visibly reboot afterward (no config re-stream). Either field 97 is still off, or
newer firmware requires an admin session passkey even over local serial, or the USB serial
stayed open through the 2s reboot so no reconnect was logged. **Needs on-device verification.**
### The hard evidence (why "nothing works")
- Directed DM tests `.198→.228` AND `.116→.228` (neither path reflashed): sender logs
`Sent plain native DM dest=30d258436d65 part=1 total=1` and RPC returns `sent:true,
encrypted:true`, but `.228` logs **nothing** — packet never reaches archy from the radio.
- A raw broadcast from `.198` (`mesh.broadcast`) was accepted by its radio but **not heard**
by `.228`/`.116`.
- In an 8-minute window, **all three nodes received 0 inbound OTA packets from any other node.**
Each only logs its OWN once-a-minute `Broadcast Meshtastic NodeInfo advert` + local TX
`field=11` queue-status. `.228 mesh.status` = `messages_received:1` total.
- `.198`'s radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that
**reception is broken on the receivers.** A radio cannot drop a broadcast AND a unicast to
its own node number while config matches, unless it simply isn't on the same airwaves.
- archy provisioning is correct & identical across nodes (read back from device): PRIMARY =
public LongFast (`name="" psk_len=1`), SECONDARY = `archipelago`, region=3 (EU_868). Admin
field constants verified. The send path hands the radio a correct unicast MeshPacket
(`to`=node, want_ack, hop_limit=3, plaintext `decoded` for the firmware to PKC-encrypt).
### PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch
archy only ever writes `region` + `use_preset` and **never explicitly pins `modem_preset`**
(it parses region but not preset; `set_lora_region` relies on the LongFast default). If ANY
radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic
app, or a different factory default after the `.198` reflash), the radios are on **different
airwaves despite identical channel name + region**, and archy would never correct it.
### DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)
Open the **Meshtastic phone app over Bluetooth** (works alongside archy's USB serial) on each
of `.116/.198/.228` and check:
1. Do the 3 nodes **see each other** in the node list (recent "heard")? → if NO, they're not
RF-reaching (preset/freq/antenna/range).
2. Do all 3 show the **same** Modem preset (LongFast), Region (EU_868), Frequency slot, and
the same PRIMARY channel? → any difference = the cause.
This single test separates "archy misconfigures the radios" from "radios physically can't
reach each other."
### THEN — the archy fix to apply (if preset/config differs)
Make archy **authoritatively write the full LoRaConfig** and force re-provision so all radios
converge: in `core/archipelago/src/mesh/meshtastic.rs::set_lora_region` (and its
caller/guard `ensure_lora_region` ~L304), explicitly set `modem_preset = LONG_FAST (0)` as a
field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision
path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new
`mesh.reboot-radio`). Also verify the `mesh.reboot-radio` actually reboots the radio
on-device (the caveat above).
### TEST RECIPE (works on each node)
- RPC helper used this session: a node-side `rpc.sh` that logs in (password
`<FLEET_PW>`), grabs the `csrf_token` cookie, echoes it as `X-CSRF-Token`, and POSTs to
`http://127.0.0.1:5678/rpc/v1`. Recreate it or run archy's RPC directly. Methods:
`mesh.peers`, `mesh.status`, `mesh.messages`, `mesh.send {contact_id,message}`,
`mesh.broadcast`, `mesh.reboot-radio {seconds}`.
- **LoRa contact ids:** `.116=1135977788` (prefix `3ca5b543`), `.198=3677050140` (`db2b551c`),
`.228=1129894448` (prefix `30d25843`), stock `3ccc=1128152268`.
- **Link health check (run on each node):** look for inbound `from=Some("!...")` lines in
`journalctl -u archipelago` that are NOT the node's own `Broadcast ... NodeInfo advert`. If
zero across all nodes → RF link is down (the current state).
- **E2E success criteria:** send `.198→.228`, the marker appears in `.228` `mesh.messages` as
an inbound row with `encrypted:true` / `transport:"lora"`, AND `.116↔.228` likewise.
### DEPLOY / BUILD RECIPE
- Build: from `core/`, `CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo
build --release -p archipelago --bin archipelago`. (If `rust-lld: undefined hidden symbol`,
it's incremental cache — `CARGO_INCREMENTAL=0` fixes it.)
- SSH key `~/.ssh/archipelago-deploy` is authorized on `.116/.198/.228`. SSH/UI/RPC password
`<FLEET_PW>`. Per node: scp the binary, `sudo systemctl stop archipelago` →
`kill -9 $(pgrep -x archipelago)` → `install -m0755` to `/usr/local/bin/archipelago` →
`systemctl start archipelago`. Verify by `sha256sum` match + `systemctl is-active`.
- **Current deployed sha on all 3 = `ba4aed590027690d`** (the reboot-enabled build).
### Fleet state (as of 2026-06-30 PM)
- All 3 nodes on binary `ba4aed59`, active. Off-grid mode currently OFF (`mesh_only:false`).
- `.198` radio was reflashed to factory `firmware-heltec-v3-2.7.26` (recovered from corrupt
NVS); region EU_868 persists. Its archy identity is NOT re-bound on `.228` (`.228` shows
`.198` as raw radio "Meshtastic 551c", `arch_pubkey_hex` absent) because `.228` hasn't heard
`.198`'s identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
- The radios are powered & each transmitting; they are simply not hearing each other.
### Deferred UI (after LoRa works)
- Device-tab **settings panel** (gear/desktop) — host the "Reboot radio" button there; calls
`mesh.reboot-radio`. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid
buttons) + a `rebootRadio()` method in `neode-ui/src/stores/mesh.ts`. See `Mesh.vue` ~L1484
actions row and `mesh.ts` ~L373 `broadcastIdentity()` pattern.
- Device-onboarding modal (detect plugged-in radio).
---
Current scope:
- Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
- Take over the `3ccc` stock Meshtastic peer bug: LoRa text from `3ccc` to Archipelago `.116` does not surface in `mesh.messages`.
- Keep release-gate fixes already made in this session.
Local gate status so far:
- `cargo test -p archipelago --bin archipelago`: green, 849/849 after Meshtastic fixes.
- `python3 scripts/check-app-catalog-drift.py --release --strict`: green.
- `npm run type-check`: green.
Key changes made so far:
- Added cascade uninstall progress truthfulness assertion to `tests/lifecycle/bats/cascade-uninstall.bats`.
- Fixed release catalog drift filters and regenerated catalog metadata.
- Fixed invalid `apps/fedimint-clientd/manifest.yml` `cpu_limit` schema value.
- Updated stale/tight Rust tests without changing production behavior.
Remaining non-automatable / operational gates:
- Workstream B signing is blocked on the offline `RELEASE_MASTER_MNEMONIC`; code + runbook exist, but the publisher must pin/sign the release-root catalog.
- Phase-3 Quadlet backend rollout is implemented behind `use_quadlet_backends` and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
- `.116` read-only `use-quadlet-backends-install.bats`: 6/6 skip-clean; no backend `.container` units, so Phase-3 is not active on that node.
- Release metadata still says `1.7.99-alpha` in `releases/manifest.json`; changelog top is `v1.8.00-alpha`. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.
Do not discard:
- `core/archipelago/src/mesh/listener/decode.rs`
- `core/archipelago/src/mesh/listener/session.rs`
- `core/archipelago/src/mesh/meshtastic.rs`
3ccc bug current hypothesis:
- The prior attempted Meshtastic fix added a hard stale-packet filter using `rx_time`.
- Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
- That would make live `3ccc` packets look older than 10 minutes and get dropped before `mesh.messages`.
- Current patch treats implausibly early `rx_time` values as unknown rather than stale.
.116 live validation after 2026-06-30 hotfix:
- `.116` reachable by SSH; `archipelago` active; `/dev/mesh-radio -> ttyUSB0` attached.
- Current canary deploy is commit `b4531bb4`; backend sha
`4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf`; frontend bundle
`index-YOAeJF7w.js` / `Mesh-BSAo88jN.js`.
- `main` pushed to `gitea-vps2`.
- RPC on `.116`:
- `transport.status` currently reports `mesh_only:false` (off-grid mode is not enabled unless
the user toggles it).
- `mesh.status` reports Meshtastic connected: `device_type:"meshtastic"`,
`self_node_id:1135977788`, `peer_count:13`.
- Recent `.116` -> `3ccc` sent rows are stored with real 2026 timestamps and `transport:"lora"`.
- UI/backend fixes included in `b4531bb4`:
- `transportLabel("lora")` displays **LoRa**.
- mesh sends refetch messages after send so transport pills settle without browser refresh.
- off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends;
banner text is `Tor/FIPS disabled - LoRa only`.
- empty mesh-chat placeholder opacity reduced.
- Meshtastic diagnostics now identify the remaining blocker:
- 3ccc NodeInfo is discovered:
`Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32`.
- Bytes from stock Meshtastic text reach `.116`, but the custom parser rejects the packet:
`Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed`.
- Non-text packets decode and are ignored with port numbers (`portnum=3/4/5`), so the serial
read path is alive. Resume inside `core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet`.
- LoRa is therefore **not fully fixed** yet: stock `3ccc` -> `.116` text does not surface in
`mesh.messages`, and `.116` -> `3ccc` still needs user-visible confirmation in the Meshtastic app.
docs/bitcoin-rpc-relay.md
+2 -2
View File
@@ -171,7 +171,7 @@ http://<archipelago-lan-ip>:80
For the tested node the LAN upstream was:
```text
http://192.168.1.116:80
http://archipelago.local:80
```
The public proxy should serve a valid TLS certificate for the chosen subdomain.
@@ -276,7 +276,7 @@ Expected result:
The working endpoint used in this setup was:
```text
https://shard.tx1138.com/
https://<your-mempool-instance>/
```
It was verified with:
docs/bulletproof-containers.md
+1 -1
View File
@@ -306,7 +306,7 @@ Ordered by likelihood × severity:
2. Read failure-mode memory: `~/.claude/projects/-home-archipelago-Projects-archy/memory/feedback_container_lifecycle_failure_modes.md`
3. Check task list for current release (should start with v1.7.41)
4. Current state on fleet as of 2026-04-22:
- All 4 mirrors (tx1138, gitea-local, .160, .168) synced to v1.7.40-alpha
- All 4 registry mirrors synced to v1.7.40-alpha
- .116, .198, .228, .253 healed manually via `systemd-run chmod 755 /opt/archipelago/web-ui`
- .228 still has stale `bitcoin.conf` rpcauth (regenerated during triage; will drift again until v1.7.43)
- .228 UI companions (archy-bitcoin-ui, archy-lnd-ui) keep vanishing (Quadlet migration in v1.7.45+ fixes)
docs/combined-test-plan-2026-07-22.md
-173
View File
@@ -1,173 +0,0 @@
# Combined test session — 2026-07-22 batch (one sitting)
Staged on **framework-pt** (`100.65.115.109`) AND **archi thinkpad** (this
machine's node) so everything can be tested in one pass. Items marked ✅ were
already verified by the agent on a node; ❑ items need a human.
**What's in this batch:** mesh message/DM persistence across restarts ·
first-message + DM announce fix · 15s announce poll · radio hot-swap modal
(probe / keep-as-is / apply-settings) · whisper beam-1 (release-gated, see §D) ·
real-time wallet push (0-conf tx shows in seconds) · calm Lightning
"still starting" notice · external tx-explorer fallback with consent modal +
wallet-settings On-chain tab · apps open ABOVE modals with the launch
animation · mempool installs no longer blocked by a resyncing ElectrumX ·
[pending: other agents' two push sets — section F fills in when their code
lands].
## H. Wallet & explorer (new — test on the thinkpad node, it's pruned)
1. ❑ **Real-time tx display:** send a small on-chain amount to this node's
wallet → the balance and the yellow "unconfirmed" transaction appear
within a few seconds of broadcast, no refresh, no wallet action.
2. ❑ **External explorer consent:** with no local Mempool app running, tap a
transaction → amber consent modal explains it opens on another node's
mempool (default tx1138.com, placeholder mempool.guide, editable) →
Open Explorer opens `<explorer>/tx/<hash>` in a new tab. Tick "don't ask
again" and confirm the next tap opens directly.
3. ❑ **Wallet Settings → On-chain tab:** explorer URL editable, warning shown,
"don't warn" toggle; tabs now read Channels / Cashu / Fedi / Ark / On-chain
and fit on one row (check mobile too).
4. ❑ **Modal → app animation:** on a node WITH Mempool running, open
Transactions and tap a tx → the Mempool app animates in ABOVE the modal
(previously loaded invisibly underneath); closing it returns to the modal.
5. ❑ **Lightning "still starting":** right after a node restart, try opening a
channel → either it just works (silent retry) or a calm amber ⏳ notice
appears — never the red "Failed to connect to peer" error.
---
## A. Staged state (agent-verified before you start)
- ✅ Dev binary (persistence + announce seeder + hot-swap) on
`/usr/local/bin/archipelago`, service healthy, no crash-loop.
- ✅ Frontend bundle with the new device modal at `/opt/archipelago/web-ui`.
- ✅ Seeder re-ran: `automations.yaml` upgraded v1→v2 (first-message announce),
`configuration.yaml` rest block at `scan_interval: 15`, HA restarted clean.
- ✅ `mesh-messages.json` persisting + restored across a service restart.
- ✅ `mesh.probe-device` returns real firmware details for the plugged stick.
## B. Mesh history survives restarts (the "messages go missing" fix)
1. ❑ Open Mesh chat — your existing DM/channel history from today is visible.
2. ❑ Send one channel message and one DM (either direction).
3. ❑ Reboot the whole node (not just the service). After it's back: history
still there, including the two new messages, correct timestamps/senders.
4. ❑ Send a NEW message to another node right after the reboot and confirm the
other side receives it (this exercises the send-seq fix — before it, the
first post-reboot sends were silently dropped by peers as replays).
## C. Speaker announcements
1. ❑ Have another node send a **public channel** message → speaker announces
sender + text within ~15s (was ~30s).
2. ❑ Have another node send you a **DM** → speaker announces it the same way.
3. ❑ Restart Home Assistant (or the node) → the last old message is NOT
re-announced (no announce storm).
4. ❑ (First-message case — the original bug — only reproducible on a node with
an empty history: optional, covered by agent verification of the guard.)
## D. Voice (regression + speed)
1. ❑ "Hey Jarvis, what's the block height" and one fuzzy phrasing — same
correct answers as before (no behavior change is the pass condition).
2. ⓘ The ~45% faster speech-to-text (whisper beam-1) ships via the **signed
catalog in the release** — it is NOT on the node during this test session.
Benchmarked on this exact hardware: identical transcripts, 0.94s → 0.51s.
## E. Radio hot-swap modal (your Reticulum stick is already plugged in)
1. ❑ Open the web UI anywhere — within ~30s a "Mesh Radio Detected" modal
appears showing the stick on `/dev/ttyACM0`, with a card of what's on it
(firmware badge: Reticulum RNode / MeshCore / Meshtastic + current
name/region/channels where the firmware exposes them).
2. ❑ Press **Keep As Is** → mesh connects using the radio exactly as flashed
(check Mesh → Device tab: connected, firmware type correct; nothing on the
radio changed).
3. ❑ Unplug the stick, plug the old MeshCore one → the modal appears AGAIN
(every plug re-triggers, same or different /dev path).
4. ❑ This time press **Set Up with Archipelago Settings** → second screen
shows channel `archipelago`, your region, and the node's RF params (the
validated Portugal preset on this fleet) BEFORE anything is written;
confirm → radio provisions and joins the mesh.
5. ❑ Swap sticks once more with no UI interaction except "Keep As Is" — chat
still works end-to-end afterwards (hot-swap without ceremony).
## F. Companion pairing + mobile onboarding (other agent — push set #1, MERGED)
1. ❑ Companion app: pair with the node via the new named QR (device tokens) —
pairing completes instantly, device appears in the paired-devices list.
2. ❑ Remote access now rides the embedded FIPS mesh (WireGuard replaced):
with the phone OFF the node's WiFi, the companion still reaches the node.
3. ❑ The reworked mobile onboarding/intro overlay screens flow correctly on
first launch of the new APK (in-tarball APK is the 27MB build).
4. ❑ (Push set #2 from the other agents is still pending — the release waits
for it; this staged build does NOT include it yet.)
## G. Quick regressions
1. ❑ Pine launcher page (:10380) still shows the live node card; "Connect
Pine to WiFi" button loads without JS errors.
2. ❑ Mobile Home: wallet card directly under My Apps (G5 from the voice epic).
3. ❑ Mesh RF settings panel (Mesh → Device) still loads and saves.
## H. LoRa radio firmware flashing (Heltec V3/V4, new — extends Section E)
Full v1 scope is 3 firmware families × 2 boards (6 cells); mark each cell
tested on real hardware vs. code-reviewed only as this is run.
1. ❑ From the hot-swap modal's step 1 (device already probed), press
**Flash Firmware…** → new step shows firmware-family + board pickers and
the erase-confirmation checkbox; "Erase & Flash Now" stays disabled until
family, board, AND the checkbox are all set.
2. ❑ Confirm what's currently on the test stick via the existing probe
BEFORE flashing it — don't flash the only known-good device without a
fallback board on hand.
3. ❑ Prefer a spare Heltec V3/V4 for the first destructive erase+flash run;
only exercise a primary/in-use stick once the flow is proven safe.
4. ❑ MeshCore → Heltec V3: erase + write completes, progress bar and log
tail update live, ends at "Flash complete".
5. ❑ Meshtastic → Heltec V3: same, using the extracted `*.factory.bin` from
the esp32s3 release zip.
6. ❑ Reticulum/RNode → Heltec V3: `archy-rnodeconf --autoinstall` path
completes (no raw esptool erase/write step for this family — see
`mesh/flash.rs` doc comment).
7. ❑ Repeat 4-6 against a Heltec V4. Confirmed 2026-07-23 on real hardware:
V4 uses the ESP32-S3's native-USB JTAG/serial peripheral (vid:pid
303a:1001, generic to every native-USB ESP32-S3 board, not V4-specific)
— so unlike V3's CP2102 bridge chip, V4 is permanently NOT auto-matchable
by vid:pid. Board auto-detect should fail closed for it every time
(manual board selection required, "couldn't confirm automatically"
warning shown) — this is expected steady-state behavior, not a gap to
close later.
8. ❑ After a successful flash, the modal automatically re-probes and shows
the NEW firmware's badge/details — same as unplugging and replugging
(Section E item 3), but without physically touching the cable.
9. ❑ Deliberately test a failure path once (disconnect the board mid-write,
or point at a bad cached asset) — confirm the error surfaces in the
progress log AND that `docs/troubleshooting.md`'s "LoRa radio firmware
flash failed" recovery steps (BOOT+RST bootloader entry, manual esptool/
rnodeconf command) actually get the board back to a flashable state.
10. ❑ Cancel button only appears (and only works) while still in the
"Downloading firmware…" stage — once erasing/writing starts, no cancel
affordance is offered.
11. ❑ **Boot-loop regression (2026-07-23 incident)**: after a *failed* flash
(e.g. kill network access mid-download to force a failure), confirm the
mesh listener does NOT auto-resume — `journalctl -u archipelago` should
show a single `Leaving mesh listener stopped after failed flash` line
and then go quiet for that device, not a repeating `mesh::serial:
Opened serial port... Starting Meshcore handshake` cycle every few
seconds. Reconnect manually via the hot-swap modal afterward and confirm
it connects normally (the board itself should be untouched — the
download fails before esptool/rnodeconf ever runs).
12. ❑ Separately, force a device to flap connected/disconnected a few times
in under 20s each (e.g. a marginal USB connection) and confirm
`reconnect_delay` in the logs actually escalates (5s → 10s → 20s → ...)
rather than resetting to 5s on every attempt — see
`STABLE_SESSION_THRESHOLD` in `mesh/listener/mod.rs`.
---
After this passes: fold the batch + other agent's work into the next release
(OTA binary + frontend tarball + catalog regen/sign/publish for pine-whisper
3.4.2), then re-run `tests/lifecycle/run-gate.sh` on .228 (back online as
Tailscale `shorty-s`).
docs/companion-pairing-qr.md
+2 -2
View File
@@ -30,7 +30,7 @@ Query parameters:
| param | required | meaning |
|-------|----------|---------|
| `v` | yes | Payload version, currently `1`. Reject/ignore unknown majors gracefully — show "please update the app". |
| `url` | yes | Full origin the app should connect to, scheme included: `https://demo.archipelago-foundation.org`, `http://archipelago.local`, `http://192.168.1.228`, etc. No trailing slash guaranteed either way — normalize. |
| `url` | yes | Full origin the app should connect to, scheme included: `https://demo.archipelago-foundation.org`, `http://archipelago.local`, `http://192.0.2.10`, etc. No trailing slash guaranteed either way — normalize. |
| `name`| no | Display name for the server entry. Real nodes send the configured server name, or `My Archipelago` when it's still the factory default. |
| `tok` | no | **Device token** minted via `auth.createDeviceToken` when the QR is rendered. The app logs in with `{"method":"auth.login","params":{"token":"…"}}` — same endpoint, same rate limiter, skips TOTP (the token was minted from an authenticated session). Long-lived until re-minted (re-showing the pair screen replaces the `companion` token) or revoked (`auth.revokeDeviceToken`). Scan → instantly connected, no typing. |
| `pw` | no | Login password. **Only present in the public demo** (shared demo password `entertoexit`). Real nodes never embed a password — the frontend doesn't have it. |
@@ -43,7 +43,7 @@ Query parameters:
Examples the web UI actually emits:
- Demo: `archipelago://pair?v=1&url=https%3A%2F%2Fdemo.archipelago-foundation.org&pw=entertoexit`
- Real node, browsed via LAN IP: `archipelago://pair?v=1&url=http%3A%2F%2F192.168.1.228`
- Real node, browsed via LAN IP: `archipelago://pair?v=1&url=http%3A%2F%2F192.0.2.10`
- Real node kiosk (UI runs on localhost, so it advertises the mDNS name from
`system.get-hostname`): `archipelago://pair?v=1&url=http%3A%2F%2Farchipelago.local`
docs/hotfix-process.md
-52
View File
@@ -1,52 +0,0 @@
# Hotfix Process
For critical bugs discovered after a tagged release.
## Severity Classification
| Level | Response Time | Examples |
|-------|--------------|---------|
| P0 — Critical | < 4 hours | Data loss, security vulnerability, node bricked |
| P1 — High | < 24 hours | App won't start, auth broken, major UI failure |
| P2 — Medium | < 72 hours | Non-critical feature broken, performance regression |
| P3 — Low | Next release | Cosmetic, minor UX, edge cases |
## Hotfix Workflow
### 1. Triage
- Reproduce the issue on dev server (192.168.1.228)
- Classify severity (P0-P3)
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
### 2. Fix
- Create branch: `hotfix/vX.Y.Z-description`
- Fix the issue with minimal code changes
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
- Verify fix on live server
### 3. Release
- Merge hotfix branch to `main`
- Tag: `vX.Y.Z` (increment patch version)
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
`releases/manifest.json` and signs it)
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
### 4. Communicate
- Update RELEASE-NOTES with hotfix details
- Note in CHANGELOG.md
## Monitoring Dashboards
- **Uptime monitor**: `/var/lib/archipelago/uptime-monitor/summary.json`
- **Soak test**: `/tmp/stability-test-*.log` on dev server
- **Health endpoint**: `http://192.168.1.228/health`
## Rollback
If a hotfix causes regressions:
1. The updater self-verifies after applying (health check on restart) and rolls the
binary back automatically if the new one fails to come up
2. Point `releases/manifest.json` back at the last-known-good version and push
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
docs/operations-runbook.md
-366
View File
@@ -1,366 +0,0 @@
# Archipelago Operations Runbook
Quick reference for common operational tasks on Archipelago nodes.
**Primary node**: `192.168.1.228` (Arch 1)
**Secondary node**: `192.168.1.198` (Arch 2)
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
---
## 1. Check Node Health
```bash
# Quick health check (from any machine)
curl http://192.168.1.228/health # Should return "OK"
curl http://192.168.1.198/health
# Detailed system stats via RPC
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"system.stats"}' \
http://192.168.1.228:5678/rpc/v1
# Check services
ssh archipelago@192.168.1.228
sudo systemctl status archipelago # Backend service
sudo systemctl status nginx # Web server
sudo systemctl status tor # Tor hidden services
```
## 2. Check Container Status
```bash
# List all containers
podman ps -a
# Running count
podman ps --format '{{.Names}}' | wc -l
# Find exited/crashed containers
podman ps -a --filter status=exited
# Container logs
podman logs {container-name} --tail 50
# Container resource usage
podman stats --no-stream
```
## 3. Fix Crashed Containers
```bash
# Restart a specific container
podman restart {container-name}
# If container won't start, check logs first
podman logs {container-name} --tail 100
# Remove and recreate (last resort)
podman rm -f {container-name}
# Then redeploy with: ./scripts/deploy-to-target.sh --live
# The health monitor auto-restarts containers every 60s
# Check its status:
sudo journalctl -u archipelago --grep="health_monitor" --no-pager -n 20
```
## 4. Add/Remove Federation Peers
```bash
# Generate invite code (on inviting node)
# Via UI: Federation page > Generate Invite
# Via RPC:
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.invite"}' \
http://localhost:5678/rpc/v1
# Join federation (on joining node)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.join","params":{"invite_code":"{code}"}}' \
http://localhost:5678/rpc/v1
# List peers
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"federation.list-nodes"}' \
http://localhost:5678/rpc/v1
# Remove a peer
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.remove-node","params":{"did":"{peer-did}"}}' \
http://localhost:5678/rpc/v1
```
## 5. Rotate Tor Address
```bash
# Delete current hidden service keys
sudo rm -rf /var/lib/tor/hidden_service/
sudo systemctl restart tor
# Wait for new hostname
sleep 15
sudo cat /var/lib/tor/hidden_service/hostname
# The backend picks up the new address automatically (30s refresh)
# Federation peers need to re-discover via sync
```
## 6. Create/Restore Backups
```bash
# Create encrypted backup (via RPC)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.create","params":{"passphrase":"your-passphrase","description":"manual backup"}}' \
http://localhost:5678/rpc/v1
# List backups
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.list"}' \
http://localhost:5678/rpc/v1
# Verify backup integrity
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.verify","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Restore (warning: overwrites current identity/data)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.restore","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Backup files stored at: /var/lib/archipelago/backups/
```
## 7. Update the Node
```bash
# From development machine:
./scripts/deploy-to-target.sh --live # Deploy to .228
./scripts/deploy-to-target.sh --both # Deploy to both nodes
./scripts/deploy-to-target.sh --dry-run --live # Preview changes
# The deploy script:
# 1. Syncs code to target
# 2. Builds frontend (vue-tsc + vite)
# 3. Builds backend (cargo build --release)
# 4. Deploys binary, frontend, configs
# 5. Restarts services
# 6. Verifies health
```
## 8. Diagnose High CPU
```bash
# Check system load
uptime
# Find CPU-heavy processes
top -b -n 1 | head -15
# Check container CPU usage
podman stats --no-stream --format '{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
# Common causes:
# - Bitcoin IBD (initial block download): normal, takes days
# - Container crash loops: check `podman ps -a --filter status=exited`
# - mempool-electrs indexing: normal after Bitcoin sync
```
## 9. Diagnose High Memory
```bash
# Check memory
free -h
# Check swap usage
swapon --show
# Per-container memory
podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
# Check for OOM kills
dmesg --level=err,crit | grep -i oom
# Add swap if missing
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
## 10. Diagnose Disk Space
```bash
# Disk usage overview
df -h /
# Find large directories
sudo du -h --max-depth=2 /var/lib/archipelago/ | sort -rh | head -20
# Container image sizes
podman images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}'
# Clean unused images
podman image prune -a
# Clean old journal logs
sudo journalctl --vacuum-size=500M
```
## 11. Check Tor Connectivity
```bash
# Tor service status
sudo systemctl status tor
# Get onion address
sudo cat /var/lib/tor/hidden_service/hostname
# Test self-connection via Tor
curl --socks5-hostname 127.0.0.1:9050 http://$(sudo cat /var/lib/tor/hidden_service/hostname)/health
# Test cross-node Tor
curl --socks5-hostname 127.0.0.1:9050 http://{peer-onion}/health
```
## 12. Check DWN Sync
```bash
# DWN status (via RPC, needs auth)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.status"}' \
http://localhost:5678/rpc/v1
# Trigger manual sync
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.sync"}' \
http://localhost:5678/rpc/v1
# Check message count
ls /var/lib/archipelago/dwn/messages/ | wc -l
```
## 13. Restart Services
```bash
# Restart backend only
sudo systemctl restart archipelago
# Restart nginx
sudo systemctl restart nginx
# Restart Tor
sudo systemctl restart tor
# Full service restart (backend + nginx)
sudo systemctl restart archipelago nginx
# Reboot (containers auto-recover via restart policy + health monitor)
sudo reboot
```
## 14. View Logs
```bash
# Backend logs
sudo journalctl -u archipelago --no-pager -n 100
# Follow logs in real time
sudo journalctl -u archipelago -f
# Nginx access log
sudo tail -f /var/log/nginx/access.log
# Nginx error log
sudo tail -f /var/log/nginx/error.log
# Container logs
podman logs {container-name} --tail 50 -f
```
## 15. Network Diagnostics
```bash
# Check listening ports
sudo ss -tlnp
# Check firewall rules
sudo ufw status verbose
# Required ports:
# 22 - SSH
# 80 - HTTP (nginx)
# 443 - HTTPS (nginx)
# 5678 - Backend API (localhost only, proxied by nginx)
# 8332 - Bitcoin RPC (container network only)
# 9050 - Tor SOCKS proxy (localhost only)
# If ports are blocked after reboot, re-add UFW rules:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.88.0.0/16 # Podman container subnet
sudo ufw allow from 10.89.0.0/16 # Podman container subnet
```
## 16. Emergency: Node Won't Boot
If a node responds to ping but SSH/HTTP are down:
1. **Check UFW**: After reboot, UFW may block all ports
```bash
# If you have console access:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload
```
2. **Check services**: SSH or nginx may not have started
```bash
sudo systemctl start ssh
sudo systemctl start nginx
sudo systemctl start archipelago
```
3. **Check disk**: If root filesystem is full, services won't start
```bash
df -h /
sudo journalctl --vacuum-size=200M
podman image prune -a
```
## 17. Run Tests
```bash
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
# Cross-node suites (federation/mesh):
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
# E2E / post-install:
./scripts/run-e2e-tests.sh
./scripts/run-post-install-tests.sh
```
docs/pine-voice-release-test-plan.md
-60
View File
@@ -1,60 +0,0 @@
# Framework PT test plan — Pine voice epic (pre-release gate)
Target node: **framework-pt** (`100.65.115.109`, LAN 192.168.1.249). Run after
BOTH agents' work is merged, with the dev binary sideloaded and the signed
catalog (pine 1.3.0 + pine-openwakeword) published. Every ❑ must pass before
the release ritual starts. Items marked **(user)** need a human in the room.
## A. Deploy / prerequisites
- ❑ A1 Dev binary sideloaded, `archipelago` service active, no crash-loop in journal.
- ❑ A2 nginx self-heal added `location /api/pine/status` to every server block; `nginx -t` passes; nginx reloaded.
- ❑ A3 Signed catalog with pine 1.3.0 + pine-openwakeword live at the raw URL; node refreshed it (hourly sweep or "Check for updates").
## B. `/api/pine/status` endpoint
- ❑ B1 Public tier through nginx (`curl http://127.0.0.1/api/pine/status`): version, uptime, bitcoin height/sync_percent/peers, mesh peers. `lightning` null, `mesh_message` absent.
- ❑ B2 Wrong bearer token → still public-only (no balances). Correct token (from `/var/lib/archipelago/secrets/pine-status-token`) → lightning balances + latest mesh message present.
- ❑ B3 Reachable from inside the HA container via `host.containers.internal:80`.
- ❑ B4 Token file is 0600, owned by the service user.
## C. Stack / openwakeword container
- ❑ C1 Reconcile installs `pine-openwakeword` (wyoming-openwakeword 2.1.0), healthy on :10400.
- ❑ C2 Existing pine-whisper / pine-piper / pine were ADOPTED, not recreated — model data dirs untouched.
- ❑ C3 `archipelago` service restart → all four pine containers come back (crash-recovery stack spec).
- ❑ C4 UI: openwakeword listed under Services (no extra store card); Pine card shows 1.3.0.
## D. Home Assistant seeding
- ❑ D1 configuration.yaml: legacy hand-staged block (bitcoind :18332 + plaintext RPC creds) fully replaced by the bounded token-based block.
- ❑ D2 `custom_sentences/en/archy.yaml` carries all four intents.
- ❑ D3 `.storage/core.config_entries`: wyoming entry for openwakeword (:10400) + `anthropic` entry (Claude, conversation + ai_task subentries).
- ❑ D4 Pipeline: `conversation_engine = conversation.claude_conversation`, `prefer_local_intents: true`.
- ❑ D5 automations.yaml: `archy_mesh_announce` seeded.
- ❑ D6 HA restarts clean — no setup errors for anthropic / wyoming / rest / intent_script in `podman logs homeassistant`.
- ❑ D7 Sensors report real values: archy_block_height, archy_bitcoin_sync, archy_bitcoin_peers, archy_mesh_peers, archy_lightning_balance (or clean unavailable if LND absent), archy_mesh_message.
## E. Voice / intents (API level first, then live speaker)
- ❑ E1 Exact phrase "what's the block height" → answered by the LOCAL intent (correct height, no Anthropic API call in HA logs).
- ❑ E2 Fuzzy phrase (e.g. "how tall is the chain right now") → Claude routes to the ArchyBlockHeight tool; answer contains the real height.
- ❑ E3 "how many peers", "is the node synced", "what's my lightning balance" → correct spoken-length answers.
- ❑ E4 Off-topic question → Claude answers, 1–2 sentences, no markdown.
- ❑ E5 **(user)** Live speaker: "Hey Jarvis, what's the block height" → audible correct answer.
- ❑ E6 Mesh announce: new received mesh text (or manual `assist_satellite.announce` if no radio) → speaker announces sender + text; no announce storm on HA restart.
## F. Pine launcher page (1.3.0)
- ❑ F1 Page on :10380→:10381 shows the live node card (version, uptime, block, sync, peers) within ~5s.
- ❑ F2 `/node-status` proxy works (pine nginx resolves host.containers.internal at startup — container must not crash-loop).
- ❑ F3 "Connect Pine to WiFi" provisioner still intact (no JS errors on load).
## G. Cleanup / regression sweep
- ❑ G1 Both stray socat 18332 forwarders killed; sensors still work via the endpoint.
- ❑ G2 No bitcoind RPC credentials anywhere in HA config.
- ❑ G3 Pre-existing HA function intact: whisper/piper entities, PineVoice satellite pairing, other integrations.
- ❑ G4 nginx regressions: `/health`, `/bitcoin-status`, `/api/app-catalog`, `/proxy/lnd/` all still proxied post-patch.
- ❑ G5 **(user)** Mobile Home: wallet card sits directly under My Apps; desktop layout unchanged.
- ❑ G6 Other agent's changes re-verified after merge (their own checklist).
## H. Production-readiness (release ritual gate)
- ❑ H1 `cargo test` workspace green; frontend builds; drift check `--release --strict` green.
- ❑ H2 `tests/lifecycle/run-gate.sh` re-run ON .228 (stack membership changed → lifecycle gate rule applies).
- ❑ H3 Catalog regenerated → signed (ceremony) → published via gitea-ai; verified at the raw URL.
- ❑ H4 Changelog (layman-readable) + `scripts/sync-whats-new.py` + version bump; release ritual per v1.7.110 notes (push main via gitea-ai BEFORE publish; sign manifest AFTER create-release).
- ❑ H5 No secrets in any commit; frontend tarball flat + APK policy per release notes.
docs/security/KEY-02-FLEET-ROTATION.md
-244
View File
@@ -1,244 +0,0 @@
# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)
Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
build half (10-03).
10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
TLS private key that every downloader of that ISO also holds, and it will never try again.
This document records the two human decisions that govern the deployed half.
---
## D-06 rotation trigger
**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.
Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
flag").
### Why
Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
machine that has ever connected to it, and the old private key is destroyed by the swap. The
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
node needs physical console access to recover, which for `.228` means a site visit.
Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
of nodes still to rotate is a fact on a screen rather than a guess.
This also matches the project's standing policy that changes are verified on the dev pair
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
has been observed on the dev pair it has already run everywhere.
### What this decision binds
- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
invocation is inert.
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
It contains no apply path. Making the boot unit rotate would require editing the unit, which is
a deliberate act, not a default.
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
the guard against "operator runs it on the wrong node" is structural, not procedural.
### Consequence recorded honestly
Any node whose verdict comes back `shared` and which is never revisited stays exposed
indefinitely. The mitigation is the visibility, not the automation. The list under
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.
---
## How a node decides
Four on-disk signals, evaluated in this precedence order by
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
that produced it, and each evidence string names the file it was read from.
| # | Signal | Source |
|---|---|---|
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |
Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.
**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
mtime is not evidence of success.
Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.
---
## C-3 — per-node host key and TLS uniqueness
Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
there as "the highest-value check here".
### Status: **FAILED — with finding.** Recorded 2026-08-02.
> **This section names live fleet nodes that are still running shared key material.
> Review it before this repository is made public** (`docs/OPEN-SOURCE-READINESS-PLAN.md`).
> Digests below are truncated; the fingerprints of public keys are public data — every SSH
> handshake offers them — but there is no reason to make a target list convenient.
**Three distinct live fleet nodes share all three of their SSH host keys. Two of those three
also share their TLS certificate, and therefore their TLS private key.** This is not a
theoretical exposure: it is F-03 in production, today.
#### Method
Gathered **remotely and read-only** — no node was logged into, nothing was written to any node,
nothing was rotated. Host keys came from `ssh-keyscan`, which is what every SSH client does
before it decides whether to trust a host, and certificates from an anonymous TLS handshake:
```bash
ssh-keyscan -T 6 <node> | ssh-keygen -lf -
openssl s_client -connect <node>:443 </dev/null 2>/dev/null \
| openssl x509 -noout -fingerprint -sha256 -subject
```
This is a deliberately weaker instrument than the checklist's on-node commands, and it was chosen
because it needs no access and can therefore cover the whole reachable fleet rather than two
nodes. What it can prove is exactly the FAIL condition: *any fingerprint appearing on two nodes*.
#### Result
| Node label | SSH host keys (ECDSA/ED25519/RSA, truncated) | TLS cert sha256 (truncated) | Cert CN |
|---|---|---|---|
| `archipelago-1` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archy-x250-beta` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archipelago` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `7C:6B:CD:98…` | `austin-sapien` |
| `archipelago-5` | `/bmgd6jS…` / `SpaNfLLf…` / `hhVFABi3…` | `95:FE:EB:C7…` | `archipelago.local` |
| `archi-dev-box` | `8hFU7QGM…` / `GAxNAcgX…` / `Tv7AfaVp…` | (no :443 listener) | — |
| `archy-dev-pa` | `JtD/RM0a…` / `XD2A5OVL…` / `esIBpbWk…` | not probed | — |
| `framework-pt` | `oicpsj3Y…` / `zxA1/kRU…` / `oxi+tMli…` | `88:85:CE:CC…` | `framework-pt` |
| `shorty-s` (`.228`) | `YVsgrv8M…` / `D/5n851i…` / `YMFLUerk…` | `4D:98:D4:9B…` | `shorty-s` |
Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`,
`quantumterminal`.
#### That the three are genuinely different machines, not one host seen three times
The obvious alternative explanation for identical host keys is a single machine registered on the
tailnet more than once. Ruled out:
- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled`
instance serves one tailnet identity, so three simultaneously-live addresses are three hosts.
- `tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers
from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago`
answer from `45.20.199.86` on different source ports — a different continent for the first,
and two distinct machines behind one NAT for the other two.
- They are owned by different tailnet accounts.
#### Why `archipelago` has a different TLS cert but the same SSH keys
Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a
node that was **renamed** through `server.set-name`, which re-mints the TLS cert via
`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else.
This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that
a node's key material is per-node.** Any renamed node gets a unique certificate for free while
its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS
fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable
signal, and this is why the audit script treats the two classes separately and reports which one
is shared rather than issuing a single node-level verdict.
#### What this does NOT establish — UNVERIFIED
| Claim | Status | Evidence still needed |
|---|---|---|
| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. |
| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. |
| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. |
| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. |
| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. |
#### Consequence
`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**.
Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys,
and for the first two, their TLS private key as well — enough for undetectable SSH host
impersonation and transparent MITM of the web UI.
None of them was rotated as part of this verification, and that is deliberate: this checkpoint
verifies, it does not remediate, and remediating a node inside a verification task is how a
verification task takes a node offline. They are recorded below.
---
## Nodes with a `shared` verdict, deliberately not rotated
Any node that reports `shared` and is not rotated in the same session MUST be added here with the
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
list rather than an assumption.
| Node label | Date detected | Why not rotated | Next step |
|---|---|---|---|
| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. |
| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. |
| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. |
**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use;
the rotation is one-way and every existing entry for them dies with it. Sequencing that is an
operator decision, which is the whole content of D-06.
---
## Operator runbook — rotating one node
Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
any node in real use without arranging access recovery first.
```bash
# 1. Detect. Read-only; safe on any node, including production.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
cat /var/lib/archipelago/host-secrets-audit.json
# 2. Dry run. Prints the plan, touches nothing, exits 0.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply
# 3. Rotate. Only proceeds if the verdict is `shared`.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes
# 4. WITHOUT closing that session, prove it survived:
echo still-here
# 5. From a second terminal, expect a host-key mismatch warning. That is the
# correct outcome. Update known_hosts against the fingerprints printed by
# step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
# blindly accepting whatever is offered.
ssh-keygen -R <node>
ssh <node>
# 6. The web UI will present a new self-signed cert. A fresh browser trust
# prompt is expected and is the correct outcome.
```
The script reloads sshd rather than restarting it. A reload re-execs the listener while
already-forked session children keep running, which is why the operator's own SSH session
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
unrecoverable.
Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
swap, so an operator who loses access anyway can still identify what changed.
docs/security/KEY-03-SIGNING-POSTURE.md
-448
View File
@@ -1,448 +0,0 @@
# KEY-03 — Signing posture after the Bitcoin Core wallet deletion
> **What this document is.** The evidence-backed record of how Archipelago's Bitcoin signing
> posture stands after Phase 10 KEY-03. It supersedes, for the Bitcoin Core wallet specifically,
> the target state described in `docs/security/PSBT-SIGNING-ARCHITECTURE.md` §8 Phase 1 — that
> phase planned to *convert* Core's wallet to watch-only; **D-07b deleted the path instead.**
>
> **Governing decisions:** `.planning/phases/10-key-material-hardening/10-CONTEXT.md`
> **D-07b** (final KEY-03 scope — delete, do not migrate) and **D-07c** (the deferred BDK cold
> vault, recorded so it is not lost with the code). D-07b supersedes D-07 and D-07a's conditional
> migration.
>
> **Audit finding closed:** F-13 (High) —
> `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md:604`, remediation register R-04.
---
## Bitcoin Core wallet path — deleted (D-07b)
### What was deleted
| Symbol | Kind | Location before deletion |
|---|---|---|
| `handle_bitcoin_init_wallet_from_seed` | `async fn` | `core/archipelago/src/api/rpc/bitcoin.rs:161-295` |
| `"bitcoin.init-wallet-from-seed"` | JSON-RPC dispatch arm | `core/archipelago/src/api/rpc/dispatcher.rs:122-124` |
### The defect (F-13)
The handler loaded the encrypted seed, derived the **BIP-84 account extended private key**
(`crate::seed::derive_bitcoin_xprv`, `bitcoin.rs:188`), stringified it (`:189`), and imported
`wpkh(xprv/0/*)` and `wpkh(xprv/1/*)` (`:230-231`) into a Bitcoin Core descriptor wallet created
with `disable_private_keys = false` (`:203`) and an **empty** wallet passphrase (`:205`).
The result was a **second copy of the node's spending key**, persisted in Core's `wallet.dat`
inside the Bitcoin container's data volume, with no Argon2 passphrase — while the first copy sits
in the daemon's Argon2 + ChaCha20-Poly1305 envelope written `0600`
(`core/archipelago/src/seed.rs:238-269`, `:318-324`). That duplication, into weaker protection,
was the entire finding.
### Evidence that deletion was the right close (re-established for this task, not inherited)
The four D-07a evidence points, verified again against the tree before anything was removed:
**1. No caller anywhere.** Repo-wide search across `core/`, `neode-ui/src`, `scripts/`, `web/`,
`apps/`, `tests/` and `docs/`, excluding `core/target`, `node_modules` and `.git`:
```
$ grep -rn 'bitcoin\.init-wallet-from-seed' core/ neode-ui/src scripts/ web/ apps/ tests/ docs/
core/archipelago/src/api/rpc/dispatcher.rs:122: "bitcoin.init-wallet-from-seed" => {
$ grep -rn 'handle_bitcoin_init_wallet_from_seed' core/ neode-ui/src scripts/ web/ apps/ tests/ docs/
core/archipelago/src/api/rpc/bitcoin.rs:161: pub(super) async fn handle_bitcoin_init_wallet_from_seed(
core/archipelago/src/api/rpc/dispatcher.rs:123: self.handle_bitcoin_init_wallet_from_seed(params).await
docs/UNIFIED-TASK-TRACKER.md:208: §8 Phase 1). `handle_bitcoin_init_wallet_from_seed` passes
docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md:607:(`handle_bitcoin_init_wallet_from_seed`):
docs/security/PSBT-SIGNING-ARCHITECTURE.md:147: `handle_bitcoin_init_wallet_from_seed`, `core/archipelago/src/api/rpc/bitcoin.rs:161-294`).
```
Exactly one occurrence of the method name (its own dispatcher registration) and two of the symbol
in code (its definition and the dispatcher call). The three remaining symbol hits are prose in
documentation — the audit, the task tracker, and the PSBT architecture spec — not callers. No
frontend, script, test or other Rust module invoked it.
**2. LND is the wallet the product actually drives.** Across all of `neode-ui/src`, every
`bitcoin.*` RPC call is read-only status: `bitcoin.getinfo` (14 call sites),
`bitcoin.prune-status` (3), `bitcoin.onion` (1). There are **no** `bitcoin.*` wallet operations.
The wallet UI (`Web5Wallet.vue`, `SendBitcoinModal.vue`) sends via `lnd.sendcoins`, estimates via
`lnd.estimatefee`, and reads balance via `lnd.getinfo`.
**3. The wallet it creates never existed on the reference node.** Verified live on
**archi-dev-box, 2026-08-02**, against the running `bitcoin-knots` container (read-only RPCs
only — see the census section for the exact commands and the standing ban on
`listdescriptors true`):
```
listwalletdir → { "wallets": [ "gatewayd-02004b91…", "gatewayd-03443c0c…", "" ] }
listwallets → [ "" ]
```
**There is no wallet named `archipelago`** — the handler's default `wallet_name`
(`bitcoin.rs:170-173`). It has never run on this node. `getwalletinfo` on the one loaded wallet
(the unnamed default) reports:
```
walletname: "" blank: true keypoolsize: 0
txcount: 0 balance: 0.00000000
descriptors: true private_keys_enabled: true
```
`blank: true` with `keypoolsize: 0` and `txcount: 0` is Bitcoin Core's own statement that **no
key was ever imported into it and no transaction ever touched it**. The two `gatewayd-*` entries
are Fedimint gateway wallets, unrelated to the BIP-84 path. The `wallet.dat` at the datadir root
is Core's own legacy default-wallet location, not this handler's output.
**This is one node.** The same check was subsequently run across the reachable fleet — see the
census below: **4 nodes examined and clear, 6 unreachable and therefore unknown.**
**Supporting history evidence:** `git log -S "init-wallet-from-seed"` scoped to
`core/archipelago/src/api/rpc/dispatcher.rs` and `neode-ui/src` returns exactly one commit —
`19dcfd4f feat: BIP-39 master seed for unified key derivation`, the commit that **added** it. No
frontend wrapper was ever written: it was built and never wired up.
**4. It was never remotely reachable.** The endpoint is absent from `UNAUTHENTICATED_METHODS`
(`core/archipelago/src/api/rpc/middleware.rs:5-40`) — so it required an authenticated session —
**and** it additionally re-verified the user's password before touching the seed
(`self.auth_manager.verify_password(password)`, `bitcoin.rs:176-179`). **F-13 was therefore
key-at-rest duplication, not an exposed endpoint.** That is why it was rated High rather than
Critical, and why deleting it is a hardening measure rather than an incident response.
### What was *not* wrong with it
Worth stating so the record is fair, and so the next reader does not mistake the lesson. The
in-memory handling of the xprv string was **careful**: it was zeroized on the error path
(`bitcoin.rs:222`) and on the success path (`:284`), matching the standard set elsewhere in
`seed.rs`. The wallet type was also correct — `createwallet` already passed `descriptors = true`
(`:207`), which is the right foundation.
**The defect was which key went into the wallet, not how the key was held in memory or what kind
of wallet it was.** A watch-only rewrite (xpub + `[fingerprint/derivation]` key origin) would
have been a legitimate fix. Deletion was chosen over rewrite because the endpoint had no caller,
no consumer, and no product role: rewriting it would have produced a correct implementation of
something nothing uses, and left a wallet-creating code path to be maintained and re-audited
forever.
### How F-13 is closed
**By removal, not by conversion to watch-only.** After this change there is no code path in the
daemon that writes the BIP-84 account private key into Bitcoin Core. The only on-node copy of
that key is the daemon's Argon2 + ChaCha20-Poly1305 envelope.
**No migration was performed and none is planned.** D-07's parity-proof migration and its
one-way checkpoint are **withdrawn** (D-07b) — there is no wallet to migrate. If a fleet node is
ever found holding a descriptor wallet this handler created, that is a **finding to surface and
stop on**, not a trigger to auto-migrate: it would mean the endpoint was invoked by hand and that
node's spending key is duplicated in Core, which deserves a human decision rather than an
automated rewrite of a wallet that may hold funds.
### This deletion removes code, not wallets
Stated explicitly so nobody reading the change later has to wonder whether it was destructive:
> **Nothing on disk is touched.** No `wallet.dat` is modified, unloaded or removed. No funds
> move. No LND state, secret, descriptor or seed is altered. The change removes a Rust function
> and a `match` arm — the *path* by which a private key could be imported into Bitcoin Core —
> and nothing else.
This holds even on a hypothetical node where the endpoint had been invoked by hand: deleting the
handler destroys nothing there either. It closes the door; it does not clean the room. Cleaning
up such a wallet, if one is ever found, is a separate human decision (see the census below), and
CLAUDE.md's **"migrations never destroy data"** invariant is not engaged by this change because
there is no migration.
### What deletion does to D-08 and D-09
Neither decision lapses; both are satisfied by a different mechanism.
- **D-08** asked that the spending key exist in exactly one place, with an opt-in air-gapped
path. Deleting the Core import achieves the first half outright. The opt-in path is LND's
existing PSBT round trip, not a Core watch-only wallet — see the next section, including the
recorded verdict on how far that actually goes today.
- **D-09** required a `[fingerprint/derivation]` key origin on emitted descriptors so a hardware
signer can locate its key. With Core's descriptors deleted there are **no Archipelago-emitted
descriptors left to annotate**, so D-09's actual protection moves to the PSBT itself. That is
why `lnd.create-psbt` now inspects and reports the key-origin data its PSBT carries
(`psbt_key_origin_report`, `core/archipelago/src/api/rpc/lnd/wallet.rs`).
### `derive_bitcoin_xprv` is retained deliberately (D-07c)
`crate::seed::derive_bitcoin_xprv` (`core/archipelago/src/seed.rs:231`) lost its only non-test
caller and was **kept**, marked `#[allow(dead_code)]` with the reason in its doc comment. It is
covered by existing tests (`seed.rs:601-602`, `:856`) and it is the derivation **D-07c's deferred
BDK cold vault** — a descriptor wallet in the daemon using the node's own ElectrumX app
(`apps/electrumx`, `electrs_status.rs`) as chain source — will need.
D-07c was considered and deliberately deferred out of Phase 10 (it needs its own phase: a new
dependency and a new UI surface). It is recorded here, and in the function's doc comment, so the
option is not quietly lost along with the code that was deleted. The alternative shape — LND
watch-only via `importaccount` plus remote signing — was considered and rejected for coupling
cold storage to LND's upgrade path.
---
## LND PSBT round trip — what is covered
With Core's wallet deleted, LND is the only wallet Archipelago has, and its PSBT round trip is
the only external-signer path that exists. This section records what that path actually consists
of, what is tested, and — the question that decides whether any of it is an air gap — whether an
externally-held signer can sign a default node's PSBT at all.
### Per-step coverage map
Round trip: **fund → export → sign offline → import → finalize → broadcast.**
| # | Step | Where it lives | `file:line` | Automated test coverage |
|---|---|---|---|---|
| 1 | **Fund** — build a funded PSBT via LND WalletKit `/v2/wallet/psbt/fund` | `lnd.create-psbt` handler | `core/archipelago/src/api/rpc/lnd/wallet.rs:605`; dispatch arm `api/rpc/dispatcher.rs:136` | **Untested.** No LND mock exists; the handler's request/response handling is exercised only by hand. |
| 1a | **Inspect** — report BIP-32 key origin on the funded PSBT | `psbt_key_origin_report` + wiring | `lnd/wallet.rs:1186` (fn), `:1169` (struct), `:705` (call site), `:737` (response field) | **Tested.** 3 unit tests, below. |
| 2 | **Export** — hand the base64 PSBT to the user | UI renders `psbt_base64` for copy | `neode-ui/src/api/rpc-client.ts:407-423`; `neode-ui/src/views/web5/Web5SendReceiveModals.vue:308` | **Partial.** `neode-ui/src/api/__tests__/rpc-client.test.ts:319-323` asserts only that the client calls the method `lnd.create-psbt`; it does not test the payload or the rendering. |
| 3 | **Sign offline** — external signer produces a signed PSBT | **Not in this repo.** No first-party signer ships today. | — | N/A |
| 4 | **Import** — user pastes the signed PSBT back | textarea → `signedPsbtInput` | `Web5SendReceiveModals.vue:102`, `:419-424` | **Untested.** |
| 5 | **Finalize** — `/v2/wallet/psbt/finalize` | `lnd.finalize-psbt` handler | `lnd/wallet.rs:743`; dispatch arm `dispatcher.rs:137` | **Untested.** |
| 6 | **Broadcast** — `/v2/wallet/tx`, in the same handler | `handle_lnd_finalize_psbt` tail | `lnd/wallet.rs:795` | **Untested.** |
| — | **Rate limiting** — both endpoints at 5 calls / 300s | `RateLimiter` defaults | `core/archipelago/src/rate_limit.rs:68-69` | **Untested for these two methods specifically.** |
**Stated plainly, because an untested path must not be described as verified:** of the six steps,
**one** (the key-origin inspection added by this plan) has automated coverage in the Rust
crate. Steps 1, 4, 5 and 6 have **none** — no test exercises the LND REST calls, the finalize
handler, or the broadcast. Step 2's only test asserts a method name. **No end-to-end test of the
round trip exists**, and none of it has been verified against a real hardware signer.
There is also **no air-gap transport**: no animated QR encode/decode, no `.psbt` file
download/upload. Export and import are copy-paste of base64 in a textarea. The BC-UR v2 / BBQr
design in `PSBT-SIGNING-ARCHITECTURE.md` §4 is unimplemented.
### New tests added by this plan
In `core/archipelago/src/api/rpc/lnd/wallet.rs`'s `mod tests`, with fixtures built
programmatically from the `bitcoin` crate rather than pasted as opaque base64:
| Test | Asserts |
|---|---|
| `psbt_without_derivations_reports_no_key_origin` | A one-input unsigned PSBT with no `bip32_derivation` reports `inputs_with_key_origin: 0` and `all_inputs_have_key_origin: false`. |
| `psbt_with_derivations_reports_key_origin` | The same PSBT with a `(Fingerprint, DerivationPath)` inserted on input 0 reports `1/1` and `true`. |
| `malformed_psbt_is_an_error_not_a_panic` | Non-base64, truncated-PSBT and empty inputs all return `Err`, never panic. |
```
running 3 tests
test api::rpc::lnd::wallet::tests::psbt_with_derivations_reports_key_origin ... ok
test api::rpc::lnd::wallet::tests::psbt_without_derivations_reports_no_key_origin ... ok
test api::rpc::lnd::wallet::tests::malformed_psbt_is_an_error_not_a_panic ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 1014 filtered out
```
`lnd.create-psbt` now returns an additive `key_origin` field:
```json
"key_origin": { "input_count": 1, "inputs_with_key_origin": 0, "all_inputs_have_key_origin": false }
```
It is computed **best-effort**: a decode failure degrades to `null` and logs a warning, never to
an error — a user's send must not fail because an inspection helper could not parse something.
When `all_inputs_have_key_origin` is false the handler emits a `tracing::warn!` with the counts,
because that is the exact condition under which a hardware signer refuses the PSBT. Existing
response fields are unchanged; `handle_lnd_finalize_psbt` and `handle_lnd_create_raw_tx` (the
sibling that deliberately auto-signs with LND's hot keys) were not touched.
### Can an external signer actually sign a default node's PSBT? — **No, not today**
This is the question that separates "we have PSBT plumbing" from "we have air-gapped custody",
and the two must not be allowed to blur.
**Verdict: on a default Archipelago node, an externally-held signer cannot meaningfully sign a
PSBT produced by `lnd.create-psbt`.** The evidence:
1. **The PSBT is funded from LND's own wallet.** `lnd.create-psbt` POSTs to LND's WalletKit
`/v2/wallet/psbt/fund` (`lnd/wallet.rs:672`), which selects UTXOs belonging to **LND's**
wallet. The keys for those inputs are the keys LND holds.
2. **LND's wallet on every node is a full key-holding wallet, created locally.**
`container::lnd::ensure_wallet_initialized` (`core/archipelago/src/container/lnd.rs:86`) calls
`init_wallet_via_rest`, which POSTs `/v1/initwallet` with a `cipher_seed_mnemonic`
(`container/lnd.rs:504-516`) and persists the aezeed backup (`:523-525`). That is a normal
wallet with private keys, not a watch-only one.
3. **No node's `lnd.conf` carries a remote-signing block.** The config Archipelago generates
(`container/lnd.rs:64-79`) contains `bitcoin.node=bitcoind` and the bitcoind RPC settings, and
**no `remotesigner.*` keys at all**.
4. **Nothing in the repo provisions watch-only LND.** A search of `apps/`, `scripts/`,
`core/archipelago/src` and `image-recipe/` for `remotesigner`, `createwatchonly` and
`nochainbackend` returns **zero matches**. There is no code path, script or manifest that sets
any node up this way.
An external signer could only sign these inputs if LND were first provisioned **watch-only
against that signer** — `remotesigner.*` on the node plus `lncli createwatchonly` from the
signer's exported accounts, with the level-3 accounts and the p2tr import step described in
`PSBT-SIGNING-ARCHITECTURE.md` §5.1-5.2. **No fleet node is so provisioned.**
**What therefore ships today is the PSBT *transport*, not air-gapped custody.** The round trip is
real and rate-limited, and it is genuinely useful for signing a PSBT whose inputs belong to some
*other* wallet — but on a default node the signer that holds the input keys is LND itself, so
routing the PSBT out to an external device and back adds a step without moving custody anywhere.
The gap between here and D-08's opt-in air-gapped path is **provisioning, not plumbing**, and
that provisioning is out of scope for Phase 10 (it is `PSBT-SIGNING-ARCHITECTURE.md` §8 Phase 6).
Nothing in the UI currently claims otherwise, and nothing added by this plan does either. If
copy is ever written for this flow, it must not describe it as cold storage on the strength of
the PSBT round trip alone.
### Lightning channel, revocation and HTLC keys are not air-gappable — at all
This is a standing constraint, not a caveat, and it survives every change in this document.
> **A Lightning node's channel, revocation and HTLC keys must sign in real time to answer
> counterparty commitments. They cannot be air-gapped.** A routing node cannot tolerate a
> human-in-the-loop signing step: a delayed response to a commitment update risks a force-close,
> and a missing revocation risks loss. LND remote signing **relocates** these keys to a hardened
> host — it does **not** cool them. There is no configuration, present or future, in which a
> live Lightning node's channel keys are cold.
This is the same limit stated in `PSBT-SIGNING-ARCHITECTURE.md` §5.1 ("Air-gap channel /
revocation / HTLC keys — **No**") and §5.4, whose honesty table remains correct and unmodified.
The consequence for user-facing copy, quoted from §5.4 and repeated here so it cannot be lost:
> *A Lightning routing node's channel keys are necessarily hot. Remote signing moves them to a
> hardened machine; it does not make them cold. Only your on-chain balance can be genuinely
> protected by an offline signer.*
**No wording in this document, or in any document this phase touches, may imply that Lightning
funds can be held cold.** A user who believes their Lightning balance is cold will keep more in
it than they otherwise would, which is exactly the miscalibration that turns an incident into a
loss.
---
## Fleet census — Core descriptor wallets
**Status: run 2026-08-02 — 4 nodes examined and CLEAR, 6 nodes UNCHECKED. No escalation.**
This section answers one question per node: *does this node hold a Bitcoin Core descriptor wallet
that the deleted wallet-init handler created, and does it hold private keys?* It is recorded per
node rather than assumed, because deletion closes the door but does not tell us whether anyone
walked through it before.
The nodes that could **not** be examined are listed with their reasons, not omitted. A census
that quietly drops its failures is worthless — an auditor must be able to see exactly which
machines were looked at and which were not.
### Hard constraint on every command in this census
> **Never run `listdescriptors true`.** The `true` argument makes Bitcoin Core return the
> descriptors **including private keys**, which would print an xprv to a terminal and into a
> transcript — creating the exact exposure this census exists to measure.
> `listwalletdir`, `listwallets`, `getwalletinfo` and `listdescriptors` **with no second
> argument** answer the question completely.
>
> If any output unexpectedly contains a string beginning `xprv`, **stop immediately, do not
> paste it**, and report only that it occurred.
### Commands (re-runnable by an auditor)
Per node, against the Bitcoin Core / Knots container:
```bash
# 0. Does the handler's wallets directory exist at all? An absent directory is
# itself a complete answer for that node — paste the output as-is.
ls -la /var/lib/archipelago/bitcoin/wallets/ 2>&1
# bitcoin-cli is NOT on $PATH inside the container. On archi-dev-box (Knots
# 29.3) it lives at:
# /opt/bitcoin-29.3.knots20260210/bin/bitcoin-cli
# The RPC user is `archipelago`; the password is read from
# /var/lib/archipelago/secrets/bitcoin-rpc-password
# — reference that path, never the value, and prefer -stdinrpcpass so the
# password never appears in a process list or shell history.
# 1. Every wallet on disk, loaded or not.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass listwalletdir
# 2. Currently loaded wallets.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass listwallets
# 3. Per wallet returned: record walletname, private_keys_enabled, descriptors,
# blank, keypoolsize, txcount, balance.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass -rpcwallet=<name> getwalletinfo
# 4. ONLY for a wallet with private_keys_enabled: true — NOTE: no second argument.
# Record descriptor prefixes (`wpkh(...`) only, never a full key string.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass -rpcwallet=<name> listdescriptors
# 5. Which Bitcoin app and version.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass getnetworkinfo | head
```
### Results — examined, 2026-08-02 (4 nodes, all CLEAR)
Run by the operator over Tailscale, read-only RPCs only.
| Node | Tailscale IP | Container | `listwalletdir` | `listwallets` | `archipelago` wallet? | Default wallet state | Verdict |
|---|---|---|---|---|---|---|---|
| **archi-dev-box** | `100.69.68.39` | `bitcoin-knots` | 2× `gatewayd-*`, `""` | `[ "" ]` | **No** | `blank: true`, `keypoolsize: 0`, `txcount: 0`, `balance: 0.00000000`, `descriptors: true` | **CLEAR** |
| **shorty-s** (`.228`) | `100.64.204.114` | `bitcoin-knots` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
| **archy-x250-beta** | `100.72.136.5` | `bitcoin-core` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
| **archy-x250-pa** | `100.89.209.89` | `bitcoin-core` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
On every examined node there is **no wallet named `archipelago`** — the deleted handler's default
`wallet_name`. The only named wallets are Fedimint `gatewayd-*`, unrelated to the BIP-84 path.
The one loaded wallet on each node is Core's unnamed default. It does report
`private_keys_enabled: true`, but also `blank: true` with `keypoolsize: 0`, `txcount: 0` and
`balance: 0.00000000` — **Bitcoin Core's own statement that no key was ever imported into it and
no transaction ever touched it.** It is not the deleted handler's output, and it holds nothing.
**The result holds across two container vintages** — `bitcoin-knots` on two nodes and
`bitcoin-core` on two others. That matters: it is not four copies of one image behaving
identically, so the finding is a property of the fleet rather than an artefact of a single build.
**No key material appeared in any output, and `listdescriptors true` was never run.**
### Not examined, 2026-08-02 (6 nodes, with reasons)
| Node | Tailscale IP | Why not checked |
|---|---|---|
| framework-pt | `100.65.115.109` | `Permission denied (publickey,password)` — SSH password rotated, not held |
| archipelago-1 | `100.82.34.38` | `Permission denied (publickey,password)` |
| archipelago | `100.70.96.88` | `Permission denied (publickey,password)` |
| archy-dev-pa | `100.64.83.15` | `Permission denied (publickey,password)` |
| archipelago-5 | `100.114.134.21` | Timed out during SSH banner exchange |
| archy-x250-dev | `100.113.100.55` | Offline — Tailscale reports last seen 2 days prior |
**Password authentication was deliberately not attempted on any of these.** Several fleet nodes
lock PAM quickly on a wrong password, and locking an in-use production node out is a worse
outcome than an incomplete census. These are recorded as UNCHECKED, **not** as clear.
### Conclusion, at the strength the evidence supports
> **No examined node holds a wallet created by the deleted handler, and no examined node holds
> any wallet with keys or funds.** Four nodes, across two container vintages, on 2026-08-02.
**This is deliberately not a claim that "the fleet is clear."** Six nodes were not examined, and
an unexamined node is unknown, not safe. F-13 is closed **by deletion** — the code that could
create such a wallet is gone from every future build, which is true regardless of the census —
and the census adds that no such wallet was found where anyone could look.
### Standing item — finish the census
The six unchecked nodes remain open. **Homed in `docs/UNIFIED-TASK-TRACKER.md`** (the project's
canonical "what's open" list) as *"Finish the Core-wallet fleet census — 6 nodes unchecked"*,
rather than only here, so it is visible to someone who is not already reading a security
document. It is flagged there as a natural fold-in for **KEY-04's on-node work**, which needs
node access anyway — but it is tracked independently so it does not vanish if KEY-04 is
re-scoped.
Re-run the read-only procedure above when credentials or connectivity allow.
### Standing rule if a wallet is found
If any node reports a wallet named `archipelago` (or any descriptor wallet with
`private_keys_enabled: true` that this handler plausibly created), that is a **finding**:
1. **Stop.** Record it here with the node label and wallet name.
2. **Raise it as a blocker.** KEY-03 does not close until a human decides what to do about it.
3. **Do not migrate, unload, rescan or modify it.** D-07b withdrew the migration deliberately.
Rewriting a wallet that might hold funds is exactly the kind of decision that belongs to a
human, and CLAUDE.md's "migrations never destroy data" invariant applies the moment anyone
touches it.
Such a wallet would mean the endpoint was invoked manually before this plan deleted it, and that
node's spending key is duplicated in Core outside the Argon2 envelope.
tests/production-quality/TRACKER.md
-260
View File
@@ -1,260 +0,0 @@
# ▶▶ SESSION SAVE / RESUME (2026-06-16) — v1.7.97-alpha CUT, mid-rollout
**v1.7.97-alpha is BUILT + TAGGED LOCALLY but NOT yet published to the fleet.**
- Release commit `47c16971` ("chore: release v1.7.97-alpha") + tag `v1.7.97-alpha` exist on LOCAL main only. NOT pushed to gitea-vps2. Fleet still sees 1.7.96-alpha.
- Contents (14 fixes + image-opt): B5,B1,B2,B4,B14,B21,B3,B15,B7,B13,B12,B16,**B17**, B6-pruned-gate + lossless background-image optimization (bg-mesh PNG→JPEG).
- Release artifacts staged: `releases/v1.7.97-alpha/{archipelago, archipelago-frontend-1.7.97-alpha.tar.gz}` + `/tmp/archipelago-frontend-1.7.97-alpha.tar.gz` (177MB, flat layout verified, optimized images baked in, no APK).
- **Deployed (sideload, NOT fleet OTA):** .116 = on 1.7.97-alpha, healthy, B17 self-heal CONFIRMED (unit now has RequiresMountsFor, 36 containers survived restart). .198 = deploying (sideload binary+frontend).
- **Backup binaries for rollback:** `/usr/local/bin/archipelago.1.7.96-alpha.bak` on .116 and .198.
**REMAINING (this session, user wants to do WITH them):**
1. Finish .198 sideload; then **UI-confirm fixes together on .116/.198** + close passing Gitea issues (#8,#9,#10,#11,#12,#14,#19(code-only),#20,#21,#22,#23,#24,#29). Issue map below.
2. **Publish to fleet:** `scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2` + `git push gitea-vps2 main + tag` (AFTER joint confirm — user's call).
3. **Cut a fresh ISO** (bakes B13 nginx + B17 unit + all frontend). ISO builds run on a server (deploy-to-target / .228). Then test the ISO together.
⚠️ LESSON: never run the release binary to "check --version" — it has no such flag and BOOTS A FULL NODE (adopts containers, grabs mesh radio). Use `strings <bin> | grep version`. (Did this on .116; the instance exited on the :5678 port conflict, no harm.)
---
# ▶▶ SESSION SAVE / RESUME (2026-06-15)
**State:** v1.7.96-alpha SHIPPED. v1.7.97-alpha NOT cut yet — 10 fixes committed on **vps2 main** (`git remote: gitea-vps2`), nothing on the fleet yet. Validate on .116/.198 + UI-confirm BEFORE cutting .97.
**Resume command (run elsewhere):**
```
cd ~/Projects/archy && git fetch gitea-vps2 && git checkout main && git reset --hard gitea-vps2/main && cat tests/production-quality/TRACKER.md
```
Then continue from "IN PROGRESS" below.
**Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7, **B13 (fedimint CSS self-heal — main conf + HTTPS snippet, verified .198 both paths app-icon 404→200)**, **B12 (mempool bitcoin-host detect across 3 render paths — unit-tested; live bitcoin-core validation pending)**, **B16 (bitcoin sync tile retain/Updating… — unit-tested 6/6, commit 83dbd25c)**. B6 pruned-gate already live. = 13 fixes. PLUS **image-optimization** (commit 386d4bfc — all bg images losslessly optimized, bg-mesh PNG→JPEG; user asked to include it in the .97 release).
**IN PROGRESS — B16 DONE (commit 83dbd25c). Pick up at B6 no-node-present half.** B13 + B12 + B16 DONE (committed; see entries below). REMAINING:
1. **B6** no-node-present half, **B12b** (sibling bitcoin-host hardcodes: LND/BTCPay/electrumx/fedimint + mempool dep declaration — reuse `{{BITCOIN_HOST}}`; needs validation, esp. LND/fedimint), **B14b** (FIPS reachability depth), **B22/B23** (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
3. **Loose end:** 4 pre-existing prod_orchestrator test failures (generated-files/data_uid fixtures use disallowed tempdir volume sources) — see B12 NOTE; separate small fix.
Note: .198 is running a sideloaded B13-era .97-dev binary (md5 4c83803d). The B12 binary was built (`core/target/release/archipelago`) but NOT sideloaded (mempool isn't on .198; .198 is Knots so B12 is a no-op there). Reflashing/OTA replaces the dev binary.
**Ship .97 when ready:** ./scripts/create-release.sh 1.7.97-alpha (curate CHANGELOG ≥3 layman bullets first + run scripts/sync-whats-new.py; SKIP_RELEASE_TESTS=1 only for the 2 known-flaky vitest timing tests) → scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2 → git push gitea-vps2 main + tag. (gitea-local push fails: token rejected — non-blocking.)
---
# Production-Quality Bug Tracker
Living tracker for the post-v1.7.96 "no new features until production quality" push.
Updated continuously as we investigate → fix → test → pass. Kept in-repo so progress
survives a session cutoff.
## Rules (from user, 2026-06-15)
- **No new features** until the OS is production / no-bugs quality.
- **Test-harness-first**: build/extend a harness for each bug before fixing.
- **Validate every fix on `.116` + `.198`** (both 192.168.1.x, pw <FLEET_PW>) **+ the harness** BEFORE it goes into any release. (.198 still carries the LND CORS nginx duplicate → good for fix-(a) validation; .116 does not.)
- **Priority order**: cloud/federated-nodes + mesh FIRST, then app-specific, then low-pri.
## Status legend
`TODO` · `INVESTIGATING` · `ROOT-CAUSED` · `FIXING` · `TESTING` (on .116+harness) · `PASSED` · `SHIPPED`
## Release status
- **v1.7.96-alpha — SHIPPED** (2026-06-15). Live on vps2 (primary OTA): manifest v1.7.96-alpha, assets HTTP 200, `main@8c3c7954` + tag present. Contents: kiosk grid removal + FIPS TCP/UDP anchor selector. NOTE: gitea-local (localhost) mirror push failed (token rejected → /login); non-blocking, needs refreshed token.
- **v1.7.97-alpha — IN PROGRESS** (this push). Will bundle the verified fixes below.
---
## 🔴🔴 TOP PRIORITY
### B5 — LND "connect your wallet" details/QR broken fleet-wide — ROOT-CAUSED
Origin: user escalation. Symptom: LND connect screen (served on app port :18083) can't load details/QR.
Two distinct root causes (confirmed live):
- **(a) Duplicate ACAO** on `/lnd-connect-info` (seen on .103): backend sets `Access-Control-Allow-Origin` (proxy.rs:108) AND nginx `add_header` adds a second → browser rejects "multiple values". nginx config drift. Fix: bootstrap.rs nginx patch must strip the redundant `add_header` from the `/lnd-connect-info` location (backend owns CORS).
- **(b) No ACAO on `/proxy/lnd/v1/*` 401** (fleet-wide): the unauth/auth-layer 401 is produced before the CORS-adding proxy handler (proxy.rs:135 `handle_lnd_proxy`). Browser → "No 'Access-Control-Allow-Origin' header". Fix: ensure auth-layer/early-return responses for `/proxy/lnd` + `/lnd-connect-info` carry CORS headers.
- `.116` `/lnd-connect-info` returns a single correct ACAO → symptom varies by node's nginx state.
- Backend CORS helper: handler/mod.rs `app_cors_origin()` (:270) — reflects Origin when its host == request host.
- Backend change → ships in .97. **Status: ✅ PASSED — verified on .116, .198, .103 (harness 4/4 each). Ready to bundle into .97.**
- Caveat: bootstrap's nginx dup-strip runs a few seconds AFTER /health goes green (async patch+reload) — converges within ~1 min of restart; not instant. Acceptable.
- **CODE CHANGES MADE (uncommitted):**
- `core/archipelago/src/bootstrap.rs`: added `NGINX_LND_DUP_CORS` const + strip in `patch_nginx_conf()` (removes the duplicate nginx `add_header` ACAO from `/lnd-connect-info` so the backend's single header wins). Idempotent; runs on startup nginx bootstrap. → fixes (a)
- `core/archipelago/src/api/handler/mod.rs`: new `unauthorized_cors(origin)` helper (:~205) + `/proxy/lnd/` route (:~505) computes origin first and returns `unauthorized_cors` so the 401 carries ACAO. → fixes (b)
- Test on **.116** for (b); test on **.103** for (a) [.116 has no dup to strip].
- **2026-06-15 RESULT — .116 (fix b): harness 4/4 PASS** (sideloaded built binary, restarted). `/proxy/lnd/v1/*` now returns CORS on the 401. ✅
- (Correction: an earlier "LND container MISSING" reading was a FALSE alarm — `docker` isn't in the non-interactive PATH; runtime is **podman**. Verified `lnd Up 9h` — containers SURVIVED the restart cleanly.)
- Next: deploy to .103 + run harness to confirm fix (a) (nginx dup strip).
- **Harness:** `tests/production-quality/lnd-cors-test.sh <node>` — asserts single correct ACAO on /lnd-connect-info + ACAO present on /proxy/lnd/v1/{getinfo,channels}. Baseline (2026-06-15): .116 = 2 pass/2 fail (proxy missing ACAO); .103 = 1 pass/3 fail (connect-info dup + proxy missing).
- **FIX PLAN (precise):**
1. (b) handler/mod.rs:504-508 `/proxy/lnd/` returns `Self::unauthorized()` (401, NO CORS) when session check fails → browser CORS wall. Add CORS (app_cors_origin) to that 401. Same pattern for any other app-origin early-return.
2. (a) nginx `/lnd-connect-info` location double-adds ACAO (backend + nginx `add_header`). Strip the nginx `add_header Access-Control-Allow-Origin` there; backend owns CORS. Update bootstrap.rs nginx patch to remove it on existing nodes (idempotent).
- Verify: rebuild backend, deploy to .116, run harness → expect 3/3 (or 4 assertions) PASS on .116 AND .103.
---
## 🔴 PRIORITY — cloud / federation / mesh
### B1 — Trusted-node list not clean — PASSED (onion-dedup; unit test 2/2; live .198 15→13 distinct, healthy). UI visual-confirm recommended.
Dupes, erroneous names, and non-convergent group membership across nodes. Expected: trusted nodes form a transitive group (every node connects to any newly-added trusted node; all nodes show the same set). `.103` has a long/dirty list.
### B2 — Duplicate chat contact for one node — PASSED (resolved by load-dedup feeding mesh seed; unit-tested). UI visual-confirm recommended.
Federated peer "sapien" shows TWO chats: one "sapien" WITHOUT archy logo (looks non-federated) + one named by raw DID `did:key:z6MkoSbN5CM7fBaQg2nWbCymEkFXsHnuXvec9Mjo5RtJf9dQ`. Same node keyed by both federated identity and raw DID → merge to one. Code: core/archipelago/src/mesh + mesh/typed_messages.rs (note :233 — meshcore adverts don't carry archy pubkey).
### B3 — Cloud peer media won't preview/play — FIXING (code done: /api/peer-content streaming proxy + playMedia streams free content)
Music/video preview files on peer nodes' cloud don't play (streaming/range/content-type over mesh+Tor peer fetch).
### B4 — Cloud "my folders" fails (JSON parse / 502) — PASSED (content-type guard; built, guard in bundle, deployed .198). UI visual-confirm recommended.
`Unexpected token '<', "<!doctype"` when FileBrowser absent (`/app/filebrowser/api/resources` → SPA index.html), and **502** when FileBrowser is down (seen on .103). filebrowser-client.ts:102/:106. Fix: detect FileBrowser unavailable, friendly prompt; consider nginx returning JSON 404/502 for missing `/app/<app>/` instead of SPA shell. Handle BOTH absent + down.
### B14 — cloud browse transport not recorded — FIXED (record_peer_transport in 4 content handlers; build OK). NOTE: live data shows FIPS reaches only ~4/15 peers, 6 fall back to Tor genuinely → see B14b.
Browsing trusted/peer nodes in the Cloud tab connects over Tor instead of FIPS (should prefer FIPS like the rest of mesh; same for peer browsing). cf project_fips_integration, project_tor_node_to_node_works (last_transport should be fips/mesh).
---
## 🟠 APP-SPECIFIC
### B6 — ElectrumX install gate — PARTIAL (pruned-node gate already works; "no node present" half DEFERRED: false-positive risk without UI test, needs package-presence check)
Show the yellow requirement badge when no full node / only a pruned node is present (reuse existing yellow badge pattern).
### B7 — ElectrumX UI stuck loader on top — FIXED (overlay hides + iframe shows when status stale; type-check green). UI-confirm.
UI renders but a loader sits on top; possibly stale pre-sync screen not clearing.
### B9 — IndeedHub keeps stopping on nodes — TODO
Container won't stay running (crash-loop / reconcile stop). Check logs + restart policy + health.
### B10 — Immich still crashes — TODO
Recurring crash ("still" → prior attempts). Check container logs + resource limits + DB/ML deps.
### B11 — Companion app: "open in external browser" apps don't work — TODO
Apps meant to open in a new/external browser don't launch from the companion app; need the phone-default-browser request-modal pattern mobile apps use. Relates to v1.7.90 "open in new tab from companion app".
### B12 — Mempool not connecting — FIXED (mempool host detect, 3 paths; unit-tested). Live bitcoin-core validation PENDING (no core node available).
**Bigger than the original "stacks.rs:1278" framing.** `CORE_RPC_HOST=bitcoin-knots` was hardcoded in THREE env-render paths; on a bitcoin-core node the container is named `bitcoin-core`, so mempool-api can't resolve RPC. Both Knots and Core are reachable on `archy-net` by container name — only the name differs.
- **Path 1 — legacy direct-podman** (`stacks.rs::install_mempool_stack`, used when no orchestrator): now `format!("CORE_RPC_HOST={}", detect_bitcoin_rpc_host())`. FIXED.
- **Path 2 — `config.rs::get_app_config`** (install.rs legacy path): same. FIXED.
- **Path 3 — Quadlet/manifest (THE MODERN FLEET PATH, e.g. .198)**: `prod_orchestrator` renders env from `apps/mempool-api/manifest.yml` static YAML. FIXED via a new `{{BITCOIN_HOST}}` derived-env placeholder: `HostFacts.bitcoin_host` (container/manifest.rs) + `resolve_derived_env` renders it; `prod_orchestrator::bitcoin_host()` detects Knots/Core via `podman ps` (test-injectable `set_bitcoin_host_for_test`); resolved on-demand only for manifests using the placeholder (perf). mempool-api manifest moved `CORE_RPC_HOST` from static env → `derived_env: {{BITCOIN_HOST}}`.
- New helper `dependencies::detect_bitcoin_rpc_host()` + pure `pick_bitcoin_host()`.
- **TESTS (all green):** `pick_bitcoin_host` 5 cases (knots/core/plain/none/substring-safety); container-crate `resolve_derived_env` renders `{{BITCOIN_HOST}}`; orchestrator `mempool_core_rpc_host_follows_bitcoin_node` (core→bitcoin-core, knots→bitcoin-knots). No-regression verified: picker returns `bitcoin-knots` live on .198 (so Knots nodes unchanged; existing mempool installs see no env drift).
- **VALIDATION GAP:** cannot exercise on a live bitcoin-core node (none available; .198 is Knots where the fix is a no-op). Need a Core node to confirm end-to-end.
- **FOLLOW-UP (B12b, NOT done):** same hardcode exists for siblings on bitcoin-core nodes — `config.rs` lnd(:724)/btcpay(:739)/electrumx(:782), and `prod_orchestrator::resolve_dynamic_env` fedimint `FM_BITCOIND_URL=...bitcoin-knots` (~:2425). Plus mempool-api manifest `dependencies: bitcoin-knots` (line 18) is Knots-specific bookkeeping (install-time check already accepts Core via BITCOIN_NAMES, so non-blocking). All can reuse `{{BITCOIN_HOST}}`. Deferred per user (mempool-only scope) — each needs its own validation, esp. LND/fedimint.
- **NOTE (unrelated pre-existing failures):** 4 prod_orchestrator tests fail on clean HEAD too — `install_applies_data_uid_chown_before_create`, `install_writes_manifest_generated_files_before_create`, `manifest_generated_files_{do_not_overwrite_by_default,can_overwrite_when_declared}` — their fixtures pass tempdir volume sources that `validate_bind_source` rejects (only `/var/lib/archipelago/*` + 2 sockets allowed). NOT caused by B12; worth a separate fix.
mempool can't reach the Bitcoin backend on some nodes. Investigate on .116. Check mempool→electrs→bitcoind wiring + deps.
### B13 — Fedimint UI not applying CSS — FIXED + VERIFIED on .198 (both HTTP + HTTPS)
Root cause confirmed: the Fedimint Guardian page (served by :8175) is a server-rendered status page with ~7.8KB INLINE CSS plus image assets referenced root-rooted (`src="/assets/img/app-icons/fedimint.jpg"`, `url("/assets/img/bg-network.jpg")`). Without an asset rewrite those `/assets/...` URLs resolve against the archipelago SPA root: `bg-network.jpg` happens to exist there (shared design asset → loaded by luck) but `app-icons/fedimint.jpg` does NOT → **404** (the broken/visibly-missing icon). The `location /assets/` block uses `try_files $uri =404`, so missing fedimint assets 404 rather than fall through.
Fix = nginx sub_filter set that reroots every root-rooted asset URL (`href="/`, `src="/`, `url("/`, and single-quote variants) under `/app/fedimint/`, plus `proxy_set_header Accept-Encoding ""` so the upstream doesn't gzip (sub_filter can't rewrite gzipped bodies). Shipped two ways:
- **Fresh ISOs** (committed a50b6df2): templates `image-recipe/configs/nginx-archipelago.conf` (HTTP) + `image-recipe/configs/snippets/archipelago-https-app-proxies.conf` (HTTPS).
- **Already-deployed nodes** (bootstrap self-heal, this commit): `core/archipelago/src/bootstrap.rs::patch_nginx_conf` now heals BOTH the main conf (Style A — swaps the old single nostr-provider sub_filter tail for the full reroot set, byte-matches the shipped template) AND the HTTPS app-proxy snippet (Style B — anchors on the unique `:8175` proxy_pass and inserts the reroot set; robust to the snippet's varying trailing directive). `missing_*` flags now gated on their splice anchors so the healed snippet early-returns cleanly (no per-boot warn-skips). Idempotent via the `'href="/' 'href="/app/fedimint/'` marker.
VERIFIED on .198 (sideloaded built binary, restart, async self-heal converged ~15s):
- HTTP `/app/fedimint/`: live conf healed byte-identical to template; app-icon **404→200 image/jpeg (41944b)**.
- HTTPS `/app/fedimint/` (snippet): healed; same app-icon **404→200**; bg-network 200; root `/assets/img/app-icons/fedimint.jpg` returns 200 **text/html** (SPA shell) — proving the reroot is necessary.
- `nginx -t` OK both times; containers survived restart (Quadlet); both files carry the marker exactly once (idempotent steady state); no warn spam in logs.
NOTE: self-healed snippet is functionally correct but NOT byte-identical to the fresh-ISO snippet template (insert-after-proxy_pass vs full block) — acceptable; nginx ignores directive order/whitespace.
### B15 — Bitcoin UI sync progress lags — FIXED (Home.vue poll 30s→10s). UI-confirm.
Bitcoin UI doesn't update its sync progress fast enough even though the console clearly already has the block-height data. Likely a polling-interval / reactive-update gap between the status source and the UI.
### B16 — Bitcoin sync status vanishes — FIXED + UNIT-TESTED (commit 83dbd25c). UI-confirm.
The bitcoin sync status in the Home > System container disappears when it should persist/cache and show an "updating" state. Related to B15 (Bitcoin UI sync lag). Root cause: the tile is gated `v-if="stats.bitcoinAvailable===true"` (HomeSystemCard.vue:60); a transient `bitcoin.getinfo` failure (RPC busy during heavy IBD, or a route-change/scan where the packages map is momentarily empty) could blank it.
FIX (commit 83dbd25c): added a `bitcoinStale` flag to homeStatus.ts —
- getinfo fails while the bitcoin container is **Running**, OR package data is momentarily **absent** → retain last-known value + `bitcoinStale=true` (tile stays, renders **"Updating…"** instead of a frozen figure shown as live).
- container authoritatively **Stopped/Exited** → `bitcoinAvailable=false`, `stale=false` (no stale-as-live — genuinely down is reflected).
- first-ever poll times out but container Running (syncing node) → show the tile as updating rather than staying hidden.
Wired `bitcoinStale` through Home.vue `systemStats` → HomeSystemCard prop; card shows "Updating…" (dimmed) when stale.
**Harness:** `neode-ui/src/stores/__tests__/homeStatus.test.ts` (6 cases) — RED before fix (5/6 fail), GREEN after (6/6). `vue-tsc --noEmit` exit 0. Full vitest suite: only pre-existing AppIconGrid cross-test teardown flake (passes 7/7 standalone; not my change). UI-confirm on .116/.198 still recommended (hard to trigger transient failure on demand — unit test is the authoritative harness here).
### B17 — archipelago.service flaps on boot before starting — FIXED + VERIFIED on .198 (commit 34b1fdc1)
On some boots, `[FAILED] Failed to start archipelago.service` printed ~20× over ~5 min before starting. ROOT CAUSE (proven live on .198): on production nodes `/var/lib/archipelago` is a **separate `/dev/mapper/archipelago-data` ext4 volume** (systemd unit `var-lib-archipelago.mount`), and podman's **graphroot=`/var/lib/archipelago/containers/storage`** lives on it too. The unit ordered only `After=network-online.target` — NO mount dependency — so on cold boots the service (and its `ExecStartPre`) could start BEFORE the volume mounted, write to the bare mountpoint on rootfs, fail every podman call, exit, and be restarted every 5s (`Restart=on-failure RestartSec=5`) until the mount appeared. Smoking gun in .198's journal: `var-lib-archipelago.mount: Directory /var/lib/archipelago to mount over is not empty, mounting anyway` — the service had written there pre-mount. Dev laptop .116 has the data dir on rootfs → never flaps (explains "on some boots"). Diagnostic: every node showed `banners == "Server listening"` (process always succeeds once it runs) ⇒ failure is systemd-level, not a Rust crash.
FIX (commit 34b1fdc1): `RequiresMountsFor=/var/lib/archipelago` (adds `Requires=` + `After=` on the mount unit).
- `image-recipe/configs/archipelago.service`: ships the directive on fresh ISOs.
- `bootstrap::ensure_archipelago_mount_ordering()`: self-heals already-deployed nodes' installed `/etc/systemd/system/archipelago.service` + `daemon-reload` (boot-ordering only — effective next reboot; never restarts the running service). Idempotent; harmless on rootfs installs.
VERIFIED on .198: applied directive → `systemctl show -p After` includes `var-lib-archipelago.mount`, `systemd-analyze verify` clean → rebooted: mount@07:35:22, archipelago banner@07:35:35 (13s AFTER mount), `banners=1 listening=1 failed_to_start=0` (zero flap), directive persisted. `cargo check` EXIT 0. NOTE: self-heal CODE (auto-patch on deployed nodes) still to be exercised with the built binary on .228 (directive was applied manually on .198); residual rootfs shadow files under the mountpoint are benign.
### B18 — Apps stop right after install (or become unstartable) — TODO
Many apps install but immediately stop, requiring a manual Start — or become unstartable entirely. Likely the install→start handoff / reconciler doesn't bring them up (or starts then they exit). Related to B9 (IndeedHub stopping), B10 (Immich). Possibly linked to the cgroup-SIGKILL-on-archipelago.service-restart issue (feedback_no_systemctl_deploy_until_quadlet) — but NOTE: on .116 (Quadlet) containers survived a service restart cleanly, so the reconciler may be fine there; reproduce on the affected nodes. Check post-install start sequencing + boot_reconciler + container restart policy + cgroup placement.
### B19 — Failed download-update lands on Install button (should be Download) — TODO
When an update download fails, the UI sometimes shows the Install button instead of returning to the Download button — a big UX issue (user can't retry the download cleanly). Check the SystemUpdate state machine's error/failure transition.
### B20 — Surface bitcoin-headers-over-mesh broadcast (send/receive toggles) — TODO (feature-adjacent, surfacing existing work)
We previously broadcast bitcoin block headers over mesh to archipelago nodes but never fully surfaced it. Want two switches: "send headers" (you broadcast) and "receive headers" (you accept). NOTE: this is feature-adjacent — surfacing existing functionality; the user added it during the no-new-features push, so treat as low-priority polish until the bug list is clear. Code: mesh block-headers (mesh.block-headers RPC seen in logs; core/archipelago/src/mesh).
### B14b — FIPS reachability: many peers fall back to Tor — INVESTIGATED (needs FIPS-network depth)
Live (2026-06-15) federation sync last_transport on .116/.198: ~4 peers fips, ~6 tor, ~5 none. So beyond the recording fix (B14), FIPS genuinely doesn't reach many federated peers (they use Tor). Investigate WHY: is fips_npub known for those peers? are they FIPS-online? is the shared anchor connecting them? (cf project_fips_integration, project_tor_node_to_node_works). This is the real "Tor not FIPS" depth.
FINDINGS (.198, 2026-06-15): archipelago-fips ACTIVE; ALL 13 peers HAVE fips_npub; last_transport = 5 fips / 5 tor / 3 none. So it's NOT a missing-npub or service-down bug — FIPS genuinely reaches some peers and not others = DIAL-TIME reachability: the 'tor' peers aren't FIPS-reachable at dial time (offline, NAT, their FIPS not registered with the shared anchor), and 'none' = fully offline (X250 roam/beta/cellular). NEXT (deeper, needs FIPS-network debugging): verify a known-online peer (e.g. .228/.116) is reachable over FIPS from .198 right now; if an online FIPS peer still falls back to Tor → real anchor/registration bug; check fips daemon peer table + anchor connectivity. Likely partly peer-availability (not fully fixable in code).
### B21 — Show Tor/FIPS transport pill on cloud browse — FIXED (build+type-check green; deploy+UI-confirm on .116/.198)
Tag whether the peer connection is Tor or FIPS and surface it as a small pill on the cloud browse screens / connection loader. Data source: federation node last_transport (now recorded by B14) exposed via federation.list-nodes; frontend renders a pill (FIPS=fast/green, Tor=slower) on PeerFiles.vue / Cloud peer view + the connection loader. Frontend-only-ish. FINDINGS: PeerFiles.vue:46 loader HARDCODES 'Connecting via Tor...' even when FIPS used (bug). Frontend types already have last_transport ('fips'|'tor'|'mesh'|'lan') federation/types.ts:31; NodeList.vue:167 already renders a transport indicator. PLAN: have content.browse-peer RETURN the transport used (B14 already computes it) → frontend shows a pill (FIPS green / Tor amber) on PeerFiles header + fix the loader text to reflect actual/attempted transport. Small backend (add transport to browse response) + frontend pill.
### B22 — Peer cloud download/audio errors (.228→.198) — TODO (pairs with B3)
Observed 2026-06-15 browsing .228's cloud from .198: (a) downloading a peer cloud file → "Operation failed. Check server logs for details." (b) playing a peer AUDIO file → "Could not play audio. File Browser may not be running." (misleading — it's a peer file, not File Browser; that's the OLD base64/blob path B3 replaces). ACTION: (a) check content.download-peer backend error on .198 logs while downloading (likely the same Range/transport/timeout path as B3, or a peer-side 4xx); (b) verify B3 streaming fixes peer audio once deployed, and fix the misleading audioPlayer error string. Get server logs: ssh .198, journalctl -u archipelago | grep -i 'content\|peer\|download'.
### B23 — Archipelago group chat (all nodes) broken/slow over Tor — TODO (PRIORITY, mesh)
The all-nodes "Archipelago group" chat (over Tor) doesn't seem to work. Facets:
- (a) Group delivery unreliable / "doesn't work" over Tor.
- (b) Messages may just be VERY SLOW (latency — likely Tor-only path; should use FIPS+Tor per the new transport method like B14, preferring FIPS).
- (c) Add the SENDER CONTACT NAME to each message so you can differentiate who sent what (group messages lack attribution).
- (d) Messages sometimes DUPLICATED (dedup by message id / sender_seq — cf mesh.ts:73 cross-transport identity (sender_pubkey, sender_seq); duplicate likely from receiving same msg over both transports or re-broadcast).
Code: core/archipelago/src/mesh (typed_messages, listener), frontend Mesh.vue/stores/mesh.ts. Relates to B2 (identity), B14/B14b (transport). Test on .116/.198 (+ a Tor-only peer like .228).
### B8 — netbird app doesn't work — TODO (LOW / much later)
(RETRACTED: CryptPad placeholder-icon — user says cryptpad is fine.)
---
## 📋 vps2 Gitea issues (lfg2025/archy) — imported 2026-06-15
- G#1 [Bug] Strange peer request behaviour — TODO (likely related to B1/federation)
- G#2 [Bug] Fix flashing USB from kiosk — TODO
- G#3 [Feature] VPN Configuration — DEFERRED (feature; no new features until production quality)
- G#4 [Bug] Bitcoind is slow — TODO
- G#5 [Feature] OpenWRT and TollGate integration — DEFERRED (feature)
- G#6 [Feature] Move dashboard/monitoring link to home screen — DEFERRED (feature)
- G#7 [Bug] Scrolling with Companion app — TODO
---
## Gitea issue mapping (vps2 lfg2025/archy)
All backlog bugs now mirrored as Gitea issues: B1→#8, B2→#9, B3→#10, B4→#11, B5→#12, B6→#13, B7→#14, B8→#15, B9→#16, B10→#17, B11→#18, B12→#19, B13→#20, B14→#21, B15→#22, B16→#23, B17→#24, B18→#25, B19→#26. (Pre-existing G#1–7 remain; some overlap, e.g. G#1 strange-peer ≈ B1.) Close the Gitea issue when a bug is verified+shipped.
## INVESTIGATION FINDINGS 2026-06-15 (B1/B2/B3/B4/B14) — cutoff insurance
**B1 trusted-node divergence** — ROOT-CAUSED. `federation/sync.rs` `merge_transitive_peers()` (~:140) dedupes ONLY by DID; the SAME physical node appears under multiple DIDs (same `onion` + `fips_npub`) → duplicate entries ("Arch Dev" ×2, "Sapien" ×2). No background convergence → lists diverge (.103=16 nodes, .116/.198=15). Model: `federation/types.rs:24` FederatedNode (PK=did); storage `federation/storage.rs` nodes.json; add_node dedupes by DID only (:125). FIX: in merge_transitive_peers add a SECOND match arm — if no DID match, match by normalized `onion` (trim .onion); if found, treat as same node (merge fips_npub/name, don't add). Same dedup on add_node. Plus a one-time cleanup of existing dup DIDs (remove-node the stale one). TEST: after sync, all 3 nodes have identical node set, no two entries share an onion.
**B2 duplicate chat contact** — ROOT-CAUSED (same root as B1). Two federation DIDs (same onion/fips_npub, e.g. "Sapien" dids z6MkoSbN… + z6MkeYMU…) get seeded as TWO mesh contacts: `mesh/mod.rs` `seed_federation_peers_into_mesh()` (~:94) upserts per-pubkey contact_id; frontend `Mesh.vue` `mergeKeyForPeer()` (~:492) keys by DID so two DIDs = two rows. FIX: (backend) in seed, skip a node whose onion was already seeded (HashSet of onions); (frontend) Mesh.vue merge by onion when DIDs differ but onion matches. Fixing B1's onion-dedup largely resolves this too. TEST: one "Sapien" row; `mesh.peers` has one contact for the shared onion.
**B3 peer media won't play** — ROOT-CAUSED. `PeerFiles.vue` `playMedia()`/`loadPreview()` (~:358,:508) fetch the WHOLE file via RPC `content.preview-peer`/`content.download-peer` (`api/rpc/content.rs` :393,:213) which base64-encodes the entire file; frontend makes a Blob URL → browser can't Range-seek → video/large-audio won't play (+ 30/120s timeouts truncate big files). The peer's HTTP `/content/<id>` handler (`api/handler/content.rs` :49) ALREADY supports Range/206 + Accept-Ranges. FIX (bigger): add a local streaming proxy endpoint `/api/peer-content/{onion}/{id}` in `api/handler/mod.rs` that forwards the browser's Range header to the peer's `/content/<id>` (via fips::dial PeerRequest) and streams back 206 + Content-Range + Content-Type; frontend sets `<video>/<audio>` src to that URL (not a blob). TEST: curl Range on the new endpoint → 206 + Content-Range; video seeks/plays.
**B4 cloud my-folders <!doctype/502** — ROOT-CAUSED. `filebrowser-client.ts` `listDirectory()` (:99) does `res.json()` (:106) after only an `res.ok` check; when FileBrowser is ABSENT nginx serves SPA index.html (200, '<!doctype') → JSON crash; when DOWN → 502. FIX (frontend, low-risk): guard res content-type !== application/json → throw typed "FileBrowser unavailable" handled by Cloud.vue/CloudFolder.vue empty-state; same guard in login() (:71) + getUsage() (:215). OPTIONAL nginx: add `error_page 502 503 = @filebrowser_unavailable` returning JSON in the /app/filebrowser/ block (image-recipe/configs/nginx-archipelago.conf ~:411). TEST: stop filebrowser on .116/.198 → Cloud shows friendly state, no doctype crash.
**B14 cloud browse Tor-not-FIPS** — ROOT-CAUSED (nuance). FIPS-first logic WORKS (`fips/dial.rs` send_get :331 tries FIPS, falls back to Tor on 404/5xx; v1.7.94 fix). BUT the 4 content handlers in `api/rpc/content.rs` (browse :297, download :237, download_paid :356, preview :421) capture `_transport` and NEVER call `record_peer_transport()` → UI badge shows Tor/null even when FIPS used. FIX: add `record_peer_transport(data_dir, None, Some(onion), &transport.to_string())` after each successful send_get (storage.rs:84 has the fn). ⚠️ VERIFY on nodes whether FIPS is ACTUALLY used or genuinely falling back to Tor (if genuinely Tor, deeper FIPS-reachability issue beyond recording). TEST: after browse, last_transport = fips (when peer FIPS-reachable).
## INVESTIGATION FINDINGS 2026-06-15 (B6/B7/B12/B13/B15/B16) — cutoff insurance
**B13 Fedimint CSS** — app HTML (docker/fedimint-ui/index.html) uses absolute /assets/* paths; under /app/fedimint/ the browser requests /assets/* which hit the main SPA, not :8175 → unstyled. FIX: nginx sub_filter rewrite (same proven pattern as indeedhub/botfights blocks) in image-recipe/configs/nginx-archipelago.conf (/app/fedimint/ ~:641) + snippets/archipelago-https-app-proxies.conf (~:164) + bootstrap patch for existing nodes. Rewrites href/src/url '/' → '/app/fedimint/'. TEST: curl .../app/fedimint/assets/...css → 200 real CSS.
**B6 ElectrumX archival gate** — electrs needs a NON-pruned full node; install card doesn't warn at a glance. /bitcoin-status returns blockchain_info.pruned. Yellow badge pattern exists (MarketplaceAppCard.vue). FIX (frontend, simple): show a yellow "Requires a full archive Bitcoin node (not pruned)" note on the electrumx card (MarketplaceAppCard.vue ~:53). catalog.json electrumx already has requires.
**B7 ElectrumX stuck loader** — sync overlay gated by electrsSync (useElectrsSync.ts syncing = status!=='synced'); if status never flips to 'synced' (stale/crash) the overlay blocks the UI forever. AppSessionFrame.vue:44 iframe gate `!electrsSync`. FIX (frontend): fail-open — allow iframe when electrsSync?.stale (and add a timeout in useElectrsSync.ts so a slow/stale status stops blocking after ~5min).
**B15 bitcoin sync UI lag** — Home.vue:485 polls every 30s. FIX: faster bitcoin refresh (~5-10s) (separate interval for bitcoin vs system stats).
**B16 bitcoin status vanishes** — homeStatus.ts refreshBitcoin clears/leaves bitcoinAvailable null on a failed/transitional poll → HomeSystemCard.vue:60 v-if hides the card. FIX: retain last-known bitcoinAvailable on transient failure + show an "Updating…" badge instead of disappearing.
**B12 mempool not connecting** — stacks.rs:1278 + apps/mempool-api/manifest.yml:50 hardcode CORE_RPC_HOST=bitcoin-knots; on nodes running bitcoin-core (not knots) mempool-api gets getaddrinfo ENOTFOUND bitcoin-knots. Also ELECTRUM_HOST=electrumx absent on pruned nodes (docs/CONTAINER_LIFECYCLE_HANDOFF.md:654). FIX: detect which bitcoin container runs (knots vs core) + set CORE_RPC_HOST dynamically; qualify the mempool stack so it doesn't half-start without electrumx. Backend (stacks.rs) — medium risk, test on .116.
- 2026-06-15 (cont. 2): **B15 ✅** (poll 30s→10s) + **B7 ✅** (ElectrumX loader fail-open on stale) — committed `c0d41cf8`, type-check green. **B6 PARTIAL** (pruned gate already works; no-node-present half deferred). Fanned out investigations for B6/B7/B12/B13/B15/B16 — all root-caused with fix plans in FINDINGS above.
- **DEFERRED with ready plans (need a backend build + careful patch, or UI test, or live repro):** B13 (fedimint CSS — nginx sub_filter asset rewrite; bootstrap exact-match patch is fragile, do carefully), B12 (mempool host — dynamic bitcoin-knots/core detect in stacks.rs), B16 (bitcoin status retain — UI-test to avoid stale-as-live), B6 no-node-present half, B14b (FIPS net depth), B22/B23 (need live repro).
- **NEXT options:** (a) continue backend batch B13+B12 (one build); (b) do UI confirms on .116/.198 + cut v1.7.97-alpha with the ~10 committed fixes (LND incident + cloud/federation/mesh).
- **Committed fixes awaiting .97:** B5, B1, B2, B4, B14, B21, B3, B15, B7 (+ B6 pruned-gate already live). All on vps2 main; NOT on fleet yet.
## Progress log
- 2026-06-15: tracker created. v1.7.96-alpha shipped. All 19 bugs filed as Gitea issues #8–#26. vps2 feature issues (G#3/5/6) deferred (no new features).
- 2026-06-15: **B5 (LND CORS) ✅ DONE** — root-caused, both fixes implemented, verified on .116/.198/.103 (harness 4/4 each), committed `1db720af`, pushed to vps2 main. Will bundle into .97 (Gitea #12 to close on .97 ship).
- Validation nodes: .116 + .198 (pw <FLEET_PW>). Runtime is podman (docker not in non-interactive PATH). Sideload binary → /usr/local/bin/archipelago + restart (containers survive on these nodes).
- 2026-06-15 (cont.): **B1,B2,B4 ✅** dedup+guard — committed `ed493106`, unit-tested 2/2, live .198 healthy. **B14 ✅** transport recording — committed `1c6dc153` (after build-repair: used private `crate::federation::storage::` path → E0603; fixed to re-exported `crate::federation::`). **B21 ✅** Tor/FIPS pill — committed `0801dd66`. All pushed to vps2 main; builds verified EXIT 0.
- **Discovered B14b** (FIPS reaches only ~4/15 peers; rest genuinely Tor) and **B21** (pill) during the block.
- ⚠️ LESSON: a backgrounded build "completed" notification does NOT mean success — grep the EXIT code before committing (a broken commit reached main once; repaired by 1c6dc153; no release cut from it → fleet unaffected).
- **NEXT: B3 (peer media streaming — big), then B14b (FIPS reachability), then app-specific (B6,B7,B9–B13,B15–B19).** None deployed to fleet yet — all on vps2 main awaiting the .97 release after full .116/.198 + UI verification.
## New backlog issues filed 2026-06-16 (this session)
- #32 Tor chat: message stuck on spinner though peers received it (task #8)
- #33 Message toast: click-to-open chat + close icon (task #9)
- #34 Local UI images never rebuild on source change — orchestrator gap (task #7); blocks OTA of bitcoin-ui relay + fedimint CSS to existing fleet
- #35 Paid 10% video previews unplayable — truncated MP4 (task #6)
NOTE: bitcoin RPC relay UI + fedimint guardian CSS now LIVE on .116 (image rebuilds); .198 deploy in progress. Bitcoin app launches host-net UI at <node>:8334 (not /app/bitcoin-ui/ proxy).