Merge main into archy-hwconfig — reconcile probe/dedup/name work

Both sides independently fixed the serial-alias dedup and the ESP32
boot-reset races; kept the branch's defer-to-auto-detect for unpinned
preferred paths (single probe pass per cycle) on top of main's
advert-name threading, Reticulum name propagation and radio-first
routing. Modal keeps main's 'Set Recommended' naming + probe progress
bar alongside the branch's in-app firmware flasher step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-28 19:03:19 -04:00
co-authored by Claude Fable 5
173 changed files with 10967 additions and 1925 deletions
+300
View File
@@ -0,0 +1,300 @@
# FIPS near-100% uptime + optimistic UI state — implementation plan
**Date:** 2026-07-27. **Status:** researched + root-caused live on the fleet; ready to
implement for the next release. Two workstreams: (A) make node↔node FIPS transport
succeed whenever a FIPS path physically exists, (B) stop the UI reloading everything
on every navigation (optimistic/cached cards, stale-while-revalidate) while keeping
data fresh.
**Honesty note on "100%":** if a node's network blackholes every anchor (the .116
WiFi case, `docs/HANDOFF-2026-07-20-fips-peer-files.md:117-133`), Tor fallback is
*correct*. The achievable target is: **FIPS wins whenever a FIPS path exists, and
fallback frequency is measured in-product so regressions are visible.** Today several
paths are 0% FIPS *by construction* regardless of network health — that's the bug.
---
## Part A — why Cloud/FIPS "commonly falls back to Tor": ranked root causes
All verified live on 2026-07-27 (.116 local, .198, .228, Framework PT, x250s) plus a
full code audit of `core/archipelago/src/{fips,transport,federation,server.rs}`.
### RC0 — 🔥 The hardening firewall drops the peer-API port on every hardened node (PROVEN)
The fips0 default-deny baseline (`/etc/fips/fips.nft`) is opened by archipelago's
drop-in `80-web-ui.nft` (`fips/config.rs:236-255`) for **80 + 8443 + app ports only**.
The peer-API listener — which carries *all* federation sync, cloud browse/download,
mesh envelopes, DWN, invoices — is **`PEER_PORT = 5679`** (`fips/dial.rs:35`).
**5679 is not in the allowlist.** The drop-in's own comment claims "web UI + peer
API" but the peer API port was never added.
Live proof (2026-07-27):
- .116 nft chain: 5,965 dropped packets; .198: **28,670 dropped packets** — that's
peers' FIPS dials dying at the firewall.
- .198 → .116 `GET :5679/health`: **timeout (6s)** before; **HTTP 200 in 0.35s**
after `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`.
Same result in reverse direction (200 in 0.64s).
- Explains the exact fleet split in `federation/nodes.json`: hardened-baseline nodes
(Framework PT, .198, .228, x250-dev, x250-mad2) = `last_transport: tor`;
non-hardened nodes (Austin Sapien, X250-Beta, X250-PA) answer :5679 (404 from the
path allowlist = listener reachable) = `last_transport: fips`.
- Every dial to a hardened peer pays the 8s FIPS connect timeout
(`dial.rs:114`) ×2 (retry, `dial.rs:128-140`) → then Tor. That's the "Cloud takes
forever / shows Tor" experience.
**Fix (one line + reload):** add `tcp dport 5679 accept` to the drop-in in
`fips/config.rs` (use a constant shared with `dial.rs::PEER_PORT`, not a literal).
The drop-in reinstalls on every daemon config install, so it heals fleet-wide on OTA.
⚠️ Transient manual rules were inserted on .116 and .198 during diagnosis (2026-07-27)
— they vanish on the next `nft -f /etc/fips/fips.nft` reload or reboot; the code fix
makes them permanent.
### RC1 — .228 (Shorty's) runs fips 0.3.0-dev; the 0.4.1 fleet can't reach it
.228's daemon: `0.3.0-dev (rev 34e00b9f6e)`, both anchor links "connected", but its
ULA is 100% unreachable from 0.4.1 nodes (ping loss 100%). FIPS wire format is not
stable across revs (`docs/HANDOFF-2026-07-23-companion-apk-deploy.md:78`). Everything
to/from .228 rides Tor no matter what else we fix.
**Fix:** fleet fips-version audit + upgrade to v0.4.1 everywhere (in-product updater
exists: `fips/update.rs`; .deb path per `reference_vps2_fips_anchor`). Add a version
check to `fips.status` and surface a "peer daemon outdated" warning.
### RC2 — Direct LAN/endpoint peering is dead code + wrong port + stale seed anchors
Without direct links, all peer traffic hairpins through the vps2 anchor spanning
tree (observed: .116→.198 cold RTT 1.53.5s on the same LAN; also the wedged-anchor
latency-rot incident, `HANDOFF-2026-07-23:141-160`).
- **G1 — `lan_fips_anchors()` has never run.** It needs `PeerRecord.fips_npub`, but
`PeerRegistry::set_fips_npub` (`transport/mod.rs:302`) has **zero callers** — mDNS
TXT records only carry `did`/`pubkey`/`version` (`transport/lan.rs:50-54`). So the
"co-located peers form a direct link" feature (`anchors.rs:294-305`,
`server.rs:761-766`) is a fleet-wide no-op.
- **G2 — wrong UDP port.** `anchors.rs:293` dials `8668`, but the generated
fips.yaml binds UDP **2121** (`fips/config.rs:187`, `fips/mod.rs:130`). Even if G1
ran, it would dial a dead port. `.116`'s live `seed-anchors.json` still carries
`.198@192.168.1.198:8668` — **stale IP (LAN renumbered to 192.168.63.x) AND dead
port**; both manual entries are useless today.
- No Tailscale/alternate endpoint fallback when LAN is unreachable (the .116↔.198
fix of 2026-07-20 was hand-applied per-node config, never productized).
**Fix:** (a) `FIPS_UDP_PORT``crate::fips::PUBLISHED_UDP_PORT` + drift-guard test;
(b) hydrate `fips_npub` into the registry from federation storage (did-keyed join) so
`lan_fips_anchors` goes live with no wire change; (c) advertise the npub in the mDNS
TXT + `set_fips_npub` on resolve as the proper fix; (d) teach the LAN-anchor tick to
also try a peer's Tailscale/last-known-good endpoint when LAN fails (reviewed change
— this area got handoffs wrong twice, per memory).
### RC3 — No fast-fail on the hottest call sites; retry silently doubles every budget
- `content.browse-peer`**the Cloud page** — has NO `fips_timeout`
(`api/rpc/content.rs:363-366`): a cold FIPS path burns up to ~16.6s (8s connect +
600ms + 8s retry) before Tor even starts, against a UI deadline of 30s
(`Cloud.vue:720`) — and the frontend then retries ×3. Users see errors, not
fallback. 12 call sites total lack `fips_timeout` (browse/download/preview-peer,
`/blob`, DWN, node_message, rotation notifies).
- `dial.rs:128-140` runs 2 full-budget attempts, so `fips_timeout(6s)` really means
~12.6s everywhere.
**Fix:** wrap `send_with_retry` in a single `tokio::time::timeout(fips_attempt_timeout())`
(call sites `dial.rs:455`, `dial.rs:488`; halve per-attempt client timeout), then add
`.fips_timeout(...)`: `content.rs:366` (6s), `content.rs:281` (8s), `content.rs:1139`
(6s), `typed_messages.rs:822` (8s), `dwn_sync.rs:188/213/272` (6s),
`node_message.rs:376` (8s), `node_message.rs:412` (4s), `tor/mod.rs:501` (6s),
`federation/handlers.rs:869` (6s). **Skip the three 900s streaming downloads**
(`content.rs:552/870/1061`, `proxy.rs:236`) — `dial.rs:311-319` documents why; the
retry-budget wrap covers their connect phase.
### RC4 — Two features are 100% Tor by construction (allowlist 404)
The peer listener path allowlist (`server.rs:1219-1239`) omits `/blob/<cid>` (mesh
file sharing, `typed_messages.rs:813-822`) and `/dwn/health` (step 1 of DWN sync,
`dwn_sync.rs:186`) → deterministic 404 over FIPS (`dial.rs:44-46` treats 404 as
fall-back) → deterministic Tor, after paying the full FIPS cost. Both endpoints are
already cryptographically gated, so they meet the allowlist's stated criterion.
**Fix:** add `|| path.starts_with("/blob/") || path.starts_with("/dwn/")`; extend the
existing test block at `server.rs:1935-1945` (assert `/blob/abc` + `/dwn/health`
allowed, `/blobber` + `/dwnx` denied).
### RC5 — Inbound listener can't heal; anchor flap = 5-minute Tor window; probe overhead
- `peer_late_bind_loop` returns after first successful bind (`server.rs:1203`) and
`accept_loop` `continue`s on errors forever (`server.rs:1249-1258`): a fips0
teardown/re-key leaves the node inbound-dead until process restart → **every peer**
falls back to Tor against it.
- Nothing reacts to anchor-link drops: anchors re-apply only on the 300s tick
(`server.rs:731`); worst-case 5min Tor-only after a flap (the historic "link dead
timeout 30s" flapping made this chronic).
- `is_service_active()` spawns up to 2 `systemctl` per FIPS attempt *and* per peer
per 25s warm tick (`dial.rs:284-294`); `warm_path` skips peers without
`fips_npub` in federation storage (`fips/mod.rs:88-95`); `anchors::apply` is
serial with unbounded subprocess waits (`anchors.rs:234-283`).
**Fix:** rebindable listener; a ~25s connectivity watcher (reuse
`service::peer_connectivity_summary`, `fips/service.rs:178-207`) that re-applies
anchors immediately on a connected→disconnected edge with bounded backoff; 10s TTL
cache for `is_service_active` (mirror `transport/fips.rs:24-107`); warm the union of
federation+registry peers; make `apply()` concurrent with per-connect timeouts.
### RC6 — Zero observability: fallbacks are invisible, so "uptime" is unfalsifiable
Fallbacks log at `debug!` only (`dial.rs:458,491`); no counters; `last_transport` is
written by only 7 of ~20 call sites and **never read** to influence anything
(`storage.rs:120-147`). The parallel `TransportRouter` system can't even see FIPS
(`FipsTransport` is never constructed — `server.rs:422-442` registers Tor/Mesh/LAN
only).
**Fix:** per-reason fallback counters (F1 no-npub / F2 service-inactive / F3
DNS-fail / F4 connect-fail / F5 404 / F6 5xx) surfaced in `fips.status` + `info!`
logs with a `reason` field; call `record_peer_transport` from all peer-dial sites;
UI: per-peer transport badge on Cloud (the response already carries `transport`
`content.rs:392-400` — Cloud.vue currently throws it away at `:716-721`).
---
## Part A — execution phases
### Phase A0 — fleet triage (no release needed; do first, validates everything)
1. Fleet audit: `fipsctl --version` + `nft list table inet fips` + `ss -tlnp | grep 5679`
on every node (roster: `reference_test_deploy_roster`).
2. Transient `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`
on hardened nodes (already done on .116 + .198, 2026-07-27) — instant fleet-wide
FIPS recovery while the code fix rides the OTA.
3. Upgrade .228 (and any other 0.3.x) fips daemon to v0.4.1.
4. Regenerate/clean stale `seed-anchors.json` on .116 (dead 192.168.1.x + :8668 entries).
5. Baseline measurement: for each node pair, `content.browse-peer` time + transport.
### Phase A1 — P0 code (one commit, mechanical, offline-testable)
1. **nft drop-in: open 5679**`fips/config.rs` (share the constant with
`dial.rs::PEER_PORT`). ← RC0
2. **Allowlist `/blob/`, `/dwn/`**`server.rs:1219-1239` + tests. ← RC4
3. **`FIPS_UDP_PORT` = `PUBLISHED_UDP_PORT` (2121)** — `anchors.rs:293` + drift-guard
test against `render_config_yaml()`. ← RC2-G2
4. **Un-deaden `lan_fips_anchors`** — hydrate `fips_npub` from federation storage in
`server.rs:761-766`; then mDNS TXT `fips` key + `set_fips_npub`
(`transport/lan.rs:50-54`, `lan.rs:96-108`, `LanTransport::new` 4th arg via
`crate::identity::fips_npub(&data_dir.join("identity"))`). ← RC2-G1
5. **Retry-budget wrap + `fips_timeout` on 12 call sites** (list in RC3). ← RC3
Verify: `cd core && cargo test -p archipelago` — watch `test_rendered_yaml_exact_snapshot`
(`config.rs:419`) + `test_render_is_deterministic` (`config.rs:476`); item 3 must
not change rendered output.
### Phase A2 — telemetry BEFORE tuning (second commit)
6. Fallback counters by reason + `fips.status` exposure + `info!` reason logs;
`record_peer_transport` from all sites. ← RC6 (gives the baseline that makes A3
measurable and "100%" falsifiable)
### Phase A3 — resilience (third commit, measured against A2 baseline)
7. `is_service_active` 10s TTL cache; warm-path union + `warm_path_unchecked`.
8. Link-state watcher → immediate anchor re-apply on drop (replaces waiting for the
300s tick); concurrent `apply()` with subprocess timeouts.
9. Rebindable peer listener (`server.rs:1203`, `1249-1258`).
10. (Reviewed, separate PR) endpoint-fallback for direct peering: LAN → Tailscale →
last-known-good, npub-keyed. Mesh-routing area — needs careful review per memory.
### Phase A4 — verification gate (on nodes, before tag)
- On .116/.198/framework-pt/.228: `content.browse-peer` to every peer must return
`transport: "fips"` with sub-second latency (LAN pairs) / <3s (WAN), 20/20 calls.
- Kill the fips daemon on one node → calls fall back to Tor gracefully within the
fast-fail budget (<8s), UI shows partial results, no errors.
- Restart daemon → FIPS recovers within one watcher tick (~25s), verified in
`fips.status` counters.
- Flap the anchor link (drop vps2 route) → direct LAN pairs keep FIPS via their
direct link (G1 fix proof).
- Add these as `tests/multinode/` cases per `docs/multinode-testing-plan.md`; also
fix the known `node_rpc()` missing `--max-time` (tracker item).
---
## Part B — optimistic loading + state management (frontend)
Full audit: Pinia exists but pages fetch-on-mount with `loading=true` spinners;
`Dashboard.vue:89` keys the router-view by `route.path`, so **every navigation
unmounts and refetches everything**; no KeepAlive/onActivated anywhere; no dedup,
no abort, no SWR layer. Four hand-rolled cache implementations already exist and
prove the pattern (`useFleetData.ts:198-231` sessionStorage hydrate;
`homeStatus.ts` sticky-ready loadState; `Home.vue:591-621` wallet localStorage
snapshot; `curatedApps.ts:21-77` TTL cache). `SkeletonCard.vue` exists, imported by
zero files.
### B1 — one shared primitive: `useCachedResource` composable + `resources` Pinia store
Semantics (generalize `homeStatus.ts` + `useFleetData.ts`):
- Keyed resource: `{ data, loadState: idle|loading|ready|error|refreshing, fetchedAt, error }`.
- **Hydrate synchronously** from memory (Pinia, survives navigation) → sessionStorage
snapshot (survives reload) → then revalidate in background.
- Sticky-ready: once `ready`, never regress to `loading`
(`loadState = loadState==='ready' ? 'ready' : 'loading'` — the `homeStatus.ts:80` idiom);
keep-last-known-value on error with a stale badge (age from `fetchedAt`).
- TTL per resource; `revalidateOnFocus` + on WS push (debounced, the
`Home.vue:539-542` pattern); explicit `invalidate(key)` for mutations.
- Optimistic mutation helper: apply → RPC → rollback on error (generalize
`TransportPrefsCard.vue:112-127`).
### B2 — rpc-client upgrades (`src/api/rpc-client.ts`)
- `AbortSignal` in `RPCOptions` (today the AbortController at `:87` is timeout-only)
→ abort-on-unmount for fan-outs.
- In-flight dedup keyed `method+JSON(params)` — collapses duplicate concurrent calls.
- Per-call `maxRetries` override; set `maxRetries: 1` for `content.browse-peer` /
`preview-peer` (retry×3 on a 30s timeout is why one slow peer = 90s spinner).
### B3 — Cloud page conversion (worst offender, the marquee win)
- Move `sectionCounts`, `peerNodes`, `myFiles`, `peerFiles`, `paidItems` out of
`Cloud.vue` component state (`:403,:476,:582,:689,:427`) into the cached store —
instant render on revisit, background refresh.
- **Incremental per-peer fan-in**: render each peer's card as its
`content.browse-peer` resolves (today `Promise.allSettled` at `:708-747` blocks on
the slowest peer). Per-peer states: cached/fresh/loading/unreachable.
- **Surface `transport` per peer** (already in the response, discarded at `:716-721`):
FIPS/Tor badge + latency — this is also the fleet-wide FIPS-uptime dashboard the
user asked for, for free.
- Skeleton cards (revive `SkeletonCard.vue`, copy `FileGrid.vue:3-19` shimmer) instead
of spinners for counts/folders/peer grids.
- Stop `CloudFolder.vue:307-319` calling `cloudStore.reset()` on every folder entry —
cache per-path listings, navigate renders cache + revalidates.
- `PeerFiles.vue`: persist catalog + preview cache in the store; cap the
`preview-peer` fan-out (`:832-841`, currently unbounded) with a small concurrency
queue + abort-on-unmount.
### B4 — roll out to remaining offenders (in audit order)
PeerFiles → Web5 wallet/ecash/LND slices → Monitoring → Lightning channels
(`LightningChannelsPanel.vue:650`) → Federation (already has `{showLoader:false}`
just adopt the store) → Server → Credentials/OpenWrtGateway/ContainerApps.
`Apps.vue`/`Marketplace.vue`/`Fleet.vue` are already good; don't touch.
### B5 — freshness via the existing push channel
`/ws/db` firehose + `sync.ts` JSON-patch already exist. Wire `useCachedResource`
revalidation to relevant WS pushes (debounced 800ms), keep the 30s staleness
reconciliation as backstop. No new backend needed for v1; a per-topic subscribe can
come later.
### Part B verification (on nodes)
- Navigate Cloud → Apps → Cloud: peer files render instantly from cache (0 spinner),
refresh indicator while revalidating, updated data lands without layout jump.
- One unreachable peer: its card shows stale/unreachable state; other peers render
immediately (no 30s all-or-nothing).
- Kill backend mid-view: stale data stays visible with age badge; recovery
revalidates automatically.
- Hard reload: sessionStorage hydrate paints before first RPC completes.
---
## Sequencing for the next release
1. **A0 now** (fleet triage + transient nft rules + .228 daemon upgrade + baseline).
2. **A1 + A2** land together (P0 fixes + telemetry) → deploy to .116/.198 →
Phase A4 checks on the pair → framework-pt → full fleet.
3. **B1 + B2 + B3** (composable + rpc-client + Cloud) in parallel with A-testing —
frontend-only, verifiable against .116 dev (`reference_neode_ui_dev_testing`).
4. **A3** after telemetry baseline exists; **B4/B5** ride the same or next OTA.
5. Gate: Phase A4 checklist green + Part B verification on-device + existing
single-node gate stays green → tag/OTA per ship ritual.
## Success criteria
- `content.browse-peer` transport = fips for ≥99% of calls between healthy 0.4.1
nodes over 24h (measured by the new counters), Tor reserved for genuinely
FIPS-unreachable peers (.116-WiFi-class networks).
- Cloud revisit paints in <100ms from cache; fresh data within one revalidate.
- Fallback counters visible in `fips.status` so regressions are caught on the
dashboard, not by users.
+108
View File
@@ -0,0 +1,108 @@
# License Compliance Audit — Open-Source Release
Audit date: 2026-07-22. Scope: entire repo (core Rust workspace, neode-ui, apps/*, Android companion, image-recipe ISO, docker/, app-catalog, reticulum-daemon, demo/) plus the external FIPS source and registry-mirrored images.
**Verdict:** the dependency graph is almost entirely permissive (MIT/Apache/BSD) and compatible with a free open-source release. But the repo is not releasable as-is: it has **no license of its own**, one **LGPL Rust dependency**, several **non-redistributable committed assets** (proprietary fonts, unknown-rights media), and **missing attribution machinery**. Everything below is ordered by severity.
---
## STATUS UPDATE — 2026-07-23
**DONE:**
- MIT adopted. Root `LICENSE` + `NOTICE` added; `license = "MIT"` in all 5 workspace crates (archy-fips-core already had it); `"license": "MIT"` (+ `"private": true`) in all 4 package.json files.
- Deleted: `Courier_New/`, `Benton_Sans/`, `Redacted/` fonts; `wireguard.apk`; `atob.s9pk`; obsolete `test-install.sh` (all git-rm'd; also removed from `web/dist`).
- Media provenance resolved: all demo music/photos/posters, UI sfx, backgrounds, and intro video are the author's original work — recorded in `demo/content/README.md` and `NOTICE`.
- Meshtastic device artwork attributed (`mesh-devices/ATTRIBUTION.md` + NOTICE); icon attribution added (`assets/icon/ATTRIBUTION.md`: game-icons.net CC BY 3.0, pixelarticons MIT).
- Reticulum decision: include + disclose (NOTICE states the Reticulum License restrictions and that it applies only to the optional daemon).
- indeedhub: deferred — partnership in place; license the submodule before/at public release.
- License inventories generated: `core/THIRD-PARTY-LICENSES.md` (649 crates) and `neode-ui/THIRD-PARTY-LICENSES.md` (runtime deps + fonts + vendored).
**REMAINING (code changes, awaiting review — see sections below for detail):**
1. Replace `zbase32` (LGPL-3.0+) with `z32` or original impl — §2.
2. Swap `redis:7.4.8` → Valkey in `scripts/image-versions.sh` and deploys — §3.
3. Delete dead StartOS-derived crates `core/{js-engine,container-init,models,helpers}` — §4.
4. Attribution build integration: cargo-about in CI → ship full license texts in ISO; vite/rollup license plugin (or UI licenses page) for the web bundle; Android OSS-licenses screen — §5.
5. Release-checklist items: per-release Debian source pointer (snapshot.debian.org), catalog `license`/`sourceUrl` fields, restrict ISO image bundling to the audited list — §6.
6. Before repo goes public: purge deleted fonts/APKs from git history (`git filter-repo`), and verify game-icons author credit.
---
## 1. BLOCKER — the project has no license
There is no `LICENSE`/`COPYING` file anywhere in the repo. No crate in `core/` declares a `license` field; none of the four `package.json` files do either (and the three `apps/*` packages aren't even `private: true`). Until fixed, the code is "all rights reserved" — publicly visible, but legally not open source and not usable by anyone.
**Do:**
- [ ] Choose a license. **Recommendation: MIT** — the Bitcoin-ecosystem norm (Bitcoin Core, LND are MIT), maximally compatible with everything found in the graph. (Alternatives: Apache-2.0 adds a patent grant; GPLv3 if copyleft is desired — nothing in the deps prevents any of these.)
- [ ] Add `LICENSE` at repo root with the year and copyright holder.
- [ ] Add `license = "MIT"` to all five workspace member `Cargo.toml`s (archipelago, container, openwrt, performance, security) and `Android/rust/archy-fips-core` (declares MIT but ships no license file — add one).
- [ ] Add `"license": "MIT"` to `neode-ui/package.json` and `apps/{morphos-server,router,did-wallet}/package.json`.
## 2. BLOCKER — copyleft dependency that must be replaced
- [ ] **`zbase32 0.1.2` — LGPL-3.0+** — the only hard copyleft blocker in all 649 resolved Rust crates. Direct dep of `archipelago`, used in `core/archipelago/src/network/did_dht.rs` for did:dht z-base-32 encoding. LGPL statically linked into a Rust binary requires shipping relinkable objects/source — impractical. **Replace with the MIT `z32` crate** or a ~30-line original alphabet-substitution implementation.
No GPL, AGPL, SSPL, or unlicensed crates exist anywhere else in the Rust graph. (`r-efi` and `self_cell` list LGPL/GPL only as options in OR-expressions — elect MIT/Apache, no action.)
## 3. BLOCKER — committed files we may not redistribute
Remove from git (and **purge from history** before the repo goes public — they're in past commits):
- [ ] `neode-ui/public/assets/fonts/Courier_New/` — Monotype proprietary font, no license, **unused in CSS**. Delete.
- [ ] `neode-ui/public/assets/fonts/Benton_Sans/BentonSans-Regular.otf` — commercial Font Bureau typeface, no license, unused. Delete.
- [ ] `neode-ui/public/packages/wireguard.apk` (17 MB) — official WireGuard Android APK containing GPL-2.0 `libwg` components; redistribution triggers GPL source-offer. **Unreferenced since the FIPS migration** — delete.
- [ ] `neode-ui/public/packages/atob.s9pk` (24 MB) — Start9 service package, unknown license, referenced only by a test script. Delete.
- [ ] `demo/content/music/` (18 full tracks, ~150 MB) and `demo/peer-media/` (17 photos/book covers/film posters) — no recorded rights. If they're your own/AI-generated work, document that in a `demo/content/README`; otherwise remove.
- [ ] `neode-ui/public/assets/video/video-intro.mp4`, `Kratter.MP3`, photographic `bg-*.jpg` backgrounds, UI/arcade sound effects in `assets/audio/` — same: document provenance (user-made per project convention) or replace. `welcome-noderunner.mp3` is ElevenLabs TTS — their commercial-use terms allow this on paid plans; note it.
- [ ] **Registry: `redis:7.4.8`** (`scripts/image-versions.sh` `REDIS_IMAGE`) — Redis ≥ 7.4 is RSALv2/SSPLv1, **not open source**; re-hosting it on your registry is redistribution under a restricted license. **Switch to Valkey** (BSD-3, already mirrored) everywhere.
## 4. VERIFY — unknown/third-party provenance
- [ ] **`neode-ui/public/assets/img/mesh-devices/` (36 SVGs)** — almost certainly Meshtastic project device artwork (meshtastic/web is GPL-3.0). Confirm source; either replace with original art or comply with the upstream license + attribution.
- [ ] **`neode-ui/public/assets/icon/`** — `barbarian.svg`, `batteries.svg` match game-icons.net (**CC BY 3.0 — visible attribution required**); pixel-style icons match pixelarticons (MIT). Confirm and add attribution, or replace.
- [ ] `Redacted/redacted.regular.ttf` — upstream is SIL OFL 1.1 but no license file is shipped. Add `OFL.txt` or delete (unused).
- [ ] **indeedhub** — submodule (private gitea) not checked out; no known license, yet `indeedhub{,-api,-ffmpeg}:1.0.0` images are distributed via registry/ISO. `indeedhub-ffmpeg` implies a bundled FFmpeg (LGPL/GPL → source-offer obligations). Must license the project and audit the ffmpeg build before public release.
- [ ] `minmoto/fmcd` v0.8.0 and `ark-bitcoin/bark` (barkd) — binaries redistributed in your images; verify upstream licenses (bark claims Apache-2.0/MIT dual) and include their notices.
- [ ] **Start9/StartOS heritage**`core/{js-engine,container-init,models,helpers}` are StartOS-derived (embassy paths, s9pk handling). start-os is MIT → attribution required if kept. **Better: delete these four crates** — they are not workspace members, cannot compile (broken `../../patch-db` path dep), and carry an unpinned `yajrc = "*"` git dep on a moving branch. Deleting removes both the attribution question and dead code.
- [ ] **Reticulum (RNS 1.3.5 + LXMF)** — verified: custom "Reticulum License" — MIT-style **plus field-of-use restrictions** (no systems designed to harm humans; no AI/ML training-dataset use). Redistribution is permitted, so shipping the PyInstaller `archy-reticulum-daemon` binary is fine **if** the license text is included with it — but the OS cannot claim to be 100 % OSI-open-source while bundling it. Options: include + disclose (recommended, matches "plan for decentralization" honesty), or make the daemon an optional download.
## 5. REQUIRED — attribution / notice machinery (currently absent)
Nearly every permissive license (MIT/BSD/ISC/Apache) requires reproducing copyright + license text **in distributed binaries** — and right now every distribution channel strips them:
- [ ] **Rust binaries** (649 crates, ~85 % MIT/Apache dual): generate `THIRD-PARTY-LICENSES` with `cargo-about` (or `cargo-license`) in CI; ship it in the ISO at e.g. `/usr/share/doc/archipelago/`. Include **ring's three license files** (LICENSE, LICENSE-BoringSSL, LICENSE-other-bits) and note the system OpenSSL (Apache-2.0) linked via `ssh2`.
- [ ] **Web bundle**: Vite/esbuild strips all `@license` comments from `web/dist`. Add `rollup-plugin-license`/`vite-plugin-license` to emit a third-party attribution file, or add an "Open-source licenses" page in the UI. Runtime deps needing notices: vue/vue-router/pinia/vue-i18n (MIT), d3 (ISC), leaflet (BSD-2), dompurify (elect Apache-2.0 of its MPL/Apache dual), fuse.js (Apache-2.0), qrcode/qr-scanner/qrloop/buffer/fast-json-patch (MIT).
- [ ] **Android APK**: `packaging.excludes` strips `META-INF` license texts and there is no licenses screen. Add an OSS-licenses screen or bundled `licenses.txt` covering AndroidX/Compose/OkHttp/ZXing (Apache-2.0), **fips © 2026 Johnathan Corgan (MIT — the core of the VPN feature)**, tokio/tracing (MIT), subtle (BSD-3), tun (WTFPL — permissive, just list it), secp256k1 family (CC0). Generate the Rust side from the committed `Cargo.lock` with cargo-about.
- [ ] **AIUI demo bundle** (`demo/aiui/` — committed minified build): bundles Mermaid, Cytoscape, KaTeX, D3, Lodash, Workbox (all MIT/BSD). Add a `THIRD-PARTY-LICENSES` file next to it (or rebuild with a license plugin).
- [ ] Keep the intact MIT headers in the two vendored `qrcode.js` copies (docker/lnd-ui, docker/electrs-ui) — already compliant, don't minify them.
- [ ] Fonts kept: Montserrat (OFL.txt present ✓), Open Sans (Apache LICENSE.txt present ✓) — keep license files adjacent to the font files in dist.
## 6. REQUIRED — distribution-level obligations (ISO & registry)
The ISO redistributes a full Debian (trixie) system plus ~29 container image tarballs; the private registry re-hosts upstream images. Re-hosting = redistribution, same obligations as bundling.
- [ ] **GPL source offer for the ISO** — kernel, GRUB, busybox/live-boot, coreutils, nftables, cryptsetup, wireguard-tools, SYSLINUX `isohdpfx.bin`, etc. Easiest compliance: keep `/usr/share/doc/*/copyright` (the build already does ✓) **and** publish, per release, either a mirror of the exact Debian source packages (`apt-get source` snapshot / snapshot.debian.org pointer) or a written offer in the docs. Add this to the release checklist.
- [ ] **AGPLv3 images redistributed** (mempool, Grafana, Vaultwarden, SearXNG, PhotoPrism, Nextcloud, Immich, CryptPad, MinIO): AGPL compliance = make corresponding source available. You ship a **modified** mempool-frontend (`docker/mempool-frontend` entrypoint patch) — the patch is in-repo, so compliance is met once the repo is public; state this in docs. For unmodified images, link upstream sources in the app catalog.
- [ ] **GPLv2/GPLv3 images** (MariaDB, Jellyfin, AdGuard Home, strfry): unmodified redistribution → provide license text + upstream source links (a `license` + `sourceUrl` field per `app-catalog/catalog.json` entry solves this catalog-wide).
- [ ] **Non-free firmware** (firmware-realtek/iwlwifi/misc/linux-nonfree, intel/amd microcode): redistributable but proprietary — disclose in docs ("includes non-free firmware for hardware support"), like Debian's own non-free-firmware ISOs do.
- [ ] The ISO build's live-server image capture (`podman save` of whatever matches on the dev server) is a compliance hazard — bundle only from the audited image list.
- [ ] FIPS daemon (jmcorgan/fips v0.4.1, MIT ✓) and nostr-rs-relay binary (MIT ✓): include their license texts in the notices bundle.
## 7. Housekeeping (supports compliance)
- [ ] Add lockfiles + pinned versions in `apps/*` (currently floating `^` ranges, violating the project's own pinning rule) — reproducibility is also what makes license audits stay true.
- [ ] `Android` fips dep is pinned to a personal fork rev (`9qeklajc/fips-native@46494a74`) — mirror or vendor it so outside contributors can build.
- [ ] Move `@types/dompurify` to devDeps; refresh stale `neode-ui/node_modules`.
- [ ] Add a `NOTICE` file at root naming: fips (Johnathan Corgan, MIT), Start9 start-os (if any derived code remains), Kazuhiko Arase qrcode.js, font licenses, icon attributions.
- [ ] Consider CI license gating: `cargo-deny` (Rust) + `license-checker` (npm) with an allowlist, so new copyleft deps are caught at PR time.
---
## Quick reference: what's already clean
- All 649 Rust crates except `zbase32`: permissive or dual-licensed.
- All 833 npm packages in neode-ui: no GPL/AGPL anywhere; only dev-tool LGPL (sharp's libvips, never distributed).
- Android Gradle deps: 100 % Apache-2.0, all pinned, no Play Services/telemetry.
- FIPS mesh: MIT (© 2026 Johnathan Corgan) — keep notice.
- js-engine binds deno_core (MIT) as a crate, nothing vendored — moot if dead crates are deleted.
- reticulum-daemon Python is original code; obligations attach only to the PyInstaller binary (see §4).
- Bitcoin Core/Knots, LND, BTCPay, Electrs, Fedimint, core-lightning, Gitea, Home Assistant, Tailscale, Portainer, Uptime-Kuma, filebrowser, ollama, penpot: MIT/Apache/BSD/Zlib/MPL — link + notice is enough.
+288
View File
@@ -0,0 +1,288 @@
# Open-Source Readiness Plan — Archipelago public launch
> Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup.
> A second agent is working the same goal concurrently — before executing any phase,
> diff against `git log` since `7e8d3314` and skip/merge what's already done.
> (Session plan file: `~/.claude/plans/resilient-moseying-reef.md`.)
## Context
The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep
exploration passes (docs/structure, code health, secrets sweep) found the repo is
fundamentally strong — README, `apps/` manifest examples, ADRs, the bats lifecycle gate,
1,104 Rust tests — but has hard blockers: **two live Anthropic API keys committed in
tracked files**, node passwords in 7 tracked files, no LICENSE (README links a 404),
5.5 GB `.git` (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea
registry `146.59.87.168:3000` that make every app image unpullable for outsiders, and
~28 internal AI-session/tracker docs mixed into `docs/`.
**Decisions made by the user:**
1. **Fresh-history publish** — new public repo with a clean initial commit; private repo keeps full history.
2. **Registry: domain + parameterize** — real domain in front of the existing registry; host configurable everywhere.
3. **Deep code cleanup** — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly).
4. **Internal docs: sanitize and keep public** — scrub creds/IPs/hostnames but publish plans/trackers for transparency.
**Invariant throughout:** the single-node production gate (`tests/lifecycle/run-gate.sh`)
is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E
especially). All cargo verification uses `--all-features` to match CI. Stage by explicit
path, never `git add -A` (shared tree).
## Current local pass status
This branch is replayed on top of `origin/main` as `public-prelaunch`.
Completed locally in this pass:
- Redacted the two tracked Anthropic API key literals from
`scripts/setup-aiui-server.sh` and
`image-recipe/_archived/build-auto-installer-iso.sh`.
- Removed `Android/app/debug.keystore` and `core/.env.production` from the
source tree; copies were preserved in
`~/Desktop/archipelago-sensitive-backup-2026-07-27/`.
- Reworked `scripts/audit-secrets.sh` to scan tracked source more aggressively
and to catch non-example env files and credential file patterns.
- Reworked `scripts/validate-app-manifest.sh` so the current `app:` manifest
schema can be audited without a Python `PyYAML` dependency.
- Updated root/community docs, CI, PR template, app developer notes, and
container/deployment docs toward public contributor expectations.
- Fixed native FIPS activation fallback: nodes that have the packaged
`fips.service` but not `archipelago-fips.service` now start the available
unit instead of repeatedly failing activation against a missing unit. This
now covers startup, supervisor self-heal, manual dashboard start/reconnect,
and post-onboarding activation. The UI now labels the action as `Start`
instead of making native FIPS look like an installable app.
- Fixed the FIPS app-port relay design so it binds relays to the node's FIPS
ULA instead of wildcard `[::]`, avoiding collisions with Podman-published app
ports such as FileBrowser `8083` and Botfights `9100`.
- Added `docs/nostr-git-source-hosting.md`, a NIP-34/ngit/GRASP source hosting
plan using a Bitcoin Core-style maintainer model: public review and easy
forks, with canonical merge rights held by a small signed maintainer set.
Verified locally:
- `./scripts/audit-secrets.sh` passes.
- Full `apps/*/manifest.yml` repository audit passes with warnings only.
- `bash -n` passes for the edited shell scripts.
- Targeted FIPS dashboard vitest passes.
- Targeted Rust tests for FIPS service unit detection and FIPS app relay
address selection pass.
Verified on a Linux Archipelago verification node:
- Native FIPS was restored by starting the already-installed packaged
`fips.service`; the daemon became active and joined the FIPS tree.
- Correct local lifecycle API endpoint is HTTP, not HTTPS
(`ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http`).
- Read-only lifecycle run progressed past login and confirmed required
containers, Bitcoin RPC, ElectrumX TCP, and manifest port-drift checks, but
did not complete cleanly: `botfights` and `filebrowser` remained in
`restarting` longer than the matrix window, and the LND `lncli getinfo`
probe hung. Do not run the destructive gate until those live-node issues are
understood.
- After the node updated to `1.7.116-alpha`, `botfights`, `filebrowser`, and
`lnd` were active/running and ports `8083`/`9100` were held by Podman's
`rootlessport` as expected. The packaged `fips.service` remained installed
and enabled but inactive, so the native FIPS service fallback should still
ship before the public launch.
Still required before public publish:
- Rotate/revoke compromised credentials listed in Phase 0.
- Finish Phase 1 password/node/token sanitization beyond the two API keys.
- Publish from fresh history after the sanitized tree is final.
- Run full Rust, frontend, Android, and lifecycle gate verification.
- Resolve the live-node lifecycle blockers above, then rerun the read-only
suite followed by the destructive gate only on an approved verification node.
- Decide the canonical Archipelago maintainer npub and merge-maintainer npub
list before publishing the Nostr Git source-hosting workflow.
---
## Phase 0 — Credential rotation (immediate, independent of the repo)
Treat all of these as already compromised; rotate even though we're doing fresh-history:
- **Anthropic API key #1**: `image-recipe/_archived/build-auto-installer-iso.sh:2837` (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (`ISO_ANTHROPIC_API_KEY`), keep the alpha-baking behavior if desired but never the literal in git.
- **Anthropic API key #2**: `scripts/setup-aiui-server.sh:28` — a *different* live key, not covered by the documented alpha exception. Revoke; parameterize the script.
- **The shared node SSH/sudo/UI password** (two variants) — in 7 tracked files + 24+ commits. Rotate fleet-wide (user task).
- **Gitea `ai` account password + 2 Gitea tokens** — embedded in `.git/config` remote URLs (not tracked, but leaks in any directory copy/tarball). Rotate; switch remotes to credential-helper storage instead of URL-embedded creds.
## Phase 1 — Secrets & sanitization of tracked files
1. Strip the password/credential lines from the 7 files:
`docs/PRODUCTION-MASTER-PLAN.md` (lines ~428429, 454457, 483, 521528, 886 — the fleet cred table),
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`, `docs/archive/HANDOVER-2026-07-02-iso-feedback.md`,
`docs/bitcoin-version-bulletproof-rollout.md`, `tests/production-quality/TRACKER.md`,
`tests/multinode/meshtastic.sh:26`, `neode-ui/test-openwrt.mjs:4` (→ env var).
2. `.gitea/workflows/post-install-tests.yml` — remove `sshpass -p '…'` + default target IP; use secrets/vars.
3. Sanitize infra identifiers repo-wide (in the *sanitize-and-keep* docs and scripts):
replace Tailscale IPs (17 unique, 14 files), LAN IPs (`192.168.1.x`, 93 files), hostnames
(`tx1138`, `shorty-s`, `archy-x250`, `archy-dev-pa`) with placeholders like `<node-a>` /
`NODE_IP`. Key script targets: `scripts/deploy-config-defaults.sh`, `scripts/deploy-tailscale.sh`,
`docs/operations-runbook.md` (opens with real node IPs), `docs/developer-guide.md`, `docs/api-reference.md`, `docs/hotfix-process.md`.
4. Fix the audit tool that let this happen: `scripts/audit-secrets.sh:28` — remove `\.md$` and
bare `test` from ALLOW_PATTERNS; add `sk-ant-` and password-table patterns; scan all
tracked files not just `*.env`. Run it clean as a Phase-1 exit check.
5. `.gitignore` additions: `.claude/`, `*.key`, `*.pem`, `id_rsa*`, `*.sqlite`, `*.db`
(`.claude/settings.local.json` with creds is currently only ignored by a machine-global rule).
6. Product-security note to raise (not fix now): `password123` is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced.
## Phase 2 — Repo restructure: deletions, binaries, layout
Delete (each its own commit):
- `loop/` (AI overnight harness w/ node SSH lines), `.agents/`, `.codex`, `.githooks/pre-push`
(the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history).
- `indeedhub/` submodule + `.gitmodules` entry (points at private HTTP Gitea, breaks `--recursive`
clones); `indeedhub-demo/` (single Dockerfile — merge or drop).
- `RELEASE-NOTES-v1.0.0.md` (superseded by CHANGELOG), `neode-ui/docs/GAMEPAD-NAV-MAP.md` (duplicate of `docs/GAMEPAD-NAV.md`).
- Stray generated HTML: `docs/container-architecture.html` (311 KB), `docs/archive/architecture-review.html`, `docs/archive/lora-functionality.html`.
- `Android/local.properties` from tracking (local absolute path); remove `Android/app/debug.keystore` (standard practice).
Move out of git (→ release assets on the Releases page, referenced by URL):
- `neode-ui/public/packages/archipelago-companion.apk` (27 MB), `wireguard.apk` (17 MB), `atob.s9pk` (23 MB).
- `Android/archipelago-0.3.0-debug.apk.zip` (16 MB, stale).
- `demo/content/music/*` + heavy `demo/aiui/assets` (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size).
- `neode-ui/dev-dist/` (generated Workbox output) → gitignore.
Rename/fix the naming lie: `image-recipe/_archived/` contains the *production* ISO builder
(`build-auto-installer-iso.sh`, referenced by `.gitea/workflows/build-iso.yml`). Move live
files up into `image-recipe/`, delete the genuinely archived rest.
## Phase 3 — Registry domain + parameterization (functional blocker)
Infra (user assists: DNS + TLS):
- Put a domain (e.g. `registry.archipelago-os.org` / `git.archipelago-os.org`) with HTTPS in
front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS.
Repo changes:
- Introduce a single source of truth for the registry host (e.g. `REGISTRY_HOST` in
`scripts/lib/` + a default in the orchestrator config). Replace `146.59.87.168:3000` in:
all 56 `apps/*/manifest.yml`, `app-catalog/catalog.json`, `releases/manifest.json`,
`release-manifest.json`, the 11 scripts (`self-update.sh`, `create-release.sh`,
`generate-app-catalog.sh`, `validate-app-manifest.sh`, `first-boot-containers.sh`, …),
both `demo-images.yml` workflows, `demo-deploy/.env.example`, and the Android sources
(`FipsPreferences.kt`, `PartyScreen.kt`).
- Because the catalog is signed: regenerate + re-sign + republish the app catalog after the
manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise).
Signing needs the user's mnemonic → schedule one ceremony after manifests are final.
- Verify: fresh machine with no LAN/tailnet access can `podman pull` one app image via the
domain and the gate node still installs apps after the re-signed catalog lands.
## Phase 4 — Documentation overhaul
### 4a. Community/legal files (missing today)
- `LICENSE` — MIT (matches existing README badge). Add `[workspace.package] license` +
`license.workspace = true` in the 5 member Cargo.tomls (also see Phase A4).
- `SECURITY.md` — disclosure address, PGP key, supported-versions; cite the March 2026 audit (`docs/archive/security-code-audit-2026-03.md`).
- `CODE_OF_CONDUCT.md` — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today).
- `CONTRIBUTING.md` edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb
the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model,
non-destructive migrations), versioning policy note for the `-alpha` scheme.
- `CLAUDE.md` — rewrite: keep invariants/build-verify (public-worthy), remove status banner,
node numbers, `gitea-ai` push mechanics, MEMORY references (those move to private notes).
### 4b. New developer docs (the three real gaps for app developers)
1. **`docs/quadlet-compilation.md`** — how a manifest becomes a Quadlet/systemd unit: naming,
`systemctl --user` lifecycle, where units land, how to inspect/debug one. (Source:
`core/archipelago/src/container/quadlet*.rs`, prod_orchestrator.)
2. **`docs/container-lifecycle.md`** — the 30 s level-triggered reconciler, install/adopt/
restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped
`docs/bulletproof-containers.md` as the current description; salvage its content.)
3. **`docs/secrets.md`** — `generated_secrets` declaration → materialisation by
`container::secrets` (0600, rootless) → injection; what developers must never do.
- Also: make every example in `docs/app-developer-guide.md` + `apps/*/manifest.yml` copy-paste
work against the new public registry host; add an end-to-end "write your first app" walkthrough
that a stranger can follow with only the public repo + an Archipelago node.
### 4c. Sanitize-and-keep internal docs (user's transparency choice)
- Keep, after Phase-1 scrubbing: `docs/PRODUCTION-MASTER-PLAN.md`, `docs/UNIFIED-TASK-TRACKER.md`,
`docs/1.8.0-RELEASE-HARDENING-PLAN.md`, `docs/RETICULUM-TRANSPORT-PROGRESS.md`, HANDOFF-*, test
plans, `docs/archive/*` — but **move all session/handoff/tracker material under
`docs/history/`** (extending the existing honest `docs/archive/README.md` pattern) so the
top-level `docs/` reads as current reference only. Add a banner to each: "historical working
document, sanitized; not maintained."
- Remove dangling agent-memory references in tracked docs (`docs/bulletproof-containers.md`,
`docs/RETICULUM-TRANSPORT-PROGRESS.md`, `docs/registry-manifest-design.md`,
`docs/bitcoin-multi-version-design.md` progress block).
- De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status
field; e.g. `docs/APP-PACKAGING-MIGRATION-PLAN.md` → public app-platform design doc).
- Extract North-Star narrative from PRODUCTION-MASTER-PLAN into `docs/ROADMAP.md`; extract
the "run the gate ON the node" philosophy from `docs/multinode-testing-plan.md` into
`tests/lifecycle/TESTING.md`.
- Add `docs/README.md` index (bitcoin/bitcoin `doc/` style): Getting started / Architecture /
App development / Operations / Design docs (ADRs) / History.
- README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized
docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at
CONTRIBUTING.md only.
## Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit)
### A. Zero-risk deletions & metadata (S each, own commits)
- **A1** Delete orphan non-compiling StartOS crates: `core/models`, `core/helpers`,
`core/js-engine` (incl. 2 committed `JS_SNAPSHOT.*.bin`), `core/container-init` (~4,100 LOC,
zero references). Verify: `cargo build --workspace && cargo test --all-features`.
- **A2** Delete unreferenced Vue components: `neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue`. Verify: `npm run type-check && npm run build`.
- **A3** Fix `.gitignore` lockfile lines (7: `Cargo.lock`, 15: `package-lock.json`) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles.
- **A4** LICENSE + Cargo license fields (see 4a). Verify with `cargo metadata`.
- **A5** `core/rust-toolchain.toml` pinning `1.95.0`; align `.github/workflows/ci.yml` (remove explicit `toolchain: stable` input so the file wins). Upgrades become deliberate PRs.
- **A6** `core/rustfmt.toml` codifying **defaults only** (`edition = "2021"` + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify `cargo fmt --all -- --check` yields no diff.
### B. CI guards (zero runtime risk)
- **B1** Enable vitest in CI: run `cd neode-ui && npm run test` locally; fix trivial failures, `.skip`+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration).
- **B2** Raw podman/systemctl **ratchet, not migration**: the 132 raw `Command::new("podman"/"systemctl")` sites use subcommands the `core/container/src/podman_client.rs` wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical `install.rs`, and the prod path intentionally uses Quadlet+systemctl. Add `scripts/ci/raw-podman-ratchet.sh` (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design.
### C. Clippy suppression trim (`core/archipelago/src/main.rs:8-18`, per-lint commits)
- Remove cheaply: `assertions_on_constants`, `drop_non_drop`, `wildcard_in_or_patterns`, `doc_lazy_continuation`, `enum_variant_names` (targeted allows on serde enums — never rename wire variants).
- Own careful commit: `unused_io_amount` — a **correctness** lint; fix sites with `read_exact`/`write_all` or documented targeted allows (`mesh/serial.rs:456,496` has raw partial reads; serial framing may be intentional). Full test suite + gate after.
- Keep crate-wide with justifying comment: `too_many_arguments`, `type_complexity`; attempt `ptr_arg` (`&Vec<T>``&[T]`, mechanical) if time allows — first to cut.
- Verify each: `cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features`.
### D. dead_code lift — Tiers 12 pre-launch, Tier 3 → commented allows + issues
Per-module procedure (one file per commit): remove `#![allow(dead_code)]``cargo check
--all-targets --all-features` → triage each warning: (a) genuinely dead → delete;
(b) future-feature/protocol-mandated → targeted `#[allow(dead_code)] // TODO(#NNN): …`;
(c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) →
clippy `-D warnings` + tests → commit.
- **Tier 1 (small/leaf, S each):** `swarm/seed_advert.rs`, `transport/{mesh_transport,lan,chunking,delta}.rs`, `mesh/{crypto,alerts,types,outbox}.rs`, `streaming/mod.rs`, `wallet/mod.rs`.
- **Tier 2 (M each):** `fips/{mod,iface,dial}.rs` (41 external refs → little residual deadness), `mesh/{x3dh,ratchet,steganography,message_types}.rs` — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why).
- **Tier 3 (defer, riskiest):** `mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs`, `transport/mod.rs` — change each blanket allow to `#![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN`.
- Optional S/M win: move `prod_orchestrator.rs`'s 5,034-line `#[cfg(test)]` module to a sibling file via `#[path]` (pure move, halves the 6,291-line file).
### E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated)
Legacy installers for immich/btcpay/mempool/indeedhub (`core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498`, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging `INSTALL ORCH SKIP` (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920).
1. Run the full gate on the node; grep install logs for `INSTALL ORCH SKIP`.
2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + `adopt_stack_if_exists`; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue.
3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers".
### Explicitly deferred → public tracked issues at launch
PodmanClient API extension + call-site migration; god-module splits (`install.rs`, `update.rs`, `mesh/mod.rs`); Playwright in CI; Tier-3 dead_code triage; `password123` default hardening.
## Phase 6 — Fresh-history publish
1. Freeze: all phases merged on internal `main`, gate green, catalog re-signed.
2. Build the public tree: `git archive`-style export of HEAD (never copy `.git/` — it holds
credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"),
optionally preserving CHANGELOG.md as the human-readable history.
3. Pre-publish gate on the export: `scripts/audit-secrets.sh` (fixed version) clean; grep-zero for
`sk-ant-`, rotated-password strings, `146.59.87.168`, tailnet `100.` IPs, `192.168.1.`,
internal hostnames; `du -sh .git` sanity (< ~100 MB); fresh `git clone` + `cd core && cargo build`
+ `cd neode-ui && npm ci && npm run build` on a clean machine/container; one app image pull
from the public domain.
4. Publish to GitHub; enable issue templates (already present in `.github/`); file the deferred-work
issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points.
5. Internal repo remains the private full-history remote; decide sync direction post-launch
(recommend: public repo becomes canonical, private keeps only ops/infra notes).
## Verification (end-to-end)
- `tests/lifecycle/run-gate.sh` green on the node after Phases 3 + 5E (and after any lifecycle-touching commit).
- CI green on every phase commit: `cargo fmt --check`, `clippy -D warnings`, `cargo test --all-features`, frontend type-check + build + (new) vitest.
- Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer.
- Docs acceptance: a reader following `docs/app-developer-guide.md` + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra.
## Sequencing / cut-line
Order: 0 → 1 → 2 → (3 ∥ 4) → 5 (A→E) → 6. Phases 02 are non-negotiable security; Phase 3 is the
functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any
commit (minimum viable: A1A6, B1B2, unused_io_amount fix); Phase 6 last. If the timeline
compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds.
+27
View File
@@ -12,6 +12,33 @@ Full plan: `.claude/plans/enchanted-strolling-rocket.md`. Memory pointer:
avoiding `mesh/listener/session.rs` transport plumbing + `mesh/mod.rs` routing, which this work
owns. Stay out of `meshtastic.rs`/`protocol.rs` to avoid collisions.
## Checkpoint 2026-07-28 — RNode connect + names FIXED, live-verified E2E (read this first)
The fleet reflash back to RNode firmware exposed a stack of bugs that made Reticulum
unusable on CP2102-bridged boards (Heltec V3 etc.) and left every archy node nameless on
RNS. All fixed in `a8c4694c` (backend) + `3f76b496` (UI), live-verified on archi-dev-box
and archy-x250-dev with a real RNode-to-RNode LXMF message (`transport: "reticulum"` in
mesh-messages) plus a cross-transport reply:
1. **probe_rnode boot race** — serial open pulses DTR/RTS via the USB-UART bridge → ESP32
power-cycles → KISS DETECT written 300ms later is eaten during ~2.5-3s of boot. Fix:
immediate probe (fast path) + drain-until-quiet boot settle + second DETECT window.
2. **configure() was a no-op on a running listener** (only enable/disable restarted it) —
the setup modal's apply/keep-as-is and every rename did nothing until process restart.
3. **Name propagation**`config.advert_name` had no reader; `server.set-name` never
reached mesh; daemon display name fixed at spawn to the "Archy" default; the ARCHY:2
announce blob REPLACED the LXMF name. Now: announces carry msgpack
`[name, stamp_cost, sf, ARCHY-blob]` (Sideband-compatible, blob invisible to stock
clients), daemon has a `set_name` verb, renames bounce the session live.
4. **Daemon-death detection** (was invisible up to the 30-min RX-stall watchdog),
**modal re-trigger loop** (plugged_at used tty mtime → bumps on every open; now
btime/ctime), **ARCHY:2 federation-name clobber**, **mesh.refresh RPC** (Refresh button
now actually re-queries the radio), **Meshtastic mesh.broadcast now sends NodeInfo**.
Still open here: legacy-format peers (old fleet builds) show as `Reticulum <hex4>` until
they OTA; RNode RF params still daemon-hardcoded (EU-868 869.525/125k/SF8/CR5); Phase 4
multi-radio; duty-cycle guard.
## Status at a glance
| Phase | What | Status |
+20
View File
@@ -15,6 +15,26 @@ those are marked ✅ below with the commit that did it, so we stop re-litigating
## Tier 0 — Quick / mechanical, no blockers
- [ ] **Ship the lightning payment false-failure fix in the next release** (fixed
on main 2026-07-27, needs OTA). Slow multi-hop payments (>15s) surfaced as
"Payment failed" while LND settled them in the background — the shared LND
REST client's 15s timeout aborted the synchronous `/v1/channels/transactions`
wait. Now: payinvoice decodes the invoice first for its payment hash, waits
up to 120s on a dedicated client, returns `status: "pending"` (never a
failure) on timeout, and the new `lnd.paymentstatus` RPC + frontend
`payLightningInvoice()` helper poll to a real terminal state (all 5 UI call
sites migrated). Verify on Framework PT with a real multi-hop payment.
- [ ] **Show the app version on the companion mobile-app banner in the app store
and on its install/pairing modal** (user request 2026-07-27) — so it's
obvious at a glance whether the node is serving the latest APK build.
- [ ] **Optimise the companion QR scan — quicker + better** (user request
2026-07-27; deferred to a later session on purpose). The pairing/scan QR
flow works (user-verified on-device 2026-07-27) but should get faster and
smoother: quicker camera start + decode (scan resolution/framerate,
continuous autofocus), more forgiving in low light / at an angle, and
snappier feedback once the code locks. Touch the native-scan path from
PR #104 and the in-app scan modal together so both benefit.
- [ ] **Update `tests/lifecycle/TESTING.md`'s stale Release Gates checklist** (lines
289296) — several boxes are unchecked but actually true now:
- #1 bitcoin-stops: covered by `tests/lifecycle/bats/bitcoin-knots.bats` stop/restart
+2 -1
View File
@@ -162,7 +162,8 @@ All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`.
| `lnd.openchannel` | `{ pubkey: string, amount: number }` | `{ funding_txid: string }` | Yes |
| `lnd.closechannel` | `{ channel_point: string }` | `{ closing_txid: string }` | Yes |
| `lnd.newaddress` | — | `{ address: string }` | Yes |
| `lnd.sendcoins` | `{ addr: string, amount: number }` | `{ txid: string }` | Yes |
| `lnd.sendcoins` | `{ addr: string, amount?: number, send_all?: bool, target_conf?: number, sat_per_vbyte?: number }` | `{ txid: string }` | Yes |
| `lnd.estimatefee` | `{ addr: string, amount: number, target_conf?: number }` | `{ fee_sat: number, sat_per_vbyte: number }` | Yes |
| `lnd.createinvoice` | `{ amount: number, memo?: string }` | `{ payment_request: string }` | Yes |
| `lnd.payinvoice` | `{ payment_request: string }` | `{ preimage: string }` | Yes |
| `lnd.create-psbt` | `{ outputs: object, ... }` | `{ psbt: string }` | Yes |
+1 -1
View File
@@ -146,7 +146,7 @@ app:
path: /health
interfaces:
main:
type: web
type: ui
port: 8090
```
+16 -12
View File
@@ -74,9 +74,10 @@ archy/
### Prerequisites
- **macOS** (development machine): Node.js 20+, npm
- **Linux server** (`192.168.1.228`): Rust toolchain, Podman, Nginx, Debian 13
- SSH key: `~/.ssh/archipelago-deploy`
- Node.js 20+ and npm for frontend development.
- Rust stable for backend development.
- Linux with Podman, systemd, and Nginx for host integration work.
- Debian 13 is the target runtime for release validation.
### Local Frontend Development
@@ -86,17 +87,18 @@ npm install
npm start # Vite dev server on :8100, mock backend on :5959
```
The dev server at `http://localhost:8100` uses a mock backend. Login with `password123`.
The dev server at `http://localhost:8100` uses a mock backend.
### Deploying Changes
**Never build Rust on macOS.** The deploy script rsyncs source to the Linux server and builds there.
Release and host-integration builds should run on Linux. The deploy script rsyncs
source to a configured Linux target and builds there.
```bash
# Deploy to live server (builds backend + frontend, restarts services)
# Deploy to the configured primary target (builds backend + frontend, restarts services)
./scripts/deploy-to-target.sh --live
# Deploy to both servers
# Deploy to both configured targets
./scripts/deploy-to-target.sh --both
```
@@ -114,14 +116,16 @@ The deploy script:
# Frontend tests
cd neode-ui && npm test
# Backend tests (on dev server via SSH)
ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228 \
"cd ~/archy/core && cargo test --all-features"
# Backend tests
cd core && cargo test --all-features
# Both
./scripts/run-tests.sh
```
`scripts/run-tests.sh` can run backend tests on a Linux target when
`ARCHIPELAGO_SSH_HOST` and `ARCHIPELAGO_SSH_KEY` are set.
## Adding a New RPC Endpoint
### 1. Create the Handler
@@ -200,7 +204,7 @@ async myAction(params: { name: string }): Promise<{ ok: boolean; result: string
```bash
./scripts/deploy-to-target.sh --live
curl -X POST http://192.168.1.228/rpc/v1 \
curl -X POST http://<node-host>/rpc/v1 \
-H "Content-Type: application/json" \
-b "archipelago_session=YOUR_SESSION" \
-d '{"method":"mymodule.action","params":{"name":"test"}}'
@@ -309,5 +313,5 @@ mod tests {
2. Make changes following the standards above
3. Test locally: `cd neode-ui && npm test`
4. Deploy to dev server: `./scripts/deploy-to-target.sh --live`
5. Verify at `http://192.168.1.228`
5. Verify on your configured development target
6. Commit with conventional format: `feat: add my feature`
+252
View File
@@ -0,0 +1,252 @@
# Nostr Git Source Hosting Plan
This plan describes how Archipelago can publish and accept contributions to its
source code through `ngit`, NIP-34, and GRASP while keeping the developer
experience inside Archipelago.
## Goals
- Publish Archipelago source from a sanitized, fresh-history repository.
- Make the in-app registry the primary onboarding path for contributors.
- Let contributors clone, branch, push PR branches, open PRs, and discuss issues
with a Nostr identity from their Archipelago node.
- Follow the Bitcoin Core development model: broad public review and easy forks,
with canonical merge authority held by a small maintainer set.
- Give contributors full read, fork, and proposal rights, but no direct merge
rights on the canonical repository.
- Keep the official maintainer identity and merge authority separate from user
node identities.
## Current Building Blocks
Archipelago already has most of the primitives needed for this:
- App manifests and the app registry already install developer tooling as
rootless Podman apps.
- The `gitea` app provides a conventional fallback Git UI and package registry.
- The app launcher already exposes a consent-gated NIP-07 bridge for launched
apps using `getPublicKey`, `signEvent`, NIP-04, and NIP-44 requests.
- The backend exposes node and identity Nostr signing RPC methods.
- FIPS gives nodes a stable mesh identity and private transport path, but repo
announcements and PRs should remain NIP-34 compatible on normal Nostr relays.
- DWN protocol registration exists and can be used later for local contribution
metadata/cache, but should not be required for the first public workflow.
## Protocol Basis
Use existing Nostr Git conventions rather than inventing an Archipelago-only
protocol:
- NIP-34 repository announcement events identify repositories with kind `30617`.
- NIP-34 repository state events publish branch/tag refs with kind `30618`.
- NIP-34 patches, pull requests, PR updates, issues, and status events use kinds
`1617`, `1618`, `1619`, `1621`, and `1630`-`1633`.
- `ngit` provides the `git-remote-nostr` helper for `nostr://` clone URLs and PR
branches.
- GRASP servers provide Git Smart HTTP storage while Nostr events remain the
authority for repository identity, refs, PRs, issues, and maintainer state.
Primary references:
- https://nips.nostr.com/34
- https://docs.rs/crate/ngit/latest/source/README.md
- https://ngit.dev/grasp/
## Recommended Architecture
### Apps
Create two first-party apps:
- `ngit`: CLI/runtime package containing `ngit` and `git-remote-nostr`.
- `archipelago-source`: web UI for cloning Archipelago source, viewing NIP-34
issues/PRs, opening branches, and submitting PR events.
The `archipelago-source` app should depend on `ngit`. It can also recommend
Gitea for users who want a conventional local web Git UI, but Gitea should not
be the source of truth for public contribution permissions.
### Contributor Onboarding
When the user installs `archipelago-source` from the registry:
1. Show a modal before first launch: "Contribute to Archipelago".
2. Explain that the app will use their Archipelago Nostr identity to clone and
sign contribution events.
3. Display the maintainer repository announcement, clone URL, maintainer npub,
and relay/GRASP endpoints.
4. Ask for consent to:
- fetch repository metadata from configured relays,
- clone source through `nostr://`,
- create local branches,
- sign NIP-34 issue/PR/comment events,
- push PR branches to approved GRASP servers.
5. Store approval per app origin, identity id, repository id, and relay set.
This should build on the existing NIP-07 app-launcher bridge, but use a more
specific permission scope than the generic sign-event approval.
### Identity And Permissions
Use four identity classes:
- `archipelago-maintainer`: an offline or tightly controlled Nostr key that
signs the canonical kind `30617` repo announcement and status/merge events.
- `archipelago-merge-maintainer`: one of the small set of maintainer npubs
allowed to advance canonical refs and publish valid merged/applied status.
- `archipelago-build`: release automation key for signed release artifacts and
CI status events. It must not have merge authority.
- `contributor`: user node or app-specific identity used for PRs, issues, and
comments.
Contributor rights:
- Clone the repository.
- Open issues.
- Push proposal branches using `pr/<npub>/<short-topic>` or `pr/<event-id>`.
- Publish NIP-34 PR/update/comment events.
- Rebase and update their own PR branch.
- Run local validation and attach status evidence.
Contributor restrictions:
- Cannot update `refs/heads/main` or release branches in canonical state.
- Cannot publish maintainer-valid merge/applied status.
- Cannot alter the canonical repository announcement.
- Cannot publish release catalog signatures.
Maintainer rights:
- Publish/update the canonical repo announcement.
- Publish canonical `refs/heads/main` state.
- Mark PRs merged/closed/draft via NIP-34 status events.
- Sign release tags and catalog updates.
Fork rights:
- Any contributor can create their own NIP-34 kind `30617` repository
announcement for a fork.
- Fork announcements should use the NIP-34 `u` tag to point back to the
canonical `archy` repository.
- The source app should make forking a first-class path: "Fork on Nostr", clone
the fork locally, push branches to the contributor's GRASP list, and open PRs
back to canonical Archipelago when they want review.
- Forks can have their own maintainer npubs, relays, policies, and release
cadence, but the app should clearly label them as forks unless signed by the
canonical maintainer set.
The GRASP server policy should enforce this by accepting pushes to maintainer
refs only when backed by signed maintainer state, while allowing contributor PR
refs from their own npubs.
## Repository Layout
Canonical repo announcement:
- repo id: `archy`
- display name: `Archipelago`
- clone URLs:
- `nostr://<maintainer-npub>/<relay-hint>/archy`
- `https://<grasp-host>/<maintainer-npub>/archy.git`
- relays:
- Archipelago-operated relay
- at least two public Nostr relays that support the event load
- GRASP servers:
- Archipelago-operated GRASP instance
- one public GRASP-compatible mirror
Keep the existing HTTP Git remote as a mirror during launch. The docs can
present `nostr://` as the preferred contribution path once the workflow is
proven.
## UI Requirements
The source app should provide:
- A first-run contribution modal with a real Archipelago source graphic, not a
generic text-only dialog.
- Current clone status and local path.
- Branch list, changed files, commit form, and push/open-PR flow.
- PR inbox, issue list, maintainer status, and relay health.
- Explicit identity indicator showing which npub will sign events.
- A merge rights indicator that clearly says contributors can propose changes
but cannot merge them.
- A fork flow that creates a user-owned NIP-34 repo announcement and remote,
then offers "Open PR to Archipelago" from any fork branch.
- Maintainer badges based only on pinned canonical maintainer npubs, not relay
metadata or server-side account names.
- Links to container docs, deployment docs, manifest spec, and open-source
readiness tasks.
## Backend Work
Add an RPC module for source contribution workflow:
- `source.repo-info`: returns canonical announcement, clone URL, relay set,
maintainer npubs, and local clone state.
- `source.ensure-ngit`: verifies the `ngit` app/runtime is installed.
- `source.clone`: clones or updates the local source checkout.
- `source.status`: returns branch, dirty files, ahead/behind, and PR state.
- `source.commit`: creates a local commit from selected files.
- `source.fork`: creates a contributor-owned NIP-34 fork announcement and local
remote.
- `source.open-pr`: pushes a PR branch and publishes a kind `1618` event.
- `source.update-pr`: updates the branch and publishes kind `1619`.
- `source.issue`: publishes a kind `1621` event.
Backend must shell out through a narrow command wrapper, never arbitrary user
commands. The wrapper should set an isolated working tree under
`/var/lib/archipelago/source/archy`, run as the Archipelago service user, and
deny operations outside that path.
## Security Model
- Never expose maintainer private keys to an Archipelago node.
- Prefer app-specific contributor identities over the node's default identity.
- Require per-action consent for first PR push, issue creation, and signing any
event that tags the canonical repository.
- Pin the canonical maintainer npub in the app manifest and backend config.
- Keep the canonical merge-maintainer allow list signed by the
`archipelago-maintainer` key; never infer merge rights from GRASP server
accounts.
- Verify the canonical kind `30617` event signature before displaying clone
instructions.
- Treat GRASP servers as untrusted storage; verify Git refs against signed
Nostr state.
- Do not use destructive git operations from the UI without an explicit modal.
- Store local clones and generated patches outside app container writable roots
unless the user exports them.
## MVP
1. Package `ngit` as a first-party app.
2. Stand up one Archipelago-operated GRASP server and one Nostr relay.
3. Publish sanitized fresh-history `archy` through `ngit init`.
4. Add a simple `archipelago-source` app that clones source and links out to the
preferred Nostr Git browser.
5. Add app-launcher consent scopes for repository-specific NIP-34 signing.
6. Allow issues and PR branch submission from contributor npubs.
7. Add a one-click fork flow that publishes a contributor-owned fork
announcement referencing canonical Archipelago.
8. Keep maintainer merge/status publication manual.
## Later
- Native PR review UI with file diffs and inline comments.
- CI status events signed by the build identity.
- FIPS-first source sync between trusted Archipelago nodes.
- Private prerelease repositories using NIP-42 allow lists and/or protected
events if the ecosystem support is mature enough.
- Multi-maintainer policy with threshold signatures or explicit maintainer-list
rotation events.
## Open Questions
- Which maintainer npub should become canonical for `archy`?
- Should contributor identities be node-default or app-specific by default?
- Which GRASP implementation should be deployed first: `ngit-grasp` or another
NIP-34/GRASP-compatible relay?
- Should the source app include a full web Git UI in v1, or launch Gitea/ngit
browser links for review while keeping signing/submission native?
- What exact license and contribution certificate should contributors accept
before submitting PR events?