lfg2025/archy
1
0
Fork 0
Code Issues 27 Pull Requests 5 Actions Packages Projects Releases 3 Wiki Activity

Archipelago — open-source initial import

Browse Source
This commit is contained in:
Archipelago
2026-08-12 10:55:49 +00:00
commit c7930b6e38
1578 changed files with 332819 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/1.8.0-RELEASE-HARDENING-PLAN.md
+417
View File
@@ -0,0 +1,417 @@
# Archipelago 1.8.0 — Release Hardening Plan & Tracker
> **The one living checklist for shipping 1.8.0.** Derived from a full-system deep
> audit (2026-07-02): backend security, backend code-quality, frontend, mesh,
> tests/release pipeline, and the ISO build. Supersedes nothing — it *sits above*
> `docs/UNIFIED-TASK-TRACKER.md` (day-to-day) as the release exit-criteria list.
> **Keep it updated: tick a box the moment an item lands, with the commit sha.**
**Definition of done for 1.8.0:** the supply chain is authenticated end-to-end
(§A), OTA self-update is safe and rollback-proven on real hardware (§B), no
secrets ship in the image (§F), and the single-node gate stays 5/5 green through
all of it. Everything else is polish that should not block the tag.
**Legend:** `[ ]` open · `[~]` in progress · `[x]` done · 🔴 critical · 🟠 high ·
🟡 medium · 🟢 low/polish · ⛔ blocked on you.
---
## 🎯 The single most important insight
The **release signing ceremony (Workstream B) is the linchpin.** ✅ The ceremony
KEY was generated (user confirmed 2026-07-02) — the hard offline part is done. But
the outputs are **not yet wired into the repo**: `anchor.rs:21` is still `None` and
`releases/app-catalog.json` carries no `signature`/`signed_by` (its `image_signature`
fields are literal `"cosign://..."` placeholders). Three mechanical steps remain,
split by who can run them: **(1)** pin the pubkey — needs only the *public* hex, can
be done in-repo now; **(2)** sign the catalog with the `RELEASE_MASTER_MNEMONIC` —
only the publisher, secret never touches a host; **(3)** implement + flip cosign
enforcement on the pull path. Until (1)+(2) land, every "verify the signature" task
below is written but not enforced. **This is still the critical path; §A converges on it.**
---
## §A — Supply-chain authentication (🔴 THE release blocker)
Today an attacker who controls the mirror IP (or any MITM on the plaintext HTTP
path) can ship an arbitrary root binary, arbitrary container images, and an
arbitrary app catalog to the entire fleet — fully unattended under
`auto_apply`. These four items are one story and must land together.
- [x] 🔴 **Pin `RELEASE_ROOT_PUBKEY_HEX` + sign the catalog** — DONE 2026-07-02.
`anchor.rs` pinned to `5d15cbee…d469951` (signer
`did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur`); trust tests updated (16/16
green). `releases/app-catalog.json` signed in place (`signed_by` matches, 64-byte sig);
two blocking floats fixed en route (`archy-btcpay-db` version→string, `cpu_limit` 0.25→1).
Ship order (backward-compatible): signed catalog goes out first (old binaries still accept
it), pinned-anchor binary follows in the next build/OTA. **Still ahead:** (a) the
pinned-anchor binary must actually be built + shipped for enforcement to be live on nodes;
(b) flip "accept unsigned" → "reject unsigned" only after the whole fleet is on the pinned
binary (`container/app_catalog.rs:397`, the `Unsigned` arm) — see the next item.
- [~] 🔴 **Enforce a signature on the OTA manifest before trusting it.** Signature
verification LANDED 2026-07-02: `check_for_updates` now fetches raw JSON and runs
`trust::verify_detached` — a present-but-invalid/wrong-signer signature hard-rejects
the mirror; unsigned manifests are offered for MANUAL apply only (`manifest_signed`
surfaced in `UpdateState`) and **auto-apply refuses them**. Publisher side:
`create-release.sh` signs the manifest inline (ceremony), `publish-release-assets.sh`
hard-refuses to ship unsigned (grep + `ceremony verify` crypto gate), and
`scripts/sign-manifest.sh` exists for re-signs. **Still open:** move the mirror
to HTTPS + pinned cert (tracked with the next item); flip unsigned-manual-apply →
hard-reject once the fleet is on a pinned-anchor binary.
- [x] 🔴 **Implement container image signature verification (cosign).** DONE 2026-07-04
(code path; enforcement dormant until the ceremony): new `container::image_verify`
gates BOTH pull sites (`PodmanClient::pull_image` + the dev-only `DockerRuntime`).
Claims classify as None / the literal `cosign://...` placeholder (every fleet
manifest today → pull proceeds, logged) / Declared → `cosign verify --key
/etc/archipelago/cosign.pub --insecure-ignore-tlog=true` (+ both insecure-registry
flags for the HTTP mirror; flags verified against cosign docs), hard-fail on missing
key, missing cosign binary, timeout, or bad signature — a declared signature can
never be skipped, on either runtime. Key path overridable via
`ARCHIPELAGO_COSIGN_PUBKEY`. Deleted the caller-less, blocking, wrong-CLI
`security::ImageVerifier`. **Activation = ceremony work**: pin cosign.pub on nodes +
install cosign + publish real `image_signature` values (in that order); tracked with
the Workstream B signing ceremony item.
- [ ] 🟠 **Move the image mirror to HTTPS; drop `--tls-verify=false`.**
`podman_client.rs:641` `INSECURE_REGISTRY_HOSTS = ["146.59.87.168:3000"]` +
`config.rs:104,124` allowlist pull images over unauthenticated HTTP. Remove the raw-IP
entries; give the mirror a valid/pinned cert. (Same host also baked insecurely into
the ISO — see §F.)
- [x] 🟠 **Validate every image string at the pull site, not just the RPC boundary.**
DONE 2026-07-03: policy extracted to `container::image_policy` (single source of truth;
RPC-boundary check delegates to it) and BOTH orchestrator pull sites (`install_fresh` +
`ensure_resolved_source_available`) hard-bail on refs that fail it. Policy accepts
trusted-registry refs + registry-less Docker Hub shorthand (`grafana/grafana` — used by
8 manifests, can't name an attacker host); rejects any explicit non-allowlisted
registry host, shell metachars, malformed refs. 4 new unit tests; container 159 /
package 46 green.
---
## §B — OTA self-update safety (🔴 1.8.0's headline feature is untested live)
The apply path itself is well-built (resumable download, staged-complete marker,
atomic swap, single-depth backup). The gaps are **authenticity** (§A) and
**verification depth** — plus the fact that the upgrade path has never run
end-to-end on real hardware.
- [x] 🔴 **Deepen the post-OTA health check.** DONE 2026-07-03: `verify_pending_update`
now requires, in the same attempt, (1) frontend 2xx/3xx via nginx, (2) backend RPC
liveness — unauthenticated POST `/rpc/v1`; 401/403 = alive, 5xx/404/refused = dead,
so a 502-behind-static-files release now rolls back, (3) rootless `podman ps`
reachability; plus a pre-loop binary-version==marker assertion that catches a silent
or half swap (new frontend + old binary) deterministically. Per-app container
assertions deliberately EXCLUDED — the pre-Quadlet service restart legitimately kills
containers and the reconciler can need minutes (false-rollback risk); revisit after
the Phase-3 flip. LND-unlock-level checks remain out of scope for the 90s window.
- [ ] 🟠 **Run one real upgrade-from-vN-1 soak on hardware before tagging.**
No test installs the previous version, points it at a staged 1.8.0 manifest, applies,
and asserts health + rollback. This is the top release risk for an OTA release. A
two-VM (or two-node) harness is enough.
- [x] 🟡 **Guard the frontend-build-no-op in the *actual* release path.** DONE 2026-07-08
(`e77ccff0`): the grep guard is folded directly into `create-release.sh` right after its
own `npm run build` (post-version-bump, pre-packaging — calling `run.sh --with-build` at
stage 0 would have checked the pre-bump version). Verified both ways against the real
dist: passes on the current version, trips on a missing one.
- [x] 🟢 **publish-release-assets verifies size, not sha256.** DONE 2026-07-08 (`e77ccff0`):
the verify loop now downloads each published asset and compares sha256 (and size)
against the manifest. Verified live against the v1.7.99-alpha assets on the vps2 mirror.
---
## §C — Backend robustness (🟠 stability, mostly low-effort/high-ROI)
Note: the `.unwrap()`/`panic!` worry is a **non-issue** — nearly all are in test
modules; production request/boot paths are essentially panic-free. The real risks:
- [x] 🟠 **Log swallowed persistence writes.** DONE 2026-07-02 (full-workspace re-inventory
found 19 production sites): 16 converted to `if let Err(e) = … { warn!(…) }` — mesh
config (`server.rs`), relay tor endpoint (`bitcoin_relay.rs`), update mirrors/state +
staging flush/sync (`update.rs`), registry config, radio-contact blocklist, mesh outbox
sweep (`scheduler.rs`), block-header cache (`mesh/mod.rs`), 7× peer-transport badge
(`sync.rs` + `content.rs`). Federation tombstone/untombstone upgraded to hard errors
(see §I). Install-log line write left fire-and-forget with an explanatory comment.
- [x] 🟠 **Remove blocking `std::process::Command` from async handlers.** DONE 2026-07-03:
converted to `tokio::process` — `published_host_port` (install), `detect_disk_gb`
(dependencies), factory-reset restart (system/handlers), `config.rs detect_host_ip`,
the orchestrator host-facts helpers (`detect_host_ip/mdns/disk_gb`, `bitcoin_host`,
`resolve_dynamic_env` now async through all 6 call sites), and `AutoRuntime::new`
probes. `transport/fips.rs is_available()` (sync trait method on the async route path)
now serves the cached value and refreshes via a background thread (stale-while-
revalidate) instead of blocking on systemctl. `image_verifier.rs` cosign sites have no
callers yet — handled with the §A cosign item. Tests: container 155 / transport 29 /
config 29 / package 46 all green.
- [x] 🟡 **Restrict Bitcoin RPC exposure.** DECIDED (user, 2026-07-08: break external
wallets) + DONE `dd61a204`: manifest port mappings grew a validated `bind` field;
bitcoin-knots/-core publish 8332 on `127.0.0.1` + the archy-net gateway `10.89.0.1`
only (in-node consumers dial host.archipelago/host.containers.internal → 10.89.0.1,
unaffected; P2P 8333 stays public); legacy config.rs strings get the same incl.
unauthenticated ZMQ 28332/28333. Unbound ports render byte-identical (no false-drift
wave). **Still to roll out:** catalog regen + re-sign for catalog-covered nodes;
each node's bitcoin container recreates once on deploy → restart lnd after (IP cache).
- [x] 🟡 **Move secret env out of plaintext channels → podman secrets.** DONE 2026-07-05,
**VERIFIED ON .228 2026-07-08**: recreate wave settled (only fedimint-gateway lagged —
restart-sensitive, converged via a controlled quadlet restart, healthy, 0 plaintext
password vars in inspect); btcpay inspect carries no secret values; 17 quadlet unit
files use `Secret=` lines. Original notes:
secret env no longer merges into `environment` — it would land in `podman inspect`
AND as plaintext `Environment=` lines in Quadlet unit files on disk (the worse leak).
New pipeline: `expand_and_partition_env` taints plain entries that interpolate
secrets (btcpay's `Password=${BTCPAY_DB_PASS}` connection strings travel as secrets
too), values register as podman secrets (stdin, `--replace`, content-hash label,
per-app cache so steady-state reconciles are podman-free), containers reference
them via `secret_env` (API) / `Secret=…,type=env` (Quadlet). Verified empirically
on fleet podman 5.4.2: value absent from inspect, runtime injection works. Rotation
drift via `io.archipelago.secret-env-hash` container label; pre-upgrade containers
lack the label → ONE-TIME recreate wave on first reconcile after deploy (by design —
scrubs plaintext secrets from existing container configs). Docker dev fallback keeps
plain env (no secret store). `/proc/<pid>/environ` inside the container is unchanged
(env is the app-compat contract); the closed leaks are inspect output + unit files.
- [x] 🟡 **Harden rate-limit IP extraction.** DONE 2026-07-03: the accept loop injects the
TCP `PeerAddr` into request extensions; `extract_client_ip` honors
`X-Real-IP`/`X-Forwarded-For` ONLY when the connection is from loopback (our nginx,
which sets `X-Real-IP $remote_addr`) — direct connections (e.g. the FIPS peer
listener) bucket under their socket IP, so per-request header rotation no longer
defeats the login limiter. 3 unit tests.
- [x] 🟢 **Include `seq` in the mesh signed preimage.** DONE 2026-07-04 (receiver half):
`verify_signature` accepts a v2 preimage `(t,v,ts,seq)` alongside legacy v1 `(t,v,ts)`;
`signed_with_seq()` is the v2 sender path, deliberately NOT yet wired — receivers
hard-drop bad signatures, so senders stay on v1 until the whole fleet verifies v2.
The seq-tampering window closes only when the v1 arm is removed (track as a
post-fleet-rollout follow-up). Unit tests cover v2 verify, v2 seq-tamper rejection,
and v1 sign-then-set-seq compatibility.
- [x] 🟢 **Guard the short-DID slice panic** (`mesh/listener/decode.rs:566`) and gate the
dev-mode `password123` bypass (`auth.rs:18`) behind `#[cfg]`. DONE 2026-07-04:
advert_name uses `.get()` fallback (malformed radio-supplied DID can't panic the
listener); the pre-setup dev-password login + the constant itself are
`#[cfg(debug_assertions)]` — no release binary carries the bypass regardless of
runtime config.
- [ ] 🟢 **Apply the seccomp/apparmor profile** — `security/src/container_policies.rs:71` is a
TODO; the profile is defined but never applied to podman.
- [ ] 🟡 **Manifests that hardcode secrets in plain `environment:` bypass the whole secret
pipeline** (found during the 2026-07-08 .228 leak check): indeedhub-api/-ffmpeg ship
`AES_MASTER_SECRET=0123456789abcdef…` and photoprism `PHOTOPRISM_ADMIN_PASSWORD=archipelago`
as literal plaintext in quadlet unit files; grafana's `GF_SECURITY_ADMIN_PASSWORD=$${…}`
reference never expands. Fix = declare them as `generated_secrets`/`secret_env` in the
manifests — which means catalog regen + re-sign + republish (catalog overlay supremacy).
- [x] 🟠 **Legacy `mempool` umbrella id destroys the split stack on stop→start (quadlet).**
FOUND by the first quadlet-mode gate run on .228 (2026-07-08), FIXED `161a6e4d`:
orchestrator start/stop/restart now alias `mempool` → the split members whenever the
umbrella manifest was dropped (same alias install already used); podman-5
`no such object` phrasing added to all 3 `is_missing_container_error` classifiers.
Follow-up (open): reconciler-side cleanup of an orphan umbrella `mempool` container
when the split stack owns the frontend, so the stale-tile state can't arise at all.
- [x] 🟠 **Transitional package state sticks past the gate window on legacy apps**
(vaultwarden:stop run C, jellyfin:stop run D, uptime-kuma:start run E — .228
2026-07-09), FIXED `dd3afbba`: the scanner already saw the settled container every
60s but `merge_preserving_transitional` refused to report it until the RPC worker
wrote back — and workers legitimately trail the container by minutes (stop workers
queue behind the orchestrator app_lock that reconcile's host-port repair holds
through multi-minute stability waits; start workers hold `Starting` through
readiness budgets up to 420s for uptime-kuma against a 240s gate window, with the
20-minute Installing stuck-timeout as the only escape). New merge rules:
(Stopping, Stopped)+user-stop-marker → Stopped; (Starting, Running) → Running;
Restarting deliberately unresolved. Follow-up (open, same family as the op-lock
known-limit): repair/readiness waits should abort early when a user-stop marker
appears mid-wait, so an explicit stop is never queued behind a multi-minute repair.
- [x] 🟢 **`install_log()` has been a no-op since April** — `/var/log/archipelago/
container-installs.log` is 0 bytes: the service sandbox leaves /var/log read-only,
the open() fails, fire-and-forget drops every line. FIXED `c3f0a306`: every line
now mirrors to tracing/journald; file append stays best-effort.
- [x] 🟢 **Gate tests 123/124 false-fail on user-stopped apps with lingering quadlet
units** (run E: the inactive bitcoin-core of the multi-version pair), FIXED
`2683ad4f0`: `use-quadlet-backends-install.bats` active-state asserts now honour
`user-stopped.json`.
- [ ] 🟠 **Backend recreate must cascade to dependent apps** (found on .228 2026-07-08):
when bitcoin-knots was recreated mid-gate (one-time secret-env recreate) it got a new
archy-net IP; **lnd caches the resolved backend IP** and kept dialing the dead one
("no route to host") for 30+ min — chain-blind with open channels, silently (container
"running", health green). Repair was a manual lnd restart. Fix = reconciler/health
monitor restarts (or at least alerts on) apps whose declared backend container was
recreated; same class applies to electrumx/btcpay/nbxplorer → bitcoin links.
---
## §D — Frontend security & performance (🟠)
The untrusted mesh/LoRa chat path is **safe** (interpolation, no `v-html` — good).
The real issues are the app-bridge origin model and a bloated bundle.
- [x] 🟠 **Validate `event.origin` + add consent gates in the NIP-07 nostr bridge.**
DONE 2026-07-02: `handleNostrRequest` rejects senders whose `event.origin` doesn't match
the open app's URL origin, and ALL identity-sensitive methods (`getPublicKey`, `signEvent`,
`nip04`/`nip44` encrypt+decrypt) now go through the consent/approved-origins gate, not just
`signEvent`. Verified present in the built bundle.
- [x] 🟠 **Origin-check the `share-to-mesh` handler.** DONE 2026-07-02: `App.vue`
`onShareToMeshMessage` now requires `ev.origin === window.location.origin` (matching
`Chat.vue`).
- [ ] 🟡 **Decide the app-iframe isolation model.** `AppSessionFrame.vue:54` /
`AppLauncherOverlay.vue:79` embed apps same-origin with no meaningful `sandbox`; a
same-origin app can read the CSRF cookie + `localStorage`. Ideal fix (serve apps from a
per-app subdomain origin) is architectural — at minimum decide + document for 1.8.0.
- [ ] 🟡 **Shrink the 93 MB dist.** `assets/video/video-intro.mp4` is **14.7 MB**
(precached by the service worker → blocks PWA install), plus ~18 MB of ~1 MB full-screen
JPEGs. Convert backgrounds to WebP/AVIF at responsive sizes, lazy/stream the intro video,
and exclude video/audio from the Workbox precache. Biggest, easiest perf win.
- [x] 🟢 **DOMPurify the `Server.vue` QR SVG / guard `Mesh.vue` pollInterval / surface
`curatedApps.ts` fetch failures.** DONE 2026-07-03: WireGuard peer QR now sanitized with
the same `USE_PROFILES: {svg}` call as TwoFactorSection; Mesh poll interval guarded +
nulled on unmount; catalog fetch failures log per-URL console.warn incl. the
all-sources-failed fallback. Bundle-verified.
---
## §E — Mesh transports (🟢 mostly done — verify & polish)
Confirmed **fixed in HEAD:** B8 (1970 timestamps), B6 (inbound RX surfacing), the
per-message transport pill, and the archy↔archy plain-TEXT-DM E2E fix. Remaining:
- [ ] 🟠 **Active Reticulum daemon-death detection.** `reticulum.rs:589` only `warn!`s on
socket EOF and `try_recv_frame` then returns `Ok(None)` forever; nothing calls
`child.try_wait()`. On an idle link a crashed daemon is invisible for up to 30 min (the
RX-stall timeout). Treat socket EOF as `Err` → immediate respawn. (Pairs with the current
`fix/reticulum-daemon-pdeathsig` branch work.)
- [ ] 🟡 **Persist chat history across restarts.** `state.messages` boots empty
(`listener/mod.rs:283`) while outbox/scheduler/peers survive — inconsistent; bubbles
vanish on restart. Add `mesh-messages.json` mirroring the `scheduler.rs`/`outbox.rs`
pattern (or explicitly accept the loss).
- [ ] 🟡 **Tighten the 30 s legacy dedup** (`listener/mod.rs:383-389`) — it silently drops a
peer legitimately sending identical text twice within 30 s.
- [ ] 🟢 **Wire the PyInstaller daemon binary into the release tarball / deploy script**
(Rust expects `/usr/local/bin/archy-reticulum-daemon`, `reticulum.rs:80`); add the RNode
udev rule; finish `ARCHY:2:` announce→`arch_pubkey_hex` binding (`reticulum.rs:119`).
- [ ] 🟢 **Duty-cycle guard for LoRa TX** — none exists; EU 868 is legally 1%. At minimum an
airtime budget/warning.
---
## §F — ISO / image build (🔴 one secret leak; otherwise 🟠 hardening)
`image-recipe/_archived/build-auto-installer-iso.sh` (3604 lines) is the real
builder; OTA is the normal update path but the ISO is what produces installable
media (latest artifact only one minor behind).
- [ ] ⛔🔴 **Anthropic API key — INTENTIONAL for alpha/beta, hard GO-LIVE gate.**
`build-auto-installer-iso.sh:2645` bakes a live `sk-ant-…` key into `claude-api-proxy.service`
so alpha/beta testers get frictionless AI (deliberate — per user 2026-07-02). **Do NOT
remove for alpha/beta.** Before public GA it MUST be removed + rotated + injected at runtime
(a second copy also exists in a worktree). Track it here so it can't be forgotten at launch.
- [x] 🔴 **Per-device secrets on first boot.** DONE 2026-07-13 (`caf9e6d3`):
`archipelago-first-boot-secrets.service` (enabled on the installed target, ordered
Before=ssh/nginx/archipelago, marker-guarded) regenerates the self-signed TLS keypair
with the device hostname in the SAN and all SSH host keys via staging-first swap —
a failed regeneration keeps the baked keys instead of leaving the device keyless.
**Unverified on hardware**: needs one RC-ISO install to confirm the service fires
and sshd/nginx pick up the new keys.
- [ ] 🟠 **Kill default credentials.** `archipelago`/`archipelago` (SSH+root), web `password123`,
and SSH `PasswordAuthentication yes` (`:411`) all ship. Lock root, force credential
creation in onboarding, disable SSH password auth (or force-change on first login).
- [~] 🟠 **Sign + checksum the ISO.** Checksums DONE 2026-07-13 (`caf9e6d3`): the builder
emits `<iso>.sha256` after xorriso, and `scripts/sign-iso-checksums.sh` signs
`{artifact, sha256, size}` as a JSON doc with the release-root ceremony (verify with
`archipelago ceremony verify` against the pinned anchor; build host never holds the
key). **Still open:** Secure Boot — `BOOTX64.EFI` is unsigned though
`grub-efi-amd64-signed` is installed.
- [ ] 🟠 **Registries over HTTPS in the image too** — `146.59.87.168:3000`
are baked `insecure=true`/`tls_verify:false` (`:216`, `:2308`). (Ties to §A.)
- [ ] 🟡 **Add `unattended-upgrades` + a default-deny nftables firewall** (allow 22/80/443 +
mesh/WG). Neither exists today; OS packages drift until reflash and there is no host
firewall.
- [ ] 🟡 **Pin the build for reproducibility.** FIPS daemon is built from unpinned upstream
`main`, Tailscale from its live apt repo, and `scripts/image-versions.sh` uses many
`:latest`/`stable` tags (+ `bitcoin-ui:1.7.84-alpha`, 15 behind). Pin to commits/versions;
snapshot apt. Wire ISO version to `Cargo.toml` so it can't drift.
- [ ] 🟢 **Harden LUKS + roadmap A/B partitioning.** The LUKS data key sits in plaintext on the
unencrypted root (`:2137`); add TPM2/passphrase binding. Longer-term: A/B (or
factory-reset) partitions for safe OTA rollback, and a real install-time TUI
(`docs/archive/INSTALL-SCREENS-DESIGN.md` exists but the installer is headless "press Enter").
---
## §G — Refactor & code health (🟢 not release-blocking; do after the tag or opportunistically)
- [ ] 🟢 **Manifest-drive per-app special-casing.** App names are branched on across 5-7 Rust
files (`config.rs` 36 match arms, `runtime.rs` 17, `install.rs:275-287` dispatch,
`prod_orchestrator.rs:54-83` baseline/restart-sensitive lists). Move `baseline`,
`restart_sensitive`, `stack_members`, `multi_container` into the manifest schema; collapse
the five near-identical `install_*_stack()` wrappers into one generic call. **Biggest
maintainability win.** (Grew again 2026-07-09: `stack_member_app_ids` in
`package/dependencies.rs` — the quadlet stack-resurrection fallback — is a fifth
per-app map that must fold into the same manifest field.)
- [ ] 🟢 **Route all podman/systemctl through `podman_client`.** 113 raw `Command::new("podman")`
+ 32 `systemctl` calls bypass the existing 952-LOC wrapper → untestable + the blocking-call
risk (§C). Consolidating also unlocks unit tests for the thinly-tested `package/` handlers
(`stacks.rs` 1 test, `config.rs` 2, `runtime.rs` 3, `install.rs` 7).
- [ ] 🟢 **Split the god-modules.** `prod_orchestrator.rs` (5,263 LOC) → `orchestrator/{reconcile,
host_ports,ownership,hooks}.rs`; `Mesh.vue` (2,485 LOC / 241 KB chunk) → sub-components.
Both are well-tested, so safe.
- [ ] 🟢 **Delete dead code.** ~4,100 LOC of orphan StartOS crates (`js-engine`, `models`,
`helpers`, `container-init`) not in the workspace or linked; the committed AppleDouble
`._*.rs` files; the committed `.venv/`/`build/`/`__pycache__` under the duplicate
`reticulum-daemon/` tree; promote `MeshRadioDevice` enum → trait.
- [ ] 🟢 **Resolve the Quadlet flag & dep hygiene.** Decide `use_quadlet_backends`' fate
(flip default + delete the legacy `create_container` branch, or freeze as experimental —
don't ship both half-maintained). Consolidate the mixed hyper 0.14/1.x ecosystem; bump
stale majors (reqwest, base64, thiserror, tokio-tungstenite).
---
## §H — Testing gaps that gate confidence (🟠)
- [ ] 🟠 **Add the OTA upgrade soak** (same as §B item 2) — the highest-value missing test.
- [ ] 🟡 **Add a host-reboot survival tier** — every app is `○` (untested) for reboot in
`TESTING.md:138`; the gate can't reboot the node it runs on. Run SSH-`reboot`-then-reprobe
out-of-band per node.
- [ ] 🟡 **Make the release gate run the full Rust suite** (or hard-require a green CI sha).
`tests/release/run.sh:101` runs only a 6-module slice because the full 1000-test suite
hangs PTYs on the dev box → 994 tests unverified at release time if CI is stale.
- [x] 🟡 **Add `--max-time` to `node_rpc()`.** DONE 2026-07-08 (`380f4f19`): login + rpc get
`--connect-timeout 10 --max-time 120` (override `MULTINODE_RPC_TIMEOUT`). Verified live:
.116 login/rpc OK; an unroutable node fails in 10s instead of hanging.
- [x] 🟢 **De-hardcode creds in tests.** DONE 2026-07-08 (`380f4f19`): multinode suites no
longer commit node passwords — `*_PW` env required, auto-loaded from git-ignored
`tests/multinode/.env` (`.env.example` documents the shape). Still open from this
bullet: snapshot/restore node baseline between destructive iterations
(teardown currently only clears `/tmp` session files).
---
## §I — Carried-over open items (from `UNIFIED-TASK-TRACKER.md`, still valid)
- [~] 🟠 **Multinode gate pass** — 5× destructive gate was launched on node `.5`; bring the
rest of the fleet to precondition, then run the existing (undocumented-but-present)
`tests/multinode/{smoke,meshtastic}.sh` cross-node suites.
- [~] 🟠 **Federation `remove-node` tombstone regression.** Code fix DONE 2026-07-02:
`remove_node` now tombstones BEFORE trimming the node list and propagates the write
error (idempotent, so retries are clean); `add_node`'s untombstone likewise propagates
before mutating. **Still open: `tests/multinode/smoke.sh` re-verify on real nodes.**
- [ ] 🟠 **Phase-3 Quadlet default-flip** — validated + opt-in on .228/.198; flip
`config.rs:256` once the .5 gate reports clean.
- [ ] 🟠 **Developer CLI suite** (`archy app validate/render/install/test`) — gates external
app publishing (`APP-PACKAGING-MIGRATION-PLAN.md` step 5).
- [ ] 🟡 **Version bump + tag** — DECIDED (user, 2026-07-08): the release ships as
**`1.8.0-alpha`**. Remaining work is the mechanical bump + `create-release.sh` run
when the gate criteria are met.
- [ ] 🟢 **Bitcoin multi-version fleet OTA** — DECIDED (user, 2026-07-08): timing doesn't
matter; fold the branch into the next fleet OTA (`docs/bitcoin-version-bulletproof-rollout.md`).
- [x] ~~⛔🟢 **3ccc stock-Meshtastic RF validation**~~ — DROPPED per user 2026-07-08; the
code fix stays in, no live-radio validation will be scheduled.
---
## Suggested order of attack
1. **The critical path:** §A signing ceremony → then turn on manifest/catalog/image
signature enforcement (§A) + OTA HTTPS/signature + deeper health check (§B).
2. **Cheap high-ROI stability:** §C swallowed-writes + blocking-calls; §D nostr-bridge
+ share-to-mesh origin checks; §H OTA soak + reboot tier.
3. **Image hardening:** rest of §F (per-device secrets, default creds, ISO signing,
firewall/unattended-upgrades, pinning).
4. **Polish, post-tag:** §G refactors, §E mesh persistence/dedup, §D bundle shrink.
5. **Decisions you own (⛔):** version name, signing mnemonic, bitcoin OTA timing, 3ccc test.
6. **Before public GA only (NOT alpha/beta):** remove + rotate the Anthropic key (§F) —
intentionally left in for frictionless AI during alpha/beta.
*Last updated: 2026-07-13 (hardening session 4: §F per-device first-boot
secrets + ISO checksum emission/ceremony signing `caf9e6d3` — both need one
RC-ISO install to verify on hardware; Secure Boot remains the open half of
ISO signing). Update this line + tick boxes with commit shas as items land.*
docs/APP-PACKAGING-MIGRATION-PLAN.md
+451
View File
@@ -0,0 +1,451 @@
# App Packaging Migration Plan
## Goal
Turn Archipelago into a serious app platform while preserving the fundamentals that drove the original architecture:
- Rootless Podman and security-first execution.
- Managed node-OS behavior: health, repair, backups, updates, secrets, and routing.
- Bitcoin/LND/Tor/Web5/mesh integration where the platform genuinely needs deep awareness.
- A developer-friendly app packaging model that avoids app-specific Rust installers as the normal path.
## Current Contract
The runtime contract is manifest-first. App packages live at `apps/<app-id>/manifest.yml` and are validated by the shared container manifest parser.
The current canonical manifest fields are:
- `app`: identity and app-level metadata.
- `container`: image or build source, pull policy, network, entrypoint, custom args, derived env, secret env, and data UID.
- `dependencies`: storage and app dependencies.
- `resources`: CPU, memory, disk.
- `security`: capabilities, read-only root, no-new-privileges, network policy, optional AppArmor profile.
- `ports`, `volumes`, `files`, `environment`, `health_check`, and `devices`.
- `metadata`: current catalog-facing presentation data such as category, tier, icon, repo/source, author, and features.
- extension keys may exist temporarily, but they are transitional and should not become a second contract.
The historical `archy-app.yml` name should be treated as superseded. The active local package filename is `manifest.yml`.
## Current Progress
As of the current `1.8-alpha` workstream:
- `apps/*/manifest.yml` is the source of truth for runtime app definitions.
- The Rust manifest parser validates app identity, image-vs-build source selection, safe environment/secrets, safe ports, safe bind/named/tmpfs volumes, generated files under declared bind mounts, devices, and security/network policy values.
- Manifest-owned generated files exist through `app.files` and have been used for app config material (e.g. strfry, netbird config regeneration).
- Local image builds are represented with `container.build`; pulled images are represented with `container.image`.
- Data ownership repair is represented with `container.data_uid`.
- Derived host facts and secret-file-backed environment variables are represented with `container.derived_env` and `container.secret_env`.
- Catalog metadata generation is implemented by `scripts/generate-app-catalog.py`.
- App-session launch ports/titles and new-tab launch behavior now have a generated TypeScript metadata path from manifests, with manual overrides preserved for companion UIs and aliases that do not have manifest-owned metadata yet.
- Runtime package listings now derive LAN launch URLs from manifest-owned `interfaces.main` declarations or HTTP app ports before falling back to legacy compatibility aliases.
- Release drift checking is implemented by `scripts/check-app-catalog-drift.py --release --strict`.
- The canonical catalog and the UI public catalog are expected to remain byte-for-byte synced after generation.
- Runtime validation has already moved many simple and moderate apps into the manifest/orchestrator path, including Filebrowser, Vaultwarden, Portainer, Uptime Kuma, Grafana, Gitea, Nextcloud, SearXNG, Nostr Relay, PhotoPrism, Jellyfin, and several Bitcoin-adjacent apps.
The remaining migration work is mostly orchestration quality: post-reboot adoption, progress reporting, stale scanner-state handling, update policy, multi-container stack ownership, proxy route generation, and cleanup of obsolete legacy installers/fallbacks.
## Target Architecture
Use a StartOS-inspired package model with Umbrel-like app folders.
```text
apps/example-commerce/
manifest.yml
Dockerfile
icon.svg
screenshots/
instructions.md
hooks/
post-install.sh
pre-start.sh
repair.sh
health.sh
backup.sh
restore.sh
proxy/
routes.yml
```
Archipelago becomes the secure compiler/runtime for these packages. The manifest declares what it needs; Archipelago validates it, injects secrets, creates rootless Podman containers, generates nginx/Tor/public routes, registers health checks, displays credentials, and manages lifecycle.
## Core Principles
- App packages are declarative by default.
- Hooks are allowed only as controlled, reviewed escape hatches.
- Rootless Podman stays.
- Arbitrary privileged Compose execution is not allowed.
- Each app has one source of truth.
- Catalog, launch URLs, mobile behavior, credentials, backup paths, and public routes come from the app package or its generated catalog entry.
- Rust backend owns orchestration, not app-specific business logic.
- Core infrastructure can remain special-case where justified.
## What Stays
- Rootless Podman.
- Archipelago orchestrator.
- Health/reconcile/repair loops.
- Host nginx.
- Nginx Proxy Manager integration.
- Tor/public routing goals.
- Bitcoin/LND/mesh/Web5/FIPS/security direction.
- OTA update system.
- App-session/mobile shell.
- Managed secrets and credentials display.
## What Changes
- Complex app stacks stop living in Rust.
- `app-catalog/catalog.json` becomes generated.
- Frontend fallback marketplace data is removed or generated.
- App-session port maps and new-tab launch behavior become generated.
- Public proxy routes become app-declared.
- Install/start/restart/backup/restore become package-driven.
- App updates become app package changes where possible, not full backend code changes.
## Package Schema Direction
Example `manifest.yml`:
```yaml
app:
id: example-commerce
name: Example Commerce
version: 3.23.0
description: Composable commerce platform
container:
image: docker.io/myorg/example-commerce:1.0.0
pull_policy: if-not-present
network: archy-net
entrypoint: ["sh", "-lc"]
custom_args:
- /app/start.sh
derived_env:
- key: PUBLIC_URL
template: https://{{HOST_MDNS}}:9010
secret_env:
- key: SALEOR_SECRET_KEY
secret_file: example-commerce-secret-key
dependencies:
- storage: 20Gi
resources:
cpu_limit: 4
memory_limit: 2Gi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
network_policy: isolated
ports:
- host: 9010
container: 9000
protocol: tcp
volumes:
- type: bind
source: /var/lib/archipelago/example-commerce
target: /data
options: [rw]
environment:
- NODE_ENV=production
health_check:
type: http
endpoint: http://localhost:9000
path: /health
interval: 30s
timeout: 5s
retries: 3
```
Optional generated files, hooks, icons, and screenshots can sit beside the manifest, but the manifest stays the source of truth. Compose-style definitions are not executed directly.
## Security Model
Do not run arbitrary Compose directly. Archipelago validates:
- No privileged containers unless explicitly approved.
- No host filesystem mounts outside approved paths.
- No Docker socket mounts.
- No host network unless explicitly approved.
- No dangerous capabilities by default.
- No arbitrary device access without declaration.
- No rootful execution.
- Pinned images preferred.
- Resource limits required.
- Backup paths declared where the app stores durable data.
- Public routes explicit.
- Secrets referenced by name, not hardcoded.
When the runtime needs app-specific facts that do not belong in the manifest, prefer adding a reusable platform primitive rather than introducing another ad hoc installer path.
This preserves the reason for avoiding raw Umbrel-style Compose while still giving developers a sane package format.
## Lifecycle Model
Every app package should support:
- install
- configure
- start
- stop
- restart
- update
- repair
- health
- backup
- restore
- uninstall
- migrate
Archipelago owns the state machine.
Optional hooks:
- `post-install.sh` for migrations/admin creation.
- `pre-start.sh` for ownership repair.
- `repair.sh` for app-specific remediation.
- `health.sh` for custom health checks.
- `backup.sh` and `restore.sh` only when simple path backups are insufficient.
Hooks run with a controlled environment and restricted permissions.
## Hard Work
The hard work is not writing YAML. The hard work is safely translating app packages into reliable rootless runtime behavior:
- Build a robust package validator.
- Map a safe Compose subset to rootless Podman.
- Handle multi-container networks without hardcoded IPs.
- Handle rootless volume ownership correctly.
- Generate host nginx routes from app metadata.
- Handle public-domain apps without leaking private `192.168.x.x` or `100.x.x.x` URLs.
- Inject secrets without exposing values in logs or frontend bundles.
- Make backup/restore consistent across databases and files.
- Migrate existing hand-built containers to package-owned containers.
- Keep old alpha nodes working while introducing the new system.
- Avoid keeping two permanent systems that drift forever.
## Alpha Node Impact
Existing alpha nodes must not be broken.
Phase 1 behavior:
- Current Rust installers keep working.
- Current app manifests keep working.
- New app package loader exists beside the old system.
- No existing app is automatically migrated.
- Alpha nodes receive compatibility code only.
Phase 2 behavior:
- New installs of selected apps use package mode.
- Existing installs can be detected and adopted.
- App state is preserved.
- Migration is opt-in or happens only for low-risk apps.
Phase 3 behavior:
- Stable migrated apps switch to package mode by default.
- Existing containers are adopted if names/volumes match.
- Data directories are preserved.
- Old Rust installers remain as fallback for at least one release cycle.
Phase 4 behavior:
- Remove old installers only after live alpha validation.
- Keep migration repair code for already-deployed nodes.
## Migration Rules
For every migrated app:
- Preserve `/var/lib/archipelago/<app>` data.
- Preserve generated secrets.
- Preserve credentials shown to users.
- Preserve public ports where possible.
- Preserve container names where needed for adoption.
- Never delete volumes during migration.
- Stop/recreate containers only when necessary.
- Record migration version in app state.
- Provide rollback path to old installer for alpha builds.
## Notes For The Release
- Catalog entries should be generated from manifests so the UI and runtime agree on launch metadata.
- The developer docs should describe the manifest/runtime contract that exists today, not the older publish-model draft.
- If a new capability is needed, add one reusable manifest field or orchestrator primitive and document it here before wiring a one-off app branch.
## First Apps To Migrate
Start with low-risk apps:
- Filebrowser
- Vaultwarden
- Uptime Kuma
- Grafana
Then moderate apps:
- Gitea
- Nextcloud
- SearXNG
- Nginx Proxy Manager metadata integration
Then complex apps:
- Mempool
- BTCPay Server
- NetBird only if safe
Leave for later:
- Bitcoin
- LND
- Electrs/ElectrumX
- Tor
- System update
- Mesh/Web5/FIPS core services
## Complex Stack Reference Goal
Saleor has been removed from the supported release catalog until it has a real
manifest-owned package. A future complex stack should become the showcase
package and prove:
- Multi-container stack support.
- Generated secrets.
- Post-install migration/admin user hooks.
- Dashboard/API/storefront routes.
- Same-origin public GraphQL routing.
- Credentials display.
- Backup paths.
- Health checks.
- Public domain support.
- Alpha-node adoption.
Once a complex stack is clean, the app system is credible.
## Implementation Phases
**Status (2026-07-08):** Phases 1–3 ✅ DONE (per-member manifests won over a
compose subset; all five real multi-container stacks — btcpay, mempool,
immich, netbird, indeedhub — install via `install_stack_via_orchestrator`).
Phase 5 mostly done (orchestrator-first with legacy fallback + per-app
adoption/repair). Phase 4 (routing via `proxy/routes.yml`) NOT started —
routing is still host-nginx driven. Phase 6 (cleanup + developer CLI) NOT
started; the CLI gates external app publishing.
### Phase 1: Package Contract
- Use `apps/<app-id>/manifest.yml` as the package contract.
- Keep the Rust parser/validator as the canonical schema implementation.
- Keep generated catalog output from manifest-owned metadata.
- Finish generated app-session launch metadata so launch behavior cannot drift from manifests.
- Add/keep tests for unsafe package rejection.
### Phase 2: Single-Container Runtime
- Continue hardening package install for one-container apps.
- Compile manifests to rootless Podman/Quadlet runtime behavior.
- Support ports, env, generated files, devices, volumes, resources, health checks, data UID repair, image pull/build availability checks, and launch metadata.
- Keep Filebrowser, Vaultwarden, Portainer, Uptime Kuma, Grafana, SearXNG, Jellyfin, PhotoPrism, and similar apps as regression proofs.
### Phase 3: Multi-Container Runtime
- Decide whether multi-container stacks use a safe `compose.yml` subset or a manifest-native `services` section.
- Support app-local networks.
- Support service dependencies and readiness gates.
- Support internal service names.
- Support generated env/secrets across services.
- Support controlled hooks only where declarative primitives are insufficient.
- Adopt existing multi-container apps without deleting data.
### Phase 4: Routing
- Add `proxy/routes.yml`.
- Generate host nginx routes.
- Generate Tor/public routes.
- Fix same-origin API routing class of bugs permanently.
- Integrate with Nginx Proxy Manager sync.
### Phase 5: Migration
- Add adoption logic for existing containers.
- Add migration metadata.
- Migrate simple apps.
- Migrate a serious multi-container app once the stack model is stable.
- Keep rollback.
- Prove reboot recovery with repeated clean post-reboot lifecycle passes.
- Preserve Nostr signer bridges, Bitcoin dependency wait states, and public launch ports during adoption.
### Phase 6: Cleanup
- Remove duplicated catalog/frontend data.
- Remove migrated Rust stack installers.
- Document package format.
- Add developer tooling: validate, test, package, install locally.
- Remove stale fallback metadata, app-specific lifecycle branches, and compatibility shims only after live validation.
## Developer Tooling
Add commands like:
```bash
archy app validate apps/example-commerce
archy app render apps/example-commerce
archy app install apps/example-commerce
archy app test apps/example-commerce
```
Developers should be able to package an app without understanding Archipelago internals.
## Open Source Story
Public explanation:
> Archipelago uses rootless Podman and a validated app package format. App authors define services declaratively, while the OS enforces security, secrets, routing, backups, health, and lifecycle repair. This gives us Umbrel-like app packaging with StartOS-like managed service discipline.
## Rework Estimate
- Package schema and validator: 1-2 weeks.
- Single-container package runtime: 1-2 weeks.
- Generated catalog/frontend metadata: 1 week.
- Multi-container support: 2-4 weeks.
- Routing/public proxy integration: 1-2 weeks.
- Hooks/secrets/backups: 2-3 weeks.
- First migrations: 2-4 weeks.
- Complex stack reference migration: 1-2 weeks.
- Cleanup/docs/tooling: 2-3 weeks.
Total estimate: 8-14 weeks of serious work for an excellent system.
Minimum viable version: 3-5 weeks.
## Biggest Risks
- Rootless Podman edge cases continue to bite.
- Compose compatibility scope creeps too wide.
- Hooks become an unsafe escape hatch.
- Migration accidentally disrupts alpha nodes.
- Generated metadata drifts from old manual data during transition.
- Old and new systems remain permanently duplicated.
## Risk Controls
- Support a strict Compose subset, not all Compose.
- Validate everything.
- Keep hooks minimal and logged.
- Migrate one app at a time.
- Add live alpha-node checks before each release.
- Generate catalog/app-session data early.
- Set a deadline for deleting migrated legacy installers.
## Immediate Next Steps
1. Expand generated app-session metadata beyond ports/titles/new-tab behavior to cover proxy paths and companion UI aliases where those can be declared safely in manifests.
2. Define the app update policy and wire it into manifest/catalog metadata.
3. Finish post-reboot adoption and stale scanner-state handling for migrated apps.
4. Convert remaining multi-container legacy stacks to a manifest-owned model without deleting data.
5. Add developer tooling around the current `manifest.yml` contract: validate, render, local install, lifecycle test.
6. Migrate a serious multi-container app as the proof package once the stack model is stable.
7. Leave Bitcoin/LND/core services as managed infrastructure until the package system is proven for normal apps.
docs/COMMANDS.md
+141
View File
@@ -0,0 +1,141 @@
# Talking to your node
Every way to ask an Archipelago node a question: over the LoRa mesh, by voice, and over HTTP.
Two rules worth internalising before the tables:
- **`!ai` asks a language model. `!archy` never does.** `!archy` reads the same status caches the HTTP endpoints serve, so its answers are deterministic, cost nothing, and keep working with the assistant switched off.
- **Mesh command prefixes are exact strings** (case-insensitive). **Voice phrases are not** — they are matched by Home Assistant's intent parser, so the examples below are representative, not literal.
---
## 1. Mesh commands
Sent as ordinary text over the mesh — either as plain channel/DM text from a stock meshcore or Meshtastic client, or typed into a 1:1 chat in the Archipelago UI.
### `!archy` — node status, no AI
| Command | Aliases | Answers with |
|---|---|---|
| `!archy` | `!archy status` | OS version, chain tip, peer count, electrum progress |
| `!archy btc` | `bitcoin`, `node`, `sync` | Sync state, block height, peer count |
| `!archy electrs` | `electrum` | Electrum index progress |
| `!archy version` | `ver` | Archipelago OS version |
| anything else | — | A one-line usage hint |
Examples of what comes back:
```
!archy → Archipelago OS v1.7.99-alpha: BTC synced 957295 (12p), electrum 86%.
!archy btc → BTC: synced, block 957295, 12 peers.
!archy electrs → Electrum: syncing 86.2% (824785/957295).
!archy version → Archipelago OS v1.7.99-alpha
!archy wat → archy: !archy [status|btc|electrs|version]. Node status, no AI.
```
`!archyfoo` is not a command — the prefix must be followed by whitespace or end-of-message.
### `!ai` / `!ask` — language model
```
!ai what is the halving schedule
!ask how do I open a lightning channel
```
Requires the assistant to be **enabled** (`assistant_enabled` in `mesh-config.json`). Backend is `ollama` (default, `qwen2.5-coder`) or `claude` (`claude-haiku-4-5-20251001`), set by `assistant_backend`. The model is told to reply in at most two short sentences, because airtime is scarce.
### Who is allowed to ask
Both commands share one gate — `is_sender_allowed()` in `mesh/listener/assist.rs`. Evaluated in order:
1. **Blocked contact** → always denied.
2. **On `assistant_allowed_contacts`** → allowed, even without a signature. This is the deliberate opt-in for keyless phone clients.
3. **`assistant_trusted_only == false`** → anyone on the mesh may ask.
4. **`assistant_trusted_only == true`** → the asker must be *authenticated* **and** carry federation `Trusted` status.
"Authenticated" means the message carried an Ed25519 signature that verified against the sender's known identity key, **or** it arrived over the federation (Tor) transport, which verifies upstream. **Bare plain-text radio messages are never authenticated** — so on a `trusted_only` node, a stock meshcore client can only get an answer by being on the allowlist.
Denials are silent on the wire (no airtime is spent saying "no"); the asker is recorded so an operator can allow them from the UI.
The one asymmetry: **`!ai` additionally requires `assistant_enabled`; `!archy` does not**, because it never calls a model. Turning the LLM off should not take node status with it.
### Where the answer goes
| You asked from | Reply arrives as |
|---|---|
| Plain radio text, your pubkey is known | A private unicast DM (not the public channel) |
| Plain radio text, pubkey unresolvable | A broadcast on channel 0 |
| 1:1 chat in the Archipelago UI | A chat bubble in the same thread |
| The `AssistQuery` widget | Ordered, reassembled `AssistResponse` chunks |
Size caps: 480 characters per answer overall, 200 for plain-text channel/DM replies, and a hard 160-byte LoRa frame limit underneath both.
---
## 2. Voice
A [PineVoice](https://pine64.org) satellite speaker, wake word **"Hey Jarvis"** (or press the centre button). Speech-to-text is Whisper, text-to-speech is Piper — both run locally on the node. Nothing leaves the box.
Phrases are matched by intent, so wording is flexible. These are examples, not exact strings:
| Ask something like | You hear |
|---|---|
| "What is Archipelago OS?" · "What is this node running?" | A one-sentence description plus the running version |
| "What version am I running?" | Version and uptime |
| "Is my node synced?" · "How is my Bitcoin node doing?" · "Bitcoin node status" | Sync state, block height, peer count |
| "What's the current block height?" · "How many blocks do we have?" | The chain tip |
| "Is the electrum server synced?" · "Electrum status" | Index progress |
Optional and alternative words are part of the templates, so "how is *the* node doing" and "how is *my* Bitcoin node" both land on the same intent.
### How voice is wired
The speaker is a Wyoming satellite. Home Assistant runs it through an Assist pipeline: wake word on-device → audio streamed to Whisper → intent matched → Piper speaks the answer.
| Piece | Location (on the node) |
|---|---|
| Whisper + Piper services | `~/.config/containers/systemd/wyoming-{whisper,piper}.container` |
| Wyoming entries, Assist pipeline | `home-assistant/.storage/{core.config_entries,assist_pipeline.pipelines}` |
| Sensors + spoken answers | `home-assistant/configuration.yaml` (`rest:` and `intent_script:`) |
| Phrasings | `home-assistant/custom_sentences/en/archipelago.yaml` |
Home Assistant reaches the node's own HTTP API at `host.containers.internal` — **not** the node's LAN IP, which under rootless podman's pasta networking resolves back to the container itself.
Adding a phrase means editing `custom_sentences/en/archipelago.yaml`; adding an *answer* means adding an `intent_script` entry (and a `rest:` sensor if it needs new data).
---
## 3. HTTP
Served by nginx on port 80, proxying the backend on `127.0.0.1:5678`.
**No authentication required** (5-second cache):
| Endpoint | Returns |
|---|---|
| `GET /health` | Status, uptime, version, services |
| `GET /bitcoin-status` | `getblockchaininfo` + `getnetworkinfo` + `getindexinfo` |
| `GET /electrs-status` | Index height, progress, onion address |
```bash
curl -s http://<node>/bitcoin-status | jq '.blockchain_info.blocks'
```
**Session required** — everything else goes through JSON-RPC at `POST /rpc/v1`:
```bash
curl -s http://<node>/rpc/v1 -H 'Content-Type: application/json' \
-d '{"method":"auth.login","params":{"password":"…"}}' -c jar.txt
curl -s http://<node>/rpc/v1 -b jar.txt -H 'Content-Type: application/json' \
-d '{"method":"system.stats","params":{}}'
```
Login returns a `session` cookie. Read-only methods (`system.stats`, `system.get-metrics`, `bitcoin.getinfo`, `monitoring.current`, `bitcoin.relay-status`, `tor.status`) are CSRF-exempt, so the cookie alone is enough; state-changing calls also need the `X-CSRF-Token` header. If TOTP is enabled, follow the login with `auth.login.totp`.
---
## Adding a command
- **New `!archy` sub-command** — add a variant to `NodeCmd` and a match arm in `run_node_cmd`, both in `mesh/listener/node_cmd.rs`. Keep answers under 200 characters so a stock client sees the whole thing in one frame.
- **New mesh prefix** — add a `strip_*_trigger` alongside `strip_archy_trigger`, then hook it in `decode.rs` (plain radio text) and `dispatch.rs` (typed 1:1 chat). Both paths must be wired or the command only works from one of them.
- **New voice phrase or answer** — see the voice table above.
docs/GAMEPAD-NAV.md
+159
View File
@@ -0,0 +1,159 @@
# Gamepad / Controller Navigation Map
## Global Controls
| Button | Action |
|--------|--------|
| D-pad Up/Down | Navigate between items |
| D-pad Left | Go to sidebar (from any page) |
| D-pad Right | Enter main content from sidebar |
| Enter (A) | Activate / click focused element |
| Escape (B) | Go back one level (inner → container → sidebar → detail page back) |
## Navigation Layers
```
SIDEBAR ──Right──► CONTAINERS (or NAV BAR) ──Enter──► INNER CONTROLS
▲ ▲ │
└──Escape──────────────┘◄─────────Escape──────────────────┘
```
### Sidebar
- **Up/Down**: Move between sidebar items (wraps), auto-navigates links
- **Right**: Jump to main content (first container, or first button on container-free pages)
- **Left**: Nothing
### Nav Bar (mode-switcher tabs, category buttons)
- **Left/Right**: Move between tabs
- **Down**: Jump to first container below (remembers which tab for Up return)
- **Up**: Nothing (Escape to go to sidebar)
- **Left from leftmost**: Go to sidebar
### Container Grid (card tiles on most pages)
- **Arrows**: Spatial nav between containers
- **Enter**: Activate primary action (Install/Launch/navigate) or enter inner controls
- **Escape**: Go to sidebar
- **Left from leftmost**: Go to sidebar
- **Up from top row**: Return to remembered nav bar tab, or spatial to nearest nav item
### Inside Container (inner buttons after Enter)
- **Arrows**: Move between inner controls
- **Escape**: Exit back to the container tile
### Text Inputs
- **Up/Down**: Exit field, navigate spatially
- **Enter**: Submit (click adjacent button)
- **Left/Right**: Cursor movement (exit at edges)
### Container-Free Pages (Settings)
- **Right from sidebar**: Focus first button immediately (no 1s poll delay)
- **Up/Down**: Linear navigation through all buttons/toggles
- **Left**: Go to sidebar
- **Escape**: Go to sidebar
---
## Per-Page Mappings
### Home (`/dashboard`)
Container grid. Dashboard info cards.
### My Apps (`/dashboard/apps`)
| # | Element | Type |
|---|---------|------|
| Nav | My Apps / App Store / Services tabs | Nav bar (Left/Right) |
| 1–N | App cards (grid) | Containers — Enter to view details, inner Launch/Stop/Restart buttons |
### App Store / Discover (`/dashboard/discover`)
| # | Element | Type |
|---|---------|------|
| Nav | My Apps / App Store / Services tabs | Nav bar (Left/Right) |
| 1–2 | Sovereignty Stack featured cards | Containers (`glass-card transition-all hover:-translate-y-1`) |
| 3–N | All Applications grid cards | Containers — Enter for details, inner Install/Launch buttons |
### Network (`/dashboard/server`)
| # | Element | Type |
|---|---------|------|
| 1 | Quick Actions card | Single container — Enter to access Restart/Check Tor/View Logs buttons |
| 2 | Local Network card | Container |
| 3 | Web3 card | Container |
| 4 | Network Interfaces card | Container |
| 5 | Tor Services card | Container |
### Mesh (`/dashboard/mesh`)
| # | Element | Type |
|---|---------|------|
| 1 | Device status card | Container (left column) |
| 2 | Actions row (Enable/Broadcast/Off-Grid/Refresh) | Container |
| 3 | Peers list card | Container — Enter peer to open chat, inner peer items navigable |
| 4 | Chat panel | Container (right column) — message input + send |
| 5+ | Tool panels (Bitcoin/Dead Man/Map) | Containers |
**Chat flow**: Select peer (Enter) → focus auto-jumps to message input → type → Enter sends.
### Cloud (`/dashboard/cloud`)
Container grid. Folder/file cards.
### Settings (`/dashboard/settings`)
**Container-free page** — linear button navigation, no containers.
| # | Element | Section |
|---|---------|---------|
| 1 | Server Name input + save | Account Info |
| 2 | What's New button | Account Info |
| 3 | Copy DID button | Account Info |
| 4 | Copy Onion Address button | Account Info |
| 5 | Change Password button | Account → opens modal |
| 6 | Enable 2FA / Disable 2FA button | Account |
| 7 | Logout button | Account |
| 8 | Language selector buttons | Interface Mode |
| 9 | Login with Claude button | Claude Auth |
| 10 | Enable All / toggle per-category | AI Data Access |
| 11 | Manage Updates button | System Updates |
| 12 | Webhook URL input | Webhooks |
| 13 | Secret input | Webhooks |
| 14 | Container Crash / Update Available toggles | Webhooks |
| 15 | Disk Space Warning / Backup Complete toggles | Webhooks |
| 16 | Save Configuration / Send Test buttons | Webhooks |
| 17 | Enable Beta Telemetry button | Telemetry |
| 18 | Create Backup button | Backup |
| 19 | Export Channel Backup button | Backup |
| 20 | Network Diagnostics button | Danger Zone |
| 21 | Reboot button | Danger Zone → confirms with modal |
| 22 | Factory Reset button | Danger Zone → confirms with modal |
### Monitoring (`/dashboard/monitoring`)
Container grid. Stats/chart cards.
---
## Focus Memory
| Key | Remembers | Used When |
|-----|-----------|-----------|
| `sidebar` | Last sidebar item | Returning to sidebar via Escape/Left |
| `main` | Last focused container | Re-entering main zone |
| `navBar` | Last focused tab/button | Up from container returns to same tab |
All focus memory is cleared on route change.
## Data Attributes
| Attribute | Purpose |
|-----------|---------|
| `data-controller-zone="main"` | Main content area (on `<main>`) |
| `data-controller-zone="sidebar"` | Sidebar navigation |
| `data-controller-container` | Focusable card/tile (with `tabindex="0"`) |
| `data-controller-install` | Container has an Install button (Enter prioritizes it) |
| `data-controller-launch` | Container has a Launch button (Enter prioritizes it) |
| `data-controller-install-btn` | The actual Install button inside a container |
| `data-controller-launch-btn` | The actual Launch button inside a container |
| `data-controller-ignore` | Skip this element and descendants from navigation |
| `data-controller-focus` | Make non-standard element focusable |
## Implementation
- **File**: `neode-ui/src/composables/useControllerNav.ts`
- **Store**: `neode-ui/src/stores/controller.ts` (tracks active state + gamepad count)
- **Sounds**: `neode-ui/src/composables/useNavSounds.ts` (move/action/back)
- **Spatial nav**: `findNearestInDirection()` — filters by direction, scores by overlap + distance
docs/PRODUCTION-MASTER-PLAN.md
+1265
View File
File diff suppressed because it is too large Load Diff
docs/RELEASE_NOTES_BACKLOG.md
+15
View File
@@ -0,0 +1,15 @@
# Release Notes Backlog
## Next Release Required Work
- Backfill missing or thin historical release notes before cutting the next release.
- Audit every `CHANGELOG.md` section from `v1.7.44-alpha` through the current release.
- Replace raw commit-hash entries with user/operator-facing bullets that explain behavior changes, operational impact, validation, and known limitations.
- Ensure `releases/manifest.json` changelog entries come from curated `CHANGELOG.md` notes only.
## Release Note Policy
- Every release must have at least three curated bullets.
- Raw `git log --oneline` output is not acceptable release documentation.
- Notes should answer what changed, why it matters, what operators should expect, and any known limitations.
- `scripts/check-release-manifest.sh` is the enforcement gate before publishing artifacts.
docs/RETICULUM-TRANSPORT-PROGRESS.md
+372
View File
@@ -0,0 +1,372 @@
# Reticulum mesh transport — progress tracker
Living status doc for the Reticulum (RNS+LXMF) third-transport work. **Update this after every
meaningful step.** If a session is cut off mid-work, read this file first, then the plan, then
resume at "Next up."
Full plan: `.claude/plans/enchanted-strolling-rocket.md`. Memory pointer:
`project_reticulum_transport_plan.md` (auto-memory index).
**Coordination note (2026-06-30):** a separate agent owns concurrent Meshtastic work, scoped to
`mesh/meshtastic.rs` + `mesh/protocol.rs` (see `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`) and explicitly
avoiding `mesh/listener/session.rs` transport plumbing + `mesh/mod.rs` routing, which this work
owns. Stay out of `meshtastic.rs`/`protocol.rs` to avoid collisions.
## Status at a glance
| Phase | What | Status |
|---|---|---|
| 0 | Gate #1 — deterministic identity from Archy keys | ✅ **DONE**, verified in venv AND in the PyInstaller binary (same dest hash) |
| 0 | Gate #2 — two-node LXMF-over-LoRa on real hardware | ✅ **PASSED 2026-06-30** — real RF announce + encrypted DM exchanged between .116's Heltec V3 RNode and a phone-flashed second RNode running Sideband |
| 0 | Gate #3 — external Sideband/MeshChat interop | ✅ **PASSED 2026-06-30** — same session as gate #2; Sideband is the stock external client this gate calls for |
| 1 | `reticulum-daemon/` (Python rns+lxmf, Unix-socket RPC) | ✅ scaffolded + tested (no radio); signed-identity announce **also done** (see below) |
| 1 | Packaging — PyInstaller single binary | ✅ **DONE + verified** — `reticulum-daemon/build.sh`, 16M standalone binary, selftest passes run from `/tmp` with no venv on PATH |
| 2 | Rust wiring (`DeviceType`, `MeshRadioDevice`, `ReticulumLink`, stamp sites) | ✅ **`cargo check`/`cargo test -p archipelago` GREEN** (99 mesh tests pass) — still untested on real hardware |
| 2c | `MeshConfig.device_kind` reflashable-board pin | ✅ **DONE** this session (was the one open Phase-2 item) |
| 3 | Frontend (~8 label/CSS spots) | ✅ DONE (scoped down — see note below) |
| 4 | Multi-device (run all 3 radios at once) + per-network channels | ⏳ not started (follow-on, after 0–3) |
| 5 | Aurora interop — optional plain-TCP Reticulum interface (radio-less) | ✅ **DONE + verified 2026-07-03** — see checkpoint below. Real Aurora GUI test still open (manual follow-up). |
## Checkpoint 2026-06-30 (late session — read this first if cut off)
This session picked up after Phase 2/3 were already green, and closed out everything that didn't
need real RNode hardware:
1. **Corrected two stale tracker entries** (both were already done, just not reflected here):
- The `_announce_app_data` "TODO" was actually already implemented:
`reticulum_daemon.py`'s `_announce_app_data()` embeds `ARCHY:2:{ed}:{x25519}` when
`--archy-ed-pubkey-hex`/`--archy-x25519-pubkey-hex` are passed, and `reticulum.rs`'s
`daemon_command()`/`open()` already forward `our_ed_pubkey_hex`/`our_x25519_pubkey_hex` from
`session.rs` (`run_mesh_session` → `auto_detect_and_open`/`open_preferred_path` →
`ReticulumLink::open`). Confirmed end-to-end by reading the call chain, not just grepping.
- Phase 3 frontend was already done (see prior entry below) — tracker table above said
"not started", now corrected.
2. **Added `MeshConfig.device_kind: Option<DeviceType>`** (plan §2c, the one explicitly-listed
open Phase-2 item) — `mesh/mod.rs` (field + Default + threaded into `start()`'s
`spawn_mesh_listener` call), `listener/mod.rs` (`spawn_mesh_listener` param → `run_mesh_session`
arg), `listener/session.rs` (`run_mesh_session` param; `auto_detect_and_open` skips
non-matching probes per-path via `device_kind.is_none_or(|k| k == ...)`;
`open_preferred_path` restructured to a `match kind { ... }` that tries **only** the pinned
driver and surfaces its real error, instead of silently falling through to another firmware's
handshake on the same port). `None` (default) preserves today's strict
Meshcore→Meshtastic→Reticulum auto-detect — fully backward compatible, no config migration
needed. `cargo check` + `cargo test -p archipelago` both green after (99 mesh tests, 0 failed).
3. **Built and verified the PyInstaller packaging** (plan's Phase 1 "Packaging" + the file list's
"Ops: release packaging to include the daemon binary" item — previously undone):
- `reticulum-daemon/build.sh` (new) — reproducible build, installs `requirements-build.txt`
(new, `pyinstaller==6.21.0`, build-only/not shipped) into the existing `.venv`, runs
PyInstaller with flags discovered by trial: `--collect-submodules RNS --collect-submodules
LXMF --collect-data RNS -d noarchive`.
- **Non-obvious gotcha, written up in `build.sh`'s comments so it isn't re-discovered:**
`RNS.Interfaces/__init__.py` builds its `__all__` via `glob.glob(os.path.dirname(__file__) +
"/*.py")` at import time (`Reticulum.py` does `from RNS.Interfaces import *`). PyInstaller's
default `--onefile` zips pure-Python modules into an in-binary PYZ archive, so `__file__`
doesn't point at a real directory and the glob comes back empty → `NameError: name
'Interface' is not defined` the moment `RNS.Reticulum(...)` is constructed. `-d noarchive`
(keep modules as loose `.pyc` files on disk inside the onefile bundle's runtime-extraction
dir) fixes it — confirmed by reproducing the failure first, then fixing it.
- **Verified, not just built:** ran the resulting `dist/archy-reticulum-daemon` binary's
`--check` (dest hash matches the venv-derived `06bb31e16f4f8d46a8ae8eac23a4fd21` for the
test seed) and `--selftest` (full RNS+LXMF bring-up, no radio) **both from `/tmp` with the
binary copied away from the repo and the `.venv` not on `PATH`** — confirms it's genuinely
self-contained, not accidentally still depending on the dev venv.
- `dist/`/`build/`/`*.spec` are already gitignored (`reticulum-daemon/.gitignore`); only
`build.sh` + `requirements-build.txt` are new tracked files.
**NOT done this session (still genuinely open):**
- Everything hardware-dependent (Phase 0 gates #2/#3, real RNode probe/spawn). The .116 Heltec V3
reflash mentioned in the prior session's memory was **not** done in this session — no physical
hardware access was exercised, only software.
- `/dev/reticulum-radio` udev symlink (plan §2c) — **deliberately not added**: the existing
`99-mesh-radio.rules` keys on USB vendor/product ID (e.g. CP2102 0x10c4/0xea60), but the whole
point of `device_kind` is that the *same* chip can run any of the three firmwares — a
vendor/product udev rule can't disambiguate them, and a fabricated rule would just be
misleading. Real fix needs either a per-device `ATTRS{serial}==...` rule the operator fills in
once they know their specific board's serial (no such board exists in-repo to template from
yet), or rely on `device_kind` alone (already done, works regardless of `/dev` path naming).
Revisit once a real RNode-flashed board's serial is known.
- PyInstaller binary not yet wired into the release tarball / `scripts/deploy-to-target.sh` (the
daemon binary path is currently resolved via `ARCHY_RETICULUM_DAEMON_BIN` env or the dev venv
fallback in `reticulum.rs`'s `daemon_command()` — production default
`/usr/local/bin/archy-reticulum-daemon` is a real path convention now that `build.sh` produces
exactly that filename, but nothing copies it there yet). Left undone deliberately — wiring
release-tarball plumbing for a binary that's never been run against real RNS network traffic
felt premature; do this once Phase 0 gates #2/#3 pass.
## Phase 2 — Rust wiring detail (what's done vs left)
**Done — `cargo check -p archipelago` is GREEN:**
- `core/archipelago/src/mesh/types.rs` — `DeviceType::Reticulum` (+ `Display` arm) + a
`radio_transport_label(DeviceType) -> &'static str` helper (`"reticulum"` vs `"lora"`).
- `core/archipelago/src/mesh/mod.rs` — all 4 outbound stamp sites use
`radio_transport_label(...)`; `use_typed_envelope` (~1571) extended to
`matches!(device_type, Meshcore | Reticulum)`; `data_dir` threaded into
`spawn_mesh_listener(...)` call (was: `MeshService::start()` → `spawn_mesh_listener`).
- `core/archipelago/src/mesh/listener/mod.rs` — `spawn_mesh_listener` takes `data_dir:
PathBuf`, passes `&data_dir` into `run_mesh_session`.
- `core/archipelago/src/mesh/listener/decode.rs:406,639` and `dispatch.rs:79` — all 3 inbound
stamp sites now use `radio_transport_label(state.status.read().await.device_type)`.
- `core/archipelago/src/mesh/listener/session.rs`:
- `MeshRadioDevice` enum has `Reticulum(ReticulumLink)`; all 18 method arms wired (no-ops:
`ensure_lora_region`, `ensure_channel`, `send_keepalive`, `send_nodeinfo_advert`, `reboot`,
`reset_contact_path`; everything else forwards to `ReticulumLink`).
- `auto_detect_and_open(data_dir: &Path)` and `open_preferred_path(path, data_dir: &Path)`
both now try `ReticulumLink::open(path, data_dir)` **last**, after Meshcore/Meshtastic —
cheap raw-serial KISS-detect probe runs first; the daemon only spawns on a confirmed match.
- `reticulum_contact_id()` helper added (delegates to the canonical
`reticulum::reticulum_contact_id_from_hash`, masked `& 0x7FFF_FFFF`, avoids 0).
- `refresh_contacts()` has an `is_reticulum` branch parallel to `is_meshtastic`; `reachable`
flows through `contact.path_len != 0` unchanged (`ReticulumLink::get_contacts()` already
encodes daemon-reported reachability into `path_len`).
- `data_dir: &Path` threaded through `run_mesh_session` → both probe functions.
- `core/archipelago/src/mesh/reticulum.rs` — **created**. `ReticulumLink`: spawns/supervises the
daemon as a child process, Unix-socket RPC client (matches the tested daemon contract),
`prefix_to_hash: HashMap<[u8;6],[u8;16]>` (mandatory per the plan), synthetic
`InboundFrame` builder byte-matching `meshtastic.rs`'s layout, `Drop` impl that kills the
daemon + cleans up the socket. Has unit tests (KISS-detect byte matching, contact-id masking,
synthetic-frame layout) — **passing, see below**.
**Concurrent-edit note:** a separate in-flight change (not mine) added `MeshPeer.pkc_capable`
and `ParsedContact.pkc_capable` (Meshtastic PKI-capability tracking) while this work was in
progress. Accounted for: `reticulum.rs`'s `ParsedContact` literal sets `pkc_capable: false`
(Reticulum/LXMF is unconditionally E2E via `take_rx_encrypted()`, this field has no analogue);
two incomplete `MeshPeer` literals in `decode.rs` (lines ~330, ~548) were completed with
`pkc_capable: false` to unblock the build for everyone — not reverted, not worked around.
**Self-review fix applied:** the RPC Unix socket originally lived in the shared system temp
dir; moved to `{data_dir}/reticulum/` (0700) instead — archipelago-owned, not shared `/tmp`,
matching the security posture. Re-confirmed `cargo check -p archipelago` GREEN after the move.
**NOT yet done:**
- `MeshConfig.device_kind: Option<DeviceType>` hint (optional reflashable-board disambiguator,
plan §2c) — not added. Auto-detect ordering (Meshcore→Meshtastic→Reticulum, strict probes)
is the only disambiguator right now.
- Phase 3 frontend — **DONE**, but **smaller scope than originally inventoried**: only
`Mesh.vue`'s `transportLabel()` (per-message field) + `mesh-styles.css` `.transport-reticulum`
+ the `mesh.ts` doc comment needed the addition. `transport.ts` `TransportKind`,
`federation/types.ts` `last_transport`, `NodeList.vue` `transportBadge`, and `PeerFiles.vue`
`transportPill` are a COARSER routing-layer category (`mesh`/`lan`/`fips`/`tor`) where
`'mesh'` already covers any radio (meshcore/meshtastic/reticulum) — adding a separate
`'reticulum'` there would be inconsistent with how meshcore/meshtastic are handled. Confirmed
via `vue-tsc --noEmit` (exit 0, zero errors).
- Everything hardware-dependent: real daemon spawn/probe against an actual RNode (the .116
Heltec V3, once reflashed), two-node LXMF-over-LoRa, the `_announce_app_data` signed-identity
TODO in the daemon (currently carries only the plaintext display name, not a verified Archy
DID/pubkey — needed for `bind_federation_twins`-style auto-binding across protocols).
## Verified facts to reuse (don't re-derive)
**RNode KISS-detect handshake** (confirmed against the canonical Reticulum source, not guessed):
```
constants: FEND=0xC0 FESC=0xDB TFEND=0xDC TFESC=0xDD CMD_DETECT=0x08 DETECT_REQ=0x73 DETECT_RESP=0x46
probe tx: C0 08 73 C0 50 00 C0 48 00 C0 49 00 C0 (detect + fw_version + platform + mcu queries)
success: response contains byte sequence ... C0 08 46 ... (FEND, CMD_DETECT, DETECT_RESP)
```
Source: `RNS/Interfaces/RNodeInterface.py` (Liberated Systems mirror), `detect()`/`readLoop()`.
**Synthetic `InboundFrame` layout** for a 1:1 DM, copied exactly from
`meshtastic.rs:1031-1047` (`ReticulumLink` must build the same shape so `frames::handle_frame`
needs zero changes):
```
data = [snr(1)=0][reserved(2)=00,00][sender_prefix(6)][path(1)=0xff][type(1)=0][rx_time(4 LE)][payload…]
code = RESP_CONTACT_MSG_V3_E2E if encrypted else RESP_CONTACT_MSG_V3 (RNS/LXMF is always E2E, so always _E2E)
```
Channel/broadcast equivalent (`RESP_MESHTASTIC_CHANNEL_TEXT`, meshtastic.rs:1019-1028) — N/A for
Reticulum in single-device Phase 2 (LXMF has no shared-channel concept); revisit in Phase 4.
**`resolve_peer`** (decode.rs:316) matches inbound `sender_prefix` against
`peer.pubkey_hex.starts_with(prefix)` — so as long as `refresh_contacts`/announce-handling
populates `pubkey_hex` = full 16-byte RNS hash hex BEFORE a message arrives (same precondition
meshtastic relies on via its `peer_pubkeys` map), no Reticulum-specific fallback is needed there.
**`ParsedContact.public_key_hex`** for Reticulum = hex of the 16-byte RNS dest hash (32 hex
chars, NOT 32 bytes) — the `hex::decode(...).len()==32` checks elsewhere (e.g. the auto-heal
`reset_contact_path` loop in `refresh_contacts`) will naturally skip Reticulum contacts since
their key decodes to 16 bytes, not 32. That's fine — no special-casing needed, just don't "fix"
it to be 32 bytes.
**`data_dir.join("identity").join("node_key")`** is the 32-byte raw Ed25519 seed file — this is
exactly what `reticulum_daemon.py --identity-key <path>` expects (confirmed against
`identity.rs` `NODE_KEY_FILE`/`load_or_create`). The daemon reads the file itself — Rust should
pass the **path**, not pipe the raw key bytes through more hops than already exist.
## Hardware update (2026-06-30)
**.116 has a Heltec V3 available to reflash with RNode firmware.** This unblocks Phase 0 gates
#2/#3 (previously marked blocked — `.198`'s radio is dead, but .116's Heltec V3 is a real path
forward without needing new hardware). Next concrete step once reflashed: run
`reticulum-daemon/reticulum_daemon.py` pointed at the RNode's serial path, confirm `--check`
hash matches `--selftest`, then bring up two instances (.116 + .228, after .228 also gets an
RNode-capable board) for the real two-node LXMF-over-LoRa gate.
## Daemon contract (already built + tested — Phase 2 codes against this, no changes needed)
`reticulum-daemon/reticulum_daemon.py`, RPC over Unix socket (0600), one JSON object per line:
- in: `{"cmd":"send","dest_hash":hex16,"content":...}` / `{"cmd":"announce"}` /
`{"cmd":"status"}` / `{"cmd":"shutdown"}`
- out: `{"event":"ready",...}` / `{"event":"recv",...}` / `{"event":"announce",...}` /
`{"event":"delivered",...}` / `{"event":"status",...}`
Verified: `--check` (hash only), `--selftest` (boots real RNS+LXMF, no radio), and a live
socket round-trip (`ready`→`status`→`shutdown`, clean exit) — see `reticulum-daemon/README.md`.
## Checkpoint 2026-06-30 (hardware session — gates #2/#3 PASSED)
Picked up after a session pipe-break; the live system (archipelago.service + the spawned
`archy-reticulum-daemon`) had kept running uninterrupted the whole time, so nothing was lost.
**What happened, in order:**
1. .116's Heltec V3 (CP2102, USB vendor/product `10c4:ea60`, serial `0001`) was reflashed with
RNode firmware and plugged into `/dev/mesh-radio` (generic udev symlink → `ttyUSB0`, not a
per-serial rule). `mesh-config.json` has `device_path: null` — pure auto-detect, no
`device_kind` pin needed.
2. Auto-detect correctly tried Meshcore → Meshtastic → Reticulum and found it: journal shows
`Found Reticulum (RNode) device via auto-detect path=/dev/mesh-radio` — but only **after**
~4 min of `Failed to spawn reticulum-daemon — is it installed/packaged?` retries, because
`/usr/local/bin/archy-reticulum-daemon` hadn't been copied into place yet from
`reticulum-daemon/dist/` (built via `./build.sh`). Once copied (sha256-verified match to the
`dist/` build), auto-detect succeeded on the very next retry.
3. `mesh.status` RPC confirmed live: `device_type: "reticulum"`, `device_connected: true`,
`dest_hash: 5d146f6e1c9707f89468b5016ed6dfad`. Periodic self-advert (`send_self_advert` →
`{"cmd":"announce"}` → real RNS `Identity.announce()`) firing every ~30s — confirmed this is
**not** the `send_nodeinfo_advert` no-op arm (that one's still legitimately a no-op for
Reticulum; the real announce path is `send_self_advert`, wired correctly).
4. Second RNode flashed onto a phone running **Sideband**. First attempt showed RF energy
(`interference_last_dbm` climbing) but `rxb: 0` — a parameter mismatch, **not** a frequency
problem (energy was detected, just not demodulated). Root cause: Spreading Factor mismatch
in Sideband's manual RNode interface config (frequency display rounds to one decimal so
"869.5" silently passed at first glance — bandwidth/SF/CR are separate fields and SF was
wrong). Once SF was corrected to match (freq `869525000`, BW `125000`, **SF `8`**, CR `5`),
`rxb` went non-zero immediately and a real `{"event":"announce","dest_hash":"1870744d...",
"app_data":"7a617a61"}` (hex for "zaza") arrived over the air.
5. **Gate #2 + gate #3 both passed in the same exchange**: `zaza` shows up as a real, reachable
`mesh.peers` contact; an inbound encrypted LXMF message ("Yoooo") arrived and was correctly
stamped `encrypted: true, transport: "reticulum"`; a reply was sent back and round-tripped.
Sideband is exactly the stock external client gate #3 calls for, so one real RNode-to-RNode
LoRa link covered both gates — no need for a second dedicated archy node.
6. **Two real bugs found from this, both fixed:**
- `record_sent_typed`'s `encrypted` flag was hardcoded `false`/`archy || pkc_capable` on the
Reticulum send path (both the native-text path in `send_message` and the typed-envelope
path in `send_typed_wire`) — correct for Meshcore/Meshtastic (where E2E really is
conditional on PKI/session state not yet threaded through), **wrong** for Reticulum: LXMF
encrypts every send to the destination identity key unconditionally, archy peer or not.
Fixed: both call sites now OR in `device_type == DeviceType::Reticulum`.
- `radio_transport_label()` collapsed Meshcore **and** Meshtastic into one generic `"lora"`
string, so the per-message pill couldn't distinguish them. User asked for 3 distinct pill
colors (Meshtastic mint, Meshcore orange, Reticulum blue) — extended the label fn to
return `"meshtastic"`/`"meshcore"`/`"reticulum"` distinctly, updated `Mesh.vue`'s
`transportLabel()` switch and `mesh-styles.css` (`.transport-meshtastic` `#3eb489`,
`.transport-meshcore` `#fb923c`, `.transport-reticulum` `#60a5fa`; kept `.transport-lora`
`#f59e0b` as a fallback for any already-stored legacy-labelled messages). `cargo check` +
`vue-tsc --noEmit` both green after.
**NOT yet done:**
- The Rust-side fix above (`encrypted` flag, transport-label split) is built but **not yet
deployed to .116's running binary** — the live daemon/auto-detect verification above was all
against the binary already running before this session's edits. Rebuild + redeploy to see the
fix live.
- `tests/lifecycle/run-gate.sh` not re-run after these mesh changes yet (project convention:
run after backend changes land).
- Multi-device (3 radios at once, Phase 4) and the release-tarball/udev-rule wiring (originally
"Next up" #6 below) are both still untouched.
## Next up (resume here)
Phase 0 gates #1–#3 are now **all passed**. What's left:
1. Rebuild the backend + frontend and redeploy to .116 so the `encrypted`-flag fix and the
3-way transport-pill color split actually take effect on the live node (currently only
checked in with `cargo check`/`vue-tsc`, not deployed).
2. Re-verify on-device after redeploy: send another Sideband↔archy DM, confirm the Sent bubble
now shows E2E + a blue "Reticulum" pill, and confirm Meshtastic/Meshcore pills (if any
messages exist) render mint/orange instead of the old generic amber "LoRa".
3. Exercise the rest of the plan's "Verification (definition of done)" items: hot-swap
detection (unplug the RNode mid-session, confirm fallback to FIPS/Tor on the same contact;
replug, confirm it picks Reticulum back up), and `device_kind: Some(Reticulum)` pin path
(currently only auto-detect has been exercised on real hardware).
4. Run `tests/lifecycle/run-gate.sh` to confirm no regression from the mesh changes landing.
5. Only after the above: wire `dist/archy-reticulum-daemon` into the release tarball /
`scripts/deploy-to-target.sh` (target path `/usr/local/bin/archy-reticulum-daemon`, matching
`reticulum.rs`'s default) and add a per-serial-number `/dev/reticulum-radio` udev rule now
that a real board's serial number (`0001` on the CP2102, .116's board) is known — though a
second board will likely report the same `0001` stock serial since CP2102 modules commonly
ship with an unprogrammed default, so this may still need a different disambiguator.
6. Phase 4 (run all 3 radios at once) — still not started, follow-on after the above.
## Checkpoint 2026-07-03 — Phase 5: Aurora interop via plain-TCP Reticulum (radio-less)
**Why:** `~/aurora` (a separate Flutter off-grid messenger) already runs real RNS + LXMF
(`LxmfRouter`, comment "interop with Sideband/NomadNet/MeshChat" in `rns_service.dart`), and its
**default** connectivity mode is plain TCP (`RnsTcpInterface`/`RnsTcpServerInterface`), not radio —
it ships a static bootstrap list of public RNS hubs on port 4242. Archy's daemon could previously
only bring up a serial-RNode interface, so it was unreachable by Aurora (or any TCP-based RNS/LXMF
client) at all, and every interop proof was bottlenecked on scarce LoRa hardware. This phase adds
an **optional, additive, loopback-only plain-TCP interface**, proves interop with a scripted
RNS/LXMF stand-in (the same class of proof the Sideband gate already established), and leaves the
serial/RNode path completely unchanged.
**Done, all verified:**
1. `reticulum-daemon/reticulum_daemon.py` — `_write_rns_config()` gained a third branch
(`--tcp-listen HOST:PORT` → `TCPServerInterface`, `--tcp-connect HOST:PORT` repeatable →
`TCPClientInterface`), mutually exclusive with `--serial-port`. `--tcp-listen` is hard-gated to
loopback (`_require_loopback`) — archy is otherwise Tor-first for inter-node traffic, so a
WAN/LAN-exposed Reticulum port is a deliberate future decision, not something this phase does
silently. Verified: `--selftest` regression still passes; two daemon processes (server +
client, throwaway identities) reached `connected: true` on both sides via `mesh.status`-daemon
RPC, live `TCPServerInterface`/`TCPClientInterface` visible in `get_interface_stats()`.
2. **Bidirectional LXMF DM gate against a scripted Aurora stand-in** (Python RNS+LXMF client
dialing as a `TCPClientInterface` + running its own `LXMRouter` — a legitimate protocol-level
proxy for Aurora's Dart stack, same wire format): forward (stand-in → archy daemon) and reverse
(archy daemon → stand-in) both delivered with matching content and correct source/dest hashes,
confirmed via the daemon's own `recv`/`delivered` RPC events. Direct TCP analogue of the
already-passed Sideband gate (RF → TCP, Sideband → scripted stand-in).
3. **Rust wiring**, fully additive — the serial/RNode path is byte-for-byte unchanged:
- `mesh/reticulum.rs`: new `ReticulumInterface` enum (`Serial`/`TcpServer`/`TcpClient`) threads
through `daemon_command()`/`spawn()`; `open()` (serial) now just wraps
`ReticulumInterface::Serial` — same `probe_rnode` gate as before. New
`open_tcp_server()`/`open_tcp_client()` associated fns skip `probe_rnode` entirely (the
"spawn without a physical RNode" path); `open_tcp_server` hard-enforces
`is_loopback_host()` (mirrors the Python-side guard).
- `mesh/types.rs`: new `ReticulumTcpConfig` enum (`Server { bind }` / `Client { connect }`).
- `mesh/mod.rs`: `MeshConfig.reticulum_tcp: Option<ReticulumTcpConfig>` (`#[serde(default)]`,
`None` by default — no migration, zero behavior change when unset); threaded into
`start()` → `spawn_mesh_listener`.
- `listener/mod.rs` / `listener/session.rs`: `reticulum_tcp` param threaded through
`spawn_mesh_listener`/`run_mesh_session`; new leading branch — if set, a new
`open_reticulum_tcp()` helper dispatches to `open_tcp_server`/`open_tcp_client`; otherwise
falls through to the **untouched** existing `preferred_path`/`auto_detect_and_open` logic.
- Deliberately **not** wired into `mesh.configure`/the frontend — dev/verification-only surface
for now (hand-edit `mesh-config.json`), consistent with how narrowly scoped this phase is.
- `cargo check -p archipelago` + `cargo test -p archipelago` (mesh module): **108 passed, 0
failed, 1 ignored** (the pre-existing hardware-gated `probe_rnode_detects_real_hardware`) —
zero regression to the serial/RNode path, provable without any hardware.
4. **End-to-end Rust integration test** (`mesh::tests::mesh_service_connects_over_reticulum_tcp_client`,
`#[ignore]`d — spawns real subprocesses, skipped in the default `cargo test` run the same way
the rest of the mesh suite skips hardware-gated tests): a real `MeshService::start()` spawns the
daemon in TCP **client** mode (no serial probe at all), dials a second stand-alone daemon
instance in TCP **server** mode (the Aurora-side role), and reaches `device_connected: true` /
`device_type: Reticulum` via the exact `MeshService::status()` call the `mesh.status` RPC uses.
Passed in ~2.6s. Run manually: `cargo test -p archipelago -- --ignored
mesh_service_connects_over_reticulum_tcp` (needs `reticulum-daemon/.venv`, see below).
**Environment note:** this session's Rust toolchain drift — system `rustc` (apt, 1.85.0) is too
old for code already on `main` (`u32::is_multiple_of` in `health_monitor.rs`, stabilized upstream
after 1.85); a pre-installed rustup toolchain at
`~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu` (1.96.0) builds clean. Not something this
phase's changes caused — pre-existing, just newly hit. Put that toolchain's `bin/` first on `PATH`
if `cargo check`/`test` reports `E0658 unsigned_is_multiple_of`.
**Explicitly NOT done (out of scope for this phase, see plan non-goals):**
- Real Aurora Flutter GUI verification — this dev sandbox has no `flutter`, no `$DISPLAY`, and no
`reticulum-dart` sibling checked out (Aurora's actual RNS implementation lives in that separate
repo; Aurora's CI clones it fresh at build time). The scripted-stand-in gate above is the
protocol-level substitute. **Manual follow-up**: point a real Aurora build's TCP hub list (or an
ad hoc connect) at an archy node's `--tcp-listen` address and confirm an LXMF DM in the actual
app UI.
- Any non-loopback (LAN/WAN) TCP bind — hard-gated off on purpose; a real "Aurora hub" deployment
needs its own security review given archy's Tor-first posture for inter-node traffic.
- LXMF propagation-node / always-on-hub role for archy (bridging Aurora's offline BLE peers) —
bigger architectural + storage commitment.
- Identity unification between archy's and Aurora's independent Nostr/secp256k1 keys — both
already have separate Nostr identities with no derivation link; out of scope here.
- `mesh.configure` RPC / frontend exposure of `reticulum_tcp` — stays hand-edit-only until/unless
it becomes user-facing.
docs/ROADMAP.md
+97
View File
@@ -0,0 +1,97 @@
# Archipelago Roadmap
_Last updated: 2026-07-08. This is the public-facing summary. The live,
priority-ordered engineering list is [`UNIFIED-TASK-TRACKER.md`](UNIFIED-TASK-TRACKER.md);
the narrative plan behind it is [`PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md)._
## North star
A world-class, **developer-ready app platform**: every app manifest-driven,
manifests distributed via a **signed registry**, and third-party developers
publishing through an **external/decentralized marketplace** — all rootless,
secure, robust, and 100%-uptime-capable.
Five pillars every app must satisfy: Quadlet-everywhere · level-triggered
reconciler · lifecycle-bulletproof (full test matrix, repeatedly green) ·
data-driven (no host changes, no per-app binary code) · rootless +
security-first.
## ✅ Shipped
- **Single-node production gate GREEN** (2026-06-23) — the full destructive
lifecycle matrix (install / UI / stop / start / restart / reinstall /
reboot-survive / backend-restart-survive / uninstall) passed 5 consecutive
times with zero failures on real hardware. This was the first exit criterion.
- **Manifest-driven app platform** — 50+ apps as declarative manifests; all
multi-container stacks (BTCPay, Mempool, Immich, NetBird, IndeeHub) install
through the orchestrator; generated-secrets system replaces per-app secret code.
- **Rust orchestrator + level-triggered reconciler** — the bash-script era is
retired; drift self-heals on a 30-second loop.
- **Registry-distributed manifests** — the signed catalog embeds full manifests
per app; nodes verify against the pinned release-root key and overlay
catalog manifests over disk files (catalog wins).
- **Release signing ceremony** (2026-07-02) — release-root Ed25519 key pinned
in the binary; OTA release manifests and the app catalog are signed;
auto-apply refuses unsigned manifests.
- **1.8.0 hardening batches** — supply-chain signature verification, deepened
post-OTA health checks, async-executor and secret-handling fixes, browser
origin checks, dist shrink.
- **Reticulum third mesh transport** — RNS/LXMF over real LoRa hardware
(RNode), interop verified against Sideband; joins Meshtastic and MeshCore
behind one chat UI with X3DH + double-ratchet E2E, attachments, and mesh AI.
- **Bitcoin multi-version** — Core and Knots with per-app version pinning and
safe switching (fleet rollout pending below).
- **Decentralized marketplace backend** — Nostr NIP-78 discovery, DID-signed
manifests, federation-weighted trust scoring, Lightning purchase invoices.
- **Quadlet migration validated** — all backends run as `user.slice` Quadlet
services on the canary node (default flip pending below).
- **Public demo** — multi-visitor sandbox deployed.
## 🔄 In progress
- **Multinode pass** — run the same production gate across the whole test
fleet, plus cross-node federation/mesh suites
([`multinode-testing-plan.md`](multinode-testing-plan.md)). This is the
current exit criterion.
- **Quadlet default flip** — flip the validated Quadlet path from opt-in to
default fleet-wide; eliminates the last container-flapping root cause.
- **Container-flapping elimination** — reconciler churn and failed-unit
self-healing gaps observed on live nodes.
- **Reticulum tail** — ship the daemon binary inside the release tarball;
final fleet redeploys.
## ⏳ Release-blocking for 1.8.0
Tracked in detail in [`1.8.0-RELEASE-HARDENING-PLAN.md`](1.8.0-RELEASE-HARDENING-PLAN.md):
- **OTA upgrade-from-previous-version soak** on real hardware — the top
untested release risk.
- **ISO/image hardening** — per-device TLS/SSH keys on first boot, remove
default credentials and SSH password auth, signed + checksummed ISO,
registries over HTTPS, unattended-upgrades and firewall defaults.
- **Bitcoin multi-version fleet OTA** — code done; rollout timing is a
deliberate hold.
- Version bump to `1.8.0-alpha` + tag once the pre-tag items close.
## 🔭 Planned (post-1.8.0)
- **Developer CLI** — `archy app validate / render / install / test`; the gate
for opening third-party app publishing.
- **External marketplace maturation** — publishing tooling, trust UX, and
reputation surfaces on top of the shipped backend
([`marketplace-protocol.md`](marketplace-protocol.md)).
- **DHT/P2P distribution** — releases and app images over iroh-based swarm
([`dht-distribution-design.md`](dht-distribution-design.md); feature-gated
skeleton exists).
- **P2P encrypted voice/video** over Tor between federated nodes.
- **Dual ecash** — Fedimint + Cashu phases 2–6, networking-sats
([`dual-ecash-design.md`](dual-ecash-design.md)).
- **Paid streaming** — streaming ecash for content
([`phase4-streaming-ecash-plan.md`](phase4-streaming-ecash-plan.md)).
- **Hardware signer** support ([`hardware-signer-design.md`](hardware-signer-design.md)).
- **Code health** — split god-modules, remove dead crates, route all
podman/systemctl calls through the wrapper.
## Release pipeline
Feature Testing (internal) → User Testing (controlled hardware) → Beta Live (public).
docs/SEED-VERIFICATION.md
+443
View File
@@ -0,0 +1,443 @@
# Archipelago Seed Verification
Independently verify that your 24-word BIP-39 mnemonic produces the correct
Nostr keys and DID identifiers — using only standard cryptographic primitives,
no Archipelago code.
```
24-word mnemonic
|
v
PBKDF2-HMAC-SHA512 (2048 rounds, salt = "mnemonic")
|
v
64-byte master seed
|
+-- HKDF-SHA256 (info="archipelago/node/ed25519/v1")
| --> Node Ed25519 keypair --> did:key:z...
|
+-- HKDF-SHA256 (info="archipelago/nostr-node/secp256k1/v1")
| --> Node Nostr key --> npub1...
|
+-- HKDF-SHA256 (info="archipelago/identity/{i}/ed25519/v1")
| --> Identity[i] Ed25519 --> did:key:z...
|
+-- BIP-32 m/44'/1237'/0'/0/{i} (NIP-06)
| --> Identity[i] Nostr key --> npub1...
|
+-- BIP-32 m/84'/0'/0'
| --> Bitcoin HD wallet
|
+-- HKDF-SHA256 (info="archipelago/lnd/entropy/v1")
--> 16 bytes LND aezeed entropy
```
Source: [`core/archipelago/src/seed.rs`](../core/archipelago/src/seed.rs) and
[`core/archipelago/src/identity.rs`](../core/archipelago/src/identity.rs)
---
## Setup
```bash
pip3 install cryptography ecdsa
```
Two packages, both pure crypto, no network calls. Python 3.9+.
---
## The Verification Script
Save as `verify-seed.py` and run with your mnemonic:
```bash
MNEMONIC="word1 word2 ... word24" python3 verify-seed.py
```
```python
#!/usr/bin/env python3
"""
Archipelago seed derivation verifier.
Re-derives every key from a BIP-39 mnemonic using the exact same algorithms
as the Rust backend (seed.rs), so you can compare outputs independently.
Dependencies: cryptography, ecdsa (pip3 install cryptography ecdsa)
No network calls. No file writes. Safe to run air-gapped.
"""
import hashlib, hmac, os, sys
# ── BIP-39: mnemonic --> 64-byte master seed ─────────────────────────────
def mnemonic_to_seed(mnemonic: str) -> bytes:
"""PBKDF2-HMAC-SHA512, 2048 rounds, salt = 'mnemonic', no passphrase."""
return hashlib.pbkdf2_hmac(
"sha512",
mnemonic.encode("utf-8"),
b"mnemonic", # BIP-39 salt prefix + empty passphrase
2048,
)
# ── HKDF-SHA256 (RFC 5869) ──────────────────────────────────────────────
def hkdf_sha256(ikm: bytes, info: bytes, length: int = 32) -> bytes:
"""
HKDF-Extract(salt=None, ikm) then HKDF-Expand(PRK, info, L).
Salt=None means 32 zero bytes per RFC 5869 section 2.2.
Matches: hkdf::Hkdf::<Sha256>::new(None, ikm).expand(info, &mut okm)
"""
# Extract
prk = hmac.new(b"\x00" * 32, ikm, hashlib.sha256).digest()
# Expand (32 bytes = 1 block, only T(1) needed)
t1 = hmac.new(prk, info + b"\x01", hashlib.sha256).digest()
return t1[:length]
# ── Ed25519 ──────────────────────────────────────────────────────────────
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
def ed25519_keypair(secret_32: bytes) -> tuple[bytes, bytes]:
"""Returns (private_32, public_32) from a 32-byte seed."""
sk = Ed25519PrivateKey.from_private_bytes(secret_32)
pk = sk.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
return secret_32, pk
# ── secp256k1 ────────────────────────────────────────────────────────────
from ecdsa import SECP256k1, SigningKey as ECDSASigningKey
def secp256k1_xonly(secret_32: bytes) -> bytes:
"""32-byte x-only pubkey (Schnorr/Nostr format) from private key bytes."""
sk = ECDSASigningKey.from_string(secret_32, curve=SECP256k1)
point = sk.get_verifying_key().pubkey.point
return point.x().to_bytes(32, "big")
# ── BIP-32 HD derivation (secp256k1) ────────────────────────────────────
import struct
SECP256K1_N = SECP256k1.order
def _bip32_master(seed: bytes) -> tuple[bytes, bytes]:
"""BIP-32 master key: HMAC-SHA512(key='Bitcoin seed', data=seed)."""
I = hmac.new(b"Bitcoin seed", seed, hashlib.sha512).digest()
return I[:32], I[32:] # (secret, chain_code)
def _bip32_ckd(key: bytes, chain: bytes, index: int) -> tuple[bytes, bytes]:
"""Child key derivation (private -> private)."""
if index >= 0x80000000:
data = b"\x00" + key + struct.pack(">I", index)
else:
# Compressed pubkey for non-hardened
sk = ECDSASigningKey.from_string(key, curve=SECP256k1)
pt = sk.get_verifying_key().pubkey.point
prefix = b"\x02" if pt.y() % 2 == 0 else b"\x03"
data = prefix + pt.x().to_bytes(32, "big") + struct.pack(">I", index)
I = hmac.new(chain, data, hashlib.sha512).digest()
child = (int.from_bytes(I[:32], "big") + int.from_bytes(key, "big")) % SECP256K1_N
return child.to_bytes(32, "big"), I[32:]
def bip32_derive(seed: bytes, path: str) -> bytes:
"""
Derive private key for a BIP-32 path like 'm/44h/1237h/0h/0/0'.
Matches: bitcoin::bip32::Xpriv::new_master + derive_priv
"""
key, chain = _bip32_master(seed)
for part in path.lstrip("m/").split("/"):
hardened = part.endswith("'") or part.endswith("h")
idx = int(part.rstrip("'h"))
if hardened:
idx += 0x80000000
key, chain = _bip32_ckd(key, chain, idx)
return key
# ── Bech32 encoding (NIP-19: npub / nsec) ───────────────────────────────
_BECH32 = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
def _bech32_polymod(values):
GEN = [0x3B6A57B2, 0x26508E6D, 0x1EA119FA, 0x3D4233DD, 0x2A1462B3]
chk = 1
for v in values:
b = chk >> 25
chk = ((chk & 0x1FFFFFF) << 5) ^ v
for i in range(5):
chk ^= GEN[i] if ((b >> i) & 1) else 0
return chk
def bech32_encode(hrp: str, data: bytes) -> str:
"""Bech32 encode (NIP-19 for npub1.../nsec1...)."""
# Convert 8-bit to 5-bit
acc, bits, vals = 0, 0, []
for byte in data:
acc = (acc << 8) | byte
bits += 8
while bits >= 5:
bits -= 5
vals.append((acc >> bits) & 31)
if bits:
vals.append((acc << (5 - bits)) & 31)
# Checksum
hrp_exp = [ord(c) >> 5 for c in hrp] + [0] + [ord(c) & 31 for c in hrp]
polymod = _bech32_polymod(hrp_exp + vals + [0]*6) ^ 1
checksum = [(polymod >> 5*(5-i)) & 31 for i in range(6)]
return hrp + "1" + "".join(_BECH32[d] for d in vals + checksum)
# ── did:key encoding ────────────────────────────────────────────────────
_B58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
def base58_encode(data: bytes) -> str:
n = int.from_bytes(data, "big")
result = ""
while n > 0:
n, r = divmod(n, 58)
result = _B58[r] + result
for b in data:
if b == 0:
result = "1" + result
else:
break
return result
def to_did_key(ed25519_pub_32: bytes) -> str:
"""did:key:z<base58btc(0xed01 + pubkey)> — W3C did:key method, Ed25519."""
return "did:key:z" + base58_encode(b"\xed\x01" + ed25519_pub_32)
# ── Main ─────────────────────────────────────────────────────────────────
def main():
mnemonic = os.environ.get("MNEMONIC", "").strip()
if not mnemonic:
print("Enter your 24-word mnemonic (space-separated):")
mnemonic = input("> ").strip()
words = mnemonic.split()
if len(words) != 24:
print(f"Error: expected 24 words, got {len(words)}", file=sys.stderr)
sys.exit(1)
seed = mnemonic_to_seed(mnemonic)
W = 72
print()
print("=" * W)
print(" ARCHIPELAGO SEED DERIVATION VERIFICATION")
print("=" * W)
print()
print(f" Seed fingerprint (SHA-256): {hashlib.sha256(seed).hexdigest()[:16]}...")
print(f" Seed length: {len(seed)} bytes")
# ── 1. Node Ed25519 + DID ────────────────────────────────────────────
print()
print("-" * W)
print(" 1. NODE ED25519 KEY")
print(f" HKDF-SHA256(seed, info='archipelago/node/ed25519/v1')")
print("-" * W)
node_ed_priv, node_ed_pub = ed25519_keypair(
hkdf_sha256(seed, b"archipelago/node/ed25519/v1")
)
node_did = to_did_key(node_ed_pub)
print(f" Private: {node_ed_priv.hex()}")
print(f" Public: {node_ed_pub.hex()}")
print(f" did:key: {node_did}")
# ── 2. Node Nostr key ────────────────────────────────────────────────
print()
print("-" * W)
print(" 2. NODE NOSTR KEY")
print(f" HKDF-SHA256(seed, info='archipelago/nostr-node/secp256k1/v1')")
print("-" * W)
node_nostr_priv = hkdf_sha256(seed, b"archipelago/nostr-node/secp256k1/v1")
node_nostr_pub = secp256k1_xonly(node_nostr_priv)
print(f" Private: {node_nostr_priv.hex()}")
print(f" X-only: {node_nostr_pub.hex()}")
print(f" nsec: {bech32_encode('nsec', node_nostr_priv)}")
print(f" npub: {bech32_encode('npub', node_nostr_pub)}")
# ── 3. Identity[0..2] Ed25519 + DID ─────────────────────────────────
print()
print("-" * W)
print(" 3. IDENTITY ED25519 KEYS + DID")
print(f" HKDF-SHA256(seed, info='archipelago/identity/{{i}}/ed25519/v1')")
print("-" * W)
for i in range(3):
info = f"archipelago/identity/{i}/ed25519/v1".encode()
priv, pub = ed25519_keypair(hkdf_sha256(seed, info))
did = to_did_key(pub)
print(f" [{i}] Public: {pub.hex()}")
print(f" did:key: {did}")
# ── 4. Identity[0..2] Nostr (NIP-06 BIP-32) ────────────────────────
print()
print("-" * W)
print(" 4. IDENTITY NOSTR KEYS (NIP-06)")
print(f" BIP-32 m/44'/1237'/0'/0/{{i}}")
print("-" * W)
for i in range(3):
priv = bip32_derive(seed, f"m/44'/1237'/0'/0/{i}")
pub = secp256k1_xonly(priv)
print(f" [{i}] X-only: {pub.hex()}")
print(f" nsec: {bech32_encode('nsec', priv)}")
print(f" npub: {bech32_encode('npub', pub)}")
# ── 5. Bitcoin BIP-84 ───────────────────────────────────────────────
print()
print("-" * W)
print(" 5. BITCOIN WALLET (BIP-84)")
print(f" BIP-32 m/84'/0'/0'")
print("-" * W)
btc_acct = bip32_derive(seed, "m/84'/0'/0'")
btc_pub = secp256k1_xonly(btc_acct)
print(f" Account key: {btc_acct.hex()}")
print(f" Account pub: {btc_pub.hex()}")
# ── 6. LND Entropy ──────────────────────────────────────────────────
print()
print("-" * W)
print(" 6. LND AEZEED ENTROPY")
print(f" HKDF-SHA256(seed, info='archipelago/lnd/entropy/v1') [16 bytes]")
print("-" * W)
lnd = hkdf_sha256(seed, b"archipelago/lnd/entropy/v1", 16)
print(f" Entropy: {lnd.hex()}")
# ── Done ─────────────────────────────────────────────────────────────
print()
print("=" * W)
print(" Compare these values with your Archipelago node:")
print(" UI: Settings > Identity")
print(" SSH: xxd -p /var/lib/archipelago/identity/node_key.pub")
print(" RPC: curl -s http://<ip>/api/rpc \\")
print(" -d '{\"method\":\"identity.get-node\"}' | jq .")
print("=" * W)
print()
if __name__ == "__main__":
main()
```
---
## How to Run
```bash
# Install (two packages, pure crypto, no telemetry)
pip3 install cryptography ecdsa
# Option A: environment variable (doesn't persist in shell history)
read -rs MNEMONIC && export MNEMONIC
# (type or paste your 24 words, press Enter)
python3 verify-seed.py
unset MNEMONIC
# Option B: interactive prompt
python3 verify-seed.py
# Enter your 24-word mnemonic (space-separated):
# > abandon abandon ... art
```
---
## What to Compare
| Output field | Where to find on your node |
|---|---|
| Node Ed25519 public | `xxd -p /var/lib/archipelago/identity/node_key.pub` |
| Node did:key | Settings > Identity > Node DID |
| Node npub | Settings > Identity > Nostr Public Key |
| Identity[0] did:key | Settings > Identity > first identity DID |
| Identity[0] npub | Settings > Identity > first identity Nostr key |
RPC alternative (from any machine on the LAN):
```bash
# Node identity
curl -s http://192.168.1.228/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.get-node"}' | jq .
# All identities
curl -s http://192.168.1.228/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.list"}' | jq .
```
---
## Cryptographic Reference
### HKDF-SHA256 (RFC 5869)
Used for Ed25519 and node-level Nostr keys. Domain separation via unique `info` strings
prevents key reuse across contexts.
```
Extract: PRK = HMAC-SHA256(salt=0x00*32, ikm=64_byte_seed)
Expand: OKM = HMAC-SHA256(PRK, info || 0x01) [first 32 bytes]
```
The Rust backend uses `hkdf::Hkdf::<Sha256>::new(None, ikm)` where `None` salt = 32 zero bytes.
### BIP-32 (secp256k1 HD derivation)
Used for per-identity Nostr keys (NIP-06) and Bitcoin wallet.
```
Master: HMAC-SHA512(key="Bitcoin seed", data=64_byte_seed)
Child: HMAC-SHA512(key=chain_code, data=0x00||key||index) [hardened]
HMAC-SHA512(key=chain_code, data=pubkey||index) [normal]
```
The Rust backend uses the `bitcoin` crate: `Xpriv::new_master()` + `derive_priv()`.
### did:key (W3C)
```
did:key:z + base58btc( 0xED 0x01 || 32_byte_ed25519_pubkey )
```
Multicodec prefix `0xED 0x01` identifies Ed25519 public keys.
The Rust backend uses `bs58::encode()` over a 34-byte buffer.
### NIP-19 Bech32 (npub/nsec)
```
npub1... = bech32(hrp="npub", data=32_byte_x_only_pubkey)
nsec1... = bech32(hrp="nsec", data=32_byte_private_key)
```
X-only pubkey = just the x-coordinate of the secp256k1 point (Schnorr format).
---
## Security
- Run on an air-gapped machine or at minimum a private terminal session
- The script makes zero network calls and writes zero files
- After verification, clean up:
```bash
rm verify-seed.py
unset MNEMONIC
history -c # bash
# or: fc -W /dev/null # zsh
```
- Never paste your mnemonic into a web tool, online REPL, or shared terminal
docs/UNIFIED-TASK-TRACKER.md
+248
View File
@@ -0,0 +1,248 @@
# Unified Task Tracker — OTA 1.8.0 + Master Plan
Single working list for everything left before 1.8.0 ships and the next master-plan
exit criteria (multinode + workstreams B/C/D) are met. Supersedes the open-task
sections of `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md`
as the day-to-day tracker — those docs remain the historical record / detailed
narrative and are still linked from here where useful. **Ordered fastest/simplest
first** so we work top-down instead of hunting across docs.
Verified against actual code state on 2026-07-01 (not just doc text — several
items the source docs still listed as "open" turned out to already be shipped;
those are marked ✅ below with the commit that did it, so we stop re-litigating them).
---
## Tier 0 — Quick / mechanical, no blockers
- [ ] **Update `tests/lifecycle/TESTING.md`'s stale Release Gates checklist** (lines
289–296) — several boxes are unchecked but actually true now:
- #1 bitcoin-stops: covered by `tests/lifecycle/bats/bitcoin-knots.bats` stop/restart
tier, included in the 5/5 green gate run.
- #2 `ARCHY_ITERATIONS=5` on .228: **GREEN 2026-06-23 per CLAUDE.md** — check the box.
- #5 cargo 0 warnings: confirmed 0 warnings on `cargo build --release` (2026-07-01).
- #7 layman changelog: `CHANGELOG.md` is backfilled with layman-readable entries
through v1.8.00-alpha — check the box.
- Leave #3 (multinode), #4 (backend-survives-restart / Phase-3 default-on), #6
(LoC decision), #8 (tag pushed) unchecked — genuinely still open, see Tier 2/3.
- [x] ~~Finish the archival/full-node manifest generalization~~ — investigated 2026-07-01:
the hardcoded fallback names in `dependencies.rs:48-52` (`electrs`, `mempool-electrs`,
`mempool-web`) are legacy **alias** ids for `electrumx`/`mempool`, resolved via
id-mapping in a dozen other places (`install.rs`, `runtime.rs`, `config.rs`, etc.),
not separate un-migrated apps with their own manifests. `electrumx` and `mempool`
themselves already declare `bitcoin:archival`. The fallback is correct as-is —
not tech debt, closing this item rather than risk breaking alias resolution.
- [x] ~~Confirm/close the Portainer image-pin item~~ — confirmed 2026-07-01:
`146.59.87.168:3000/lfg2025/portainer:2.19.4` is present in `podman images` on
all 3 LAN nodes (.116/.198/.228), i.e. actually resolvable/pulled from the mirror.
Not a live bug.
- [x] ~~grafana Quadlet "stuck activating"~~ — checked live on .116 (2026-07-01):
`grafana.service` is `active (running)`, container `Up 2 hours (healthy)`. The
2026-06-21 report is stale for grafana. **strfry still unconfirmed** — not
installed on any of .116/.198/.228 to check directly; low priority until someone
actually needs it installed.
## Tier 1 — Medium effort, unblocked
- [x] ~~immich → Quadlet migration~~ — investigated 2026-07-01, turned out already done:
immich uses the same `install_stack_via_orchestrator` primitive as netbird/btcpay
(`immich_stack_app_ids()` in `stacks.rs:690`), and is confirmed running as real
Quadlet units live on .228 (`immich_server.container`, `immich_postgres.container`,
`immich_redis.container`, all active). Not a legacy in-cgroup app — the only
remaining piece is the fleet-wide Phase-3 default-flip, already tracked in Tier 2.
- [x] ~~Netbird reinstall adoption path~~ — investigated 2026-07-01, **not a bug, by
design.** `adopt_stack_if_exists()` (`stacks.rs:140-198`) is only used as a
fallback when the orchestrator has no manifest for the app — there's nothing to
render certs/config from in that case, so skipping rendering is correct. When
the orchestrator *does* have the manifest (the normal path), the reconcile loop
already re-renders certs even for adopted-running containers, fixed in
`4519dbf0` (`prod_orchestrator.rs:1707-1708`).
- [x] ~~TanStack Query (or equivalent) investigation~~ — spike complete 2026-07-01,
**recommendation: don't adopt / close as not needed.** Only 3 stores actually fetch
data, WebSocket push already handles hot data (server-info/package-data), no
cache-invalidation or stale-data bugs found, migration would touch 62 RPC call
sites for no concrete payoff. If boilerplate ever bothers us, extract a
`usePolling()` composable instead — much cheaper than a query-cache migration.
## Tier 2 — High effort, mostly unblocked (the actual next exit criteria)
- [~] **Multinode test pass** (`docs/multinode-testing-plan.md`) — worked the
preconditions on .198 2026-07-01:
- ✅ cleared 2 stale failed-unit records (`archy-mempool-db.service`,
`meshtastic.service` — both `not-found`/dead since 6 and 5 days ago, harmless
bookkeeping, `systemctl --user reset-failed`).
- ✅ nginx `/app/lnd/` proxy target confirmed correct (→ `18083`, matches the
running `archy-lnd-ui` port) — the plan's "stale proxy target" concern doesn't
apply here.
- ⛔ .198 disk (448GB) is below the 1TB archival threshold + was only 21%
through IBD — user chose to **swap in a different node** rather than wait/add
storage. **.116 ruled out** (no bitcoin container installed at all, just the
UI companion). **.120 ruled out** (reserved for another developer). **.5**
(archy-x250-beta, Tailscale `100.72.136.5`) chosen: also sub-1TB (472GB, so
still pruned — that ceiling is shared by every non-.228 node), but **fully
synced** (`ibd:false`, blocks==headers 956,240). Bootstrapped bats 1.11.1 +
jq 1.7.1 onto it 2026-07-01 and **launched the 5× destructive gate
(`ARCHY_ITERATIONS=5 ARCHY_ALLOW_DESTRUCTIVE=1`) — running now**, log at
`/tmp/gate.log` on .5, background poller watching for the `RESULTS` banner.
- Once .5's gate reports: bring the rest of the fleet to precondition, then the
cross-node federation/mesh/transport suites. This is the literal
"next exit criterion" called out in `CLAUDE.md`.
- [ ] **Phase-3 Quadlet default-flip** — code is validated + opt-in via
`ARCHIPELAGO_USE_QUADLET_BACKENDS=true` on .228/.198 already (confirmed live
2026-07-01). Ready to flip (`config.rs:256` + its test) the moment the .5 gate
reports clean — deliberately NOT staged uncommitted in the tree (a prior attempt
left an uncommitted flip sitting around and that caused confusion; it's a 2-line
change, faster to just do it fresh once confirmed).
- [x] ~~Per-app test coverage for the ~30 apps with zero automated coverage~~ —
**reframed 2026-07-01, mostly a non-issue.** `all-apps-matrix.bats` +
`all-apps-lifecycle.bats` already give EVERY installed app generic baseline
coverage (no stuck state, no error state, stop/start/restart survives, UI
reachable). The real gap is narrower: **34 apps lack app-specific assertions**
(health endpoints, API queryability, data integrity) beyond that baseline —
aiui, bitcoin-core, botfights, core-lightning, did-wallet, fedimint-clientd,
fedimint-gateway, fips-ui, gitea, grafana, home-assistant, indeedhub (+5
sub-containers), jellyfin, lightning-stack, lnd-ui, morphos-server, netbird
(+2 sub-containers), nextcloud, nostr-rs-relay, photoprism, portainer, router,
searxng, strfry, uptime-kuma, vaultwarden. Not urgent — baseline coverage is
real safety net; treat as a backlog "nice to harden further," not a gate item.
- [x] ~~Convert remaining multi-container legacy stacks to the manifest-owned model~~ —
**investigated 2026-07-01, DONE, nothing left.** All 5 real multi-container
stacks (btcpay, mempool, immich, netbird, indeedhub) are on the
`install_stack_via_orchestrator` pattern (`stacks.rs`). saleor was removed from
the codebase; portainer/home-assistant/grafana are single-container
manifest-driven apps, never stacks; fedimint/fedimint-gateway/fedimint-clientd
are 3 separate single-container apps with manifest dependency edges, not a
coordinated stack. Workstream A's stack-migration tail is fully closed.
- [ ] **Container thrashing/flapping + reconciler churn** (added 2026-07-04 — was
implicit across other tracks, now an explicit pre-tag concern). The root cause
of restart-storm flapping is pre-Quadlet architecture: restarting
`archipelago.service` SIGKILLs every container in its cgroup, then the
reconciler rebuilds the world over several minutes (the post-OTA health check
deliberately skips per-app container assertions because of exactly this).
Consolidated lever list, in order of impact:
- **Phase-3 Quadlet default-flip** (tracked above) — removes the SIGKILL-the-world
behavior entirely; the single biggest fix.
- **Workstream F lifecycle items** — immich/grafana uninstall hangs + ghost
containers, grafana reinstall stops, fedimint guardian sync
(`docs/PRODUCTION-MASTER-PLAN.md` workstream F).
- **Reconciler churn observability** — no metric/log today distinguishes "settling
after restart" from "flapping"; add a per-app restart counter + log line when an
app restarts >N times in M minutes so thrash is visible instead of anecdotal.
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
never revived it (it repairs missing/drifted containers but doesn't
`reset-failed`+start failed .services). Same for the indeedhub trio after the
gate run. The health monitor also can't help (container is gone when the unit
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
not user-stopped → reset-failed + start, with backoff.
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
removes executor stalls that made the API janky under reconcile load),
quadlet entrypoint-split false-drift fix (2026-07-08 — `container_command_drifted`
compared entrypoint/cmd halves separately, but quadlet folds `sh -lc` into
`Entrypoint=sh` + `Exec=-lc …`, so every quadlet-created app with a
multi-element entrypoint read as permanently drifted; electrumx on .228
recreated 114×/6h until the comparator was switched to concatenated argv).
- Perf polish riding along: 93 MB frontend dist shrink (hardening plan §D 🟡).
- [ ] **Developer tooling CLI suite** (validate/render/local-install/lifecycle-test) —
APP-PACKAGING-MIGRATION-PLAN.md step 5, needed before external devs can publish.
- [x] ~~**Consolidated deploy 2026-07-01**: merged PR #67 (reticulum daemon
process-group fix, `469b0203`), the UI/UX work (`8256fde1` — mesh/web5/apps
layout, modal, search UX), and `archy-openwrt` (TollGate/OpenWrt gateway
integration — new `core/openwrt` crate, RPC surface, `OpenWrtGateway.vue`)
into `main`, alongside the indeedhub self-heal fix~~ — all merged clean, no
conflicts. **Found + fixed 2 real build-breaking issues during
verification, not caught by whoever authored them**: a vestigial unused
`ref` in `Web5ConnectedNodes.vue` that broke `vue-tsc`, and a stale
`MeshMap.test.ts` mock missing `federatedPositions` (predated this
session's Mesh Map feature) that crashed on mount. Full test suite green
(667 passed) after fixes. **Deployed fleet-wide 2026-07-01, all 5 nodes
sha256-verified**: .116, .198, .228, .5 (recovered cleanly from one
truncated-transfer hiccup, caught via checksum before it hit the live
service), 100.82.34.38 (non-Quadlet node — all containers survived the
restart intact, unlike the worst-case risk flagged beforehand). Also
built an unbundled installer ISO from this same merged source
(`archipelago-installer-1.7.99-alpha-unbundled-x86_64.iso`, 2.4GB) —
the ISO pipeline was archived from the release process at v1.7.43-alpha
(OTA tarballs are now primary) but the wrapper script still works.
- [ ] **⚠️ NOT YET DEPLOYED — start here next session.** After the fleet deploy
above, found that PR #67 ("kill whole daemon process group on drop",
branch `fix/reticulum-daemon-process-group`, head `be50c886`) is a
**different, separate** reticulum-daemon fix from the one already
deployed (`469b0203` on `fix/reticulum-daemon-pdeathsig`) — I'd
conflated the two by topic similarity and only merged/deployed the
Python-level `pdeathsig` fix, missing PR #67's Rust-level
kill-whole-process-group-on-`Drop` fix entirely. Merged PR #67 into
`main` (`7a7fec21`, clean, `cargo check` green, complementary not
conflicting with the already-deployed fix) and separately fixed a real
bug found live: `OpenWrtGateway.vue`'s back button had no `@click`
handler at all (`7d7ba573`, `vue-tsc` clean). **Both committed + pushed
to `main` but genuinely NOT deployed to any node** — user asked to hold
off deploying to restart their computer. Also spot-checked
`openwrt.scan` live on .116: RPC plumbing works, but no physical
OpenWrt router was available to confirm true-positive detection, and
`detect::scan_subnet` does blocking TCP/SSH calls inside an `async fn`
with no `.await` — untested at scale, worth hardening. **Next steps**:
build release binary + frontend from current `main`, deploy to all 5
fleet nodes (.116/.198/.228/.5/100.82.34.38) the same way as the
earlier consolidated deploy, then verify the back button + (if a real
OpenWrt router is available) router detection live.
- [~] **Cross-node federation/mesh/transport suites** — **big find 2026-07-01: these
already exist**, just aren't wired into the gate or documented as existing:
`tests/multinode/smoke.sh` (federation pairing/sync, FIPS anchor, peer content
browse, tombstone-removal regression tests), `tests/multinode/meshtastic.sh`
(8-stage on-air mesh test), harness in `tests/multinode/lib/multinode.bash`.
**Actually ran `smoke.sh` live against .116↔.228 2026-07-01: 14 passed, 1
failed, 1 skipped.** Confirms federation pairing (both directions), FIPS
anchor connectivity (both nodes), and peer-content-browse-over-mesh (the
v1.7.95 fix) all genuinely work node-to-node right now.
- ⚠️ **Real robustness gap found**: `node_rpc()` in `tests/multinode/lib/multinode.bash`
has no `--max-time` on its curl calls — a slow server-side RPC hangs the whole
suite with zero feedback (this is what looked like a hang before it eventually
completed on its own). Cheap fix, not yet applied.
- 🐛 **Real regression found and root-caused**: removing a federation node
(`federation.remove-node`) doesn't reliably stick — B reappeared in A's peer
list after removal in the live test. Root cause: `remove_node()`
(`core/archipelago/src/federation/storage.rs:187`) does
`let _ = tombstone_did(data_dir, did).await` — **silently swallows the
tombstone write's errors.** If that write fails (disk I/O, permission,
transient issue), the peer is removed from `nodes.json` but never actually
tombstoned, so the next background sync/notify-join re-adds it — the
tombstone check at `handlers.rs:592-599` passes because the DID was never
recorded as removed. Diagnosed as a **pre-existing logic gap**, not a fresh
regression from the v1.7.95 fix. **Not fixed yet** — this is federation/trust
code, deliberately not touching it blind; needs a careful fix (surface the
tombstone-write failure instead of swallowing it, and/or retry) plus
re-verification with `smoke.sh` before considering it closed.
## Tier 3 — Blocked on a decision or resource only you can supply
- [x] ~~Version naming decision~~ — **decided 2026-07-08: `1.8.0-alpha`.** Remaining
work is the mechanical bump + tag + push once the pre-tag items above close.
- [x] ~~Workstream B signing ceremony~~ — **done 2026-07-02.** `anchor.rs` pins
`RELEASE_ROOT_PUBKEY_HEX = 5d15cbee…9951` (signer
`did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur`); mnemonic held
offline per `docs/workstream-b-signing-runbook.md`.
- [ ] **Bitcoin multi-version fleet-wide OTA** — `.228` fully working on branch,
per your prior gating this rollout is explicitly held for your decision on
timing (`docs/bitcoin-version-bulletproof-rollout.md`).
- [ ] **3ccc stock-Meshtastic RF validation** — needs a live send/receive test with
physical radios in your hands; code fix is in place, just unverified live.
## Backlog — deferred, no scope decided, low priority
- [ ] **Marketplace protocol (workstream C)** — design-only (`docs/marketplace-protocol.md`),
no tooling/trust UX built. Future work, not urgent.
- [ ] **DHT distribution (workstream D)** — confirmed design-only, no code
(`docs/dht-distribution-design.md` explicitly says "Status: Design (no code yet)");
an experimental iroh provider skeleton exists behind a feature flag for future
PoC measurement, nothing fleet-facing.
- [ ] **Custom live voice-call protocol** — deprioritized 2026-07-01 per user request;
scope not yet decided. Revisit after the tiers above are worked down.
---
*Historical narrative and detailed per-session logs remain in
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md` §6/§8b —
this doc is the live "what's left, in priority order" list. Update it (don't just
append to the old docs) as items close or new ones surface.*
docs/adr/001-podman-over-docker.md
+32
View File
@@ -0,0 +1,32 @@
# ADR-001: Podman Over Docker
**Status**: Accepted
**Date**: 2026-03
## Context
Archipelago needs a container runtime for running applications. Docker and Podman are the two main options.
## Decision
Use Podman as the container runtime instead of Docker.
## Consequences
### Positive
- **Rootless by default**: Containers run without root privileges, reducing attack surface
- **Daemonless**: No persistent daemon process; containers are managed as individual processes under systemd
- **Docker-compatible**: Supports Docker images and most Docker CLI commands
- **Systemd integration**: Podman containers can be managed as systemd services natively
- **No vendor lock-in**: OCI-compliant, works with any container registry
### Negative
- **Smaller ecosystem**: Some Docker-specific tools and compose features require adaptation
- **Docker Compose differences**: Podman Compose exists but has occasional compatibility gaps
- **Documentation**: Most container documentation assumes Docker; developers need to translate
- **Networking**: Podman networking (CNI/netavark) differs from Docker's bridge networking
### Mitigation
- Use `podman` CLI wrapper that provides Docker-compatible interface
- Document Podman-specific commands in developer guide
- Use `archy-net` custom network for inter-container DNS
docs/adr/002-did-key-method.md
+31
View File
@@ -0,0 +1,31 @@
# ADR-002: DID Key Method for Node Identity
**Status**: Accepted
**Date**: 2026-03
## Context
Each Archipelago node needs a cryptographic identity for peer authentication, federation, and verifiable credentials. Multiple DID methods exist (did:web, did:ion, did:key, did:peer).
## Decision
Use `did:key` as the primary DID method.
## Consequences
### Positive
- **Self-contained**: The DID document is derived entirely from the public key — no external resolution needed
- **Offline-capable**: Works without internet, aligning with sovereignty principles
- **Simple**: No registration, no blockchain, no web server required
- **Fast**: DID resolution is a local computation, not a network request
- **Ed25519**: Uses Ed25519 keys which are fast, compact, and well-supported
### Negative
- **No key rotation**: The DID is bound to a single key; rotating requires a new DID
- **No service endpoints in DID**: Cannot embed service URLs in the DID document itself
- **No revocation**: Cannot revoke a did:key without out-of-band mechanisms
### Mitigation
- Use federation trust lists for key management and revocation
- Store service endpoints (onion address, pubkey) separately in federation state
- Support migration to did:peer or did:web in future versions if key rotation is needed
docs/adr/003-nostr-for-discovery.md
+35
View File
@@ -0,0 +1,35 @@
# ADR-003: Nostr Relays for Node and App Discovery
**Status**: Accepted
**Date**: 2026-03
## Context
Archipelago nodes need to discover peers and community apps without a central registry. Options: custom P2P protocol, DHT, BitTorrent tracker, Nostr relays, IPFS.
## Decision
Use Nostr relays (NIP-78, kind 30078) for both node discovery and marketplace app manifests.
## Consequences
### Positive
- **Decentralized**: Multiple independent relays; no single point of failure
- **Existing infrastructure**: Thousands of Nostr relays already running globally
- **Censorship-resistant**: If one relay censors, others still serve events
- **Simple protocol**: WebSocket + JSON — easy to implement without heavy dependencies
- **Key management**: Nostr uses secp256k1, same curve as Bitcoin — natural fit
- **NIP-33 replaceable events**: Latest event replaces previous — clean update model
- **Tor-compatible**: WebSocket over Tor SOCKS proxy works natively
### Negative
- **Relay availability varies**: Some relays may be down or rate-limited
- **No guaranteed persistence**: Relays may prune old events
- **Spam potential**: Open publishing means anyone can publish junk manifests
- **Latency**: Querying multiple relays adds latency to discovery
### Mitigation
- Query multiple relays in parallel; deduplicate results
- Cache results locally with 15-minute TTL
- Use trust scoring to rank manifests (DID verification, relay consensus, federation trust)
- Use hashtag filtering (`archipelago-marketplace`) to narrow queries
docs/adr/004-tor-for-peer-communication.md
+35
View File
@@ -0,0 +1,35 @@
# ADR-004: Tor Hidden Services for Peer Communication
**Status**: Accepted
**Date**: 2026-03
## Context
Federated nodes need to communicate directly for state sync, app deployment, and peer verification. Options: direct IP, VPN tunnel, Tor hidden services, I2P.
## Decision
Use Tor hidden services (.onion addresses) for all inter-node communication.
## Consequences
### Positive
- **NAT traversal**: Works behind any firewall or NAT without port forwarding
- **IP privacy**: Nodes never expose their real IP addresses to each other
- **End-to-end encryption**: Tor provides encryption without additional TLS setup
- **Censorship resistance**: Onion routing makes traffic analysis difficult
- **Stable addressing**: .onion addresses persist across IP changes and network migrations
- **No central infrastructure**: No VPN server, STUN/TURN server, or relay needed
### Negative
- **Latency**: Tor adds 200-500ms per hop; 3 hops per direction = noticeable delay
- **Bandwidth**: Tor network has limited bandwidth; not suitable for bulk data transfer
- **Reliability**: Tor circuits can break; connections may need retry logic
- **Setup complexity**: Requires running a Tor daemon (`archy-tor` container)
- **Blocked networks**: Some networks block Tor; bridges can help but add complexity
### Mitigation
- Use Tor only for RPC/control plane; bulk data (container images) pulled from registries
- Implement retry with backoff for Tor connections
- Container `archy-tor` runs automatically with host networking for hidden service access
- Federation sync interval (5 min) tolerates occasional connection failures
docs/adr/005-chacha20-backup-encryption.md
+32
View File
@@ -0,0 +1,32 @@
# ADR-005: ChaCha20-Poly1305 for Backup Encryption
**Status**: Accepted
**Date**: 2026-03
## Context
Backups contain sensitive data (keys, credentials, app state) and must be encrypted at rest. Options: AES-256-GCM, ChaCha20-Poly1305, XChaCha20-Poly1305.
## Decision
Use ChaCha20-Poly1305 (AEAD) with Argon2id key derivation for backup encryption.
## Consequences
### Positive
- **Software performance**: ChaCha20 is faster than AES on hardware without AES-NI (common on ARM/SBCs)
- **Constant-time**: No timing side channels, unlike some AES implementations
- **AEAD**: Authenticated encryption ensures both confidentiality and integrity
- **Widely audited**: Used in TLS 1.3, WireGuard, and Signal Protocol
- **Simple implementation**: No padding, no CBC/CTR mode complexity
- **Argon2id KDF**: Memory-hard key derivation resists GPU/ASIC brute force attacks
### Negative
- **96-bit nonce**: Must ensure nonce uniqueness per encryption (random generation with collision check)
- **Not FIPS-certified**: Some enterprise environments require AES (not relevant for personal nodes)
- **Less hardware acceleration**: AES-NI on x86 can make AES faster on desktop CPUs
### Mitigation
- Generate random nonce per backup; store nonce alongside ciphertext
- Argon2id with high memory cost (64MB) and iterations (3) for password-to-key derivation
- Target hardware is mixed x86/ARM; ChaCha20's consistent performance is an advantage
docs/adr/006-nostr-marketplace-discovery.md
+57
View File
@@ -0,0 +1,57 @@
# ADR-006: Nostr Relays for Marketplace Discovery
## Status
Accepted
## Context
Archipelago needs a mechanism for users to discover and install third-party applications. The traditional approach is a centralized app store (like Apple App Store, Google Play, or Umbrel's marketplace). However, a centralized store introduces:
- A single point of failure and censorship
- A trust dependency on the store operator
- Barriers to entry for app developers (gatekeeping)
- Privacy concerns (the store operator knows what every user installs)
As a sovereign computing platform, Archipelago should align with decentralized principles.
## Decision
Use **Nostr relays** (NIP-78 application-specific data, kind 30078 events) for decentralized app manifest discovery instead of a centralized marketplace server.
### How It Works
1. **App developers** publish signed manifests as Nostr events to public relays
2. **Archipelago nodes** query multiple relays for available app manifests
3. **Trust scoring** uses verification count across relays, developer reputation (DID-linked), and optional community endorsements
4. **Users** see a merged, deduplicated list of available apps with trust indicators
### Trust Tiers
- **Verified**: Published by known developers, seen on 3+ relays, DID-verified
- **Community**: Seen on 2+ relays, valid manifest, unsigned or new developer
- **Unverified**: Single relay, new developer, use at own risk
## Consequences
### Positive
- No single point of failure — apps remain discoverable even if relays go offline
- No gatekeeping — any developer can publish apps
- Privacy-preserving — no central server tracking installs
- Censorship-resistant — apps can't be removed by a single entity
- Aligns with Nostr ecosystem already used for node identity
### Negative
- Discovery can be slower (querying multiple relays)
- Quality control relies on trust scoring rather than human curation
- Spam/malicious manifests require robust filtering
- Users need to understand trust tiers (not a simple "everything is safe" model)
### Mitigations
- Cache relay responses locally for fast subsequent loads
- Built-in curated app list for essential apps (Bitcoin, LND, etc.)
- Container security model (readonly_root, capability dropping) limits damage from malicious apps
- Manifest signature verification before installation
docs/adr/007-did-federation-trust.md
+54
View File
@@ -0,0 +1,54 @@
# ADR-007: DID-Based Federation Trust
## Status
Accepted
## Context
Archipelago supports federation — multiple nodes forming a trusted group for remote monitoring, app deployment, and state synchronization. Federation requires a trust establishment mechanism:
- **Centralized PKI** (Certificate Authorities): requires internet access, introduces third-party trust
- **Pre-shared keys**: simple but doesn't scale, no identity verification
- **DID-based bilateral verification**: each node verifies the other's cryptographic identity directly
## Decision
Use **bilateral DID-based verification** with single-use invite codes for federation trust establishment.
### How It Works
1. **Node A** generates a single-use invite code containing its DID, .onion address, and a shared secret
2. **Node B** receives the code (out-of-band: QR code, message, etc.) and submits it
3. **Both nodes** verify each other's DIDs by exchanging signed challenges over Tor
4. **Trust is established** — each node stores the other's DID and public key
5. **Ongoing communication** uses DID-authenticated messages over Tor hidden services
### Trust Levels
- **Trusted**: Full access — can view status, deploy apps, sync state
- **Observer**: Read-only access — can view status but not modify
- **Untrusted**: Blocked from federation operations
## Consequences
### Positive
- No third-party trust dependency (no CA, no central server)
- Works fully offline/air-gapped for the verification step
- Strong cryptographic identity (Ed25519 keys)
- Granular trust levels for different access patterns
- Invite codes are single-use (no replay attacks)
### Negative
- Requires out-of-band code exchange (can't auto-discover peers for federation)
- No revocation mechanism beyond removing the peer from the local trust store
- Key rotation requires re-establishing trust with all peers
- Trust is bilateral — each node maintains its own trust decisions
### Mitigations
- Nostr-based node discovery (ADR-003) handles finding nodes; federation handles trusting them
- Tor hidden services provide transport encryption and anonymity
- State sync includes heartbeat/health checks to detect unreachable peers
docs/adr/008-dual-key-strategy.md
+62
View File
@@ -0,0 +1,62 @@
# ADR-008: Dual Key Strategy (Ed25519 + Secp256k1)
## Status
Accepted
## Context
Archipelago operates at the intersection of two cryptographic ecosystems:
- **Web5 / DIDs**: The W3C DID specification and Verifiable Credentials ecosystem predominantly uses **Ed25519** (EdDSA) for digital signatures
- **Nostr / Bitcoin**: The Nostr protocol and Bitcoin ecosystem use **secp256k1** (ECDSA/Schnorr) for signatures
A single key type cannot serve both ecosystems without conversion layers or compatibility issues.
## Decision
Maintain **two key pairs per node identity**:
1. **Ed25519** — Primary identity key for DID documents, verifiable credentials, federation authentication, and backup encryption
2. **Secp256k1** — Nostr-compatible key for relay publishing, node discovery, and Lightning Network interactions
### Key Derivation
- Both keys are derived from the same master seed during node initialization
- The Ed25519 key is the canonical identity (stored in the DID document)
- The secp256k1 key is linked to the DID via the Nostr profile (NIP-05 verification)
### Usage Matrix
| Operation | Key Used |
|-----------|----------|
| DID document signing | Ed25519 |
| Verifiable credentials | Ed25519 |
| Federation auth | Ed25519 |
| Backup encryption | Ed25519 (via X25519 DH) |
| Nostr event publishing | secp256k1 |
| Node discovery | secp256k1 (Nostr) |
| Lightning channel auth | secp256k1 |
## Consequences
### Positive
- Full compatibility with both Web5 and Nostr ecosystems
- No conversion layers or compatibility hacks needed
- Each key type is used in its native context (maximum security)
- Both keys from same seed — single backup protects both
- Future-proof: can add new key types without breaking existing ones
### Negative
- Two keys to manage instead of one
- Users need to understand which pubkey is which (mitigated by UI)
- Key rotation must update both key types
- Slightly larger DID documents (two verification methods)
### Mitigations
- UI presents a unified identity view — users see "My Identity" not "My Ed25519 Key"
- Backup system captures the master seed, from which both keys derive
- DID document includes both verification methods with clear purpose labels
docs/adr/009-manifest-container-security.md
+77
View File
@@ -0,0 +1,77 @@
# ADR-009: Manifest-Level Container Security Enforcement
## Status
Accepted
## Context
Archipelago runs third-party applications as containers. Without enforcement, containers could:
- Run as root and escalate privileges
- Access the host filesystem
- Modify their own binaries (persistence of malicious code)
- Acquire unnecessary Linux capabilities
- Use unverified or tampered container images
Other node OS projects (Umbrel, Start9) vary in their security enforcement. Archipelago targets a higher security bar suitable for handling Bitcoin private keys and personal data.
## Decision
Enforce security constraints at the **manifest level**, applied automatically during container creation. Every container MUST comply with these non-negotiable defaults:
### Mandatory Security Defaults
| Constraint | Value | Rationale |
|-----------|-------|-----------|
| `readonly_root` | `true` | Prevents runtime filesystem modification (anti-persistence) |
| `no_new_privileges` | `true` | Prevents privilege escalation via setuid/setgid |
| `user` | UID > 1000 | Never run as root |
| `capabilities` | Drop ALL, add only required | Principle of least privilege |
| `image_tag` | Pinned version | No `latest` tags — reproducible deploys |
| `seccomp_profile` | Default | Blocks dangerous syscalls |
### Manifest Enforcement
The `core/container/` module validates manifests before container creation:
1. **Parse** the YAML manifest
2. **Validate** all required security fields are present
3. **Reject** manifests that violate mandatory defaults (e.g., `readonly_root: false` without explicit override)
4. **Apply** security context during `podman create`
### Optional Overrides
Some apps legitimately need elevated privileges:
- `readonly_root: false` — Only for apps that must write to their root filesystem (documented reason required)
- Additional capabilities (e.g., `NET_ADMIN` for VPN apps) — must be explicitly listed and justified
## Consequences
### Positive
- Defense in depth — even if a container image is compromised, damage is limited
- Consistent security posture across all apps
- Transparent — users can inspect any app's security manifest
- Aligns with industry best practices (CIS Benchmarks, NIST)
### Negative
- Some apps may not work without modifications (e.g., apps expecting root)
- Read-only root requires explicit volume mounts for writable directories
- Developers must understand and comply with the security model
- Slightly more complex manifest format than competitors
### Mitigations
- Clear documentation in `docs/app-manifest-spec.md`
- Example manifests for common app patterns
- Build-time validation catches issues before deployment
- Override mechanism for legitimate exceptions (with audit trail)
## References
- `docs/app-manifest-spec.md` — Full manifest specification
- `core/container/src/` — Container security implementation
- `core/security/src/` — AppArmor profiles and secrets management
docs/adr/011-dwn-deprioritization.md
+31
View File
@@ -0,0 +1,31 @@
# ADR-011: DWN Deprioritization
## Status
Accepted
## Context
TBD/Block shut down in November 2024, donating Web5 code to the Decentralized Identity Foundation (DIF). The DWN (Decentralized Web Node) specification was heavily backed by TBD — without their engineering team, the spec has lost momentum:
- No maintained Rust DWN SDK exists (the `dwn` crate by unavi-xyz is v0.4.0 with 323 downloads)
- TBD's reference implementation was TypeScript-only
- DIF has not allocated resources to continue DWN development
- The spec itself is complex (personal data stores with protocol-based access control)
Meanwhile, Archipelago's federation over Tor + Nostr relays already serves the core peer data sync use case that DWN was intended for.
## Decision
1. **Keep existing DWN store code** in `core/archipelago/src/network/dwn_store.rs` — it works for peer file catalogs and federation state
2. **Stop calling it "Web5 DWN"** in user-facing text — it's our custom implementation, not a full DWN spec implementation
3. **Do not invest in DWN spec compliance** — the spec is stalled and may not stabilize
4. **Prioritize Nostr + federation** for peer discovery and data exchange
5. **Re-evaluate if DIF produces a viable Rust SDK** or the spec gains new maintainers
## Consequences
- DWN functionality remains available but is not actively developed
- Peer sync uses federation + Nostr instead of DWN protocols
- Reduces maintenance burden — no need to track a stalled spec
- If DWN resurfaces with strong tooling, we can adopt it later
docs/api-reference.md
+397
View File
@@ -0,0 +1,397 @@
# Archipelago API Reference
All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`.
**Request format:**
```json
{
"method": "namespace.action",
"params": { ... }
}
```
**Response format:**
```json
{
"result": { ... }
}
```
**Error format:**
```json
{
"error": { "message": "Error description" }
}
```
**Authentication:** All endpoints require a valid session cookie (`archipelago_session`) except those marked "No Auth".
---
## Authentication
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `auth.login` | `{ password: string }` | `{ ok: bool, totp_required?: bool }` | No Auth |
| `auth.logout` | — | `{ ok: bool }` | Yes |
| `auth.changePassword` | `{ current: string, new: string }` | `{ ok: bool }` | Yes |
| `auth.isOnboardingComplete` | — | `{ complete: bool }` | No Auth |
| `auth.onboardingComplete` | — | `{ ok: bool }` | Yes |
| `auth.resetOnboarding` | — | `{ ok: bool }` | Yes |
### TOTP 2FA
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `auth.totp.setup.begin` | `{ password: string }` | `{ secret: string, qr_uri: string, backup_codes: string[] }` | Yes |
| `auth.totp.setup.confirm` | `{ code: string }` | `{ ok: bool }` | Yes |
| `auth.totp.disable` | `{ password: string }` | `{ ok: bool }` | Yes |
| `auth.totp.status` | — | `{ enabled: bool }` | Yes |
| `auth.login.totp` | `{ code: string }` | `{ ok: bool }` | No Auth |
| `auth.login.backup` | `{ code: string }` | `{ ok: bool }` | No Auth |
---
## Container Orchestration
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `container-install` | `{ image: string, name?: string }` | `{ ok: bool, container_id: string }` | Yes |
| `container-start` | `{ id: string }` | `{ ok: bool }` | Yes |
| `container-stop` | `{ id: string }` | `{ ok: bool }` | Yes |
| `container-remove` | `{ id: string }` | `{ ok: bool }` | Yes |
| `container-list` | — | `{ containers: Container[] }` | Yes |
| `container-status` | `{ id: string }` | `{ status: string, ... }` | Yes |
| `container-logs` | `{ id: string, lines?: number }` | `{ logs: string }` | Yes |
| `container-health` | `{ id: string }` | `{ healthy: bool }` | Yes |
## Package Management
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `package.install` | `{ id: string, dockerImage?: string, url?: string, version?: string }` | `{ ok: bool }` | Yes |
| `package.start` | `{ id: string }` | `{ ok: bool }` | Yes |
| `package.stop` | `{ id: string }` | `{ ok: bool }` | Yes |
| `package.restart` | `{ id: string }` | `{ ok: bool }` | Yes |
| `package.uninstall` | `{ id: string }` | `{ ok: bool }` | Yes |
## Bundled Apps
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `bundled-app-start` | `{ id: string }` | `{ ok: bool }` | Yes |
| `bundled-app-stop` | `{ id: string }` | `{ ok: bool }` | Yes |
---
## Node Identity & P2P
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `node.did` | — | `{ did: string }` | Yes |
| `node.signChallenge` | `{ challenge: string }` | `{ signature: string }` | Yes |
| `node.tor-address` | — | `{ address: string }` | Yes |
| `node.nostr-publish` | — | `{ ok: bool, event_id: string }` | Yes |
| `node.nostr-pubkey` | — | `{ pubkey: string }` | Yes |
| `node-nostr-verify-revoked` | — | `{ revoked: bool, nostr_pubkey: string }` | Yes |
| `node-nostr-discover` | — | `{ nodes: DiscoveredNode[] }` | Yes |
| `node-add-peer` | `{ did: string, address: string }` | `{ ok: bool }` | Yes |
| `node-list-peers` | — | `{ peers: Peer[] }` | Yes |
| `node-remove-peer` | `{ did: string }` | `{ ok: bool }` | Yes |
| `node-send-message` | `{ to: string, message: string }` | `{ ok: bool }` | Yes |
| `node-check-peer` | `{ did: string }` | `{ online: bool }` | Yes |
| `node-messages-received` | — | `{ messages: Message[] }` | Yes |
| `node.createBackup` | `{ password: string }` | `{ path: string }` | Yes |
---
## Identity Management
### Multi-Identity
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `identity.list` | `{}` | `{ identities: Identity[] }` | Yes |
| `identity.create` | `{ label: string }` | `{ identity: Identity }` | Yes |
| `identity.get` | `{ id: string }` | `{ identity: Identity }` | Yes |
| `identity.delete` | `{ id: string }` | `{ ok: bool }` | Yes |
| `identity.set-default` | `{ id: string }` | `{ ok: bool }` | Yes |
| `identity.sign` | `{ id: string, data: string }` | `{ signature: string }` | Yes |
| `identity.verify` | `{ id: string, data: string, signature: string }` | `{ valid: bool }` | Yes |
| `identity.resolve-did` | `{ did: string }` | `{ document: DIDDocument }` | Yes |
| `identity.resolve-remote-did` | `{ did: string }` | `{ document: DIDDocument }` | Yes |
| `identity.verify-did-document` | `{ document: object }` | `{ valid: bool }` | Yes |
### Nostr Keys
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `identity.create-nostr-key` | `{ id: string }` | `{ pubkey: string, npub: string }` | Yes |
| `identity.nostr-sign` | `{ id: string, event: object }` | `{ signed_event: object }` | Yes |
### Bitcoin Names (NIP-05)
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `identity.register-name` | `{ name: string, pubkey: string }` | `{ ok: bool }` | Yes |
| `identity.remove-name` | `{ name: string }` | `{ ok: bool }` | Yes |
| `identity.resolve-name` | `{ name: string }` | `{ pubkey: string }` | Yes |
| `identity.list-names` | `{}` | `{ names: NameEntry[] }` | Yes |
| `identity.link-name` | `{ name: string, identity_id: string }` | `{ ok: bool }` | Yes |
### Verifiable Credentials
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `identity.issue-credential` | `{ subject: string, claims: object }` | `{ credential: VC }` | Yes |
| `identity.verify-credential` | `{ credential: object }` | `{ valid: bool }` | Yes |
| `identity.list-credentials` | `{ id?: string }` | `{ credentials: VC[] }` | Yes |
| `identity.revoke-credential` | `{ credential_id: string }` | `{ ok: bool }` | Yes |
| `identity.create-presentation` | `{ credentials: string[] }` | `{ presentation: VP }` | Yes |
| `identity.verify-presentation` | `{ presentation: object }` | `{ valid: bool }` | Yes |
---
## Bitcoin & Lightning
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `bitcoin.getinfo` | — | `{ blocks: number, connections: number, ... }` | Yes |
| `lnd.getinfo` | — | `{ identity_pubkey: string, num_active_channels: number, ... }` | Yes |
| `lnd.listchannels` | — | `{ channels: Channel[] }` | Yes |
| `lnd.openchannel` | `{ pubkey: string, amount: number }` | `{ funding_txid: string }` | Yes |
| `lnd.closechannel` | `{ channel_point: string }` | `{ closing_txid: string }` | Yes |
| `lnd.newaddress` | — | `{ address: string }` | Yes |
| `lnd.sendcoins` | `{ addr: string, amount: number }` | `{ txid: string }` | Yes |
| `lnd.createinvoice` | `{ amount: number, memo?: string }` | `{ payment_request: string }` | Yes |
| `lnd.payinvoice` | `{ payment_request: string }` | `{ preimage: string }` | Yes |
| `lnd.create-psbt` | `{ outputs: object, ... }` | `{ psbt: string }` | Yes |
| `lnd.finalize-psbt` | `{ psbt: string }` | `{ signed_psbt: string }` | Yes |
---
## Ecash Wallet
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `wallet.ecash-balance` | — | `{ balance: number, mint_url: string }` | Yes |
| `wallet.ecash-mint` | `{ amount: number }` | `{ ok: bool }` | Yes |
| `wallet.ecash-melt` | `{ amount: number, invoice: string }` | `{ ok: bool }` | Yes |
| `wallet.ecash-send` | `{ amount: number }` | `{ token: string }` | Yes |
| `wallet.ecash-receive` | `{ token: string }` | `{ amount: number }` | Yes |
| `wallet.ecash-history` | — | `{ transactions: EcashTx[] }` | Yes |
| `wallet.networking-profits` | — | `{ total_sats: number, ... }` | Yes |
---
## Network
### Interfaces & WiFi
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `network.list-interfaces` | — | `{ interfaces: Interface[] }` | Yes |
| `network.scan-wifi` | — | `{ networks: WifiNetwork[] }` | Yes |
| `network.configure-wifi` | `{ ssid: string, password: string }` | `{ ok: bool }` | Yes |
| `network.configure-ethernet` | `{ interface: string, mode: "dhcp"\|"static", ip?: string, gateway?: string, dns?: string }` | `{ ok: bool }` | Yes |
| `network.diagnostics` | — | `{ wan_ip: string, nat_type: string, upnp_available: bool, tor_connected: bool, wifi_count: number }` | Yes |
### DNS
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `network.dns-status` | — | `{ provider: string, servers: string[], doh_enabled: bool, doh_url: string?, resolv_conf_servers: string[] }` | Yes |
| `network.configure-dns` | `{ provider: "system"\|"cloudflare"\|"google"\|"quad9"\|"mullvad"\|"custom", servers?: string[] }` | `{ ok: bool, provider: string, servers: string[], doh_enabled: bool }` | Yes |
### Network Overlay
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `network.get-visibility` | — | `{ visibility: string }` | Yes |
| `network.set-visibility` | `{ visibility: string }` | `{ ok: bool }` | Yes |
| `network.request-connection` | `{ target_did: string }` | `{ request_id: string }` | Yes |
| `network.list-requests` | — | `{ requests: ConnectionRequest[] }` | Yes |
| `network.accept-request` | `{ request_id: string }` | `{ ok: bool }` | Yes |
| `network.reject-request` | `{ request_id: string }` | `{ ok: bool }` | Yes |
### Router / UPnP
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `router.discover` | — | `{ router: RouterInfo }` | Yes |
| `router.list-forwards` | — | `{ forwards: PortForward[] }` | Yes |
| `router.add-forward` | `{ port: number, protocol: string, description: string }` | `{ ok: bool }` | Yes |
| `router.remove-forward` | `{ port: number, protocol: string }` | `{ ok: bool }` | Yes |
| `router.detect` | `{ ... }` | `{ detected: bool, ... }` | Yes |
| `router.info` | — | `{ ... }` | Yes |
| `router.configure` | `{ ... }` | `{ ok: bool }` | Yes |
---
## Tor
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `tor.list-services` | — | `{ services: TorService[] }` | Yes |
| `tor.create-service` | `{ name: string, port: number }` | `{ onion_address: string }` | Yes |
| `tor.delete-service` | `{ name: string }` | `{ ok: bool }` | Yes |
| `tor.get-onion-address` | `{ name: string }` | `{ address: string }` | Yes |
## Nostr Relays
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `nostr.list-relays` | — | `{ relays: RelayConfig[] }` | Yes |
| `nostr.add-relay` | `{ url: string }` | `{ ok: bool }` | Yes |
| `nostr.remove-relay` | `{ url: string }` | `{ ok: bool }` | Yes |
| `nostr.toggle-relay` | `{ url: string }` | `{ ok: bool }` | Yes |
| `nostr.get-stats` | — | `{ total_relays: number, connected: number, enabled: number }` | Yes |
---
## VPN
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `vpn.status` | — | `{ connected: bool, provider?: string, ip_address?: string, hostname?: string, peers_connected: number }` | Yes |
| `vpn.configure` | `{ provider: "tailscale"\|"wireguard", auth_key?: string, address?: string, dns?: string, peer?: object }` | `{ ok: bool }` | Yes |
| `vpn.disconnect` | — | `{ disconnected: bool }` | Yes |
## Mesh Networking
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `mesh.status` | — | `{ enabled: bool, device: string?, nodes: MeshNode[] }` | Yes |
| `mesh.discover` | `{ timeout_secs?: number }` | `{ nodes: MeshNode[] }` | Yes |
| `mesh.broadcast` | — | `{ ok: bool }` | Yes |
| `mesh.configure` | `{ enabled: bool, device?: string }` | `{ ok: bool }` | Yes |
---
## Federation
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `federation.invite` | — | `{ code: string }` | Yes |
| `federation.join` | `{ code: string }` | `{ ok: bool, node: FederatedNode }` | Yes |
| `federation.list-nodes` | — | `{ nodes: FederatedNode[] }` | Yes |
| `federation.remove-node` | `{ did: string }` | `{ ok: bool }` | Yes |
| `federation.set-trust` | `{ did: string, trust: "trusted"\|"observer"\|"untrusted" }` | `{ ok: bool }` | Yes |
| `federation.sync-state` | — | `{ results: SyncResult[] }` | Yes |
| `federation.get-state` | — | `{ state: NodeStateSnapshot }` | Federation peer |
| `federation.peer-joined` | `{ did: string, onion: string, pubkey: string }` | `{ accepted: bool }` | Federation peer |
| `federation.deploy-app` | `{ target_did: string, app_id: string, version?: string }` | `{ ok: bool }` | Yes |
---
## Marketplace
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `marketplace.discover` | — | `{ apps: DiscoveredApp[], relay_count: number }` | Yes |
| `marketplace.publish` | `{ app_id, name, version, description, author, container, category, ... }` | `{ ok: bool, event_id: string }` | Yes |
| `marketplace.get-manifest` | `{ app_id: string }` | `DiscoveredApp \| { error: string }` | Yes |
| `marketplace.list-published` | — | `{ manifests: AppManifest[] }` | Yes |
| `marketplace.verify` | `{ ... manifest fields ... }` | `{ valid: bool, issues: string[], trust_score: number }` | Yes |
---
## DWN (Decentralized Web Node)
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `dwn.status` | — | `{ running: bool, message_count: number, protocol_count: number }` | Yes |
| `dwn.sync` | — | `{ synced: number }` | Yes |
| `dwn.register-protocol` | `{ uri: string, definition: object }` | `{ ok: bool }` | Yes |
| `dwn.list-protocols` | — | `{ protocols: Protocol[] }` | Yes |
| `dwn.remove-protocol` | `{ uri: string }` | `{ ok: bool }` | Yes |
| `dwn.query-messages` | `{ protocol?: string, limit?: number }` | `{ messages: DwnMessage[] }` | Yes |
| `dwn.write-message` | `{ protocol: string, data: object }` | `{ ok: bool, message_id: string }` | Yes |
---
## Content Catalog
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `content.list-mine` | — | `{ items: ContentItem[] }` | Yes |
| `content.add` | `{ title: string, type: string, data: object }` | `{ ok: bool, id: string }` | Yes |
| `content.remove` | `{ id: string }` | `{ ok: bool }` | Yes |
| `content.set-pricing` | `{ id: string, price_sats: number }` | `{ ok: bool }` | Yes |
| `content.set-availability` | `{ id: string, available: bool }` | `{ ok: bool }` | Yes |
| `content.browse-peer` | `{ peer_did: string }` | `{ items: ContentItem[] }` | Yes |
---
## System
### Monitoring
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `system.stats` | — | `{ cpu_percent: number, ram_used: number, ram_total: number, disk_used: number, disk_total: number, uptime_secs: number, load_avg: number[] }` | Yes |
| `system.processes` | — | `{ processes: Process[] }` | Yes |
| `system.temperature` | — | `{ celsius: number? }` | Yes |
| `system.detect-usb-devices` | — | `{ devices: UsbDevice[] }` | Yes |
### Updates
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `update.check` | — | `{ available: bool, version?: string, changelog?: string }` | Yes |
| `update.status` | — | `{ state: string, progress?: number }` | Yes |
| `update.dismiss` | — | `{ ok: bool }` | Yes |
| `update.download` | — | `{ ok: bool }` | Yes |
| `update.apply` | — | `{ ok: bool }` | Yes |
| `update.rollback` | — | `{ ok: bool }` | Yes |
| `update.get-schedule` | — | `{ auto_check: bool, auto_install: bool, schedule: string }` | Yes |
| `update.set-schedule` | `{ auto_check?: bool, auto_install?: bool, schedule?: string }` | `{ ok: bool }` | Yes |
### Backup & Restore
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `backup.create` | `{ password: string, include?: string[] }` | `{ path: string, size: number }` | Yes |
| `backup.list` | — | `{ backups: BackupEntry[] }` | Yes |
| `backup.verify` | `{ path: string, password: string }` | `{ valid: bool }` | Yes |
| `backup.restore` | `{ path: string, password: string }` | `{ ok: bool }` | Yes |
| `backup.delete` | `{ path: string }` | `{ ok: bool }` | Yes |
| `backup.list-drives` | — | `{ drives: UsbDrive[] }` | Yes |
| `backup.to-usb` | `{ drive: string, password: string }` | `{ ok: bool }` | Yes |
### Security
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `security.rotate-secrets` | `{ app_id?: string }` | `{ rotated: string[] }` | Yes |
| `security.list-expiring` | `{ days?: number }` | `{ secrets: ExpiringSecret[] }` | Yes |
---
## Utility
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `echo` | `{ message: string }` | `{ message: string }` | No Auth |
| `server.echo` | `{ message: string }` | `{ message: string }` | No Auth |
---
## Example: cURL
```bash
# Login
curl -c cookies.txt -X POST http://192.168.1.228/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"auth.login","params":{"password":"password123"}}'
# Get system stats (authenticated)
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"system.stats"}'
# Get DID
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"node.did"}'
```
docs/app-developer-guide.md
+416
View File
@@ -0,0 +1,416 @@
# Archipelago App Developer Guide
Build and package containerized apps for Archipelago.
## Overview
Apps run as rootless Podman containers on user nodes. You describe an app in `apps/<app-id>/manifest.yml`; the backend validates that manifest, compiles it into rootless container/runtime behavior, and the release pipeline generates catalog surfaces from the same manifest-owned metadata.
Archipelago's app contract is deliberately manifest-first. A developer should be able to describe images or local builds, ports, volumes, generated files, dependencies, health/readiness, data ownership, networking, secrets, and supported bridge integrations in the app manifest without asking for a custom OS image or app-specific backend patch. When a real app needs a capability that is not represented yet, the preferred path is to add a reusable manifest/orchestrator primitive that other apps can use too.
The historical marketplace-publish design is not the active local developer contract for `1.8-alpha`. For this release, local manifests are the source of truth and catalog JSON is generated from them.
## App Manifest
Every app needs a manifest at `apps/<app-id>/manifest.yml`. The root key is `app`; runtime, catalog, and integration fields live below that key.
### Template Manifest
```yaml
# apps/my-app/manifest.yml
app:
id: my-app # Unique, lowercase kebab-case
name: My App
version: 1.0.0 # Semantic versioning
description: My App does one thing well.
container:
image: docker.io/myorg/my-app:1.0.0
pull_policy: if-not-present
network: archy-net
entrypoint: ["sh", "-lc"]
custom_args:
- /app/start.sh
derived_env:
- key: PUBLIC_URL
template: https://{{HOST_MDNS}}:8180
secret_env:
- key: APP_PASSWORD
secret_file: my-app-password
dependencies:
- storage: 1Gi
resources:
cpu_limit: 2
memory_limit: 512Mi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
network_policy: isolated
ports:
- host: 8180
container: 8080
protocol: tcp
volumes:
- type: bind
source: /var/lib/archipelago/my-app
target: /data
options: [rw]
environment:
- APP_MODE=production
health_check:
type: http
endpoint: http://localhost:8080
path: /health
interval: 30s
timeout: 5s
retries: 3
files:
- path: /var/lib/archipelago/my-app/config.yml
content: |
bind: 0.0.0.0:8080
overwrite: false
metadata:
icon: /assets/img/app-icons/my-app.svg
category: tools
tier: optional
repo: https://github.com/myorg/my-app
launch:
open_in_new_tab: false
```
### Required Fields
| Field | Description |
|-------|-------------|
| `app.id` | Unique identifier, lowercase, kebab-case only |
| `app.name` | Human-readable name |
| `app.version` | Version string containing at least one digit; semantic versions are preferred |
| `container.image` or `container.build` | Exactly one image source must be present |
| `security.readonly_root` | Should remain `true` for normal apps |
| `security.no_new_privileges` | Should remain `true` for normal apps |
### Current Manifest Fields
| Field | Purpose |
|-------|---------|
| `app.id`, `app.name`, `app.version`, `app.description` | App identity and release metadata |
| `app.container.image` | Registry image to pull |
| `app.container.build` | Local build definition with `context`, `dockerfile`, `tag`, and optional `build_args` |
| `app.container.pull_policy` | Pull behavior, usually `if-not-present` |
| `app.container.network` | Podman network setting such as `archy-net` or `pasta`; dangerous namespace-sharing modes are rejected |
| `app.container.entrypoint` / `custom_args` | Entrypoint and command override |
| `app.container.derived_env` | Environment values rendered from allowed host facts such as `HOST_IP`, `HOST_MDNS`, and `DISK_GB` |
| `app.container.secret_env` | Environment values read from `/var/lib/archipelago/secrets/<secret_file>`, injected as podman secrets (never visible in `podman inspect` or unit files) |
| `app.container.generated_secrets` | Secrets the orchestrator creates on first use (`hex16`/`hex32`/`base64`/`bcrypt`) — self-healing, 0600, no host provisioning |
| `app.container.generated_certs` | Self-signed TLS certs materialised before create; CN/SANs rendered from host facts |
| `app.container.network_aliases` | Extra DNS names on the app network so stack members answer to short baked-in hostnames (`api`, `minio`, `relay`) |
| `app.container.data_uid` | UID:GID ownership repair for app data directories |
| `app.hooks` | Allow-listed lifecycle hooks (`post_install`: `exec` inside the app's own container, `copy_from_host` from allow-listed roots) — see `manifest-hooks-design.md` |
| `app.dependencies` | Storage requirements and app dependencies |
| `app.resources` | CPU, memory, and disk limits |
| `app.security` | Capabilities, read-only root, no-new-privileges, network policy, optional AppArmor profile |
| `app.ports` | Host-to-container port mappings |
| `app.volumes` | `bind`, `volume`, or `tmpfs` mounts |
| `app.files` | Generated files under declared bind-mounted host paths |
| `app.environment` | Static `KEY=value` environment entries |
| `app.health_check` | HTTP or TCP health check settings |
| `app.devices` | Explicit device paths |
| `app.metadata` | Catalog-facing presentation metadata such as icon, category, tier, repo/source, author, feature bullets, and launch hints |
| `app.interfaces.main` | Optional primary UI launch surface with `port`, `protocol`, and `path` |
Additional extension keys may exist for current integrations, for example Bitcoin, Lightning, or app-specific launch/interface metadata. Treat extension keys as transitional unless they are documented as reusable platform primitives.
Use `metadata.launch.open_in_new_tab: true` when the app UI is known to reject iframe embedding with headers such as `X-Frame-Options` or restrictive CSP. The frontend app-session metadata is generated from this flag during release work.
### Launch Interfaces
If an app exposes a user-facing web UI, declare its primary launch surface in
`interfaces.main`. Runtime package listings prefer this interface over inferred
port mappings, which matters for apps that expose non-UI service ports or use a
companion wait/proxy UI.
```yaml
interfaces:
main:
name: Web UI
description: Primary app interface
type: ui
port: 8180
protocol: http
path: /
```
For simple HTTP apps without `interfaces.main`, Archipelago can still infer the
launch URL from the first declared TCP host port when the app has an HTTP health
check. TCP-only service ports, such as Bitcoin RPC/P2P, are not treated as UI
launch URLs.
Interface keys must use lowercase ASCII letters, digits, hyphens, or
underscores. Supported interface types are `ui`, `api`, and `metrics`; only
`type: ui` is treated as a launchable app surface. Supported protocols are
`http` and `https`, and `path` must start with `/`.
### Nostr Signer Bridge (NIP-07)
Apps embedded in the Archipelago iframe can use the node's Nostr identity to sign
events without managing their own keys. Archipelago injects a **NIP-07 provider**
(`window.nostr` with `getPublicKey()` / `signEvent()` / `nip04` / `nip44`) that bridges
to the host. Your app code uses standard NIP-07 — no Archipelago-specific API.
**How injection works.** After install, the host copies `nostr-provider.js` into the
app container and patches the app's web server so every page loads it and the app is
iframe-embeddable. This is **best-effort** and depends on your server config exposing
the right hooks. For an **nginx-served SPA** (the supported reference shape, e.g.
IndeeHub) your `nginx.conf` must satisfy this contract:
1. **Be iframe-embeddable.** Do not send a hard `X-Frame-Options: DENY`. The host
strips a `SAMEORIGIN`/`DENY` `X-Frame-Options` header line if present; restrictive
CSP `frame-ancestors` will still block embedding.
2. **Keep an exact-match `location = /sw.js {` block.** The provider's no-cache
`location = /nostr-provider.js` block is inserted immediately before it.
3. **Keep an SPA fallback line `try_files $uri $uri/ /index.html;`.** A
`sub_filter` that injects `<script src="/nostr-provider.js"></script>` before
`</head>` is inserted right after it. (nginx must have `ngx_http_sub_module` —
stock `nginx:alpine` does.)
4. **If you proxy an API that does NIP-98 URL verification**, expose
`proxy_set_header X-Forwarded-Prefix /api;`; the host rewrites it to honor the
outer reverse proxy's prefix.
The patch is **idempotent** (it checks for an existing `nostr-provider` reference
before editing) and re-runs on reinstall. If you rename or remove any of the anchor
strings above, injection silently no-ops and `window.nostr` will be undefined in your
app — so guard those lines in your config (see the contract comment block at the top of
IndeeHub's `nginx.conf` for a template).
> Non-nginx servers (Next.js `node server.js`, etc.) are not auto-patched today. Either
> serve via nginx, or ship `nostr-provider.js` yourself and reference it in your HTML;
> the canonical script lives at `/opt/archipelago/web-ui/nostr-provider.js` on the node.
Declare iframe intent in the manifest so the launcher embeds (vs. opens a new tab):
```yaml
metadata:
launch:
open_in_new_tab: false # default; set true only if the app cannot be iframed
```
## Security Requirements
These are enforced by the marketplace/catalog pipeline and the node. Non-compliant apps are flagged.
### Mandatory
1. **No `:latest` tag** — Pin a specific version: `myapp:1.0.0`
2. **Read-only root filesystem** — `security.readonly_root: true` (use volumes for writable data)
3. **No privilege escalation** — `security.no_new_privileges: true`
4. **Minimal capabilities** — Drop all caps, only add required ones
5. **No host network unless explicitly approved** — keep `security.network_policy` isolated or bridge
### Allowed Capabilities
The parser currently accepts this allow-list. Keep capability requests minimal; some accepted capabilities still require release review before a public package should depend on them.
| Capability | When Needed |
|-----------|-------------|
| `CHOWN` | App needs to change file ownership |
| `DAC_OVERRIDE` | App needs to bypass file permissions |
| `FOWNER` | App needs ownership-related file operations |
| `NET_ADMIN` | Network administration; requires extra scrutiny |
| `NET_BIND_SERVICE` | App binds to ports below 1024 |
| `NET_RAW` | Raw network sockets; requires extra scrutiny |
| `SETUID`, `SETGID` | App manages user switching |
| `SYS_ADMIN` | Broad administrative capability; avoid for normal apps |
### Forbidden
- Namespace-sharing network modes such as `container:<name>` or `ns:<path>`
- Mounting system paths: `/`, `/etc`, `/var`, `/usr`, `/proc`, `/sys`
- `SYS_PTRACE`, privileged containers, Docker socket mounts, or rootful execution
- Hardcoded secrets in environment variables or images
## Container Best Practices
### Volumes
```yaml
volumes:
- type: bind
source: /var/lib/archipelago/my-app
target: /data
options: [rw]
```
Data is stored at `/var/lib/archipelago/{app-id}/` on the host.
Generated files must live under a declared bind-mounted host path:
```yaml
files:
- path: /var/lib/archipelago/my-app/config.yml
content: |
bind: 0.0.0.0:8080
overwrite: false
```
Use `overwrite: false` for first-run defaults that users or the app may later modify. Use `overwrite: true` only for generated files the platform must own.
### Health Checks
Define a health check endpoint in your container:
```dockerfile
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -f http://localhost:8080/health || exit 1
```
### Logging
- Log to stdout/stderr (Podman captures container logs)
- Never log secrets, passwords, or keys
- Use structured logging (JSON) for machine parsing
### Networking
Apps get their own network namespace. To connect to other Archipelago apps:
```yaml
# If your app needs to talk to Bitcoin
dependencies:
- bitcoin-knots
container:
network: archy-net
derived_env:
- key: BITCOIN_RPC_HOST
template: bitcoin-knots
- key: BITCOIN_RPC_PORT
template: "8332"
```
The `archy-net` Podman network provides DNS resolution between containers. Use `derived_env` for host facts like `HOST_MDNS` instead of hardcoding node-specific URLs.
## Catalog Generation
Catalog JSON is generated from manifests during release work. Do not manually edit generated fields in `app-catalog/catalog.json` or `neode-ui/public/catalog.json` when the same value belongs in the manifest.
Manifest-owned catalog fields currently include:
- app title from `app.name`;
- version from `app.version`;
- description from `app.description`;
- Docker image from `app.container.image`;
- category from `app.category` or `app.metadata.category`;
- tier from `app.metadata.tier`;
- icon from `app.metadata.icon`;
- repo URL from `app.metadata.repo`, `repoUrl`, or `source`.
### 1. Build and Push Your Image
```bash
podman build -t docker.io/myorg/my-app:1.0.0 .
podman push docker.io/myorg/my-app:1.0.0
```
### 2. Generate Catalogs
```bash
python3 scripts/generate-app-catalog.py
```
### 3. Verify Drift
```bash
python3 scripts/check-app-catalog-drift.py --release --strict
```
Before release, the canonical catalog and UI public catalog should match:
```bash
cmp -s app-catalog/catalog.json neode-ui/public/catalog.json
```
## Testing Your App
### Local Testing
```bash
# Run your container locally
podman run -d --name my-app \
-p 8180:8080 \
--read-only \
--security-opt no-new-privileges \
--user 1000:1000 \
docker.io/myorg/my-app:1.0.0
# Verify it works
curl http://localhost:8180/health
# Check logs
podman logs my-app
```
### On an Archipelago Node
1. Install via the marketplace UI or RPC:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
-d '{"method":"package.install","params":{"id":"my-app","dockerImage":"docker.io/myorg/my-app:1.0.0"}}'
```
2. Verify the container is running:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
-d '{"method":"container-list"}'
```
3. Check the UI at `http://192.168.1.228/app/my-app/`
### Validate Manifest
```bash
cargo test --manifest-path core/Cargo.toml -p archipelago-container
python3 scripts/check-app-catalog-drift.py --release --strict
```
## Updating Your App
1. Build and push the new version: `docker.io/myorg/my-app:1.1.0`.
2. Update `app.version` and `app.container.image` or `app.container.build.tag`.
3. Run catalog generation and drift checks.
4. Validate install/start/stop/restart/uninstall/reinstall behavior before shipping.
The broader app update policy for `1.8-alpha` is still being finalized. Until that policy is locked, app manifests should be explicit and pinned so update detection compares concrete image/tag metadata rather than mutable tags.
## App Icon
- Provide a URL to your app icon (PNG, WebP, or SVG)
- Recommended size: 256x256 pixels
- Square aspect ratio
- If no icon URL, a generic placeholder is shown in the marketplace
## Release Validation Expectations
Every supported app must satisfy the lifecycle contract:
- install
- launch
- stop
- start
- restart
- uninstall while preserving data
- reinstall with preserved data
- report truthful health/status
- survive backend restart
- survive host reboot
For apps with special dependencies, launch must explain dependency wait states instead of showing a dead iframe. Examples include Bitcoin sync/IBD, Lightning wallet readiness, Nostr signer bridge injection, Tailscale login/auth, and app-specific setup screens.
Runtime changes should be validated with focused tests first, then the release lifecycle harness on the validation host when host access is intentionally resumed.
docs/app-manifest-spec.md
+155
View File
@@ -0,0 +1,155 @@
# App Manifest Specification
_Accurate as of 2026-07-08. The canonical schema is the Rust parser in
`core/container/src/manifest.rs` (`AppManifest` → `AppDefinition`); if this
document and the code disagree, the code wins. See
[`app-developer-guide.md`](app-developer-guide.md) for the authoring workflow._
Every app is a directory `apps/<id>/` containing a `manifest.yml` with a single
top-level `app:` block. Apps are purely declarative — the orchestrator owns the
entire lifecycle; there is no per-app installer code.
## Top-level fields (`app:`)
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `id` | string | ✅ | Lowercase alphanumeric + `-`/`_`. Must match the directory name. |
| `name` | string | ✅ | Display name. |
| `version` | string | ✅ | App version shown in the UI. |
| `description` | string | — | One-line description. |
| `container` | ContainerConfig | — | Image/build source + runtime shape (below). |
| `dependencies` | list | — | `- storage: "10GB"`, `- { app_id: bitcoin, version: … }`, or a bare string. |
| `resources` | ResourceLimits | — | `cpu_limit` (int), `memory_limit` (e.g. `"512m"`), `disk_limit`. |
| `security` | SecurityPolicy | — | See [Security](#security). |
| `ports` | list of PortMapping | — | `- { host: 8080, container: 80, protocol: tcp }`. |
| `volumes` | list of Volume | — | See [Volumes](#volumes). |
| `files` | list of GeneratedFile | — | Config files written before create: `{ path, content, overwrite }`. `path` must sit under a declared bind mount. |
| `environment` | list of string | — | `- KEY=value` pairs (static). |
| `health_check` | HealthCheck | — | `{ type, endpoint/path, interval, timeout, retries }`. `type` is free-form today; `http` is what the monitor exercises. |
| `devices` | list of string | — | Host device paths; must start with `/dev/`. |
| `interfaces` | map | — | Launch surfaces, keyed by name (`main`): `{ name, description, type, port, protocol, path }`. |
| `hooks` | LifecycleHooks | — | Allow-listed lifecycle hooks. See [Hooks](#hooks). |
| _anything else_ | — | — | Unknown keys are absorbed into an `extensions` map (serde flatten) and treated as transitional metadata — e.g. `container_name`, `metadata`, `category`, `bitcoin_integration`, `lightning_integration`. These are **not** typed schema; do not rely on them being validated. |
## `container:` (ContainerConfig)
Exactly **one** of `image` or `build` must be present (image XOR build).
| Field | Type | Notes |
|-------|------|-------|
| `image` | string | Registry reference. Pull source. |
| `image_signature` | string | Optional signature reference for image verification. |
| `pull_policy` | string | Default `if-not-present`. |
| `build` | BuildConfig | Local build: `{ context, dockerfile (default "Dockerfile"), tag, build_args }`. |
| `network` | string | Literal podman `--network` value (`archy-net`, `host`, a stack network, …). Omitted = rootless default isolated network. |
| `network_aliases` | list of string | Extra DNS names on `network` (podman `--network-alias`) — lets stack members answer to short baked-in hostnames (`api`, `minio`, `relay`). |
| `entrypoint` | list of string | Entrypoint override. |
| `custom_args` | list of string | Extra positional args appended after the image. |
| `derived_env` | list | `- { key, template }` — template rendered against host facts at apply time. Allowed placeholders: `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}` (plus dependency-resolved facts such as the active bitcoin host). Never hard-code host specifics. |
| `secret_env` | list | `- { key, secret_file }` — value read from `/var/lib/archipelago/secrets/<secret_file>` and injected as a **podman secret**, so it never appears in `podman inspect` or unit files. `secret_file` must be a bare filename (no `/`, no `..`). |
| `generated_secrets` | list | `- { name, kind }` — orchestrator materialises the secret on first use (0600, rootless service user, idempotent + self-healing). `kind ∈ hex16 | hex32 | base64 | bcrypt` (bcrypt writes `<name>` = hash and `<name>.pw` = plaintext). |
| `generated_certs` | list | `- { crt, key, common_name?, sans? }` — self-signed TLS materialised before create; CN/SANs rendered against host facts. |
| `data_uid` | string | `"UID:GID"` applied to the app's bind-mounted data dir before create (rootless subuid mapping, e.g. Postgres). |
## Security
```yaml
security:
readonly_root: true # default true
no_new_privileges: true # default true
capabilities: [CHOWN] # default [] (cap-drop ALL, add back only these)
network_policy: isolated # isolated | bridge | host (default isolated)
apparmor_profile: null # optional profile name
```
Validation (enforced at `AppManifest::validate()`):
- Capabilities must come from the reviewed allow-list (CHOWN, DAC_OVERRIDE,
FOWNER, NET_ADMIN, NET_BIND_SERVICE, NET_RAW, SETGID, SETUID, SYS_ADMIN).
- `network_policy` must be exactly `isolated`, `bridge`, or `host`.
- No `container:`/`ns:` network modes; devices must be `/dev/*`.
- Bind-mount sources are confined to `/var/lib/archipelago` (reviewed
exceptions: the rootless podman socket and dbus).
- `derived_env` templates may only use the placeholder allow-list;
`secret_env`/`generated_secrets` names must be bare filenames.
- Hook steps are validated against the hook allow-list (below).
## Volumes
```yaml
volumes:
- type: bind # bind | volume | tmpfs
source: /var/lib/archipelago/myapp/data
target: /data
options: [rw] # allow-list: rw, ro, z, Z, shared, …
- type: tmpfs
target: /tmp
tmpfs_options: "rw,noexec,nosuid,size=256m"
```
## Hooks
Declarative, allow-listed operations that run against the app's **own
container** — never the host (design: `manifest-hooks-design.md`).
```yaml
hooks:
post_install: # runs once after install, container running
- copy_from_host: # src relative to an allow-listed root (data dir / web-ui);
src: web-ui/nostr-provider.js # no absolute paths, no '..'
dest: /usr/share/nginx/html/nostr-provider.js
- exec: ["sh", "-c", "nginx -s reload"] # podman exec inside the container
pre_start: [] # reserved in the schema; executor not yet wired
```
## Installation semantics
The orchestrator compiles the manifest into a rootless Podman **Quadlet unit
under `user.slice`** — the container survives backend restarts and reboots, and
a level-triggered reconciler converges drift every 30 seconds. Multi-container
apps are sets of per-member manifests installed together via the stack
orchestrator (`api/rpc/package/stacks.rs`) on an app-local network.
## Distribution
Manifests ship two ways:
1. **Signed catalog** (primary): `releases/app-catalog.json` embeds the full
manifest per app and carries an Ed25519 detached signature verified against
the pinned release-root anchor. Nodes overlay catalog manifests over disk
files — **catalog wins** for image-only apps; `apps/<id>/manifest.yml` on
disk remains the fallback and is still required for build-source apps.
2. **Decentralized marketplace**: Nostr NIP-78 discovery with DID-signed
manifests ([`marketplace-protocol.md`](marketplace-protocol.md)). Note the
marketplace uses its own flatter manifest schema, not this one.
## Minimal example
```yaml
app:
id: myapp
name: My App
version: 1.0.0
description: Does something useful
container:
image: docker.io/vendor/myapp:1.0.0
generated_secrets:
- { name: myapp-admin-password, kind: hex16 }
secret_env:
- { key: ADMIN_PASSWORD, secret_file: myapp-admin-password }
ports:
- { host: 8090, container: 8080 }
volumes:
- { type: bind, source: /var/lib/archipelago/myapp, target: /data, options: [rw] }
health_check:
type: http
path: /health
interfaces:
main:
type: web
port: 8090
```
Validate with `scripts/validate-app-manifest.sh` and regenerate the catalog
with `scripts/generate-app-catalog.py` (drift-checked in CI by
`scripts/check-app-catalog-drift.py`).
docs/architecture.md
+233
View File
@@ -0,0 +1,233 @@
# Archipelago — Architecture
> **Bitcoin Node OS** — Flash to USB, install on hardware, manage via web UI.
**Stack**: Rust backend + Vue 3 + TypeScript (strict) + Vite + Tailwind CSS + Pinia + rootless Podman (Quadlet)
**Target OS**: Debian 13 (Trixie) — x86_64 and ARM64
**Status**: 1.8.0-alpha — single-node production gate green; multinode pass + release hardening in progress (see [`ROADMAP.md`](ROADMAP.md))
---
## System Layers
```
┌──────────────────────────────────────────────────────┐
│ YOUR BROWSER │
│ Vue 3 SPA (Composition API + Pinia) │
└──────────────────────┬───────────────────────────────┘
│ HTTP / WebSocket
┌──────────────────────┴───────────────────────────────┐
│ NGINX │
│ /rpc/v1 → backend /app/{id}/ → container │
└──────────────────────┬───────────────────────────────┘
│ port 5678 (127.0.0.1)
┌──────────────────────┴───────────────────────────────┐
│ RUST BACKEND (core/) │
│ Auth, ~380 RPC methods, orchestrator + reconciler, │
│ federation, mesh, identity, wallet, updates │
└──────────────────────┬───────────────────────────────┘
│ Podman REST API socket + systemd Quadlet units
┌──────────────────────┴───────────────────────────────┐
│ ROOTLESS PODMAN CONTAINERS │
│ 50+ manifest-driven apps as user.slice Quadlet │
│ units — survive backend restarts, self-heal │
└──────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────┐
│ DEBIAN 13 (Trixie) │
│ systemd, UFW, Tor, AppArmor, Reticulum daemon │
└──────────────────────────────────────────────────────┘
```
## Codebase Stats
| Component | Lines | Files |
|-----------|-------|-------|
| Rust backend (`core/`) | ~117,000 | ~334 |
| TypeScript/Vue (`neode-ui/src/`) | ~69,000 | ~325 |
| Shell scripts (`scripts/`) | — | ~51 |
| Packaged apps (`apps/*/manifest.yml`) | — | 51 |
## Backend Crates (`core/`)
Workspace members (root `core/Cargo.toml`):
| Crate | Purpose |
|-------|---------|
| `archipelago` | Main binary — API (~380 RPC methods), container orchestrator + boot reconciler, mesh, identity/federation, wallet, updates, marketplace |
| `container` (`archipelago-container`) | Podman REST client, canonical manifest schema, Quadlet compiler, health monitor, signed app catalog, image verification |
| `security` (`archipelago-security`) | AppArmor/seccomp container policy generation, secrets manager |
| `openwrt` (`archipelago-openwrt`) | TollGate gateway provisioning over SSH/UCI |
| `performance` (`archipelago-performance`) | Resource limits |
Also on disk but **not** workspace members (standalone/legacy, cleanup tracked in the hardening plan §G): `models`, `helpers`, `js-engine`, `container-init`.
### Key Backend Modules
```
core/archipelago/src/
├── api/handler/ — HTTP routing (/rpc, /health, /dwn, /ws)
├── api/rpc/dispatcher.rs — RPC dispatch (~380 method arms)
├── api/rpc/package/ — App install/lifecycle/stacks (multi-container)
├── container/ — prod_orchestrator, boot_reconciler, quadlet,
│ app_catalog (signed, embedded manifests),
│ version_config, crash_recovery, secrets
├── trust/ — release-root anchor (pinned Ed25519 pubkey),
│ detached-signature verify, did:key
├── mesh/ — Meshtastic + MeshCore + Reticulum transports,
│ X3DH/double-ratchet crypto, outbox/scheduler,
│ mesh AI assistant, bitcoin relay
├── federation/ — multi-node federation over Tor, state sync
├── identity.rs / identity_manager.rs — Ed25519 did:key, multi-identity
├── credentials/ — W3C Verifiable Credentials
├── nostr_discovery.rs — Nostr presence (NIP-33 kind 30078)
├── nostr_handshake.rs — NIP-44 encrypted peer comms
├── marketplace.rs — decentralized app marketplace (Nostr NIP-78,
│ DID-signed manifests, trust scoring)
├── wallet/ — LND integration, ecash (Fedimint/Cashu)
├── update.rs — signed OTA: resumable download, rollback,
│ post-update self-verify window
├── session.rs / auth.rs — sessions (persisted), Argon2id, TOTP
├── transport/ / network/ — Tor transport, DWN store/sync
└── fips/ / swarm/ / streaming/ — federation IPS anchor, P2P swarm (gated), streaming (WIP)
```
## App Platform (as built)
An app is a directory `apps/<id>/manifest.yml` parsed by the canonical schema
in `core/container/src/manifest.rs`. A manifest declares identity, a container
source (**image XOR build**), and runtime shape: ports, volumes (confined to
`/var/lib/archipelago`), generated config files, environment, devices,
resources, health checks, and the launch interface. Ergonomics are declarative
too: `derived_env` (host-fact templating), `secret_env` (podman secrets — values
never appear in `podman inspect` or unit files), `generated_secrets` /
`generated_certs` (self-healing), `network_aliases`, `data_uid`, and
allow-listed `post_install` hooks that run inside the app's own sandbox.
**Install** compiles the manifest to a rootless **Quadlet unit under
`user.slice`** — containers survive backend restarts and reboots.
Multi-container apps (BTCPay, Mempool, Immich, NetBird, IndeeHub) are sets of
per-member manifests installed via the stack orchestrator on an app-local
network with readiness gates and generated cross-service secrets. A
level-triggered **boot reconciler** converges actual state to desired state
every 30 seconds.
**Distribution**: the signed catalog (`releases/app-catalog.json`, Ed25519
detached signature over canonical JSON, verified against the pinned
release-root anchor in `trust/anchor.rs`) embeds the full manifest per app;
nodes overlay catalog manifests over disk files (catalog wins), so apps can
ship without OTA disk files. A curated subset (27 apps) powers the store UI
(`app-catalog/catalog.json`). A parallel **decentralized marketplace**
(Nostr NIP-78 discovery, DID-signed manifests, federation-weighted trust
scoring, Lightning purchase invoices) is implemented as a second,
community-distribution channel.
**Security invariants** enforced at manifest validation: read-only root and
no-new-privileges by default, capability allow-list, `network_policy ∈
{isolated, bridge, host}`, bind mounts confined to `/var/lib/archipelago`, no
privileged containers, rootless only.
## Frontend (`neode-ui/src/`)
```
├── api/ — RPC client, WebSocket, container client
├── views/ — Dashboard, Apps, Marketplace, Cloud, Server,
│ Mesh, Web5, Settings, Monitoring, Fleet, Chat,
│ onboarding flow (11 screens), kiosk, recovery
├── components/ — EasyHome, ModeSwitcher, BootScreen, SpotlightSearch, …
├── stores/ — Pinia: app, install, mesh, cloud, goals, uiMode,
│ controller (gamepad), aiPermissions, …
├── composables/ — useControllerNav, useToast, useNavSounds, …
├── router/ — ~51 routes
└── style.css — global glassmorphism theme
```
Three UI modes (Pro/Easy/Chat), gamepad navigation, i18n, PWA. Tested with
Vitest + Playwright. AIUI is a separate external app surfaced via nginx.
## Mesh Networking
Three LoRa transports behind one chat UI and a common `MeshRadioDevice`
surface:
- **Meshtastic** — in-process async serial driver (protobuf over SLIP)
- **MeshCore** — framed-serial protocol; phone companion apps speak this
- **Reticulum (RNS/LXMF)** — host-supervised Python daemon
(`reticulum-daemon/`, PyInstaller-packaged, one per RNode radio) speaking
Unix-socket JSON-RPC to the backend; `archy-rnodeconf` ships as an OS-level
radio config tool
End-to-end encryption uses X3DH key agreement + double-ratchet. Extras: image
and voice attachments, mesh AI assistant (`!ai`), Bitcoin balance relay over
mesh, steganography, store-and-forward outbox.
## Networking
- **Container DNS**: app-local Podman networks with `network_aliases`; aardvark-dns resolution
- **Tor**: system daemon, SOCKS5 on 9050, hidden services per node; all inter-node federation traffic
- **Federation**: invite-based joining, DID-based trust levels, state sync, cross-node app deploy
- **UFW**: `DEFAULT_FORWARD_POLICY="ACCEPT"` required for LAN container access
- **OpenWrt/TollGate**: gateway provisioning via the `openwrt` crate
## Security Model
| Layer | Measures |
|-------|----------|
| OS | Debian hardening, AppArmor, minimal packages |
| Nginx | CSP headers, rate limiting, auth_request, session validation |
| Backend | Input validation, CSRF, session auth, bind 127.0.0.1 only |
| Containers | Rootless Podman, cap-drop ALL + reviewed allow-list, readonly root, no-new-privileges, memory limits |
| Supply chain | Ed25519-signed release manifests + app catalog against a pinned release-root anchor; auto-apply refuses unsigned |
| Crypto | Ed25519 signatures, ChaCha20-Poly1305 encryption, Argon2id password hashing (transparent bcrypt upgrade), constant-time comparisons |
| Network | Tor hidden services, UFW firewall, SSRF prevention |
## Data Paths
| Data | Path |
|------|------|
| App data | `/var/lib/archipelago/{app-id}/` |
| Identity | `/var/lib/archipelago/identity/` |
| Multi-identity | `/var/lib/archipelago/identities/` |
| Federation | `/var/lib/archipelago/federation/` |
| DWN messages | `/var/lib/archipelago/dwn/messages/` |
| Credentials | `/var/lib/archipelago/credentials/` |
| Backups | `/var/lib/archipelago/backups/` (ChaCha20-Poly1305) |
| Secrets | `/var/lib/archipelago/secrets/{app-id}/` (0600, service-user-owned) |
| Sessions | `/var/lib/archipelago/sessions.json` |
| Marketplace cache | `/var/lib/archipelago/marketplace/` |
| Frontend | `/opt/archipelago/web-ui/` |
| Backend binary | `/usr/local/bin/archipelago` |
## Key Features (Working)
- 50+ containerized apps with one-click install/manage; full lifecycle matrix repeatedly green on real hardware
- Bitcoin Core **and** Knots with per-app version pinning and safe switching; LND + Core Lightning
- Multi-node federation with invite-based joining and trust levels
- W3C DID identity (did:key, DID Documents, Verifiable Credentials)
- Nostr: NIP-33 node discovery, NIP-44/NIP-04 encryption, NIP-07 signer bridge for iframe apps, relay hosting
- Decentralized marketplace (NIP-78 discovery, trust scoring, Lightning purchases)
- File sharing with access controls (free/peers-only/paid via LN, on-chain, ecash)
- Encrypted backups (Argon2 + ChaCha20-Poly1305)
- Health monitoring + level-triggered reconciler with tiered auto-restart
- Tri-protocol LoRa mesh (Meshtastic / MeshCore / Reticulum) with E2E crypto
- Signed OTA updates with rollback and post-update self-verification
- Three-mode UI (Pro/Easy/Chat), gamepad navigation, real-time WebSocket updates
- Bootable ISO installer (`image-recipe/`), Android companion app
## Further Documentation
| Doc | Purpose |
|-----|---------|
| [`ROADMAP.md`](ROADMAP.md) | Shipped / in-progress / planned |
| [`developer-guide.md`](developer-guide.md) | Dev setup, workflow, code conventions |
| [`api-reference.md`](api-reference.md) | RPC endpoint reference |
| [`app-developer-guide.md`](app-developer-guide.md) | Building and publishing apps |
| [`app-manifest-spec.md`](app-manifest-spec.md) | The `manifest.yml` schema |
| [`user-walkthrough.md`](user-walkthrough.md) | End-user installation and usage guide |
| [`troubleshooting.md`](troubleshooting.md) | Diagnostic scenarios and solutions |
| [`operations-runbook.md`](operations-runbook.md) | Ops commands and emergency recovery |
| [`multi-node-architecture.md`](multi-node-architecture.md) | Federation protocol design |
| [`marketplace-protocol.md`](marketplace-protocol.md) | Decentralized app discovery via Nostr |
| [`PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md) | North star and workstream narrative |
| [`UNIFIED-TASK-TRACKER.md`](UNIFIED-TASK-TRACKER.md) | Live, priority-ordered open items |
| [`archive/`](archive/) | Historical audits, session logs, shipped designs |
docs/archive/HANDOVER-2026-07-02-iso-feedback.md
+151
View File
@@ -0,0 +1,151 @@
# Handover — fresh-ISO feedback bug-bash (2026-07-02)
**For: the agent building the next ISO + fleet deploy.** All fixes below are
**merged and pushed: gitea-ai main = `f5d24796`** (merge of `c375ecc4`,
65 files; branch `iso-feedback-fixes-2026-07-02` also pushed). Source
feedback: user's fresh ISO install on a Framework (11th-gen Tiger Lake)
machine, node `192.168.1.81` (SSH `archipelago` / `archipelago`).
Diagnostic bundle: `/home/archipelago/incoming-logs/node-logs-192.168.1.81/`.
**⚠️ Known-red tests on main (NOT from this work):** `trust::anchor::
unset_constant_is_none` + 2 `trust::signed_doc` tests fail because a prior
commit pinned `RELEASE_ROOT_PUBKEY_HEX` without updating them. The signing/
audit agent's uncommitted changes in the shared tree fix exactly these —
coordinate with them; don't "fix" it independently or you'll collide. This
bug-bash branch alone was 898/898 green; merged with main it's 894/898 with
only those three.
## ⚠️ Outstanding user request for the deploy
- **Change .81's web-UI password to `ThisIsWeb54321@`** — the user forgot the
current one. Node was unreachable from .116 during this session (flaky WiFi
AP, IP flapped .68↔.81). Do this during deploy (SSH works from the user's
machine; `archipelago`/`archipelago`).
## What changed (by file)
### Backend (core/archipelago/src) — builds clean, targeted tests pass
- `api/handler/websocket.rs` — **subscribe BEFORE initial snapshot** (the
"everything needs ctrl-r" root cause: broadcasts in the snapshot→subscribe
gap were silently lost; a stale client never learned containers-scanned).
- `main.rs` — crash check now runs BEFORE writing the PID marker (**crash
recovery had never run on any node** — it always saw its own PID and
skipped); tracing default demoted debug→info (journal volume).
- `crash_recovery.rs` — PID-reuse guard (`process_is_archipelago`); new
**pending-boot-starts registry** (names queued for recovery/reconcile) with
writers in `recover_containers` + stack recovery.
- `server.rs` — scanner overlays Stopped/Exited → **Restarting** for
pending-boot-start ids (user ask: "status should be restarting if they are
being restarted"); `SCANNER_RESTARTING` ownership set so scanner-authored
Restarting resolves immediately instead of wedging in the 20-min
transitional-preserve.
- `container/prod_orchestrator.rs` — reconcile pass + `adopt_existing`
register/deregister pending boot-starts; LND pre-start hook passes detected
`bitcoin_host()` (Knots vs Core) into `lnd::ensure_config`; new
`fedimint-clientd` pre-start hook (mkdir + chown 1000:1000 of
`/var/lib/archipelago/fmcd` — self-heals the crash-loop).
- `container/lnd.rs` — `ensure_config(paths, rpc_pass, bitcoin_host)`;
bitcoind.rpchost no longer hardcoded `bitcoin-knots`; drift check rewrites
host changes; +unit test `ensure_config_repairs_bitcoin_host_drift`.
- `api/rpc/package/dependencies.rs` — bounded **dependency wait**
(`wait_for_install_deps`, 36×5s): installed-but-starting deps wait with
"Waiting for Bitcoin to start…" on the card; not-installed deps fail fast
with `DependencyGateError` marker; +5 unit tests.
- `api/rpc/package/install.rs`, `stacks.rs` — call sites wired to
`gate_install_deps` (lnd/electrumx/mempool/btcpay).
- `api/rpc/package/async_lifecycle.rs` — `DependencyGateError` removes the
optimistic entry (**no more phantom "Stopped" LND tile**) + pushes an Error
notification with the reason.
- `api/rpc/package/progress.rs` — `set_install_message` helper.
- `api/rpc/seed_rpc.rs` — `save_pending_seed_encrypted`; seed.restore also
stashes the mnemonic; `auth.rs` — **auth.setup persists the encrypted seed
backup** (recovery-phrase reveal previously failed on EVERY node because
nothing ever wrote `master_seed.enc`).
- `api/rpc/middleware.rs` — sanitizer allowlist extended (seed/2FA/auth
errors reach the user instead of "Check server logs"); +2 tests.
- `bitcoin_status.rs` — friendly status for "connection reset" (bitcoind
starting); raw URL/os-error chains no longer shown; +3 tests.
- `bootstrap.rs` — journald drop-in self-heal (OTA nodes get log caps);
bitcoin.conf printtoconsole heal. (Log-spam agent's work; verified.)
- `api/rpc/package/config.rs` — bitcoin args `-printtoconsole=0`.
### Manifests / scripts / configs
- `apps/lnd/manifest.yml` — BITCOIND_HOST now `derived_env {{BITCOIN_HOST}}`.
- `apps/bitcoin-knots/manifest.yml`, `apps/bitcoin-core/manifest.yml` —
`-printtoconsole=0` (90.6% of the journal was IBD UpdateTip spam;
debug.log in the datadir keeps full logs).
- `scripts/first-boot-containers.sh` — chown 1000:1000 of
`/var/lib/archipelago/fmcd` in BOTH fmcd blocks (root-owned dir was the
fedimint-clientd "Permission denied os error 13" crash-loop);
printtoconsole=0.
- `scripts/container-doctor.sh`, `scripts/reconcile-containers.sh` —
printtoconsole=0.
- `image-recipe/configs/journald-archipelago.conf` (NEW) — SystemMaxUse=500M,
rate limits; baked by ISO builder + bootstrap self-heal.
- `image-recipe/configs/nginx-archipelago.conf` — `/assets/` 404s no longer
cacheable (the `always` immutable header could pin a missing background for
a YEAR); HTTPS block gained the missing `/assets/` location (was silently
serving index.html as images).
- `image-recipe/configs/archipelago-kiosk.service` — MemoryMax 1500→2800M,
MemoryHigh 1200→2200M (kiosk was riding reclaim-throttle = the lag).
- `image-recipe/_archived/build-auto-installer-iso.sh` — kiosk launcher/service
now spliced from `image-recipe/configs/` at build time (was a stale inline
heredoc that force-disabled GPU); **+ `firmware-intel-graphics` +
`firmware-amd-graphics`** (Debian trixie split the i915 DMC blobs out of
firmware-misc-nonfree; the .81 kernel logged tgl_dmc missing).
### Frontend (neode-ui) — vue-tsc clean, vitest green
- `views/Login.vue` — Enter in field 1 → focus confirm; Enter in confirm →
submit; submit button always clickable (shows inline mismatch/length error
instead of being silently disabled); errors clear on input; **Restart
Onboarding needs a confirming second click** (5s window) — this button is
the likely cause of the "onboarding restarted after mismatch" report.
+`login.restartConfirm` key in en/es locales.
- `stores/sync.ts` — 30s staleness reconciliation (server.get-state) while
connected; already-connected fast path now refetches too.
- `composables/useContainersScanTimeout.ts` (NEW, +tests) — 20s escape hatch;
wired into `Apps.vue` / `Discover.vue` / `Marketplace.vue`; fresh empty node
reaches the real "no apps yet" empty state; "Checking…" can never persist.
- Backgrounds: 10 heaviest bg JPEGs → **WebP q90** (9.4MB→6.6MB; refs updated
in OnboardingWrapper/Dashboard/useRouteTransitions); 7 remaining images
stayed JPEG (WebP came out LARGER on those — noisy sources; deliberate).
- `public/assets/video/video-intro.mp4` — re-encoded CRF20 (SSIM 0.988) with
**+faststart** (moov was at EOF → browser had to download all 15MB before
playing = the intro lag). 12.7MB now, streams immediately.
- LND icon: stale dist artifact; any fresh `npm run build` ships
`app-icons/lnd.png` correctly.
## Verification done here
- `cargo build -p archipelago` + `cargo check` clean; targeted tests
(bitcoin_status, middleware sanitize, dep_wait, lnd, crash_recovery,
boot_reconciler, bitcoin_host, prod_orchestrator lnd hooks): **52 passed,
0 failed**. Full suite: **898 passed, 0 failed, 1 ignored** (22s).
- `npm run build` green; dist verified: 10 bg-*.webp present, `lnd.png`
icon present, `restartConfirm` string in bundle, optimized faststart
video (12,740,782 bytes) in place. Note: main had a latent build breaker
(unused template ref in `Web5ConnectedNodes.vue` from commit 8256fde1,
vue-tsc TS6133) — fixed here by removing the dead ref/binding; without
this fix `npm run build` fails on current main.
- vitest: new composable tests + related suites pass.
- `bash -n` clean on all touched scripts; nginx conf live-verified by agent
(200/404/cache headers on both HTTP+HTTPS blocks).
- ISO kiosk splice byte-verified against configs/ by agent simulation.
## NOT done / left for you
1. **Full test-suite run + gate**: run the complete `cargo test` and (after
deploy) `tests/lifecycle/run-gate.sh` ON .228 per CLAUDE.md before any tag.
2. **Frontend bundle grep before shipping** (per memory/feedback): verify new
strings (e.g. `restartConfirm`, `bg-home.webp`) in the built tarball.
3. **Diagnostics collector** (`data-dir-listing.txt` = 15MB of podman overlay
internals; dmidecode empty) — collector script wasn't found in this repo
(likely lives on-node or in the user's collection script); fix when found.
4. **podman healthcheck cgroup EPERM spam** (1,250 journal errors, healthchecks
unreliable fleet-wide) — real open bug, Quadlet-phase territory, NOT fixed.
5. **DP link-training failures on .81** (display corruption) — likely
cable/dock/port hardware; firmware fix may help; tell user to try another
cable/port if corruption recurs.
6. **LoRa/RNode onboarding surface** — never scoped; user may want it as a
feature (mesh device-found modal exists only on Mesh page post-login).
7. The concurrent audit agent's files (`docs/1.8.0-RELEASE-HARDENING-PLAN.md`,
`core/.../trust/*`, parts of `bootstrap.rs`) are ALSO uncommitted here —
coordinate before committing; don't mix attribution.
docs/archive/INSTALL-SCREENS-DESIGN.md
+117
View File
@@ -0,0 +1,117 @@
# Archipelago Installer — Screen Designs
Edit these screens to match your vision. I'll implement exactly what you specify.
Each screen is what the user sees at that moment on the console (80 columns wide).
Constraints: bash TUI only (no ncurses). ANSI colors available:
- `\033[1;37m` = bold white, `\033[1;33m` = bold yellow/orange
- `\033[32m` = green, `\033[31m` = red, `\033[37m` = dim gray
- `\033[0m` = reset. Box-drawing chars: ━ ─ │ ╭ ╮ ╰ ╯ ╔ ╗ ╚ ╝ █ ▓ ░ ▌▐
- Spinners possible: ⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏ or ◐◓◑◒ or |/-\
---
## Screen 1: Welcome / Press Enter
```
(clear screen, centered)
a r c h i p e l a g o
━━━━━━━━━━━━━━━━━━━━━
automatic installer
Press Enter to install | Ctrl+C for shell
```
---
## Screen 2: Detecting Disk
```
a r c h i p e l a g o
━━━━━━━━━━━━━━━━━━━━━
[1/7] Checking tools .............. ✓
[2/7] Detecting disks
Found: /dev/sda (465.8G) — TOSHIBA MQ01ACF0
──────────────────────────────────────────
⚠ All data on /dev/sda will be erased.
Press Enter to install | Ctrl+C to cancel
```
---
## Screen 3: Installing (progress)
```
a r c h i p e l a g o
━━━━━━━━━━━━━━━━━━━━━
[1/7] Checking tools .............. ✓
[2/7] Detecting disks ............. ✓
[3/7] Creating partitions ......... ✓
[4/7] Formatting .................. ✓
[5/7] Installing system ........... ✓
[6/7] Encrypting data partition ◐
AES-256-XTS (AES-NI detected)
──────────────────────────────────────────
```
---
## Screen 4: Bootloader
```
a r c h i p e l a g o
━━━━━━━━━━━━━━━━━━━━━
[1/7] Checking tools .............. ✓
[2/7] Detecting disks ............. ✓
[3/7] Creating partitions ......... ✓
[4/7] Formatting .................. ✓
[5/7] Installing system ........... ✓
[6/7] Encrypting data ............. ✓
[7/7] Installing bootloader ....... ✓
──────────────────────────────────────────
```
---
## Screen 5: Complete
```
a r c h i p e l a g o
━━━━━━━━━━━━━━━━━━━━━
Installation Complete
After reboot, open the Web UI from any device:
http://192.168.1.198
SSH: ssh archipelago@192.168.1.198
Password: archipelago
Web Login: password123
──────────────────────────────────────────
>>> REMOVE THE USB DRIVE NOW <<<
Press Enter to reboot
```
---
## Notes for Dorian
- Edit any screen above to match what you want to see
- Add/remove steps, change wording, change layout
- Specify colors per line if you want (e.g. "this line in yellow")
- I can add a spinner animation on the active step
- Box-drawing, progress bars, anything bash can render is fair game
- Once you're happy with the designs I'll implement them exactly
docs/archive/README.md
+26
View File
@@ -0,0 +1,26 @@
# docs/archive — historical records
Documents here are **finished history**: completed session logs, handovers,
point-in-time status snapshots, security audits of past versions, and design
docs whose feature has since shipped. They are kept for provenance and are
**not** maintained — nothing in this directory describes the current system.
For current state, start at:
- `docs/UNIFIED-TASK-TRACKER.md` — what's open, priority-ordered
- `docs/PRODUCTION-MASTER-PLAN.md` — north star and workstream narrative
- `docs/architecture.md` — as-built system architecture
- `docs/ROADMAP.md` — public-facing roadmap
| File | What it was | Why archived |
|------|-------------|--------------|
| `SESSION-1.8.0-OTA-PROGRESS.md` | Session narrative of the 1.8.0 OTA work | Superseded by the unified task tracker |
| `HANDOVER-2026-07-02-iso-feedback.md` | One-shot handover for the ISO feedback bug-bash | All fixes merged |
| `rust-orchestrator-migration.md` | Design for migrating container lifecycle from bash to Rust | Migration complete — `prod_orchestrator.rs` + `boot_reconciler.rs` are the live system |
| `demo-deployment-design.md` | Design for the public demo sandbox | Demo shipped; `docs/demo-build-info.md` is the live ops doc |
| `app-registry-status-2026-06-21.md` | Per-app migration snapshot from node .228 @ v1.7.99-alpha | Point-in-time snapshot; headline findings (immich legacy, meshtastic present) no longer true |
| `security-code-audit-2026-03.md` | March 2026 security audit of v0.1.0 (33 findings) | Historical record; top findings since remediated (Argon2id, persisted sessions, image verification) |
| `architecture-review.html` | Generated interactive architecture guide (2026-03) | Stale generated artifact; describes an early crate/app layout |
| `lora-functionality.html` | Generated LoRa/mesh guide (2026-04) | Predates X3DH/double-ratchet, Reticulum transport, and mesh AI |
| `INSTALL-SCREENS-DESIGN.md` | Installer screen design solicitation | Installer implemented in `image-recipe/` |
| `three-mode-ui-design.md` | Design for the Pro/Easy/Chat three-mode UI | Fully implemented (`stores/uiMode.ts`, `EasyHome.vue`, `Chat.vue`, goals system) |
docs/archive/SESSION-1.8.0-OTA-PROGRESS.md
+344
View File
@@ -0,0 +1,344 @@
# 1.8.0 OTA Session Progress
Updated: 2026-06-30
> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc is kept
> as the historical session-by-session log; open items were consolidated into the
> unified tracker on 2026-07-01 (several turned out already shipped — see that doc for
> current status instead of re-deriving it from the log below).
---
## ▶️▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (evening) — #17 deployed + verified on .198/.228
**#17 (3ccc / stock-peer E2E pill) is now built, deployed, and live-verified** on `.198` and
`.228` only (`.116` skipped per the hardware notice below — its radio is mid-reflash to RNode).
- Built release binary **sha `b1d695fc626a7382`** from the working tree (`cargo check` +
`cargo test -p archipelago mesh::` both green, 99 passed/0 failed/1 ignored, right before
building — tree was settled, no collision with the Reticulum agent's concurrent edits).
- Deployed via stop/swap/start to `.198` (192.168.1.198) and `.228` (192.168.1.228), sha256
confirmed matching on both, `systemctl is-active` = `active` on both (`.228` took its usual
~couple-minute convergence — heavy resilience node, unrelated bitcoind/fedimint container
startup noise in the logs during that window, no mesh errors).
- **Live-verified the actual fix**, not just deploy: on `.198`, `mesh.peers` shows
`"advert_name":"Meshtastic 3ccc", "pkc_capable":true`, and `mesh.send` to 3ccc
(`contact_id:1128152268`) now returns **`"encrypted":true`** — confirms the
`archy || peer_pkc_capable(contact_id)` TX fix is live, not just compiled.
- `.228`'s RPC password in memory (`password123`) was stale — user confirmed the correct
password is `ThisIsWeb54321@` (same as `.198`/`.116`, i.e. fully unified now). Re-verified via
RPC: `mesh.peers` shows 3ccc `pkc_capable:true`, and `mesh.send` to 3ccc returns
`"encrypted":true` — #17 confirmed live on `.228` too, not just `.198`.
**NOT yet done:** push commit to gitea-vps2 (still uncommitted in the working tree, by design —
shares the tree with the Reticulum agent's uncommitted work); user on-device confirmation that
the E2E pill actually renders in the Mesh UI for 3ccc.
---
## 🛠️ HARDWARE NOTICE 2026-06-30 (~16:30) — .116's Heltec V3 is being repurposed
**The Reticulum agent is reflashing .116's Heltec V3 (the board on `/dev/ttyUSB0`, currently
.116's live Meshtastic radio) to RNode firmware**, with explicit user approval, to unblock the
Reticulum Phase-0 hardware gates (real RNode needed; see `docs/RETICULUM-TRANSPORT-PROGRESS.md`).
This was user-confirmed specifically because it takes .116 offline as a Meshtastic radio.
**Effect on this workstream: do all on-device Meshtastic testing on .198 and .228 only — .116 no
longer has a Meshtastic-firmware radio attached once this lands.** `cargo check`/`cargo test
-p archipelago` were both confirmed clean (99/99 mesh tests) right before the reflash started, so
the earlier "wait for their edit to settle" blocker above is cleared — software-side it's safe to
build/test/deploy; only .116's *physical radio role* changed.
---
## ▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (later PM, ~15:50) — READ THIS FIRST IF RESUMING
**#17 (3ccc / stock-peer E2E pill) is CODE-COMPLETE in the working tree**, isolated
to `meshtastic.rs`/`protocol.rs`/`types.rs`/`mod.rs` as planned (no `session.rs`
transport-plumbing changes from this side):
- `ParsedContact.pkc_capable` (`protocol.rs`) + `MeshPeer.pkc_capable` (`types.rs`),
both `#[serde(default)]`/defaulted `false` at every construction site.
- `MeshtasticDevice::get_contacts()` now stamps `pkc_capable` per contact from the
existing `peer_is_pkc_capable(node_num)` seam (de-`allow(dead_code)`'d).
- `listener/session.rs::refresh_contacts` ORs the new value into `MeshPeer.pkc_capable`
(capability only grows, never cleared by a transient refresh) — this IS a touch of
session.rs, but additive/non-colliding with the Reticulum device-enum match arms
already there; did not touch transport plumbing/routing.
- `mod.rs::MeshService::send_message` now does `archy || self.peer_pkc_capable(contact_id)`
for the Sent-row `encrypted` flag (was `archy`-only before).
- Verified via `cargo check -p archipelago --bin archipelago` (clean, exit 0) **before**
the other agent's latest edit landed.
**NOT YET DONE:** rebuild release binary → redeploy 5 nodes → push → user on-device test
(same as #16, both still pending live verification).
**⚠️ BLOCKED right now — do not build/deploy/push until this clears:** the Reticulum
agent is actively mid-edit in the *same* working tree. A `cargo test` run right after
the clean `cargo check` above failed with a real (but transient, not mine) signature
mismatch: `session.rs::auto_detect_and_open` / `run_mesh_session` were observed with a
new `device_kind: Option<DeviceType>` param that `listener/mod.rs`'s call site didn't
have yet — a normal in-flight snapshot of their work, not a regression to fix here.
**Action on resume: re-run `cargo check` first; if it's clean, the other agent's edit
has settled and it's safe to proceed to build/test/deploy. If still broken, wait —
do not stash, revert, or patch their in-progress session.rs/listener/mod.rs changes**
(see memory `feedback_concurrent_agent_tree.md`). Also: building/deploying right now
would bundle their not-yet-finished `reticulum.rs` wiring into the binary — confirm
with the user before shipping a combined build, since only the meshtastic `#17` piece
has been asked for/owned by this session.
---
## ▶️▶️ LIVE CHECKPOINT 2026-06-30 (late PM) — READ THIS FIRST
**Fleet state:** all **5 test nodes** on binary **`38c456b0bacec3c4`** + frontend
**`Mesh-CAkPgvLo.js`**, `archipelago` active on each:
`.116`, `.198`, `.228` (LAN, archipelago@ + `~/.ssh/archipelago-deploy`),
`100.72.136.5`, `100.89.209.89` (Tailscale, same key — installed this session;
SSH user `archipelago` / pw `ThisIsWeb54321@`; NOPASSWD sudo on all 5).
**Shipped this session (commit `12e7990b` on `main`, pushed to gitea-vps2):**
- ✅ **#16 public-channel routing** — inbound Meshtastic text to `BROADCAST_NUM`
now files under the **public channel thread** (contact_id `u32::MAX - idx`),
attributed to its real sender, instead of polluting per-sender DM threads.
Directed text (`to == our node`) still routes to the DM thread (regression test
`packet_to_inbound_frame_directed_dm_stays_a_contact_message`). `send_channel_text`
now sets `MeshPacket.channel` so archy TX's on channel 0 (public).
Code: `meshtastic.rs` (`packet_to_inbound_frame`, `parse_mesh_packet` to/channel,
`send_channel_text`), `protocol.rs` (`RESP_MESHTASTIC_CHANNEL_TEXT = 0x70`),
`listener/frames.rs` (handler + sender attribution), `Mesh.vue` (`senderLabelFor`).
Tests green (95 mesh tests). **Pending: user on-device test with the radios.**
**Push access:** `main` is a PROTECTED branch on gitea-vps2. Direct push uses the
dedicated **`ai`** account via remote **`gitea-ai`** (`git push gitea-ai main`).
See memory `reference_gitea_ai_push_account.md`.
**Coordination:** another agent owns **Reticulum** (`reticulum-daemon/` + Rust
transport wiring). DO NOT touch `mesh/listener/session.rs` transport plumbing or
`mod.rs` routing in ways that collide. Keep #17 work isolated to `meshtastic.rs`
RX/TX + (if needed) the sent-row encrypted flag.
### ✅ CODE-COMPLETE (not yet deployed/tested live) — #17 (3ccc / stock-peer E2E pill)
Goal: DMs **to and from** a PKC-capable stock peer (3ccc, NodeInfo public_key
key_len=32 confirmed) must show the E2E pill.
- **RX side is already correct:** `parse_mesh_packet` reads `public_key` (field 16)
+ `pki_encrypted` (field 17) per the MeshPacket proto; the directed-DM RX path
promotes to `RESP_CONTACT_MSG_V3_E2E` when `pki_encrypted`. (Verify live.)
- **TX bug (root cause) — FIXED:** `mod.rs::send_message` now records the Sent row
with `encrypted = archy || peer_pkc_capable(contact_id)`. `peer_is_pkc_capable`
(meshtastic.rs) is wired out via `get_contacts()` → `ParsedContact.pkc_capable` →
`refresh_contacts` (session.rs) → `MeshPeer.pkc_capable` → `MeshService::peer_pkc_capable`.
See the LIVE CHECKPOINT at the top of this file for the exact touch points.
- NEXT STEP when resuming: confirm `cargo check` is clean (the other agent's
Reticulum work shares this tree and may be mid-edit — see top checkpoint), then
rebuild → redeploy 5 nodes → push → user test (same pending step as #16).
**Remaining open after #17:** #12 (provisioning robustness — HOLD, session.rs churn
risks reticulum collision), #8 (Device-tab settings panel + reboot button — RPC
`mesh.reboot-radio` already exists), #6 (onboarding modal), #7 (.116 re-verify),
#14 (RSSI/SNR per-contact indicator), #15 (peer-location map, POSITION_APP portnum=3).
---
## ▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST
**Goal:** archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill,
identical in off-grid and normal mode. Test bed = `.116` / `.198` / `.228` (all EU_868).
Don't touch the federation/FIPS path.
### ✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)
VERIFIED: `.198→.228` directed DM → `.228` row `RECEIVED enc=True peer="Arch Optiplex"`.
All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary
**`737b16c3235b`** active on all three. Fix source **COMMITTED as `a57ae388`** on `main`
(not yet pushed to gitea-vps2/origin).
**THE fix (receive stream):** archy ignored `FromRadio.rebooted` (field 8). Every config
write reboots the radio → firmware PhoneAPI resets to `STATE_SEND_NOTHING` and stops
streaming received packets until the client re-sends `want_config`. archy never did →
went deaf to inbound (that's why old messages only arrived after a full restart = fresh
want_config). Fix: handle `FROM_RADIO_REBOOTED` → set `pending_reinit` → re-send
want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and
a pinned `modem_preset=LONG_FAST` so all radios share frequency. Combined with the earlier
E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.
**Open follow-ups:** #A surface received msgs under archy identity in all UI views; #6
device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation;
#12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog;
#14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr);
#15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3
lat/lon). See the resume memory `project_session_resume_2026_06_30_lora.md` for the full
task list.
### (historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix
The **archy software is correct and deployed.** The blocker was at the
**radio/RF layer: the three radios are not hearing each other over the air at all.** No
amount of archy code change will fix that until the radios actually RF-link. **Resume by
testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS
FIRST AT HOME" below.** ← this turned out to be the want_config resubscribe bug above.
### What is DONE and deployed (commit pending — see below)
- **E2E send fix** (`core/archipelago/src/mesh/mod.rs` `send_message`, ~L1542): archy↔archy
plain chat text is now sent as a **native `TEXT_MESSAGE_APP` DM** (firmware PKC-encrypts
it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked
`encrypted=true` so the pill shows. Rich typed msgs still use `send_typed_wire`. This was
the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
- **NEW: software radio-reboot** end-to-end, so a wedged/RX-deaf radio can be rebooted
without physical access (and for the Device-tab settings panel the user requested):
- `meshtastic.rs`: `reboot(seconds)` driver method + `ADMIN_REBOOT_SECONDS_FIELD = 97`
(verified vs meshtastic/protobufs admin.proto — `set_owner=32/set_channel=33/set_config=34`
matched our existing constants, confirming the proto read).
- `listener/mod.rs`: `MeshCommand::RebootRadio { seconds }`.
- `listener/session.rs`: device-enum `reboot()` dispatch (Meshtastic only) + handler arm.
- `mesh/mod.rs`: `MeshService::reboot_radio(seconds)`.
- `api/rpc/mesh/messaging.rs`: `handle_mesh_reboot_radio` → RPC **`mesh.reboot-radio`**
`{seconds?}` (default 2); dispatcher arm in `api/rpc/dispatcher.rs`.
- `cargo check` passes. Built release **sha `ba4aed590027690d`** and DEPLOYED + active on
`.116/.198/.228`. The RPC works (`{"reboot":true,"seconds":2}`).
- ⚠️ **Caveat:** when called, archy logged "Sent Meshtastic radio reboot" but the radio did
**not** visibly reboot afterward (no config re-stream). Either field 97 is still off, or
newer firmware requires an admin session passkey even over local serial, or the USB serial
stayed open through the 2s reboot so no reconnect was logged. **Needs on-device verification.**
### The hard evidence (why "nothing works")
- Directed DM tests `.198→.228` AND `.116→.228` (neither path reflashed): sender logs
`Sent plain native DM dest=30d258436d65 part=1 total=1` and RPC returns `sent:true,
encrypted:true`, but `.228` logs **nothing** — packet never reaches archy from the radio.
- A raw broadcast from `.198` (`mesh.broadcast`) was accepted by its radio but **not heard**
by `.228`/`.116`.
- In an 8-minute window, **all three nodes received 0 inbound OTA packets from any other node.**
Each only logs its OWN once-a-minute `Broadcast Meshtastic NodeInfo advert` + local TX
`field=11` queue-status. `.228 mesh.status` = `messages_received:1` total.
- `.198`'s radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that
**reception is broken on the receivers.** A radio cannot drop a broadcast AND a unicast to
its own node number while config matches, unless it simply isn't on the same airwaves.
- archy provisioning is correct & identical across nodes (read back from device): PRIMARY =
public LongFast (`name="" psk_len=1`), SECONDARY = `archipelago`, region=3 (EU_868). Admin
field constants verified. The send path hands the radio a correct unicast MeshPacket
(`to`=node, want_ack, hop_limit=3, plaintext `decoded` for the firmware to PKC-encrypt).
### PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch
archy only ever writes `region` + `use_preset` and **never explicitly pins `modem_preset`**
(it parses region but not preset; `set_lora_region` relies on the LongFast default). If ANY
radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic
app, or a different factory default after the `.198` reflash), the radios are on **different
airwaves despite identical channel name + region**, and archy would never correct it.
### DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)
Open the **Meshtastic phone app over Bluetooth** (works alongside archy's USB serial) on each
of `.116/.198/.228` and check:
1. Do the 3 nodes **see each other** in the node list (recent "heard")? → if NO, they're not
RF-reaching (preset/freq/antenna/range).
2. Do all 3 show the **same** Modem preset (LongFast), Region (EU_868), Frequency slot, and
the same PRIMARY channel? → any difference = the cause.
This single test separates "archy misconfigures the radios" from "radios physically can't
reach each other."
### THEN — the archy fix to apply (if preset/config differs)
Make archy **authoritatively write the full LoRaConfig** and force re-provision so all radios
converge: in `core/archipelago/src/mesh/meshtastic.rs::set_lora_region` (and its
caller/guard `ensure_lora_region` ~L304), explicitly set `modem_preset = LONG_FAST (0)` as a
field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision
path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new
`mesh.reboot-radio`). Also verify the `mesh.reboot-radio` actually reboots the radio
on-device (the caveat above).
### TEST RECIPE (works on each node)
- RPC helper used this session: a node-side `rpc.sh` that logs in (password
`ThisIsWeb54321@`), grabs the `csrf_token` cookie, echoes it as `X-CSRF-Token`, and POSTs to
`http://127.0.0.1:5678/rpc/v1`. Recreate it or run archy's RPC directly. Methods:
`mesh.peers`, `mesh.status`, `mesh.messages`, `mesh.send {contact_id,message}`,
`mesh.broadcast`, `mesh.reboot-radio {seconds}`.
- **LoRa contact ids:** `.116=1135977788` (prefix `3ca5b543`), `.198=3677050140` (`db2b551c`),
`.228=1129894448` (prefix `30d25843`), stock `3ccc=1128152268`.
- **Link health check (run on each node):** look for inbound `from=Some("!...")` lines in
`journalctl -u archipelago` that are NOT the node's own `Broadcast ... NodeInfo advert`. If
zero across all nodes → RF link is down (the current state).
- **E2E success criteria:** send `.198→.228`, the marker appears in `.228` `mesh.messages` as
an inbound row with `encrypted:true` / `transport:"lora"`, AND `.116↔.228` likewise.
### DEPLOY / BUILD RECIPE
- Build: from `core/`, `CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo
build --release -p archipelago --bin archipelago`. (If `rust-lld: undefined hidden symbol`,
it's incremental cache — `CARGO_INCREMENTAL=0` fixes it.)
- SSH key `~/.ssh/archipelago-deploy` is authorized on `.116/.198/.228`. SSH/UI/RPC password
`ThisIsWeb54321@`. Per node: scp the binary, `sudo systemctl stop archipelago` →
`kill -9 $(pgrep -x archipelago)` → `install -m0755` to `/usr/local/bin/archipelago` →
`systemctl start archipelago`. Verify by `sha256sum` match + `systemctl is-active`.
- **Current deployed sha on all 3 = `ba4aed590027690d`** (the reboot-enabled build).
### Fleet state (as of 2026-06-30 PM)
- All 3 nodes on binary `ba4aed59`, active. Off-grid mode currently OFF (`mesh_only:false`).
- `.198` radio was reflashed to factory `firmware-heltec-v3-2.7.26` (recovered from corrupt
NVS); region EU_868 persists. Its archy identity is NOT re-bound on `.228` (`.228` shows
`.198` as raw radio "Meshtastic 551c", `arch_pubkey_hex` absent) because `.228` hasn't heard
`.198`'s identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
- The radios are powered & each transmitting; they are simply not hearing each other.
### Deferred UI (after LoRa works)
- Device-tab **settings panel** (gear/desktop) — host the "Reboot radio" button there; calls
`mesh.reboot-radio`. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid
buttons) + a `rebootRadio()` method in `neode-ui/src/stores/mesh.ts`. See `Mesh.vue` ~L1484
actions row and `mesh.ts` ~L373 `broadcastIdentity()` pattern.
- Device-onboarding modal (detect plugged-in radio).
---
Current scope:
- Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
- Take over the `3ccc` stock Meshtastic peer bug: LoRa text from `3ccc` to Archipelago `.116` does not surface in `mesh.messages`.
- Keep release-gate fixes already made in this session.
Local gate status so far:
- `cargo test -p archipelago --bin archipelago`: green, 849/849 after Meshtastic fixes.
- `python3 scripts/check-app-catalog-drift.py --release --strict`: green.
- `npm run type-check`: green.
Key changes made so far:
- Added cascade uninstall progress truthfulness assertion to `tests/lifecycle/bats/cascade-uninstall.bats`.
- Fixed release catalog drift filters and regenerated catalog metadata.
- Fixed invalid `apps/fedimint-clientd/manifest.yml` `cpu_limit` schema value.
- Updated stale/tight Rust tests without changing production behavior.
Remaining non-automatable / operational gates:
- Workstream B signing is blocked on the offline `RELEASE_MASTER_MNEMONIC`; code + runbook exist, but the publisher must pin/sign the release-root catalog.
- Phase-3 Quadlet backend rollout is implemented behind `use_quadlet_backends` and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
- `.116` read-only `use-quadlet-backends-install.bats`: 6/6 skip-clean; no backend `.container` units, so Phase-3 is not active on that node.
- Release metadata still says `1.7.99-alpha` in `releases/manifest.json`; changelog top is `v1.8.00-alpha`. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.
Do not discard:
- `core/archipelago/src/mesh/listener/decode.rs`
- `core/archipelago/src/mesh/listener/session.rs`
- `core/archipelago/src/mesh/meshtastic.rs`
3ccc bug current hypothesis:
- The prior attempted Meshtastic fix added a hard stale-packet filter using `rx_time`.
- Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
- That would make live `3ccc` packets look older than 10 minutes and get dropped before `mesh.messages`.
- Current patch treats implausibly early `rx_time` values as unknown rather than stale.
.116 live validation after 2026-06-30 hotfix:
- `.116` reachable by SSH; `archipelago` active; `/dev/mesh-radio -> ttyUSB0` attached.
- Current canary deploy is commit `b4531bb4`; backend sha
`4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf`; frontend bundle
`index-YOAeJF7w.js` / `Mesh-BSAo88jN.js`.
- `main` pushed to `gitea-vps2`.
- RPC on `.116`:
- `transport.status` currently reports `mesh_only:false` (off-grid mode is not enabled unless
the user toggles it).
- `mesh.status` reports Meshtastic connected: `device_type:"meshtastic"`,
`self_node_id:1135977788`, `peer_count:13`.
- Recent `.116` -> `3ccc` sent rows are stored with real 2026 timestamps and `transport:"lora"`.
- UI/backend fixes included in `b4531bb4`:
- `transportLabel("lora")` displays **LoRa**.
- mesh sends refetch messages after send so transport pills settle without browser refresh.
- off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends;
banner text is `Tor/FIPS disabled - LoRa only`.
- empty mesh-chat placeholder opacity reduced.
- Meshtastic diagnostics now identify the remaining blocker:
- 3ccc NodeInfo is discovered:
`Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32`.
- Bytes from stock Meshtastic text reach `.116`, but the custom parser rejects the packet:
`Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed`.
- Non-text packets decode and are ignored with port numbers (`portnum=3/4/5`), so the serial
read path is alive. Resume inside `core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet`.
- LoRa is therefore **not fully fixed** yet: stock `3ccc` -> `.116` text does not surface in
`mesh.messages`, and `.116` -> `3ccc` still needs user-visible confirmation in the Meshtastic app.
docs/archive/app-registry-status-2026-06-21.md
+153
View File
@@ -0,0 +1,153 @@
# Archipelago App Registry — Status Survey
**Generated:** 2026-06-21 · **Survey node:** .228 (archi resilience node, 14-app) · **Binary:** v1.7.99-alpha
This document inventories every app in the registry and reports, per app:
manifest-based or not · installed on .228 · migration status (Quadlet/legacy) ·
automated test coverage / release-gate status.
---
## 1. Architecture context — "manifest-based or not"
**Every registry app is manifest-based.** That is the core architecture
(Pillar 4, *data-driven apps*): install/uninstall needs only the app's
`manifest.yml` + catalog entry — no host OS changes, no archipelago binary code
per app. The live registry on .228 is **40 loaded manifests**
(`Loaded 40 app manifest(s) from disk`).
The **only** non-manifest runtime units are:
- **4 companions** — `archy-bitcoin-ui`, `archy-lnd-ui`, `archy-electrs-ui`,
`archy-fedimint-ui`. Built from `docker/<name>` contexts via
`core/archipelago/src/container/companion.rs`, *not* the manifest registry.
- **Stack sub-containers** — `immich_*`, `indeedhub-*`, `netbird-*`. Spawned by
their parent manifest app.
---
## 2. Migration status (Quadlet-everywhere — Pillar 1)
"Migrated" = runs as a **Quadlet unit under `user.slice`**, so it survives an
`archipelago.service` restart (legacy in-cgroup containers get SIGKILLed on
restart and reconciled back).
On .228 migration is **effectively complete** — every installed app is
`QUADLET:running` **except one**:
| Status | Apps |
|---|---|
| ✅ Migrated (Quadlet / user.slice) | bitcoin-knots, electrumx, lnd, fedimint, fedimint-clientd, fedimint-gateway, btcpay-server (+archy-btcpay-db, archy-nbxplorer), mempool, mempool-api, archy-mempool-db, indeedhub (+7 sub-containers), netbird (+server, +dashboard), vaultwarden, jellyfin, filebrowser, portainer, botfights, nostr-rs-relay, homeassistant, + 4 companions |
| ⚠️ NOT migrated (legacy, service cgroup) | **immich_server** — still in `/system.slice/archipelago.service`. The only legacy holdout. (`immich_postgres`/`immich_redis` are pod members.) |
---
## 3. Exhaustive per-app registry table
| App (registry id) | Manifest | Installed on .228 | Migration | Test coverage |
|---|---|---|---|---|
| bitcoin-knots | yes | ✅ | QUADLET | **L1 RPC ●**, L2 UI ● |
| bitcoin-core | yes | ✗ (shares knots) | — | ◐ regression-gate |
| lnd | yes | ✅ | QUADLET | **L1 RPC ●**, L2 ● |
| electrumx | yes | ✅ | QUADLET | **L1 RPC ●**, L2 ● |
| btcpay-server | yes | ✅ | QUADLET | **L1 RPC ●**, L2 ● |
| mempool | yes | ✅ | QUADLET | **L1 RPC ●**, L2 ● |
| mempool-api | yes | ✅ | QUADLET | via mempool stack |
| archy-mempool-db | yes | ✅ | QUADLET | via mempool stack |
| archy-mempool-web | yes | ✗ | — | via mempool stack |
| archy-btcpay-db | yes | ✅ | QUADLET | via btcpay stack |
| archy-nbxplorer | yes | ✅ | QUADLET | via btcpay stack |
| fedimint (Guardian) | yes | ✅ | QUADLET | L1 ◐ container-only, L2 ● |
| fedimint-clientd | yes | ✅ | QUADLET | none |
| fedimint-gateway | yes | ✅ (this session) | QUADLET | none |
| filebrowser | yes | ✅ | QUADLET | L2 probe-only |
| indeedhub | yes | ✅ | QUADLET | none |
| jellyfin | yes | ✅ | QUADLET | none |
| vaultwarden | yes | ✅ | QUADLET | none |
| portainer | yes | ✅ | QUADLET | none |
| botfights | yes | ✅ | QUADLET | none |
| nostr-rs-relay | yes | ✅ | QUADLET | none |
| home-assistant | yes | ✅ (container `homeassistant`) | QUADLET | none |
| netbird | yes | ✅ (+server, +dashboard) | QUADLET | none |
| immich | yes | ✅ | ⚠️ **LEGACY** | none |
| grafana | yes | ✗ (unit *activating*, no container) | staged | none |
| strfry | yes | ✗ (unit *activating*) | staged | none |
| ~~onlyoffice~~ | — | removed 2026-06-21 | — | — |
| aiui | yes | ✗ | — | none |
| core-lightning | yes | ✗ | — | none |
| did-wallet | yes | ✗ | — | none |
| gitea | yes | ✗ | — | none |
| lightning-stack | yes | ✗ | — | none |
| meshtastic | yes | ✗ | — | none |
| morphos-server | yes | ✗ | — | none |
| nextcloud | yes | ✗ | — | none |
| photoprism | yes | ✗ | — | none |
| router | yes | ✗ | — | none |
| searxng | yes | ✗ | — | none |
| uptime-kuma | yes | ✗ | — | none |
| bitcoin-ui | yes | runs as companion `archy-bitcoin-ui` | QUADLET (companion) | L3 companions ● |
| lnd-ui | yes | runs as companion `archy-lnd-ui` | QUADLET (companion) | L3 companions ● |
| electrs-ui | yes | runs as companion `archy-electrs-ui` | QUADLET (companion) | L3 companions ● |
| fips-ui | yes | ✗ | — | none |
Notes:
- `home-assistant` (registry id) runs as container **`homeassistant`** — the
app-id ≠ container-name. A duplicate `home-assistant.service` quadlet unit
sits in *activating*; the live container is `homeassistant` (Up 6 days, healthy).
- `grafana` / `strfry` have Quadlet `.container` units but the units are stuck
*activating* with **no running container** — staged, not live. Worth a
separate investigation.
- `onlyoffice` was **removed from the registry on 2026-06-21**.
---
## 4. Test-gate reality
**No app has passed the formal release gate.** The gate is `run-gate.sh` green
across the full lifecycle matrix (install / UI reachable / stop / start /
restart / reinstall / reboot-survive / archipelago-restart-survive / uninstall),
**5× on .228 AND .198**. All 8 release-gate checkboxes in
`tests/lifecycle/TESTING.md` are **unchecked (☐)**.
What exists today:
| Layer | Status |
|---|---|
| L0 unit | 631 tests ● green |
| L1 RPC | ● for **6 core apps only**: bitcoin-knots, lnd, electrumx, btcpay, mempool, fedimint |
| L2 UI | ● dashboard + 7 proxy paths + bitcoin-ui:8334 |
| L3 lifecycle survival | companions ● ; backends ◐ (regression-gate only — fails until Phase-3 Quadlet flag flips by default) |
| Per-app L1+L2 matrix | **50 of 110 cells** |
| L4 browser / L5 chaos / L6 perf | ○ 0 — not started |
Regression suites added after v1.7.90-alpha (run read-only, abort releases on
failure): `bitcoin-receive.bats`, `port-drift.bats`, `secret-completeness.bats`.
**The other ~30 registry apps have zero automated coverage.**
---
## 5. Key gaps
1. **immich** is the last legacy (in-cgroup) app — migrate to Quadlet to finish Pillar 1.
2. **grafana / strfry** Quadlet units stuck *activating* with no container — investigate. (onlyoffice removed 2026-06-21.)
3. **fedimint-gateway / fedimint-clientd** (this session) now run but have no lifecycle test coverage.
4. The formal **5× release gate has never been green** — it is the blocker for the v1.7.52 tag.
---
## 6. This session's changes (2026-06-21)
- **Generated-secrets system** deployed to .228 (binary + manifests). Self-healing:
the root-owned `fedimint-gateway-hash` was regenerated archipelago-owned/readable
→ **fedimint-gateway now starts** (gatewayd webserver up on :8176). `fmcd-password`
generated for fedimint-clientd.
- **Guardian-UI CSS fix** applied on .228: rebuilt the stale `localhost/fedimint-ui:latest`
companion image (built 2026-06-12, pre-fix) from the corrected context
(`@guardian_assets` proxy fallback to :8177). Guardian's own CSS
(`/assets/bootstrap.min.css`, `/assets/style.css`) **404 → 200 text/css**.
Root cause: `companion.rs::ensure_image_present` skips rebuild when the
`:latest` image already exists, so the context fix never re-baked.
*Survey method: live `podman` cgroup inspection on .228 + `/opt/archipelago/apps`
manifest enumeration + `tests/lifecycle/TESTING.md`.*
docs/archive/architecture-review.html
+2523
View File
File diff suppressed because it is too large Load Diff
docs/archive/demo-deployment-design.md
+169
View File
@@ -0,0 +1,169 @@
# Public Demo Deployment — Design
**Status:** design (2026-06-22)
**Goal:** a public, click-to-play demo of the Archipelago UI that **auto-tracks
the real code** yet stays **separated** from the private monorepo and its
secrets/backend. Deployed via **Portainer**, mock-data driven, with working file
storage and a testnet-flavored Bitcoin sandbox so visitors can play freely.
See also: `neode-ui/mock-backend.js` (existing mock), `docker-compose.demo.yml`
(existing demo stack), `MEMORY → reference_neode_ui_dev_testing`,
`MEMORY → reference_ovh_168_mirror` (Portainer/registry host).
---
## 1. What already exists (the 70%)
The demo is mostly built. Inventory:
| Asset | Path | State |
|-------|------|-------|
| Mock backend (Node/Express + ws) | `neode-ui/mock-backend.js` (~3,862 lines) | 95+ JSON-RPC methods: auth, package lifecycle, Bitcoin/LND wallet, mesh, federation, identity, monitoring, mock filebrowser |
| Mock data | `mockData` / `walletState` / `MOCK_FILES` in `mock-backend.js` | rich; 10 pre-installed apps, 30+ marketplace apps, wallet balances, seeded files (Music/Documents/Photos/Videos) |
| Demo compose | `docker-compose.demo.yml` | `neode-backend` (mock, `:5959`) + `neode-web` (nginx, `:4848`); header already says "Deploy via Portainer" |
| Backend image | `neode-ui/Dockerfile.backend` | Node 22 Alpine → `node mock-backend.js` |
| Web image | `neode-ui/Dockerfile.web` | multi-stage `vite build` → nginx |
| Demo nginx | `neode-ui/docker/nginx-demo.conf` | proxies `/rpc/v1`, `/ws`, `/app/*` to the mock backend |
| Precedent | `indee-demo` Portainer stack | separate stack referencing a **pre-built image** — the pattern we extend |
**Gaps for a *public* (not dev) demo:** state is global (visitors collide),
uploads are no-ops, Bitcoin block height is hardcoded, no CI image pipeline, no
separated public deploy repo.
---
## 2. Architecture: source in monorepo, demo ships as images, public repo is thin
The tension — "must update as I update the real code" **and** "sort of
separated" — is resolved by separating at the **deploy layer, not the source
layer**.
```
monorepo (private — single source of truth)
neode-ui/ + mock-backend.js
│ push to main
▼
CI: build archy-demo-web + archy-demo-backend
│ push :demo / :latest
▼
registry (146.59.87.168:3000 / vps2)
│ Portainer webhook / re-pull
▼
archy-demo (public repo — tiny)
docker-compose.yml ──referencing pre-built images──▶ Portainer ▶ demo.<host>
.env.example
```
- **Single source of truth = the monorepo.** `neode-ui/` and `mock-backend.js`
stay where they are, so the demo tracks real code automatically — no fork to
sync, no drift.
- **Separation = the public repo never holds source.** `archy-demo` contains only
a `docker-compose.yml` (image refs) + `.env.example` + README. No Rust backend,
no secrets, no UI source. Safe to make public.
- **Auto-update flow:** edit code → push → CI rebuilds demo images → Portainer
redeploys. The public compose file is touched rarely (only when service shape
changes).
**Why not a true fork / `git subtree split`?** It works but needs a sync job
*and* re-exposes UI source publicly. The image pipeline gives stronger
separation (zero source leak) **and** zero manual sync. (Decided 2026-06-22.)
---
## 3. Work items
### 3.1 CI image pipeline
- On push to `main` (path filter: `neode-ui/**`), build:
- `archy-demo-backend` from `neode-ui/Dockerfile.backend`
- `archy-demo-web` from `neode-ui/Dockerfile.web` (`build:docker`)
- Tag `:demo` + `:<git-sha>`, push to the registry.
- Trigger Portainer redeploy (stack webhook) on success.
### 3.2 Public `archy-demo` repo
- `docker-compose.yml` mirroring `docker-compose.demo.yml` but **`image:`
references instead of `build:`** (pull `:demo`, no build context).
- `.env.example` (`ANTHROPIC_API_KEY`, `VITE_DEV_MODE=existing`, session TTL,
upload quota).
- README: one-paragraph "deploy in Portainer → web editor paste / deploy from
repo," access on `:4848`.
- No source. This is the only public surface.
### 3.3 Multi-user: per-session sandbox (reset on idle) ⟵ *decided*
The biggest code change. Today `mockData` / `walletState` / `MOCK_FILES` are
**global singletons** → visitors corrupt each other's view.
- Issue a `demo-session` cookie on first hit (the mock already sets a session on
login; extend it to anonymous visitors).
- Key state by session id: `sessions[sid] = { mockData, walletState, files }`,
each **deep-cloned from a pristine seed** on creation.
- Reap on idle (e.g. 30 min no activity) + hard cap concurrent sessions; on reap,
free memory + temp dir.
- RPC dispatch + WS patches resolve the per-session state instead of the global.
- Keeps the demo a true playground: install/uninstall/spend freely, reset by
reconnecting.
### 3.4 File storage: persisted per session ⟵ *decided*
Today filebrowser upload/delete/rename are 200-OK no-ops.
- Back each session with a temp dir (e.g. `/tmp/demo/<sid>/`), seeded from
`MOCK_FILES`.
- Make `POST/DELETE/PATCH /app/filebrowser/api/resources/*` and `GET …/raw/*`
read/write that dir. Enforce a per-session quota (e.g. 50 MB) and reject
oversize/odd MIME.
- Cleaned when the session is reaped — no standing public writable volume, no real
filebrowser container to harden.
### 3.5 Bitcoin: testnet-flavored mock ⟵ *decided*
- Relabel wallet/chain as **testnet/signet**: `tb1q…` addresses, "testnet" chain
in `bitcoin.getinfo`, scripted-but-plausible block height + confirmations.
- Keep `dev.faucet` as the in-UI "get test sats" button (instant, free).
- No real `bitcoind` → no sync, no disk, no public RPC attack surface.
- *Future upgrade path:* swap to a real signet node + LND in the stack if we ever
want movable real test sats (out of scope now).
### 3.6 Mock containers / app lifecycle
- The mock already simulates `package.install/uninstall/start/stop/restart`
asynchronously. For the demo, **force simulation mode** (never touch a real
Docker socket — rootless/safe and host-independent). Confirm no path in
`mock-backend.js` reaches for a real runtime when `DEMO=1`.
### 3.7 Mock-data refresh
- Update `mockData` static apps + marketplace to current app set/versions, refresh
wallet figures, seeded mesh messages, and files so the demo feels current. This
is ongoing and rides the same image pipeline.
---
## 4. Invariants / guardrails (public exposure)
- **No real secrets, no real backend, no real Docker socket** in the demo image or
public repo. Mock password stays a known demo credential, clearly labeled.
- **Per-session isolation** is a hard requirement before going public — without it
the demo is unusable for strangers.
- **Resource caps:** session count, per-session memory + upload quota, idle reap;
the box can't be DoS'd into OOM by upload spam or session churn.
- **`ANTHROPIC_API_KEY`** (chat) is injected via Portainer env, never committed;
rate-limit / budget-cap demo chat usage.
- **Read-only registry creds** for the Portainer host to pull `:demo`.
---
## 5. Files / seams
| Concern | Where |
|---------|-------|
| Per-session state, file persistence, testnet labels, sim-mode | `neode-ui/mock-backend.js` |
| Build contexts (reused as-is) | `neode-ui/Dockerfile.backend`, `neode-ui/Dockerfile.web`, `neode-ui/docker/nginx-demo.conf` |
| Demo stack (in-repo, dev) | `docker-compose.demo.yml` (keep `build:`) |
| Public stack (new repo) | `archy-demo/docker-compose.yml` (`image:` refs), `.env.example`, README |
| CI pipeline | new workflow (path filter `neode-ui/**` → build + push `:demo` → Portainer webhook) |
---
## 6. Open questions
1. **Demo host** — which Portainer instance (OVH `.168`? a dedicated VPS)? Public
DNS + TLS for `demo.<domain>`?
2. **Registry for `:demo` images** — `146.59.87.168:3000` vs vps2; public-pull or
creds baked into Portainer?
3. **Session TTL + concurrency cap** — concrete numbers (30 min / N sessions / 50 MB)?
4. **Chat in the demo** — enable Claude chat (needs key + budget cap) or stub it?
5. **Sync cadence** — rebuild `:demo` on every `neode-ui/**` push, or nightly?
docs/archive/lora-functionality.html
+899
View File
@@ -0,0 +1,899 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Archipelago — LoRa & Mesh Functionality Guide</title>
<style>
:root {
--bg: #000000;
--glass-card: rgba(0, 0, 0, 0.65);
--glass-dark: rgba(0, 0, 0, 0.35);
--glass-darker: rgba(0, 0, 0, 0.6);
--glass-border: rgba(255, 255, 255, 0.18);
--glass-highlight: rgba(255, 255, 255, 0.22);
--glass-blur: 18px;
--glass-blur-strong: 24px;
--shadow-glass: 0 8px 24px rgba(0, 0, 0, 0.45);
--shadow-glass-inset: inset 0 1px 0 rgba(255, 255, 255, 0.22);
--text: rgba(255, 255, 255, 0.9);
--text-muted: rgba(255, 255, 255, 0.6);
--accent: #fb923c;
--accent-dim: rgba(251, 146, 60, 0.15);
--green: #4ade80;
--green-dim: rgba(74, 222, 128, 0.15);
--red: #ef4444;
--red-dim: rgba(239, 68, 68, 0.12);
--blue: #3b82f6;
--blue-dim: rgba(59, 130, 246, 0.12);
--yellow: #facc15;
--yellow-dim: rgba(250, 204, 21, 0.12);
--purple: #a78bfa;
--purple-dim: rgba(167, 139, 250, 0.12);
--radius: 16px;
--radius-sm: 12px;
--transition: 0.3s cubic-bezier(0.4, 0, 0.2, 1);
}
* { margin: 0; padding: 0; box-sizing: border-box; }
html { scroll-behavior: smooth; }
body {
font-family: 'Avenir Next', system-ui, -apple-system, sans-serif;
background: var(--bg);
color: var(--text);
line-height: 1.7;
font-size: 16px;
}
nav {
position: fixed;
top: 0; left: 0;
width: 280px;
height: 100vh;
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border-right: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass);
overflow-y: auto;
padding: 24px 0;
z-index: 100;
scrollbar-width: thin;
scrollbar-color: rgba(255,255,255,0.15) transparent;
}
nav .logo { padding: 0 24px 20px; margin-bottom: 16px; }
nav .logo h1 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 18px; font-weight: 700;
color: var(--accent); letter-spacing: -0.02em;
}
nav .logo p { font-size: 12px; color: var(--text-muted); margin-top: 4px; }
nav .nav-section {
padding: 12px 16px 4px;
font-size: 10px; font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.1em;
color: var(--text-muted);
}
nav a {
display: block;
padding: 6px 24px;
color: var(--text-muted);
text-decoration: none;
font-size: 13px;
transition: all var(--transition);
border-left: 2px solid transparent;
}
nav a:hover, nav a.active {
color: var(--text);
background: rgba(255, 255, 255, 0.06);
border-left-color: var(--accent);
}
main {
margin-left: 280px;
max-width: 960px;
padding: 48px 48px 120px;
}
h2 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 28px; font-weight: 700;
margin: 64px 0 8px;
padding-top: 24px;
color: var(--text);
letter-spacing: -0.02em;
}
h2:first-of-type { margin-top: 0; }
h3 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 20px; font-weight: 600;
margin: 40px 0 12px;
color: var(--text);
}
h4 {
font-size: 16px; font-weight: 600;
margin: 24px 0 8px;
color: var(--accent);
}
p { margin: 8px 0 16px; color: var(--text); }
ul, ol { margin: 8px 0 16px 24px; color: var(--text); }
li { margin: 4px 0; }
.subtitle {
font-size: 15px;
color: var(--text-muted);
margin-bottom: 32px;
}
.hero { text-align: center; padding: 48px 0 56px; margin-bottom: 24px; }
.hero h1 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 42px; font-weight: 800;
background: linear-gradient(135deg, var(--accent), #f59e0b);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
letter-spacing: -0.03em;
}
.hero .tagline {
font-size: 18px;
color: var(--text-muted);
margin: 12px auto 0;
max-width: 640px;
}
.hero .meta {
margin-top: 20px;
display: flex; gap: 16px;
justify-content: center; flex-wrap: wrap;
}
.hero .meta span {
font-size: 12px;
padding: 4px 12px;
border-radius: 999px;
background: var(--glass-dark);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
color: var(--text-muted);
}
.card {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 24px;
margin: 16px 0;
}
.card-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 16px;
margin: 16px 0;
}
.card-sm {
background: var(--glass-darker);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 16px 20px;
transition: transform var(--transition), box-shadow var(--transition);
}
.card-sm:hover {
transform: translateY(-2px);
box-shadow: 0 12px 32px rgba(0, 0, 0, 0.6), inset 0 1px 0 rgba(255, 255, 255, 0.25);
}
.card-sm h4 { margin: 0 0 6px; font-size: 14px; }
.card-sm p { font-size: 13px; color: var(--text-muted); margin: 0; }
.badge {
display: inline-block;
font-size: 11px; font-weight: 600;
padding: 2px 10px;
border-radius: 999px;
text-transform: uppercase;
letter-spacing: 0.05em;
}
.badge-green { background: var(--green-dim); color: var(--green); }
.badge-red { background: var(--red-dim); color: var(--red); }
.badge-yellow { background: var(--yellow-dim); color: var(--yellow); }
.badge-blue { background: var(--blue-dim); color: var(--blue); }
.badge-purple { background: var(--purple-dim); color: var(--purple); }
.badge-accent { background: var(--accent-dim); color: var(--accent); }
table {
width: 100%;
border-collapse: separate;
border-spacing: 0;
margin: 16px 0;
font-size: 13px;
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
overflow: hidden;
box-shadow: var(--shadow-glass);
}
th {
text-align: left;
padding: 10px 14px;
background: rgba(0, 0, 0, 0.4);
color: var(--text-muted);
font-weight: 600;
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.05em;
border-bottom: 1px solid var(--glass-border);
}
td {
padding: 10px 14px;
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
vertical-align: top;
}
tr:last-child td { border-bottom: none; }
tr:hover td { background: rgba(255, 255, 255, 0.04); }
code {
font-family: 'Menlo', 'Monaco', 'Courier New', monospace;
font-size: 13px;
background: rgba(0, 0, 0, 0.4);
padding: 2px 6px;
border-radius: 4px;
color: var(--accent);
}
pre {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
padding: 20px;
overflow-x: auto;
margin: 16px 0;
font-size: 13px;
line-height: 1.6;
box-shadow: var(--shadow-glass);
}
pre code { background: none; padding: 0; color: var(--text); }
.diagram {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 24px;
margin: 20px 0;
overflow-x: auto;
font-family: 'Menlo', 'Monaco', monospace;
font-size: 13px;
line-height: 1.5;
color: var(--text-muted);
white-space: pre;
}
.diagram .highlight { color: var(--accent); font-weight: 600; }
.diagram .green { color: var(--green); }
.diagram .blue { color: var(--blue); }
.diagram .red { color: var(--red); }
.diagram .purple { color: var(--purple); }
.callout {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
padding: 16px 20px;
margin: 16px 0;
font-size: 14px;
border-left: 3px solid;
box-shadow: var(--shadow-glass);
}
.callout-info { border-color: var(--blue); }
.callout-warn { border-color: var(--yellow); }
.callout-danger { border-color: var(--red); }
.callout-success { border-color: var(--green); }
.callout-learn {
border-color: var(--purple);
background: rgba(167, 139, 250, 0.06);
position: relative;
padding-top: 32px;
}
.callout-learn::before {
content: 'Layman Analogy';
position: absolute;
top: 10px; left: 20px;
font-size: 10px;
font-weight: 700;
letter-spacing: 0.1em;
text-transform: uppercase;
color: var(--purple);
}
.callout strong { display: block; margin-bottom: 4px; }
.score-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
gap: 12px;
margin: 20px 0;
}
.score-card {
background: var(--glass-darker);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 16px;
text-align: center;
transition: transform var(--transition);
}
.score-card:hover { transform: translateY(-2px); }
.score-card .score {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 28px; font-weight: 800;
margin: 4px 0;
color: var(--accent);
}
.score-card .label {
font-size: 11px;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.05em;
}
hr {
border: none;
border-top: 1px solid var(--glass-border);
margin: 48px 0;
}
@media (max-width: 900px) {
nav { display: none; }
main { margin-left: 0; padding: 20px; }
.hero h1 { font-size: 32px; }
}
</style>
</head>
<body>
<nav>
<div class="logo">
<h1>Archipelago</h1>
<p>LoRa &amp; Mesh Guide</p>
</div>
<div class="nav-section">Overview</div>
<a href="#intro">Introduction</a>
<a href="#layman">What is LoRa?</a>
<a href="#why">Why Archipelago uses it</a>
<div class="nav-section">Stack</div>
<a href="#hardware">Hardware &amp; Firmware</a>
<a href="#serial">USB Serial Transport</a>
<a href="#wire">Wire Format</a>
<a href="#crypto">Encryption Layers</a>
<a href="#fragmentation">Fragmentation</a>
<div class="nav-section">Routing</div>
<a href="#dual-transport">Dual Transport</a>
<a href="#addressing">Addressing</a>
<a href="#synthetic">Federation Contacts</a>
<div class="nav-section">Messages</div>
<a href="#msg-overview">All 23 Types</a>
<a href="#msg-text">Text / Reply / Edit</a>
<a href="#msg-social">Reactions &amp; Receipts</a>
<a href="#msg-content">Content / Files</a>
<a href="#msg-bitcoin">Bitcoin &amp; Lightning</a>
<a href="#msg-safety">Alerts &amp; Presence</a>
<a href="#msg-identity">Identity &amp; Keys</a>
<div class="nav-section">Operations</div>
<a href="#rpc">RPC API</a>
<a href="#ui">User Interface</a>
<a href="#listener">Listener Loop</a>
<a href="#files">File Map</a>
</nav>
<main>
<section class="hero">
<h1>LoRa &amp; Mesh Functionality</h1>
<p class="tagline">How Archipelago sends encrypted messages, Bitcoin transactions, and emergency alerts over long-range radio when the internet is gone.</p>
<div class="meta">
<span>Meshcore Companion USB</span>
<span>Double Ratchet E2E</span>
<span>23 Message Types</span>
<span>160-byte LoRa Frame</span>
</div>
</section>
<h2 id="intro">Introduction</h2>
<p>This document explains Archipelago's mesh subsystem — the code under <code>core/archipelago/src/mesh/</code> that lets nodes talk to each other over <strong>LoRa radio</strong> instead of (or alongside) the internet. It covers every message type, the transport layer that carries it, the cryptography that protects it, and the code paths that glue it all together.</p>
<p>The goal: give you a mental model that works both ways. If you're an engineer, you can read this and know exactly which bytes get put on the wire for a given RPC call. If you're not, the purple "Layman Analogy" boxes translate each piece into familiar metaphors.</p>
<h2 id="layman">What is LoRa? <span class="badge badge-purple">Layman</span></h2>
<div class="callout callout-learn">
<strong>Think of LoRa as a whisper that travels 10 kilometers.</strong>
Normal Wi-Fi is a shout: loud, fast, lots of data, but only a few rooms away. LoRa is the opposite — a tiny, slow whisper that can cross an entire city because it's so narrow and patient that it slips through walls, trees, and hills. The tradeoff: you can only whisper about <strong>160 bytes</strong> at a time, and each whisper takes a second or two to complete.
</div>
<p>Technically, LoRa (Long Range) is a proprietary radio modulation by Semtech that uses <em>chirp spread spectrum</em> (CSS). It operates in unlicensed ISM bands (915 MHz in the Americas, 868 MHz in Europe) and trades bandwidth for sensitivity, allowing receivers to decode signals below the noise floor. Typical line-of-sight range is 5–15 km with a simple antenna; data rates are 0.3–50 kbps.</p>
<p>Archipelago does not talk to a LoRa chipset directly. Instead it delegates to a small USB-attached device running <strong>Meshcore firmware</strong>, which handles the radio, the mesh routing, and the store-and-forward queue. Archipelago speaks to that device over USB serial.</p>
<h2 id="why">Why Archipelago uses it</h2>
<div class="card-grid">
<div class="card-sm">
<h4>Off-grid safety</h4>
<p>Dead-man switch and emergency alerts reach family without cell coverage.</p>
</div>
<div class="card-sm">
<h4>Censorship resistance</h4>
<p>No ISP, no DNS, no TLS termination — just radio waves between nodes.</p>
</div>
<div class="card-sm">
<h4>Bitcoin when internet is down</h4>
<p>Relay signed transactions and Lightning payments through on-grid peers.</p>
</div>
<div class="card-sm">
<h4>Truly peer-to-peer chat</h4>
<p>Text, replies, reactions, read-receipts — Telegram-quality UX, zero servers.</p>
</div>
</div>
<hr>
<h2 id="hardware">Hardware &amp; Firmware</h2>
<p>Archipelago expects a Meshcore-compatible radio board plugged into USB. The firmware handles RF, mesh forwarding, and contact management; Archipelago handles encryption, message types, and UI.</p>
<table>
<thead><tr><th>Component</th><th>Role</th><th>Examples</th></tr></thead>
<tbody>
<tr><td><strong>MCU</strong></td><td>Runs Meshcore firmware, talks USB serial</td><td>ESP32, nRF52840</td></tr>
<tr><td><strong>Radio</strong></td><td>Semtech LoRa transceiver</td><td>SX1262, SX1276</td></tr>
<tr><td><strong>Board</strong></td><td>MCU + radio + USB + antenna</td><td>Heltec V3, T-Beam, RAK WisBlock, Station G2</td></tr>
<tr><td><strong>Firmware</strong></td><td>Mesh routing + Companion USB protocol</td><td>Meshcore</td></tr>
<tr><td><strong>Connection</strong></td><td>USB CDC-ACM serial</td><td><code>/dev/mesh-radio</code> (udev symlink), <code>/dev/ttyUSB*</code>, <code>/dev/ttyACM*</code></td></tr>
<tr><td><strong>Link params</strong></td><td>115200 baud, 8N1</td><td>Set in <code>mesh/serial.rs</code></td></tr>
</tbody>
</table>
<div class="callout callout-learn">
<strong>It's a modem.</strong> Exactly like a 56k modem from the '90s plugged into your serial port, except the other end of the wire is a radio mesh network instead of a phone line. Archipelago tells it "send this to contact X", and it figures out which radios to hop through.
</div>
<h2 id="serial">USB Serial Transport</h2>
<p>Every byte in and out of the radio is wrapped in a framed serial protocol. The host speaks with <code>'&lt;'</code> and listens for <code>'&gt;'</code>.</p>
<div class="diagram">Host → Device: <span class="highlight">0x3C</span> '&lt;' │ <span class="blue">len_lo len_hi</span> │ <span class="green">frame_bytes...</span>
Device → Host: <span class="highlight">0x3E</span> '&gt;' │ <span class="blue">len_lo len_hi</span> │ <span class="green">frame_bytes...</span>
Baud: 115200 Framing: 8N1 Source: mesh/serial.rs</div>
<p>The frame body is a Meshcore <em>Companion</em> command or response. Archipelago builds these in <code>mesh/protocol.rs</code> and parses replies in <code>mesh/listener/decode.rs</code>.</p>
<h3>Companion commands Archipelago uses</h3>
<table>
<thead><tr><th>Code</th><th>Name</th><th>Purpose</th></tr></thead>
<tbody>
<tr><td><code>0x01</code></td><td>APP_START</td><td>Handshake; device returns its node_id and name</td></tr>
<tr><td><code>0x02</code></td><td>SEND_TXT_MSG</td><td>Send payload to a contact (targeted by 6-byte pubkey prefix)</td></tr>
<tr><td><code>0x03</code></td><td>SEND_CHANNEL_TXT_MSG</td><td>Broadcast on a channel (no specific recipient)</td></tr>
<tr><td><code>0x04</code></td><td>GET_CONTACTS</td><td>Pull the device's contact table</td></tr>
<tr><td><code>0x06</code></td><td>SET_DEVICE_TIME</td><td>Sync Unix timestamp for message dating</td></tr>
<tr><td><code>0x07</code></td><td>SEND_SELF_ADVERT</td><td>Broadcast our identity onto the mesh</td></tr>
<tr><td><code>0x08</code></td><td>SET_ADVERT_NAME</td><td>Set our display name</td></tr>
<tr><td><code>0x0A</code></td><td>SYNC_NEXT_MESSAGE</td><td>Pop the next queued inbound message</td></tr>
<tr><td><code>0x0B</code></td><td>SET_RADIO_PARAMS</td><td>Frequency, spreading factor, bandwidth</td></tr>
<tr><td><code>0x0C</code></td><td>SET_RADIO_TX_POWER</td><td>Transmit power (dBm)</td></tr>
<tr><td><code>0x38</code></td><td>GET_STATS</td><td>Device statistics</td></tr>
</tbody>
</table>
<h3>Responses and push notifications</h3>
<p>Responses begin with a status byte. Codes <code>&lt; 0x80</code> are replies to a command we sent; codes <code>&gt;= 0x80</code> are asynchronous push events from the device.</p>
<table>
<thead><tr><th>Code</th><th>Name</th><th>Meaning</th></tr></thead>
<tbody>
<tr><td><code>0x00</code></td><td>RESP_OK</td><td>Command accepted</td></tr>
<tr><td><code>0x01</code></td><td>RESP_ERR</td><td>Command failed + error code</td></tr>
<tr><td><code>0x03</code></td><td>RESP_CONTACT</td><td>One contact entry (32-byte pubkey + metadata)</td></tr>
<tr><td><code>0x05</code></td><td>RESP_SELF_INFO</td><td>Our node_id and name after APP_START</td></tr>
<tr><td><code>0x10</code></td><td>RESP_CONTACT_MSG_V3</td><td>Direct inbound message (SNR + sender prefix + payload)</td></tr>
<tr><td><code>0x11</code></td><td>RESP_CHANNEL_MSG_V3</td><td>Channel broadcast inbound</td></tr>
<tr><td><code>0x83</code></td><td>PUSH_MESSAGES_WAITING</td><td>Async: new messages in queue, call SYNC_NEXT_MESSAGE</td></tr>
</tbody>
</table>
<h2 id="wire">Wire Format — the payload byte 0</h2>
<p>Once a frame reaches the message payload, Archipelago looks at the <strong>first byte</strong> to decide what kind of thing it's dealing with. This single-byte marker is the master switch of the entire mesh protocol.</p>
<div class="diagram"><span class="highlight">0x00</span> Plain text (legacy, unencrypted)
<span class="highlight">0x01</span> Identity broadcast (ARCHY:2 / ARCHY:3)
<span class="highlight">0x02</span> Typed CBOR envelope (plaintext, used for debug or intra-LAN)
<span class="highlight">0xEE</span> Encrypted typed — ChaCha20-Poly1305 w/ static shared secret
<span class="highlight">0xDD</span> Ratcheted typed — Double Ratchet, forward-secure</div>
<p>Markers <code>0xEE</code> and <code>0xDD</code> are the interesting ones — they carry real production traffic. Everything else is either debug or identity bootstrap.</p>
<h3>0xEE — static-key encrypted envelope</h3>
<pre><code>[0xEE] [nonce: 12 bytes] [ciphertext...] [auth tag: 16 bytes]</code></pre>
<ul>
<li>Key: X25519 ECDH between our Ed25519 identity (converted) and the peer's.</li>
<li>Cipher: ChaCha20-Poly1305 AEAD.</li>
<li>Max plaintext: <code>160 − 1 − 12 − 16 = 131</code> bytes (see <code>crypto::MAX_ENCRYPTED_PLAINTEXT</code>).</li>
<li>Properties: confidential + authenticated, <em>but</em> compromise of a key decrypts all history.</li>
</ul>
<h3>0xDD — Double Ratchet envelope</h3>
<pre><code>[0xDD] [RatchetHeader: 40 bytes] [nonce: 12] [ciphertext] [tag: 16]</code></pre>
<ul>
<li>Per-message keys derived via DH ratchet + symmetric-key ratchet (HKDF-SHA256).</li>
<li>Handles out-of-order delivery via a skipped-keys cache.</li>
<li>Properties: forward secrecy + post-compromise recovery. Used for <code>mesh.*</code> chat once a session is established.</li>
<li>Implementation: <code>mesh/ratchet.rs</code>, session load/save in <code>mesh/listener/session.rs</code>.</li>
</ul>
<div class="callout callout-learn">
<strong>Static key vs. ratchet = a safe vs. a self-shredding envelope.</strong>
The <code>0xEE</code> lane is like a locked safe: one key opens everything. The <code>0xDD</code> lane is like handing your friend a new envelope each time, and burning the old one — so even if someone steals next week's key, they can't read last week's messages.
</div>
<h2 id="crypto">Encryption Layers</h2>
<p>Three cryptographic primitives combine to produce the <code>0xDD</code> ratchet flow:</p>
<div class="card-grid">
<div class="card-sm">
<h4>X25519 ECDH</h4>
<p>Each Double Ratchet step generates a fresh keypair. Peers mix the new shared secret into the chain.</p>
</div>
<div class="card-sm">
<h4>HKDF-SHA256</h4>
<p>Derives root key, chain key, and message key at each ratchet step.</p>
</div>
<div class="card-sm">
<h4>ChaCha20-Poly1305</h4>
<p>Symmetric AEAD used for the actual payload encryption + authentication tag.</p>
</div>
</div>
<h3>Session bootstrap — X3DH-like handshake</h3>
<p>Before the ratchet can start, peers exchange a <strong>PrekeyBundle</strong> (type 5) and a <strong>SessionInit</strong> (type 6). Those two messages are carried by the <code>0xEE</code> static-key envelope, because the ratchet session doesn't exist yet. Once <code>SessionInit</code> is processed, subsequent traffic switches to <code>0xDD</code>. See <code>mesh/x3dh.rs</code>.</p>
<h2 id="fragmentation">Fragmentation — how a 500-byte message rides a 160-byte pipe</h2>
<p>The LoRa frame budget is <strong>160 bytes</strong> (<code>protocol::MAX_MESSAGE_LEN</code>). Subtract the marker, nonce, ratchet header, and tag and you end up with ~90 usable plaintext bytes per frame. Anything bigger gets chunked.</p>
<div class="diagram"><span class="highlight">Chunk header</span> ┌──────────┬──────────┬────────────┐
│ type (1) │ id (1) │ total (1) │
└──────────┴──────────┴────────────┘
<span class="highlight">Chunk body</span> Up to 140 bytes of Base64-encoded payload
Sender: compress → encrypt → split into 140-char chunks
→ send with tiny inter-chunk delay
Receiver: accumulate by (sender, chunk_id) → reassemble
→ decrypt → decompress → dispatch</div>
<p>For chat messages shorter than 160 bytes, none of this kicks in — the whole thing fits in one frame. For larger payloads (long messages, forwarded content, PSBTs), the sender splits and the receiver joins.</p>
<div class="callout callout-info">
<strong>Escape hatch: federation fallback.</strong> If a peer is a synthetic federation contact and the message is bigger than 160 bytes, Archipelago <em>skips LoRa entirely</em> and routes the message over Tor federation instead. See the <code>ContentRef</code> path in <code>rpc/mesh/typed_messages.rs</code>.
</div>
<h2 id="dual-transport">Dual Transport — LoRa + Tor federation</h2>
<p>Archipelago treats LoRa and Tor federation as <strong>two lanes of the same highway</strong>. A single chat window may receive some messages over radio and others over onion routing, and the UI doesn't distinguish. The mesh module picks the lane per-message based on the peer type and payload size.</p>
<div class="diagram"> ┌──────────────────┐
│ mesh.send(...) │
└────────┬─────────┘
│
┌──────────┴──────────┐
│ Is peer synthetic? │
└──────────┬──────────┘
No │ Yes
┌──────────┘ └──────────┐
▼ ▼
<span class="highlight">LoRa radio</span> <span class="blue">Tor federation</span>
(160-byte frame) (unlimited, slower setup)
│ │
│ if &gt; 160 B &amp;&amp; synth ──────┘ (fallback)
▼
Chunked over LoRa
or refused if no fallback</div>
<h2 id="addressing">Addressing</h2>
<ul>
<li><strong>Contact ID</strong> — 32-bit handle from Meshcore's contact table. Used by <code>SEND_TXT_MSG</code>.</li>
<li><strong>Pubkey prefix</strong> — first 6 bytes of the peer's Ed25519 public key. Included on the wire so receivers can deduplicate and route replies.</li>
<li><strong>DID / onion</strong> — used for federation peers; synthetic contacts carry the DID so the mesh layer can hand the message to the federation layer.</li>
</ul>
<h2 id="synthetic">Synthetic federation contacts</h2>
<p>To let the chat list show federation peers <em>before</em> any message arrives, Archipelago inserts <strong>synthetic contacts</strong> into the mesh peer list. Their contact IDs live in the upper half of the 32-bit space (<code>≥ 0x8000_0000</code>), derived deterministically from the federation node's Ed25519 pubkey. Collisions with real LoRa contact IDs are impossible by construction.</p>
<hr>
<h2 id="msg-overview">All 23 Message Types</h2>
<p>Every typed message is a CBOR envelope identified by a single <code>MeshMessageType</code> byte. The <strong>Transport</strong> column shows which marker carries it on the wire and which Companion command is used.</p>
<table>
<thead><tr>
<th>ID</th><th>Type</th><th>Purpose</th><th>Marker</th><th>Cmd</th><th>Chunked?</th>
</tr></thead>
<tbody>
<tr><td>0</td><td>Text</td><td>Plain chat message</td><td>0xDD</td><td>0x02</td><td>If &gt;160 B</td></tr>
<tr><td>1</td><td>Alert</td><td>Emergency / dead-man heartbeat</td><td>0xDD</td><td>0x02/0x03</td><td>No (short)</td></tr>
<tr><td>2</td><td>Invoice</td><td>Lightning / BOLT11 invoice</td><td>0xDD</td><td>0x02</td><td>Usually</td></tr>
<tr><td>3</td><td>PsbtHash</td><td>Unsigned tx hash for co-signing</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>4</td><td>Coordinate</td><td>GPS location share</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>5</td><td>PrekeyBundle</td><td>X3DH bootstrap (pre-session)</td><td>0xEE</td><td>0x02</td><td>No</td></tr>
<tr><td>6</td><td>SessionInit</td><td>Initial ratchet message</td><td>0xEE</td><td>0x02</td><td>No</td></tr>
<tr><td>7</td><td>BlockHeader</td><td>Bitcoin block height/hash</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>8</td><td>TxRelay</td><td>Signed Bitcoin tx for on-grid peer to broadcast</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>9</td><td>TxRelayResponse</td><td>txid or error from the relay peer</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>10</td><td>LightningRelay</td><td>BOLT11 to pay via on-grid peer</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>11</td><td>LightningRelayResponse</td><td>payment_hash or error</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>12</td><td>TxConfirmation</td><td>Depth update (1/2/3 confs)</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>13</td><td>Reply</td><td>Quoted reply to a previous message</td><td>0xDD</td><td>0x02</td><td>If long</td></tr>
<tr><td>14</td><td>Reaction</td><td>Emoji reaction on MessageKey</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>15</td><td>ReadReceipt</td><td>"Seen up to MessageKey X"</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>16</td><td>Forward</td><td>Re-forwarded original w/ provenance</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>17</td><td>Edit</td><td>In-place text replacement</td><td>0xDD</td><td>0x02</td><td>If long</td></tr>
<tr><td>18</td><td>Delete</td><td>Tombstone for earlier message</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>19</td><td>ContentRef</td><td>CID of blob held by sender (file/image)</td><td>0xDD</td><td>0x02 or Tor</td><td>Federation fallback</td></tr>
<tr><td>20</td><td>Presence</td><td>Heartbeat + last-activity epoch</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>21</td><td>ChannelInvite</td><td>Group membership announcement</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>22</td><td>ContactCard</td><td>Shareable federation node card</td><td>0xDD</td><td>0x02</td><td>Maybe</td></tr>
</tbody>
</table>
<p>The remaining sections walk through each category and explain both the sender-side code path and what the bytes look like on the air.</p>
<h2 id="msg-text">Text, Reply, Edit, Delete, Forward</h2>
<h3>Text (type 0)</h3>
<p><strong>Sender path.</strong> <code>rpc.mesh.send</code> → <code>typed_messages::send_text</code> → CBOR-encode the <code>Text{body}</code> variant → ratchet-encrypt → prefix <code>0xDD</code> → if under 160 B, send in one <code>SEND_TXT_MSG</code> frame; otherwise split into Base64 chunks and send sequentially with a small inter-frame sleep so the radio doesn't overflow its TX buffer.</p>
<h3>Reply (type 13)</h3>
<p>Same as Text, but the CBOR envelope carries a <code>MessageKey</code> pointing at the parent message (sender pubkey prefix + timestamp). The UI renders a quote banner; the wire cost is ~12 extra bytes.</p>
<h3>Edit (type 17)</h3>
<p>Envelope contains the original <code>MessageKey</code> plus the new body. Receiver updates its local store in-place and tags the entry "edited".</p>
<h3>Delete (type 18)</h3>
<p>Tombstone only: <code>MessageKey</code> with no body. Receivers keep the original bytes but mark the row deleted. Costs ~20 bytes on the wire.</p>
<h3>Forward (type 16)</h3>
<p>Wraps original <code>{sender_name, original_timestamp, body}</code> so the receiver can render "Forwarded from &lt;name&gt;". Because the body is nested, forwards are <em>almost always</em> chunked.</p>
<h2 id="msg-social">Reaction, ReadReceipt, Presence</h2>
<h3>Reaction (type 14)</h3>
<p>Envelope: <code>{target: MessageKey, emoji: String}</code>. Single-frame, single-emoji. Receiver aggregates reactions per MessageKey and shows them as inline chips (see <code>MessageActions</code> in <code>neode-ui</code>).</p>
<h3>ReadReceipt (type 15)</h3>
<p>Envelope: <code>{up_to: MessageKey}</code>. Semantically "I've seen everything up to and including this message." One receipt covers all prior unread, so traffic is O(1) per read burst rather than O(n).</p>
<h3>Presence (type 20)</h3>
<p>Periodic heartbeat carrying <code>{last_activity_epoch}</code>. Broadcast on a channel (<code>SEND_CHANNEL_TXT_MSG</code>, cmd <code>0x03</code>) rather than to a specific peer, so every listener updates their "last seen" indicator in one shot.</p>
<div class="callout callout-learn">
<strong>Like a lighthouse beacon.</strong> Presence doesn't go to anyone in particular — it's a flash that everyone in radio range can see. "I'm still here, last active two minutes ago." Cheap and unaddressed.
</div>
<h2 id="msg-content">ContentRef — files and images without bloating the radio</h2>
<p>LoRa cannot move a 500 KB image. The <code>ContentRef</code> type (19) solves this by sending only a <strong>pointer</strong> — a content ID (CID) plus a tiny thumbnail or description — and letting the receiver fetch the full blob out-of-band over Tor federation.</p>
<div class="diagram">Sender Receiver
────── ────────
store blob locally (CID)
┌──────────────────────┐
│ ContentRef {cid, │ ──ratchet──▶
│ mime, size, │ 0xDD
│ thumb_hash} │ over LoRa
└──────────────────────┘
see CID in chat
click to fetch
┌─────────────────┐
│ rpc.mesh.fetch- │
│ content(cid) │
└────────┬────────┘
▼
federation (Tor)
resolve DID → pull blob</div>
<div class="callout callout-info">
<strong>Resolution bug fix note.</strong> An earlier revision of <code>ContentRef</code> routed the fetch via a name-match on the contact list, which broke when two peers had the same display name. The fix (see commit <code>5f7ebf14</code>) resolves the owning peer by DID and falls back to name-match only if DID lookup fails.
</div>
<h2 id="msg-bitcoin">Bitcoin &amp; Lightning over LoRa</h2>
<p>Archipelago uses the mesh as a <strong>Bitcoin transport of last resort</strong>. Signed transactions travel from an offline signer, through the mesh, to a peer with internet, who then rebroadcasts them to the Bitcoin network and reports back.</p>
<h3>TxRelay (8) → TxRelayResponse (9) → TxConfirmation (12)</h3>
<div class="diagram">Offline signer On-grid relay peer Bitcoin p2p
────────────── ────────────────── ───────────
sign tx
┌─────────────┐
│ TxRelay │ ─ratchet/LoRa▶ decrypt → validate
│ {raw_tx} │ broadcast via bitcoind ───▶ mempool
└─────────────┘ │
▼
┌────────────────────────┐
◀─ratchet│ TxRelayResponse{txid} │
└────────────────────────┘
(or {error})
later, as blocks arrive:
┌────────────────────────┐
◀─ratchet│ TxConfirmation │
│ {txid, depth: 1..3} │
└────────────────────────┘</div>
<p>The binary framing in <code>mesh/bitcoin_relay.rs</code> is intentionally tight — raw binary, not CBOR — to keep a signed 1-input/1-output tx inside one or two 160-byte frames. Confirmation updates are tiny (txid + depth byte) and ride in a single frame.</p>
<h3>LightningRelay (10) → LightningRelayResponse (11)</h3>
<p>Same shape but the payload is a BOLT11 invoice string. The relay peer pays the invoice from its own node and returns <code>payment_hash</code> or an error. Invoices are often long enough to chunk.</p>
<h3>Invoice (2) and PsbtHash (3)</h3>
<p>These are <em>not</em> relays — they're peer-to-peer handoffs. <code>Invoice</code> delivers a BOLT11 to be paid by the recipient. <code>PsbtHash</code> carries just the hash of an unsigned PSBT so the recipient can retrieve the full PSBT out-of-band and co-sign.</p>
<h3>BlockHeader (7)</h3>
<p>Off-grid nodes need a recent block height to avoid being fooled by stale data. A BlockHeader broadcast (sent via <code>SEND_CHANNEL_TXT_MSG</code>) lets anyone in range learn the latest height and hash from any peer with internet. Tiny payload: 4 bytes height + 32 bytes hash.</p>
<h2 id="msg-safety">Alerts, Coordinates, Dead-Man</h2>
<h3>Alert (type 1)</h3>
<p>Envelope: <code>{kind, message, sender_contact_id}</code>. Kinds include <code>Emergency</code> and <code>Deadman</code>. Alerts can be sent direct-to-contact (for family) or channel-broadcast (for community).</p>
<h3>Dead-man switch</h3>
<p>A background task in <code>mesh/alerts.rs</code> sends a <code>Deadman</code> alert on a configurable interval (default 6 hours). If the user doesn't touch the UI within that window, the alert fires automatically and asks chosen recipients to check in. Powered off? The next peer to receive your last heartbeat notices the gap.</p>
<h3>Coordinate (type 4)</h3>
<p>Envelope: <code>{lat, lon, accuracy_m}</code> with lat/lon as fixed-point integers to stay under 16 bytes. Used for off-grid location sharing — hiking, sailing, field ops.</p>
<h3>ChannelInvite (type 21)</h3>
<p>Phase 5 group chat primitive. Announces a new channel and its membership so other nodes can subscribe. Broadcast via <code>SEND_CHANNEL_TXT_MSG</code>.</p>
<h2 id="msg-identity">Identity, PrekeyBundle, ContactCard</h2>
<h3>Identity broadcast (marker 0x01, ARCHY:2/3)</h3>
<p>The handshake. Before any ratchet session exists, a node advertises its Ed25519 public key on the mesh with an identity packet prefixed <code>0x01</code>. This is how peers discover each other. The payload encodes protocol version (<code>ARCHY:2</code> or <code>ARCHY:3</code>) and the raw pubkey. Carried by <code>CMD_SEND_SELF_ADVERT</code> (<code>0x07</code>).</p>
<h3>PrekeyBundle (type 5) and SessionInit (type 6)</h3>
<p>X3DH handshake. <code>PrekeyBundle</code> advertises a signed prekey; <code>SessionInit</code> consumes it to derive the initial ratchet root key. Both ride on <code>0xEE</code> (static-key encryption), because the ratchet session they're creating doesn't yet exist.</p>
<h3>ContactCard (type 22)</h3>
<p>A shareable card containing <code>{did, onion_address, pubkey, display_name}</code>. When a receiver taps "add" on the card, Archipelago one-click federates with that node over Tor. This is the bridge that lets LoRa-discovered peers become full federation contacts.</p>
<hr>
<h2 id="rpc">RPC API — what callers actually invoke</h2>
<p>Every user-facing action goes through the RPC dispatcher (<code>api/rpc/dispatcher.rs</code>, lines 287+) and ends in <code>api/rpc/mesh/typed_messages.rs</code>. The tables below show the public surface.</p>
<h3>Core commands</h3>
<table>
<thead><tr><th>RPC</th><th>Effect</th></tr></thead>
<tbody>
<tr><td><code>mesh.status</code></td><td>Device info, peer count, enabled state</td></tr>
<tr><td><code>mesh.peers</code></td><td>List all discovered peers with RSSI / SNR / hop count</td></tr>
<tr><td><code>mesh.messages</code></td><td>Retrieve stored mesh messages</td></tr>
<tr><td><code>mesh.send</code></td><td>Send plain text to a specific peer</td></tr>
<tr><td><code>mesh.send-channel</code></td><td>Broadcast on a channel</td></tr>
<tr><td><code>mesh.broadcast</code></td><td>Mesh-wide announcement</td></tr>
<tr><td><code>mesh.configure</code></td><td>Set device params (name, power, channel)</td></tr>
<tr><td><code>mesh.debug-dump</code></td><td>Raw state for debugging</td></tr>
</tbody>
</table>
<h3>Rich message commands</h3>
<table>
<thead><tr><th>RPC</th><th>Msg Type</th><th>Notes</th></tr></thead>
<tbody>
<tr><td><code>mesh.send-invoice</code></td><td>Invoice (2)</td><td>Deliver BOLT11 to peer</td></tr>
<tr><td><code>mesh.send-coordinate</code></td><td>Coordinate (4)</td><td>Single frame, fixed-point</td></tr>
<tr><td><code>mesh.send-alert</code></td><td>Alert (1)</td><td>Emergency or deadman</td></tr>
<tr><td><code>mesh.send-content</code></td><td>ContentRef (19)</td><td>Stores blob, sends CID</td></tr>
<tr><td><code>mesh.fetch-content</code></td><td>—</td><td>Pulls blob via federation</td></tr>
<tr><td><code>mesh.send-psbt</code></td><td>PsbtHash (3)</td><td>Hash only, full PSBT via fetch</td></tr>
<tr><td><code>mesh.send-reply</code></td><td>Reply (13)</td><td>Quoted response</td></tr>
<tr><td><code>mesh.send-reaction</code></td><td>Reaction (14)</td><td>Emoji</td></tr>
<tr><td><code>mesh.send-read-receipt</code></td><td>ReadReceipt (15)</td><td>Cumulative "seen up to"</td></tr>
<tr><td><code>mesh.forward-message</code></td><td>Forward (16)</td><td>Wraps original + provenance</td></tr>
<tr><td><code>mesh.edit-message</code></td><td>Edit (17)</td><td>In-place text replacement</td></tr>
<tr><td><code>mesh.delete-message</code></td><td>Delete (18)</td><td>Tombstone</td></tr>
</tbody>
</table>
<h2 id="ui">User Interface</h2>
<p>The Vue side lives under <code>neode-ui/src/views/mesh/</code> with state in <code>stores/mesh.ts</code>. Notable panels:</p>
<div class="card-grid">
<div class="card-sm">
<h4>Mesh chat</h4>
<p>Telegram-style UI with reply banners, inline reaction chips, forward/edit/delete action menu, read-receipts, outbox status.</p>
</div>
<div class="card-sm">
<h4>MeshBitcoinPanel</h4>
<p>UI for TxRelay / LightningRelay submission and confirmation tracking.</p>
</div>
<div class="card-sm">
<h4>MeshDeadmanPanel</h4>
<p>Configure dead-man interval, pick recipients, show last heartbeat time.</p>
</div>
<div class="card-sm">
<h4>Unified inbox</h4>
<p>Federation and mesh chats appear side-by-side; the transport is invisible to the user.</p>
</div>
</div>
<h2 id="listener">Listener loop — how inbound traffic is decoded</h2>
<p>A long-running async task in <code>mesh/listener/mod.rs</code> owns the serial device and feeds events into the rest of the system.</p>
<div class="diagram">loop {
event = await serial_read()
match event {
<span class="green">PUSH_MESSAGES_WAITING</span> → send SYNC_NEXT_MESSAGE until empty
<span class="green">RESP_CONTACT_MSG_V3</span> → decode.rs extracts payload
→ match first byte:
<span class="highlight">0x00</span> plain text
<span class="highlight">0x01</span> identity → frames::parse_identity
<span class="highlight">0x02</span> typed CBOR plaintext
<span class="highlight">0xEE</span> → crypto::decrypt_static
<span class="highlight">0xDD</span> → session::load + ratchet::decrypt
→ dispatch.rs routes typed msg
to chat store / bitcoin relay /
alerts / presence / ...
<span class="green">RESP_CONTACT</span> → contact list update
<span class="green">RESP_SELF_INFO</span> → record our node_id
}
}</div>
<p>Chunk reassembly happens in <code>listener/session.rs</code>, keyed by <code>(sender_pubkey_prefix, chunk_id)</code>. Incomplete chunks expire after a timeout so a lost frame doesn't leak memory.</p>
<h2 id="files">File Map</h2>
<table>
<thead><tr><th>File</th><th>Size</th><th>Role</th></tr></thead>
<tbody>
<tr><td><code>mesh/mod.rs</code></td><td>52 KB</td><td>Public API, send paths, federation integration</td></tr>
<tr><td><code>mesh/protocol.rs</code></td><td>26 KB</td><td>Frame encoding/decoding, command builders</td></tr>
<tr><td><code>mesh/serial.rs</code></td><td>15 KB</td><td>USB driver, device detection, handshake</td></tr>
<tr><td><code>mesh/crypto.rs</code></td><td>10 KB</td><td>X25519 ECDH, ChaCha20-Poly1305, HKDF</td></tr>
<tr><td><code>mesh/ratchet.rs</code></td><td>16 KB</td><td>Double Ratchet implementation</td></tr>
<tr><td><code>mesh/message_types.rs</code></td><td>23 KB</td><td>23 typed message discriminators + CBOR schemas</td></tr>
<tr><td><code>mesh/bitcoin_relay.rs</code></td><td>17 KB</td><td>TxRelay / LightningRelay binary framing</td></tr>
<tr><td><code>mesh/listener/dispatch.rs</code></td><td>29 KB</td><td>Typed-message routing into chat/relay/alerts</td></tr>
<tr><td><code>mesh/listener/session.rs</code></td><td>14 KB</td><td>Ratchet session persistence + chunk reassembly</td></tr>
<tr><td><code>mesh/x3dh.rs</code></td><td>—</td><td>Prekey / SessionInit bootstrap</td></tr>
<tr><td><code>mesh/outbox.rs</code></td><td>—</td><td>Retry queue for unacked sends</td></tr>
<tr><td><code>mesh/steganography.rs</code></td><td>—</td><td>Weather/sensor framing for deniable traffic</td></tr>
<tr><td><code>api/rpc/mesh/typed_messages.rs</code></td><td>—</td><td>All <code>mesh.*</code> RPC handlers</td></tr>
<tr><td><code>neode-ui/src/stores/mesh.ts</code></td><td>14 KB</td><td>Pinia store consumed by all mesh Vue views</td></tr>
</tbody>
</table>
<hr>
<h2>Summary scoreboard</h2>
<div class="score-grid">
<div class="score-card"><div class="score">23</div><div class="label">Message types</div></div>
<div class="score-card"><div class="score">160</div><div class="label">Bytes / frame</div></div>
<div class="score-card"><div class="score">2</div><div class="label">Transports</div></div>
<div class="score-card"><div class="score">5</div><div class="label">Wire markers</div></div>
<div class="score-card"><div class="score">~6k</div><div class="label">LoC in mesh/</div></div>
<div class="score-card"><div class="score">FS</div><div class="label">Forward-secure</div></div>
</div>
<div class="callout callout-success">
<strong>Bottom line.</strong> Archipelago's mesh isn't a chat toy. It's a complete off-grid transport with forward-secure end-to-end encryption, 23 typed message kinds, Bitcoin and Lightning relay, fragmentation, store-and-forward, and a seamless Tor federation fallback. From the user's perspective it looks like iMessage; from the wire's perspective it's a carefully budgeted 160 bytes of ChaCha20 ciphertext riding on a sub-kbps radio link.
</div>
</main>
</body>
</html>
docs/archive/rust-orchestrator-migration.md
+525
View File
@@ -0,0 +1,525 @@
# Rust Orchestrator Migration — Design Doc
Status: **DRAFT — pending user approval**
Author: OpenCode session, 2026-04-22
Supersedes planning in `docs/bulletproof-containers.md` v1.7.43 slot
## Problem statement
Today, the archipelago backend has **no production container orchestrator**. Production containers (bitcoin-knots, lnd, electrumx, btcpay, filebrowser, and the three custom UIs archy-bitcoin-ui / archy-electrs-ui / archy-lnd-ui) are installed by **bash scripts** at first boot (`scripts/first-boot-containers.sh`) and optionally reconciled by another bash script (`scripts/reconcile-containers.sh`) that is **not enabled by default**. The existing `DevContainerOrchestrator` (`core/archipelago/src/container/dev_orchestrator.rs`) is hardcoded to append `-dev` suffixes and gated behind `config.dev_mode`, so it has never managed a production container.
This design migrates production container management into Rust, under a single orchestrator that owns install, start, stop, restart, upgrade, uninstall, health, and self-healing for every container. The three custom UI containers are the first-class test fixture: they exercise the "build image from local Dockerfile" path (which today doesn't exist in the manifest schema) and their lifecycle was the original failure class the user asked to fix.
## Non-goals
- Backwards compatibility with `first-boot-containers.sh`: we **delete** it and its systemd unit after verifying Rust parity.
- Backwards compatibility with the existing `package-install` RPC’s podman shell-outs: those get rewritten to call the orchestrator.
- Registry signature verification: `image_signature` stays optional. Sigstore/cosign integration is out of scope.
- Network isolation improvements: existing SecurityPolicy fields stay as-is.
- Dev mode removal: `DevContainerOrchestrator` keeps existing behavior for local development; prod code path is separate.
## Scope of this migration
In scope:
1. Extend `ContainerConfig` schema with a `source:` variant supporting `{type: build, context, dockerfile, tag}` alongside `{type: pull, image, pull_policy}`.
2. Extend `ContainerRuntime` trait + `PodmanRuntime` impl with `build_image(...)` and `image_exists(...)`.
3. Introduce `ProdContainerOrchestrator` (new type) with identical public surface to `DevContainerOrchestrator` but **no `-dev` suffix**, **no port offset**, **no data-path rewriting**, **no bitcoin_simulator gate**. It is wired into `RpcHandler::orchestrator` in prod (currently `None`).
4. Add `AdoptionScan` at orchestrator startup: enumerate `podman ps -a`, match by container name against declared manifests, adopt into orchestrator state without recreating.
5. Add `BootReconciler` task spawned from `main.rs` (replacing the commented-out `run_boot_reconciliation` hook). Walks the manifest set on startup and periodically, ensures each is present-and-running, builds/pulls/creates anything missing, logs failures non-silently.
6. Ship three manifests in the repo: `apps/bitcoin-ui/manifest.yml`, `apps/electrs-ui/manifest.yml`, `apps/lnd-ui/manifest.yml`. They use the new `source: build` variant pointing at `/opt/archipelago/docker/<name>/`.
7. Delete `scripts/first-boot-containers.sh`, `scripts/reconcile-containers.sh`, `scripts/container-specs.sh`, `image-recipe/configs/archipelago-first-boot-containers.service`, `image-recipe/configs/archipelago-reconcile.service`. Remove enablement from ISO builder.
Out of scope this migration (tracked separately):
- Migrating btcpay / mempool / fedimint multi-container stacks to manifests (they currently live in `core/archipelago/src/api/rpc/package/stacks.rs`). They keep working via `package-install` RPC. Phase 2.
- Rewriting the 26 existing `apps/*/manifest.yml` files to use the new `source:` schema. They stay on `image:` for now; the schema is **additive and backwards-compatible**.
- Re-enabling signature verification; stays todo.
## Data model changes
### 1. `ContainerConfig` gets a `source` enum
File: `core/container/src/manifest.rs:58`
**Before:**
```rust
pub struct ContainerConfig {
pub image: String,
pub image_signature: Option<String>,
pub pull_policy: String,
}
```
**After:**
```rust
pub struct ContainerConfig {
// Legacy shorthand (backwards compatible with all 26 existing manifests):
// if `source` is absent, `image` + `pull_policy` are interpreted as
// `source: { type: pull, image, pull_policy }`.
#[serde(default)]
pub image: String,
#[serde(default)]
pub image_signature: Option<String>,
#[serde(default = "default_pull_policy")]
pub pull_policy: String,
// New: explicit source. If present, overrides the legacy shorthand.
#[serde(default)]
pub source: Option<ContainerSource>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "lowercase")]
pub enum ContainerSource {
/// Pull an image from a registry.
Pull {
image: String,
#[serde(default)]
image_signature: Option<String>,
#[serde(default = "default_pull_policy")]
pull_policy: String,
},
/// Build an image from a local Dockerfile.
Build {
/// Filesystem path to build context, absolute or relative to manifest dir.
context: String,
/// Dockerfile path relative to context. Defaults to "Dockerfile".
#[serde(default = "default_dockerfile")]
dockerfile: String,
/// Tag to assign to the built image, e.g. "localhost/bitcoin-ui:local".
tag: String,
/// `--build-arg` key=value pairs.
#[serde(default)]
build_args: HashMap<String, String>,
/// If true, rebuild on every reconcile. If false, only build when tag is missing.
#[serde(default)]
always_rebuild: bool,
},
}
```
Validation in `AppManifest::validate`:
- If `source` is absent AND `image` is empty → error (unchanged rule just rephrased).
- If `source` is present, legacy `image` field is ignored with a warning.
- `Build::context` must resolve to an existing directory that contains `dockerfile`.
Tests to add:
- Parse a legacy manifest → works, produces `ContainerSource::Pull` at resolution time.
- Parse a `source: { type: build, ... }` manifest → works.
- Parse a manifest with both legacy `image:` and `source:` → warning logged, `source:` wins.
- Parse a manifest with neither → rejected.
### 2. `ContainerRuntime` trait gets `build_image` + `image_exists`
File: `core/container/src/runtime.rs:10`
```rust
#[async_trait]
pub trait ContainerRuntime: Send + Sync {
// existing methods unchanged...
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()>;
async fn create_container(...) -> Result<()>;
// ...
// NEW:
/// Build an image from a local Dockerfile. Returns Ok(()) if the image now
/// exists under the given tag (whether newly built or already present and
/// `force=false`). Returns Err if the build failed.
async fn build_image(
&self,
context: &Path,
dockerfile: &str,
tag: &str,
build_args: &HashMap<String, String>,
force: bool,
) -> Result<()>;
/// Check if an image exists in the local image store.
async fn image_exists(&self, tag: &str) -> Result<bool>;
}
```
`PodmanRuntime::build_image` shells out:
```
podman build --tag <tag> \
--file <context>/<dockerfile> \
--build-arg KEY=VALUE ... \
<context>
```
Force-rebuild semantics: if `force=false`, skip when `image_exists(tag) == true`. If `force=true`, always build (podman's own layer cache handles the fast path).
Tests:
- `build_image` happy path on a minimal Dockerfile (using a throwaway context in tmpdir).
- `build_image` failure path (nonsense Dockerfile) → Err.
- `image_exists` returns false for nonexistent tag.
- `image_exists` returns true after `build_image`.
### 3. Manifest resolution: `ContainerSource::resolve(manifest_dir) -> ResolvedSource`
New method that turns the raw manifest into something the orchestrator can act on:
```rust
pub enum ResolvedSource {
Pull { image: String, signature: Option<String>, pull_policy: PullPolicy },
Build { context: PathBuf, dockerfile: String, tag: String, build_args: HashMap<String,String>, always_rebuild: bool },
}
impl ContainerConfig {
pub fn resolve(&self, manifest_dir: &Path) -> Result<ResolvedSource> {
match &self.source {
Some(ContainerSource::Pull { image, image_signature, pull_policy }) => Ok(ResolvedSource::Pull { ... }),
Some(ContainerSource::Build { context, dockerfile, tag, build_args, always_rebuild }) => {
let abs_context = if Path::new(context).is_absolute() {
PathBuf::from(context)
} else {
manifest_dir.join(context)
};
Ok(ResolvedSource::Build { context: abs_context, ... })
}
None => {
// Legacy shorthand
if self.image.is_empty() {
return Err(...);
}
Ok(ResolvedSource::Pull { image: self.image.clone(), ... })
}
}
}
}
```
## Runtime architecture
### `ProdContainerOrchestrator`
New file: `core/archipelago/src/container/prod_orchestrator.rs`
```rust
pub struct ProdContainerOrchestrator {
runtime: Arc<dyn ContainerRuntimeTrait>,
manifests_dir: PathBuf, // e.g. /opt/archipelago/apps
data_dir: PathBuf, // e.g. /var/lib/archipelago
state: Arc<RwLock<OrchestratorState>>,
config: Config,
}
struct OrchestratorState {
/// app_id → known manifest (loaded from disk at startup, refreshed on reconcile)
manifests: HashMap<String, AppManifest>,
/// app_id → current known state (from adoption scan or our own ops)
containers: HashMap<String, ContainerState>,
/// app_id → last install/health/build timestamp
last_reconciled: HashMap<String, Instant>,
}
```
Public surface mirrors `DevContainerOrchestrator` but **container name = `archy-<app_id>` for UI apps, `<app_id>` for backends, matching existing .116 naming**:
```rust
impl ProdContainerOrchestrator {
pub async fn new(config: Config) -> Result<Self> { ... }
pub async fn load_manifests(&self) -> Result<()> { /* walks manifests_dir */ }
pub async fn adopt_existing(&self) -> Result<AdoptionReport> { /* scans podman ps -a */ }
pub async fn reconcile_all(&self) -> Result<ReconcileReport> { /* ensures every manifest has a running container */ }
pub async fn install(&self, app_id: &str) -> Result<()> { /* build-or-pull + create + start */ }
pub async fn start(&self, app_id: &str) -> Result<()> { ... }
pub async fn stop(&self, app_id: &str) -> Result<()> { ... }
pub async fn restart(&self, app_id: &str) -> Result<()> { ... }
pub async fn remove(&self, app_id: &str, preserve_data: bool) -> Result<()> { ... }
pub async fn upgrade(&self, app_id: &str) -> Result<()> { /* re-read manifest, rebuild/pull, recreate */ }
pub async fn status(&self, app_id: &str) -> Result<ContainerStatus> { ... }
pub async fn list(&self) -> Result<Vec<ContainerStatus>> { ... }
pub async fn logs(&self, app_id: &str, lines: u32) -> Result<Vec<String>> { ... }
pub async fn health(&self, app_id: &str) -> Result<String> { ... }
}
```
**Container naming rule** (matches `.116` existing fixture so adoption works):
- If the manifest has `extensions["container_name"]` → use that verbatim.
- Else if the app_id starts with `bitcoin-ui` / `electrs-ui` / `lnd-ui` → `archy-<app_id>`.
- Else → `<app_id>`.
This is codified and tested; no ad-hoc naming in the codebase.
### `AdoptionScan`
On orchestrator startup, before any reconcile:
```rust
async fn adopt_existing(&self) -> Result<AdoptionReport> {
let all = self.runtime.list_containers().await?; // podman ps -a
let mut report = AdoptionReport::default();
for c in all {
// For each manifest we have loaded, check if the expected container name matches
for (app_id, manifest) in self.state.read().await.manifests.iter() {
let expected_name = compute_container_name(manifest);
if c.name == expected_name {
// This container is ours. Record its state.
self.state.write().await.containers.insert(app_id.clone(), c.state.clone());
report.adopted.push(app_id.clone());
}
}
}
Ok(report)
}
```
No recreate. No touching data volumes. Just "we now know this container belongs to app X and its current state is Y".
### `BootReconciler`
New file: `core/archipelago/src/container/boot_reconciler.rs`
```rust
pub struct BootReconciler {
orchestrator: Arc<ProdContainerOrchestrator>,
interval: Duration, // e.g. 5 minutes
shutdown: CancellationToken,
}
impl BootReconciler {
pub async fn run_forever(self) {
// Initial reconcile immediately (after adoption).
let _ = self.orchestrator.reconcile_all().await;
loop {
tokio::select! {
_ = tokio::time::sleep(self.interval) => {
let _ = self.orchestrator.reconcile_all().await;
}
_ = self.shutdown.cancelled() => break,
}
}
}
}
```
`reconcile_all`:
```rust
async fn reconcile_all(&self) -> Result<ReconcileReport> {
let manifests: Vec<_> = self.state.read().await.manifests.values().cloned().collect();
let mut report = ReconcileReport::default();
for manifest in manifests {
let app_id = &manifest.app.id;
match self.ensure_running(&manifest).await {
Ok(action) => report.record(app_id, action),
Err(e) => {
tracing::error!(app_id, error = %e, "Reconcile failed for app");
report.failures.push((app_id.clone(), e.to_string()));
}
}
}
if !report.failures.is_empty() {
// Surface via WebSocket so the UI can show a banner.
self.notify_failures(&report).await;
}
Ok(report)
}
async fn ensure_running(&self, manifest: &AppManifest) -> Result<ReconcileAction> {
let name = compute_container_name(manifest);
match self.runtime.get_container_status(&name).await {
Ok(status) if matches!(status.state, ContainerState::Running) => Ok(ReconcileAction::NoOp),
Ok(status) if matches!(status.state, ContainerState::Exited | ContainerState::Stopped) => {
self.runtime.start_container(&name).await?;
Ok(ReconcileAction::Started)
}
Ok(_) => Ok(ReconcileAction::NoOp), // Created / Paused — leave alone
Err(_) => {
// Container doesn't exist. Install it.
self.install_fresh(manifest).await?;
Ok(ReconcileAction::Installed)
}
}
}
async fn install_fresh(&self, manifest: &AppManifest) -> Result<()> {
let manifest_dir = ...; // directory of manifest.yml
let resolved = manifest.app.container.resolve(manifest_dir)?;
match resolved {
ResolvedSource::Pull { image, signature, .. } => {
self.runtime.pull_image(&image, signature.as_deref()).await?;
}
ResolvedSource::Build { context, dockerfile, tag, build_args, always_rebuild } => {
if always_rebuild || !self.runtime.image_exists(&tag).await? {
self.runtime.build_image(&context, &dockerfile, &tag, &build_args, always_rebuild).await?;
}
}
}
self.runtime.create_container(manifest, &compute_container_name(manifest), 0).await?;
self.runtime.start_container(&compute_container_name(manifest)).await?;
Ok(())
}
```
### Wire-up in `main.rs`
File: `core/archipelago/src/main.rs`
Replace the commented-out `run_boot_reconciliation` block (`main.rs:107-111`) with:
```rust
// Load manifests + adopt existing + start reconciler loop.
let orchestrator = Arc::new(ProdContainerOrchestrator::new(config.clone()).await?);
orchestrator.load_manifests().await?;
let adoption = orchestrator.adopt_existing().await?;
tracing::info!(adopted = adoption.adopted.len(), "Container adoption complete");
let reconciler = BootReconciler::new(orchestrator.clone(), Duration::from_secs(300), shutdown_token.clone());
tokio::spawn(reconciler.run_forever());
```
`RpcHandler` gets the orchestrator regardless of `dev_mode`:
```rust
// core/archipelago/src/api/rpc/mod.rs:83
let orchestrator: Option<Arc<dyn ContainerOrchestrator>> = if config.dev_mode {
Some(Arc::new(DevContainerOrchestrator::new(config.clone()).await?))
} else {
Some(Arc::new(prod_orch.clone()))
};
```
Where `ContainerOrchestrator` becomes a trait implemented by both `DevContainerOrchestrator` and `ProdContainerOrchestrator`.
### First-boot replacement
There is no separate first-boot code. The reconciler handles it: when the archipelago service starts on a fresh node, `adopt_existing` finds nothing, `reconcile_all` sees no running container for any manifest, and installs each one in dependency order (bitcoin-core first, then everything else). On subsequent boots, adoption finds existing containers and reconcile mostly no-ops.
**Removes completely**:
- `/var/lib/archipelago/.first-boot-containers-done` marker (no longer needed)
- `/var/lib/archipelago/.unbundled` handling in first-boot script (becomes a config flag in archipelago.conf if we still need it)
- `scripts/first-boot-containers.sh` (1392 lines)
- `scripts/reconcile-containers.sh`
- `scripts/container-specs.sh`
- `image-recipe/configs/archipelago-first-boot-containers.service`
- `image-recipe/configs/archipelago-reconcile.service`
- Related enable/disable in ISO builder
## The three UI manifests
Example: `apps/bitcoin-ui/manifest.yml`
```yaml
app:
id: bitcoin-ui
name: Bitcoin Knots UI
version: 1.0.0
description: Custom Archipelago UI for Bitcoin Knots
container:
source:
type: build
context: /opt/archipelago/docker/bitcoin-ui
dockerfile: Dockerfile
tag: localhost/bitcoin-ui:local
build_args:
BITCOIN_RPC_AUTH: ${BITCOIN_RPC_AUTH} # injected from host-ip.env or secrets
always_rebuild: false
dependencies:
- app_id: bitcoin-core
resources:
memory_limit: 128Mi
security:
network_policy: host
readonly_root: false
ports: [] # host networking
volumes: []
environment: []
health_check:
type: http
endpoint: http://127.0.0.1:8334
path: /
interval: 30s
extensions:
container_name: archy-bitcoin-ui
```
The `extensions.container_name` is how we match the existing running container on .116 for adoption. Same pattern for `electrs-ui` (container_name: `archy-electrs-ui`, port probe 50002) and `lnd-ui` (container_name: `archy-lnd-ui`, port probe 8081).
**BITCOIN_RPC_AUTH injection**: today `first-boot-containers.sh` `sed`s this value into `nginx.conf` (destructively). In the new world, it's a `--build-arg` — the Dockerfile gets `ARG BITCOIN_RPC_AUTH` and templates `nginx.conf` from a template file. Fixes the "sed destroys the source" bug from the mapping.
## Migration path (.116 and .228 specifically)
### .116 (all 3 UIs currently running, adopted from bash install)
1. Ship the new archipelago binary with the prod orchestrator.
2. On archipelago restart, `adopt_existing` scans `podman ps -a`, sees `archy-bitcoin-ui`, `archy-electrs-ui`, `archy-lnd-ui` already running.
3. Matches them against the new manifests by `extensions.container_name`.
4. Records state. Reconciler sees them Running → NoOp.
5. Manual test: `podman stop archy-bitcoin-ui` → within 5 minutes, reconciler starts it again. `podman rm -f archy-bitcoin-ui` → reconciler rebuilds from `/opt/archipelago/docker/bitcoin-ui/Dockerfile` and re-creates.
### .228 (no bitcoin-ui, no lnd-ui, has electrs-ui from bash first-boot)
1. Ship same binary.
2. Adoption finds only `archy-electrs-ui`.
3. Reconciler sees `bitcoin-ui` and `lnd-ui` missing → triggers `install_fresh` for each.
4. For `bitcoin-ui`: `image_exists("localhost/bitcoin-ui:local")` → false. `build_image(/opt/archipelago/docker/bitcoin-ui, Dockerfile, localhost/bitcoin-ui:local, {BITCOIN_RPC_AUTH: ...}, force=false)`. Then create + start.
5. Same for `lnd-ui`.
6. Manual test: HTTP probe ports 8334 and 8081 return 200 within ~5 minutes of service restart.
## Test plan
Unit tests (Rust, in-process):
- `manifest::tests::legacy_image_parses_as_pull_source`
- `manifest::tests::explicit_pull_source_parses`
- `manifest::tests::explicit_build_source_parses`
- `manifest::tests::source_build_requires_tag`
- `runtime::tests::build_image_happy_path` (uses a minimal Dockerfile in `tempfile::TempDir`)
- `runtime::tests::build_image_failure`
- `runtime::tests::image_exists_roundtrip`
- `prod_orchestrator::tests::install_fresh_pull`
- `prod_orchestrator::tests::install_fresh_build`
- `prod_orchestrator::tests::adopt_existing_matches_by_name`
- `prod_orchestrator::tests::reconcile_starts_exited_container` (with a mock runtime)
- `prod_orchestrator::tests::reconcile_installs_missing_container`
- `prod_orchestrator::tests::compute_container_name_ui_apps_prefixed`
- `prod_orchestrator::tests::compute_container_name_backend_apps_bare`
Integration tests (require real podman, run on archy node):
- Fresh-install path: wipe containers + images, start archipelago, verify all 3 UIs up within 60s.
- Adoption path: containers pre-running, start archipelago, verify no recreate (compare container IDs before/after).
- Reconcile-start path: `podman stop archy-bitcoin-ui`, wait, verify restart.
- Reconcile-recreate path: `podman rm -f archy-bitcoin-ui`, wait, verify rebuild+recreate.
- Rebuild-on-Dockerfile-change path: edit Dockerfile, call `upgrade` RPC, verify image rebuilt and container recreated.
Chaos matrix (bash + Playwright, the original goal):
- For each UI (bitcoin-ui, electrs-ui, lnd-ui) × each event (stop, start, restart, remove+reconcile, SIGKILL, archipelago-service-restart, host-reboot) × each node (.116, .228): assert HTTP 200 + page-title marker returns within 60s of event.
## Risks + mitigations
| Risk | Mitigation |
|------|------------|
| Adoption mismatches and re-creates a container we already had, losing its data | Adoption matches by exact name; `install_fresh` only runs when `get_container_status` returns Err (container doesn't exist), not when it returns Stopped/Exited. Unit tested. |
| Build loop: reconciler rebuilds on every tick | `always_rebuild: false` + `image_exists` check. Only rebuilds when image tag is missing OR `upgrade` RPC is called. |
| Reconciler runs while user is mid-install via the UI | Orchestrator state has per-app mutex; reconcile waits. Install path takes the same mutex. |
| Auto-rollback (v1.7.41) fires during testing | `reconcile_all` is spawned AFTER server is healthy and responding; if it fails, archipelago the service still passes verification. Individual container failures are logged, not fatal. |
| Dependency ordering: bitcoin-ui needs BITCOIN_RPC_AUTH which is generated at first boot | Reconciler handles dependency order by reading `manifest.app.dependencies` and installing in topological order. If the dep doesn't exist yet, skip and retry next tick. |
| Moving `/opt/archipelago/docker/<name>` content breaks the build context | That path is stable per the ISO builder at `image-recipe/build-auto-installer-iso.sh:1671-1685`. Manifests reference it absolutely. |
| Dropping bash scripts breaks existing ISOs in the field | Target release cycle is disposable alpha nodes. For existing alpha nodes (.116, .228) we hot-swap the binary and let the reconciler take over, then the next reboot doesn't need the systemd units; we mask them manually. |
| User wants to downgrade to v1.7.42 | Auto-rollback mechanism already handles that; binary swap is reversible. The removed bash scripts are still in git history. |
## Implementation order
1. **Schema first**: extend `ContainerConfig` + `ContainerSource` + `resolve()` + validation + unit tests. ~100 LOC Rust + ~80 LOC tests.
2. **Runtime**: `build_image` + `image_exists` in trait, `PodmanRuntime`, `DockerRuntime` (can stub), `AutoRuntime`. ~150 LOC + tests with throwaway tempdir Dockerfile.
3. **ProdContainerOrchestrator**: new type with `install/start/stop/restart/remove/status/list/logs/health/adopt_existing/reconcile_all/ensure_running/install_fresh`. ~400 LOC + unit tests with mocked runtime.
4. **ContainerOrchestrator trait**: abstract over Dev and Prod so `RpcHandler` is polymorphic. ~50 LOC refactor.
5. **BootReconciler**: task spawner with loop + cancellation. ~80 LOC + unit tests.
6. **main.rs wire-up**: adopt + spawn reconciler. ~20 LOC.
7. **3 UI manifests + Dockerfile BITCOIN_RPC_AUTH refactor** (use ARG + template file, not sed). ~60 lines of YAML + ~20 lines of Dockerfile.
8. **Remove bash scripts + services**: split into sub-steps because `first-boot-containers.sh` creates 25+ containers (only 3 ported in Step 7) AND does non-container setup (secret gen, UID-mapping chowns, Tor hostnames, WireGuard, firewall, nostr-relay dir):
- **8a** (cheap, safe): delete `image-recipe/configs/archipelago-reconcile.{service,timer}` + their ISO-builder touchpoints (the systemd enablement + `cp` into `$WORK_DIR`). `BootReconciler` fully replaces the timer-driven path — no more periodic bash invocation. **Keep** `scripts/reconcile-containers.sh` + `scripts/container-specs.sh` because `core/archipelago/src/api/rpc/package/update.rs` still shells out to reconcile-containers.sh during OTA updates; porting that call site requires manifests for every container it touches (which is Step 8b's scope). Atomic commit, low risk.
- **8b** (large, deferred): port the remaining ~25 container creations from `first-boot-containers.sh` into `apps/<id>/manifest.yml` files. One manifest per commit, validated against current bash behavior (ports, volumes, env, deps, health checks, post-create wallet/db bootstrap). Probably 1-2 days of careful porting. Includes `apps/filebrowser/manifest.yml`. Then port `update.rs`'s two `reconcile-containers.sh` call sites to the `ContainerOrchestrator` trait (`upgrade(app_id)`).
- **8c** (final, one-way door): rename `first-boot-containers.sh` → `first-boot-setup.sh`, strip out all `$DOCKER run/pull/exec` calls, keep only secret generation + dir prep + Tor/WG/firewall/nostr setup. Rename `archipelago-first-boot-containers.service` → `archipelago-first-boot-setup.service`. Delete `scripts/reconcile-containers.sh` + `scripts/container-specs.sh` (update.rs no longer needs them). Add ISO builder lines to copy `apps/*/manifest.yml` → `/opt/archipelago/apps/`. Full ISO build test on .116 required before commit.
9. **Live test on .228**: hot-swap binary, expect 3 UIs to come up within 60s of service restart.
10. **Live test on .116**: hot-swap binary, expect zero container recreation + adoption-confirmed log lines.
11. **Chaos matrix** on both nodes.
Each step is a separate commit. Steps 1–6 are independent-enough that they can each have their own test gate.
## Estimated total
~1000 LOC Rust added, ~1500 lines bash deleted, ~50 LOC Rust deleted. 8–12 hours of focused work across multiple sessions. No release pressure per user decision.
## Open questions for user
1. **Container naming**: I propose `archy-<app_id>` for UIs, `<app_id>` for backends (matches current .116 fixture). Alternative: unify on `archy-<app_id>` for everything and migrate existing backends by renaming at adoption. Which?
2. **BITCOIN_RPC_AUTH injection**: the build-arg approach rebuilds the UI image when the auth value changes. Fine during normal operation (rare). Alternative: mount the nginx.conf at runtime as a volume, never bake auth into the image. Which?
3. **Reconciler interval**: 5 minutes. Too slow for a dropped container (user sees a broken UI for up to 5 min). Alternative: 30 seconds + more expensive `podman ps` calls. Which?
4. **Concurrent reconcile + user install**: per-app mutex is the simple answer. Alternative: a single orchestrator-wide mutex (simpler, slower). Which?
5. **Delete bash scripts in this migration, or keep them around as fallback?** I recommend delete (single source of truth), but deleting `first-boot-containers.sh` is a one-way door in terms of field recovery.
docs/archive/security-code-audit-2026-03.md
+576
View File
@@ -0,0 +1,576 @@
# Archipelago Security & Code Quality Audit Report
**Date**: March 2026
**Version audited**: 0.1.0
**Auditor**: Automated code review (Claude)
**Scope**: Authentication, sessions, cryptography, container security, RPC, frontend, custom code vs libraries
---
## 1. Executive Summary
### Overall Security Posture: 7.5 / 10
Archipelago demonstrates a security-conscious design with several production-grade patterns already in place. The project makes defensible choices in cryptography, follows capability-based container hardening, and implements layered authentication with TOTP 2FA. However, gaps remain in image signature verification, some postMessage origin validation, and the use of bcrypt instead of the already-available Argon2id for password hashing.
For a v0.1.0 self-sovereign personal server, this is a strong foundation. The code reads like it was written by someone who understands the threat model (local network appliance, single admin user, potentially hostile containers).
### Top 5 Risks (by severity)
1. **Cosign image verification is a TODO** (`podman_client.rs:84`). Container images are pulled without cryptographic signature checks. A compromised registry or MITM on image pull could inject malicious containers. This is the single largest attack surface.
2. **`postMessage('*')` wildcard origin in Nostr signer** (`AppSession.vue:490`, `appLauncher.ts:262,306,309`). Responses to NIP-07 signing requests are sent with `'*'` target origin, allowing any window/iframe to intercept signed Nostr events. A malicious app loaded in an adjacent iframe could harvest signatures.
3. **bcrypt for password hashing instead of Argon2id** (`auth.rs:108,245`). bcrypt is battle-tested but vulnerable to GPU/ASIC acceleration. Argon2id is already a dependency (used in TOTP and backup encryption) and provides memory-hard resistance. Using two different password hashing schemes in the same codebase is also a maintenance smell.
4. **Sessions are in-memory only** (`session.rs`). All sessions are lost on service restart, forcing all users to re-authenticate. More critically, there is no persistence layer to support session revocation auditing or multi-instance deployments.
5. **`v-html` used for TOTP QR SVG rendering** (`Settings.vue:286`). The SVG is server-generated, but `v-html` is a known XSS vector. If the QR generation path were ever to include user-controlled input, this would become exploitable.
### Top 5 Strengths
1. **TOTP implementation is production-grade**. Envelope encryption (KEK wraps MEK wraps secret), Argon2id key derivation with strong parameters (64 MiB, t=3, p=4), ChaCha20-Poly1305 AEAD, zeroize on drop, replay protection via used time steps, bcrypt-hashed backup codes. This exceeds what most node-OS projects implement.
2. **Comprehensive rate limiting**. Login rate limiting (5 attempts / 60s per IP) plus per-endpoint rate limiting on 25+ sensitive methods including financial operations, identity creation, backup operations, and federation joins. Configurable windows per method.
3. **CSRF protection is properly implemented**. Double-submit cookie pattern: `csrf_token` cookie (readable by JS) + `X-CSRF-Token` header validated on every authenticated request. SameSite=Strict on session cookies. HttpOnly on session cookie (not accessible to JS).
4. **Container security defaults are correct**. `--cap-drop=ALL` with explicit per-app capability add-back, `--security-opt=no-new-privileges:true` on all non-privileged containers, read-only root filesystem where compatible, per-app capability documentation.
5. **Error sanitization prevents information leakage**. `sanitize_error_message()` strips internal file paths and system details, returning generic errors for anything not in the user-facing prefix allowlist. Path components like `/var/lib/archipelago/` are replaced with `[data]/`.
### Recommended Actions (ordered by impact)
1. Implement cosign image verification before any public release. This is a hard requirement for supply chain security.
2. Replace `postMessage('*')` with explicit target origins derived from the iframe's `src` URL.
3. Migrate password hashing from bcrypt to Argon2id (already a dependency) with a transparent upgrade path on next login.
4. Add `DOMPurify.sanitize()` around all `v-html` usage or replace with a component-based SVG renderer.
5. Add session persistence (SQLite) to survive restarts and enable audit logging.
---
## 2. Session & Auth
### Password Hashing
**Current**: `bcrypt` crate with `DEFAULT_COST` (cost factor 12).
| Property | bcrypt (current) | Argon2id (available) |
|----------|-----------------|---------------------|
| Algorithm | Blowfish-based, 1999 | Memory-hard, won PHC 2015 |
| GPU resistance | Moderate (small state) | Strong (memory-hard) |
| Cost factor | `DEFAULT_COST = 12` (~250ms) | m=64MiB, t=3, p=4 (already configured in `totp.rs`) |
| ASIC resistance | Weak | Strong |
| Ecosystem status | Mature, stable | Modern standard, OWASP recommended |
| Already a dependency | No (separate crate) | Yes (`argon2` crate used by `totp.rs` and `backup/identity.rs`) |
**Finding**: bcrypt at cost 12 is adequate for a local appliance where login attempts are rate-limited. However, Argon2id is already linked into the binary. Using two different password hashing algorithms in the same project increases cognitive overhead and the risk of confusion. The TOTP module already uses Argon2id with well-chosen parameters (64 MiB memory, t=3 iterations, p=4 parallelism).
**Recommendation**: Migrate to Argon2id on next password change. Store a version tag in `user.json` to allow transparent upgrade: on successful bcrypt login, re-hash with Argon2id and save.
### Session Tokens
**Current**: 32 bytes from `rand::random()` (which delegates to `OsRng`/`ChaCha20Rng`), hex-encoded (64 characters). Tokens are hashed with SHA-256 before storage, so the raw token never exists in the session map.
**Analysis**:
- 256 bits of entropy from a CSPRNG: more than sufficient. Brute-forcing 2^256 is infeasible.
- SHA-256 hashing of stored tokens: correct. A database leak would not expose session tokens.
- Hex encoding doubles the string length but is unambiguous and URL-safe.
**Verdict**: This is correct and secure for a single-instance appliance. No change needed.
### Session Storage
**Current**: In-memory `HashMap<[u8; 32], Session>` behind `Arc<RwLock<>>`.
| Feature | Status |
|---------|--------|
| TTL-based expiry | 24 hours inactivity (full), 5 minutes (pending TOTP) |
| Max concurrent sessions | 5 (oldest evicted) |
| Session rotation on password change | Yes (`rotate()` + `invalidate_all_except()`) |
| Cleanup of expired sessions | `cleanup_expired()` available, presumably called periodically |
| Persistence across restarts | **No** |
| Audit trail | **No** |
**Risk**: Medium. Session loss on restart is annoying but not a security issue (it forces re-authentication). The lack of audit trail means there is no way to retroactively determine who was authenticated when.
**Recommendation**: Add SQLite-backed session store when adding multi-user support. For now, the in-memory approach is acceptable.
### CSRF Protection
**Current**: Double-submit cookie pattern.
1. On login, server sets `csrf_token` cookie (SameSite=Strict, readable by JS) and `session` cookie (HttpOnly, SameSite=Strict).
2. Frontend reads `csrf_token` from `document.cookie` and sends it as `X-CSRF-Token` header on every RPC call.
3. Backend validates `csrf_cookie == csrf_header` on every authenticated request.
**Analysis**:
- SameSite=Strict prevents cross-origin cookie submission entirely in modern browsers.
- The double-submit pattern provides defense-in-depth for browsers that do not enforce SameSite.
- CSRF token is 32 bytes (256 bits) of randomness -- sufficient.
- Secure flag is conditionally set (production only, not dev mode) -- correct.
**Finding**: The CSRF implementation is sound. One minor note: the CSRF token is generated independently of the session token. This is fine because both are random and the cookie binding ensures they cannot be used cross-session.
**Verdict**: Correct. No changes needed.
### Rate Limiting
**Login rate limiter**: 5 failures per 60 seconds per IP. Implemented as `Vec<Instant>` per IP with sliding window.
**Endpoint rate limiter**: Per-method limits on 25+ sensitive endpoints. Examples:
| Endpoint | Max Requests | Window |
|----------|-------------|--------|
| `wallet.send` | 5 | 300s |
| `lnd.payinvoice` | 10 | 300s |
| `identity.create` | 10 | 300s |
| `backup.create` | 10 | 600s |
| `system.factory-reset` | (not rate-limited) | -- |
| `container-install` | 5 | 300s |
| `auth.changePassword` | 3 | 300s |
| `federation.join` | 5 | 60s |
| `update.apply` | 2 | 600s |
**Coverage gaps**:
- `system.factory-reset` is not rate-limited. While it requires authentication, a compromised session could rapidly trigger factory resets. Low practical risk since one reset wipes everything.
- `tor.rotate-service` is not rate-limited. Rapid rotation could burn through Tor circuits.
- No global rate limit across all endpoints -- only per-method. A compromised session could flood non-limited endpoints.
**Verdict**: Good coverage for a single-user appliance. The per-method approach is appropriate for the threat model.
### TOTP 2FA
**Implementation quality**: Excellent.
| Feature | Implementation |
|---------|---------------|
| Secret generation | 20 bytes (160 bits) from `OsRng` |
| Secret storage | Encrypted at rest: Argon2id KDF -> ChaCha20-Poly1305 envelope |
| Encryption layers | 3: password -> KEK (Argon2id) -> MEK (random) -> secret |
| Verification window | Current step +/- 1 (3 steps total, ~90s window) |
| Replay protection | Used time steps tracked and rejected |
| Code comparison | Constant-time comparison (`constant_time_eq`) |
| Backup codes | 8 codes, bcrypt-hashed, one-time use |
| Pending session | Max 5 attempts, 5-minute TTL, then forced re-login |
| Re-keying | MEK re-encrypted under new password on password change |
| Zeroize | KEK, MEK, and raw secret zeroized after use |
**Finding**: This is a textbook TOTP implementation. The envelope encryption (KEK/MEK pattern) is the same approach used by hardware security modules. The `zeroize` crate ensures secrets do not linger in memory.
One minor note: `constant_time_eq` is hand-rolled rather than using the `subtle` crate's `ConstantTimeEq`. The implementation is correct (XOR accumulation), but the `subtle` crate is specifically designed to resist compiler optimizations that could break constant-time behavior.
**Recommendation**: Consider switching to `subtle::ConstantTimeEq` for the TOTP comparison. The current implementation is likely fine in practice, but `subtle` provides stronger guarantees against compiler reordering.
---
## 3. Cryptographic Review
| Component | Our Implementation | Library Alternative | Correct? | Secure? | Verdict |
|-----------|-------------------|---------------------|----------|---------|---------|
| **Password hashing** | `bcrypt` crate, `DEFAULT_COST` (12) | `argon2` crate (already a dep) | Yes | Adequate | **Migrate to Argon2id**. Already a dependency, memory-hard, OWASP recommended. bcrypt is not broken but Argon2id is strictly better against modern attacks. |
| **Session tokens** | `rand::random::<[u8; 32]>()` + hex, SHA-256 stored hash | `tower-sessions` or signed JWTs via `jsonwebtoken` | Yes | Yes | **Keep custom**. 256-bit CSPRNG tokens with hashed storage is textbook. JWTs add complexity and stateless verification is not needed for single-instance. |
| **TOTP KDF** | `argon2` crate, Argon2id v0x13, m=64MiB, t=3, p=4 | N/A (already using the right library) | Yes | Yes | **Correct**. Strong parameters that balance security and UX on modest hardware. |
| **TOTP encryption** | `chacha20poly1305` crate, KEK/MEK envelope | `age` crate | Yes | Yes | **Keep custom**. The envelope pattern gives us re-keying without re-encrypting the secret. `age` would not support this use case without wrapping. |
| **DID signing** | `ed25519-dalek` direct usage | SpruceID `ssi` crate | Yes | Yes | **Keep custom**. Our code is 381 lines, handles did:key + DID Documents + dual-key (Ed25519+secp256k1). `ssi` would add 50+ transitive deps. |
| **VC signatures** | Custom Ed25519Signature2020 proof (credentials.rs, 796 lines) | SpruceID `ssi` VC module | Yes | Yes for our proof type | **Keep custom for issuance**. Consider `ssi` only for verifying external VCs with non-Ed25519 proof types. |
| **Backup encryption** | Argon2id KDF + ChaCha20-Poly1305 (backup/identity.rs, 132 lines) | `age` crate | Yes | Yes | **Keep custom**. Clean, minimal, well-tested. `age` is simpler API but our code is already simple. |
| **Key storage** | Raw bytes in files with `0o600` permissions | `keyring` crate or OS keychain | Yes | Adequate | **Keep current**. File-based is correct for headless Linux server. No desktop environment means no keychain daemon. Permissions are set immediately after key generation. |
| **Constant-time comparison** | Hand-rolled XOR accumulation | `subtle` crate `ConstantTimeEq` | Correct logic | Likely | **Consider `subtle`**. Hand-rolled constant-time code can be optimized away by the compiler. `subtle` uses inline assembly barriers. Low risk in practice. |
| **CSRF tokens** | `rand::thread_rng().fill()` 32 bytes | N/A | Yes | Yes | **Correct**. `thread_rng()` delegates to `OsRng`-seeded `ChaCha20Rng`. 256 bits is more than sufficient. |
### Key Observations
1. **Argon2 is used correctly in two places** (TOTP and backup) **but not for password hashing**. This is the most obvious inconsistency. The project already pays the compilation cost for `argon2`; using it for password hashing would unify the crypto stack.
2. **ChaCha20-Poly1305 is used correctly** throughout. Nonces are generated from `OsRng`, key material is zeroized after use, and the AEAD construction prevents both tampering and ciphertext manipulation.
3. **Ed25519-dalek usage is clean**. Key generation uses `OsRng`, signing and verification are straightforward, the Ed25519-to-X25519 conversion for key agreement is done correctly via `curve25519-dalek`.
4. **No custom cryptographic primitives**. All cryptographic operations use well-audited Rust crates. The project does not implement any ciphers, hash functions, or key exchange algorithms from scratch. This is the correct approach.
---
## 4. Container Security
### Capability Dropping
**Default**: `--cap-drop=ALL` applied to all non-privileged containers (`package.rs:265`).
Per-app capabilities are explicitly added back via `get_app_capabilities()`:
| App Category | Capabilities Added | Justification |
|-------------|-------------------|---------------|
| Minimal apps (searxng, filebrowser, etc.) | None | Runs with zero capabilities |
| Standard apps (photoprism, grafana) | CHOWN, SETUID, SETGID | Internal user switching |
| Bitcoin/Lightning | CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE | Data directory ownership |
| Web servers (nginx-proxy-manager, vaultwarden) | CHOWN, SETUID, SETGID, NET_BIND_SERVICE | Binding ports < 1024 |
| Tailscale | `--privileged` + NET_ADMIN, NET_RAW | VPN tunnel creation (unavoidable) |
**Finding**: The capability model is well-designed. Each app gets the minimum capabilities needed. The `--privileged` exception for Tailscale is documented and unavoidable (it needs TUN device access and network namespace manipulation).
**Risk**: DAC_OVERRIDE grants the ability to bypass file permission checks. This is a broad capability. For Bitcoin/Lightning, it is necessary because the containers need to access data directories with varying ownership. Consider whether `FOWNER` alone would suffice for some of these apps.
### Read-Only Root Filesystem
**Current**: `--read-only` is applied to apps listed in `is_readonly_compatible()`:
- searxng, grafana, filebrowser, mempool-electrs, electrs, nostr-rs-relay, ollama, indeedhub
**Not read-only**: Bitcoin, LND, Nextcloud, BTCPay, Jellyfin, HomeAssistant, and others.
With `--read-only`, tmpfs mounts are added for `/tmp` and `/run`:
```
--tmpfs=/tmp:rw,noexec,nosuid,size=256m
--tmpfs=/run:rw,noexec,nosuid,size=64m
```
**Finding**: The `noexec` and `nosuid` flags on tmpfs mounts are a good hardening measure. The list of read-only-compatible apps is conservative, which is the correct approach -- it is better to not break an app than to force read-only on an incompatible container.
**Recommendation**: Gradually test more apps with `--read-only` and expand the list. Each new addition should be validated by running the app and checking for write failures.
### No-New-Privileges
**Current**: `--security-opt=no-new-privileges:true` applied to all non-Tailscale containers (`package.rs:266`).
**Finding**: Correct. This prevents setuid binaries inside containers from escalating privileges. Combined with `--cap-drop=ALL`, this creates a strong privilege boundary.
### User Namespace / Non-Root
**Current**: Podman runs containers with rootless mode when the `archipelago` user is not root (`podman_client.rs:60-66`). However, `podman_async()` uses `sudo podman` (`podman_client.rs:71-73`), which runs containers in the root Podman context.
**Finding**: The `sudo podman` invocation means containers run in a root context, not rootless. While `--cap-drop=ALL` and `no-new-privileges` provide strong isolation, the containers themselves may run as root (UID 0) inside their namespace. Whether the container process runs as non-root depends on the container image's Dockerfile (e.g., `USER 1000`).
**Recommendation**: Add `--user` flags to container creation where the upstream image supports it. Audit each container image to determine which ones run as root internally.
### Image Pinning
**Current**: The `is_valid_docker_image()` function validates registry origin (docker.io, ghcr.io, localhost) and rejects shell metacharacters. However, there is no enforcement of digest pinning (e.g., `image@sha256:...`).
Images can be specified with tags (`:latest`, `:v1.0.0`) but tags are mutable -- a registry compromise could replace the image behind a tag.
**Finding**: No digest pinning is enforced. While registry validation limits the attack surface, a compromised registry account could push malicious images under existing tags.
**Recommendation**: For the curated app list, pin images to specific digests in the marketplace metadata. Allow tag-based pulls for user-specified images but warn about the risk.
### Cosign Verification
**Current**: `podman_client.rs:83-84`:
```rust
// TODO: Implement cosign verification
log::warn!("Signature verification not yet implemented: {}", sig);
```
**Finding**: This is the most significant security gap in the container subsystem. Without signature verification, there is no cryptographic proof that a pulled image was built by the expected author.
**Risk**: HIGH. This should be implemented before any public release.
**Recommendation**: Integrate `sigstore-rs` for cosign verification. At minimum, verify signatures for the curated app list. Third-party apps from the decentralized marketplace should also have verifiable signatures (the publisher's Nostr key can serve as the trust anchor).
### Network Isolation
**Current**: Containers are placed on either:
- `archy-net` (shared network for Bitcoin stack: bitcoin-knots, lnd, electrs, mempool, btcpay, fedimint)
- Host network (Tailscale only)
- Default isolated network (all other apps)
**Finding**: The `archy-net` shared network is necessary for the Bitcoin stack to communicate (LND needs Bitcoin RPC, Mempool needs Electrs, etc.). Other apps are properly isolated.
**Recommendation**: Consider creating separate networks for distinct app clusters (e.g., `btcpay-net` for BTCPay + nbxplorer + postgres) rather than putting everything on `archy-net`. This would limit lateral movement if a single container is compromised.
### Secrets Injection
**Current**: Secrets are passed to containers via environment variables (`-e` flag). For example, Bitcoin RPC credentials:
```rust
"--bitcoind-password".to_string(), "archipelago123".to_string(),
```
**Finding**: Environment variables are visible via `podman inspect` and `/proc/<pid>/environ` on the host. The hardcoded `archipelago123` RPC password is particularly concerning -- it should be randomly generated per installation.
**Recommendation**:
1. Generate random credentials per app installation and store them via the secrets manager.
2. Prefer bind-mounting secret files into containers (`--secret` or `-v /path/to/secret:/run/secrets/password:ro`) over environment variables.
3. Replace the hardcoded `archipelago123` Bitcoin RPC password with a per-install random password.
---
## 5. RPC Security
### Authentication Enforcement
**Unauthenticated endpoints** (from `UNAUTHENTICATED_METHODS`):
- `auth.login`, `auth.login.totp`, `auth.login.backup` -- login flow
- `auth.isOnboardingComplete`, `auth.isSetup` -- setup status checks
- `health` -- health check
- `backup.restore-identity` -- onboarding restore (before user account exists)
- `federation.peer-joined`, `federation.peer-address-changed`, `federation.get-state` -- inter-node RPC
**Finding**: The unauthenticated endpoint list is reasonable. The federation endpoints are called by peer nodes over Tor and cannot use session cookies -- they are rate-limited instead (10 requests/60s for peer-joined and peer-address-changed, 30 requests/60s for get-state).
`backup.restore-identity` is unauthenticated by design -- it is used during onboarding before a user account exists. This is the correct approach.
**Risk**: The federation endpoints accept peer assertions (e.g., "I just joined your federation") without cryptographic authentication beyond the Tor hidden service address. A future improvement would be to require DID-signed payloads for federation RPCs.
### RBAC
**Current**: RBAC is implemented and wired into the RPC dispatcher (`mod.rs:249-269`). Three roles are defined:
| Role | Access |
|------|--------|
| Admin | Everything |
| Viewer | Read-only system/node/container/federation/identity/backup methods + logout |
| AppUser | Basic system stats, container listing, health, logout, password change |
**Finding**: RBAC is operational. The `can_access()` method uses prefix matching (e.g., `method.starts_with("system.")`) which is a reasonable approach for method-based access control.
**Concern**: The Viewer role grants access to `federation.list` and `dwn.query` but not `dwn.write-message`. This is correct. However, the prefix-matching approach means that if a new method like `system.factory-reset` is added, it would be accessible to Viewers because it starts with `system.`. The current code mitigates this because `system.factory-reset` is not listed in the Viewer's allowed prefixes -- it requires an exact `system.` prefix match, and the Viewer role only allows `method.starts_with("system.")`.
**Wait** -- actually, `system.factory-reset` does start with `system.`, so Viewers WOULD have access to it under the current RBAC rules.
**Risk**: MEDIUM. Any new `system.*` method is automatically accessible to Viewers. The Viewer role should use an explicit allowlist rather than prefix matching for the `system.` namespace.
**Recommendation**: Change Viewer's `system.*` access to an explicit list: `system.stats`, `system.temperature`, `system.disk-status`, etc. Do not allow `system.factory-reset`, `system.shutdown`, `system.reboot`, or `system.disk-cleanup` for Viewers.
### Input Validation
Five critical endpoints traced from params to handler:
1. **`auth.login`**: Password extracted as string from params, passed to `bcrypt::verify()`. No injection risk -- bcrypt operates on byte arrays. Rate-limited.
2. **`package.install`**: Package ID validated by `validate_app_id()` (lowercase alphanumeric + hyphens, 1-64 chars, no leading hyphen). Docker image validated by `is_valid_docker_image()` (length check, no shell metacharacters, registry allowlist). Both validations are solid.
3. **`system.factory-reset`**: Requires `confirm: true` parameter. Authenticated and RBAC-checked. No injection risk -- the handler performs fixed system operations.
4. **`backup.restore-identity`**: Accepts a JSON blob with a base64-encoded encrypted backup. The backup is decrypted with a user-supplied passphrase. Input validation: blob must be valid base64, must contain salt + nonce + ciphertext of minimum length, decrypted key must be exactly 32 bytes. The Argon2id KDF prevents timing attacks on the passphrase.
5. **`identity.create`**: Accepts optional `label` and `type` parameters. The label is stored as-is in a JSON file. No length validation on the label. This is a low risk since the label is never used in shell commands or HTML rendering, but a maximum length should be enforced.
### Error Sanitization
**Current**: `sanitize_error_message()` in `mod.rs:72-104`:
- User-facing prefixes (Invalid, Missing, Not found, etc.) are passed through with path sanitization.
- Path components (`/var/lib/archipelago/`, `/usr/local/bin/`, `/etc/`) are replaced with `[data]/`, `[bin]/`, `[config]/`.
- Messages exceeding 200 characters are truncated.
- All other errors return: `"Operation failed. Check server logs for details."`
**Finding**: This is a good approach. The prefix allowlist ensures that validation errors remain actionable for the user while internal errors (stack traces, database errors, file system errors) are hidden.
**Minor concern**: The `contains()` check (`msg.contains(prefix)`) rather than `starts_with()` means that an internal error message containing the word "Password" anywhere would be passed through. For example, an error like "Failed to read /etc/shadow: Password file locked" would match the "Password" prefix. This is unlikely to leak sensitive information but is worth tightening.
### Path Traversal
**Frontend** (`filebrowser-client.ts`): `sanitizePath()` is not present in `rpc-client.ts`, but the filebrowser client strips `..` and `/` from filenames: `name.replace(/\.\./g, '').replace(/\//g, '')`.
**Backend**: File operations use `PathBuf::join()` which does not normalize `..` components. However, all file paths are constructed from validated app IDs (alphanumeric + hyphens) and fixed directory structures. There is no user-controlled path component that could escape the data directory.
**Verdict**: Path traversal risk is low. The app ID validation prevents directory traversal in container data paths.
---
## 6. Frontend Security
### XSS
**`v-html` usage**: Found in one location:
- `Settings.vue:286`: `<div v-html="totpQrSvg" />` -- renders server-generated SVG.
**Analysis**: The SVG is generated by the `qrcode` crate on the backend and contains only geometric shapes (rects, paths). It does not include any user-controlled content. However, `v-html` bypasses Vue's template escaping entirely.
**Risk**: LOW currently (SVG is trusted server output), but HIGH if the generation path ever changes.
**Recommendation**: Replace `v-html` with either:
1. An `<img>` tag with a data URI: `<img :src="'data:image/svg+xml;base64,' + btoa(totpQrSvg)" />`
2. `DOMPurify.sanitize(totpQrSvg)` before rendering with `v-html`.
### CSRF
**Frontend implementation** (`rpc-client.ts:18-21, 42-45`):
```typescript
function getCsrfToken(): string | null {
const match = document.cookie.match(/(?:^|;\s*)csrf_token=([^;]+)/)
return match ? match[1]! : null
}
```
The CSRF token is read from cookies and sent as `X-CSRF-Token` header on every RPC call via `fetch()` with `credentials: 'include'`.
**Finding**: Correctly implemented. The token is scoped to the session (new token issued on login, rotated on password change, expired on logout).
### Credential Storage (localStorage)
Audit of all `localStorage.setItem` calls:
| Key | Content | Risk |
|-----|---------|------|
| `neode_locale` | Language preference ("en") | None |
| `neode-auth` | Boolean flag ("true") | None -- not a credential |
| `neode_onboarding_complete` | Boolean flag | None |
| `neode_intro_seen` | Boolean flag | None |
| `neode_backup_created` | Boolean flag | None |
| `neode_did` | DID string (public identifier) | None -- DIDs are public |
| `neode_did_state` | DID + KID + pubkey (all public) | None |
| `neode_nostr_npub` | Nostr public key (public) | None |
| `archipelago-ui-mode` | "easy" / "advanced" | None |
| `archipelago-goal-progress` | UI progress state | None |
| `archipelago-spotlight-recent` | Recent search items | None |
| `federation-view` | Active federation tab | None |
| `IDENTITY_KEY + appId` | Nostr identity for app context | **LOW** -- contains public key, not private |
| `APPROVED_ORIGINS_KEY` | Set of approved iframe origins | **LOW** -- UI preference |
| `DISPLAY_MODE_KEY` | "overlay" / "tab" | None |
**Finding**: No secrets, passwords, private keys, or session tokens are stored in localStorage. All stored values are either UI preferences or public identifiers. This is correct.
### iframe postMessage Security
**Outbound `postMessage('*')` calls** (wildcard target origin):
1. `AppSession.vue:490,492` -- Nostr signing responses sent to iframe source with `'*'`
2. `AppLauncherOverlay.vue:390` -- Escape key event sent to parent with `'*'`
3. `appLauncher.ts:262,306,309` -- Nostr signing responses sent to iframe source with `'*'`
**Inbound origin validation**:
1. `contextBroker.ts:65` -- Validates `event.origin !== this.allowedOrigin` (properly restrictive)
2. `Chat.vue:110` -- Validates against expected AIUI URL origin (properly restrictive)
3. `AppSession.vue` and `appLauncher.ts` -- No origin validation on incoming `nostr-request` messages
**Finding**: The Nostr signer (NIP-07 bridge) accepts signing requests from any iframe without verifying the origin, and sends signed responses back with `'*'` target origin. This means:
- Any iframe loaded in the app launcher could request Nostr event signatures.
- The signed response could be intercepted by any window.
**Mitigation**: The user is prompted to approve signing requests (a consent dialog exists in `appLauncher.ts`), and the approved origins list is stored in localStorage. However, the initial request acceptance has no origin check.
**Risk**: MEDIUM. A malicious app loaded in an iframe could silently request signatures for crafted Nostr events.
**Recommendation**:
1. Validate `event.origin` on incoming `nostr-request` messages against the app's known URL.
2. Replace `postMessage(msg, '*')` with `postMessage(msg, expectedOrigin)` for Nostr responses.
3. The `AppLauncherOverlay.vue:390` escape event using `'*'` is lower risk since it only sends a UI event to the parent window, but should still use a specific origin.
### Dependency Audit
Note: `npm audit` was not run as part of this review (requires network access and node_modules). This should be run separately:
```bash
cd neode-ui && npm audit
```
The project uses Vue 3, Vite 7, and Pinia -- all actively maintained. The key security-relevant frontend dependencies are:
- `fetch` API (native, no third-party HTTP client)
- No `eval()` or `new Function()` usage detected
- No inline scripts or styles that would conflict with CSP
---
## 7. Custom Code vs Libraries
### Summary Table
| # | Component | Lines | Quality | Alternative | Verdict |
|---|-----------|-------|---------|-------------|---------|
| 1 | HTTP Server (`handler.rs`) | 813 | Functional but hand-rolled | `axum` | **Migrate** |
| 2 | Session Management (`session.rs`) | 595 | Solid, well-tested | `tower-sessions` | **Keep** (for now) |
| 3 | Rate Limiting (`session.rs` + `mod.rs`) | ~120 | Simple, effective | `governor` | **Keep** |
| 4 | DID Implementation (`identity.rs`) | 381 | Clean, W3C compliant | SpruceID `ssi` | **Keep** |
| 5 | Verifiable Credentials (`credentials.rs`) | 796 | W3C VC 2.0 compliant | SpruceID `ssi` VC | **Keep** (consider `ssi` for external VC verification) |
| 6 | did:dht | ~200 | Works via `mainline` | `pkarr` | **Evaluate** |
| 7 | DWN Store | ~300 | Skeletal | None mature | **Keep** (deprioritize) |
| 8 | WebSocket State Broadcasting | ~200 | Works but full-resync | `json-patch` | **Add library** |
| 9 | Form Validation (frontend) | Scattered | Inconsistent | `zod` | **Add library** |
| 10 | Container Runtime (`podman_client.rs`) | 410 | Clean abstraction | `bollard` | **Keep** |
### Detailed Assessments
**1. HTTP Server (custom `handler.rs` -- 813 lines)**
The handler manually implements routing, CORS headers, WebSocket upgrade, request body parsing, and response building using raw `hyper 0.14`. This works but is fragile -- every new route requires manual pattern matching, there is no middleware stack, and hyper 0.14 is end-of-life.
Alternative: `axum` (built by the tokio team on hyper 1.x) provides typed extractors, a middleware stack via `tower`, built-in WebSocket support, and is the de facto standard for Rust web servers.
**Verdict**: Migrate. This is the highest-impact refactoring item. `axum` would reduce `handler.rs` to approximately 200 lines while adding type safety, automatic request parsing, and tower middleware support. Risk is medium -- the RPC logic is unchanged, only the HTTP glue changes.
**2. Session Management (custom `session.rs` -- 595 lines including 300+ lines of tests)**
The session store is ~200 lines of production code with ~370 lines of comprehensive tests. It implements token hashing, TTL expiry, concurrent session limits, session rotation, and pending TOTP sessions with attempt tracking. The code uses `zeroize` for TOTP secrets.
Alternative: `tower-sessions` with `tower-sessions-sqlx-store` for SQLite-backed persistence.
**Verdict**: Keep custom for now. The implementation is correct, well-tested, and purpose-built for the two-phase TOTP flow. A library would not handle the pending/full session distinction without significant customization. Migrate to `tower-sessions` only if SQLite persistence or multi-instance deployment is needed.
**3. Rate Limiting (custom, ~120 lines)**
Simple in-memory sliding window counters per (method, IP). Not configurable at runtime but the static configuration is well-chosen for each endpoint category.
Alternative: `governor` crate or `tower::limit::RateLimitLayer`.
**Verdict**: Keep custom. The implementation is straightforward, correct, and tailored to the per-method needs. `governor` would add a dependency for minimal benefit. Revisit only if distributed rate limiting is needed (multiple backend instances).
**4. DID Implementation (`identity.rs` -- 381 lines)**
Clean implementation of `did:key` method using `ed25519-dalek`. Generates W3C DID Core v1.0 compliant DID Documents with Ed25519 verification keys and X25519 key agreement keys. Includes Ed25519-to-X25519 conversion, Nostr secp256k1 dual-key support, and roundtrip tests.
Alternative: SpruceID `ssi` crate (v0.15.0).
**Verdict**: Keep custom. The code is ~380 lines, handles exactly the features needed (did:key + dual-key DID Documents), and has good test coverage (12 tests). `ssi` would add 50+ transitive dependencies for features like did:web, did:ethr, did:ion resolution that are not needed. The maintenance burden of 380 lines of well-tested code is far lower than managing a large dependency tree.
**5. Verifiable Credentials (`credentials.rs` -- 796 lines)**
W3C VC Data Model 2.0 implementation supporting issuance, verification, revocation, and verifiable presentations. Uses Ed25519Signature2020 proof format.
Alternative: SpruceID `ssi` VC module.
**Verdict**: Keep custom for issuance and node-to-node verification. The code handles the one proof type needed for Archipelago's use case (Ed25519Signature2020). Consider `ssi` only if external VC verification is needed (verifying credentials issued by non-Archipelago systems with different proof types like BbsBlsSignature2020 or JsonWebSignature2020).
**6. did:dht (`did_dht.rs` -- ~200 lines)**
Implements did:dht resolution via the `mainline` crate (BEP-44 signed DHT records). Includes in-memory caching.
Alternative: `pkarr` crate (v5.0.3, 550K downloads) -- higher-level abstraction over mainline DHT.
**Verdict**: Evaluate `pkarr`. If it handles the BEP-44 encoding that is currently done manually, it would reduce code and benefit from upstream maintenance. If it adds unnecessary abstraction, keep custom. The current code is small and works.
**7. DWN Store (`dwn_store.rs` -- ~300 lines)**
Basic CRUD operations, filesystem-backed, protocol registration. Skeletal implementation.
Alternative: No production-ready DWN implementation exists in Rust. The `dwn` crate by unavi-xyz is v0.4.0 with 323 downloads.
**Verdict**: Keep custom. No viable alternative. Per ADR-011, DWN is deprioritized. The current skeleton is sufficient for the protocol registration feature.
**8. WebSocket State Broadcasting (`state.rs` -- ~200 lines)**
Uses tokio broadcast channels to send full state model resyncs on every change. Every WebSocket client receives the entire state JSON on every update.
Alternative: `json-patch` crate for RFC 6902 JSON diffs. The frontend already includes `fast-json-patch`.
**Verdict**: Add `json-patch`. This is one of the highest-impact improvements. On a system with 10+ containers and active monitoring, the full-state broadcast can be 50-100 KB per update. JSON patches would reduce this to a few hundred bytes per change. Both the Rust `json-patch` crate and the frontend `fast-json-patch` library are mature and actively maintained.
**9. Form Validation (manual inline in Vue components)**
Validation logic is scattered across Vue components with inconsistent patterns. Some forms validate on submit, others on blur, and error messages are not standardized.
Alternative: `zod` (TypeScript-first schema validation, 40M+ weekly npm downloads).
**Verdict**: Add `zod`. Centralize validation schemas in `src/types/schemas.ts`. This is critical for the onboarding flow where bad input (weak passphrase, malformed DID) can cause key generation failures. `zod` integrates naturally with TypeScript and can generate types from schemas, reducing duplication.
**10. Container Runtime Abstraction (`podman_client.rs` -- 410 lines)**
Clean Podman client that wraps CLI invocations for container lifecycle operations. Handles both JSON array and NDJSON output formats from Podman.
Alternative: `bollard` crate (Docker/Podman API client, 7M downloads).
**Verdict**: Keep custom. The current abstraction is clean and purpose-built for the manifest-based approach. `bollard` is Docker-first and would require wrapping for the `AppManifest`-driven container creation. The CLI approach also avoids the Podman socket configuration complexity that `bollard` would require.
---
## What To Do Next
The three most impactful changes from this audit, in priority order:
1. **Implement cosign image verification** (`podman_client.rs`). Integrate `sigstore-rs` for container image signature verification. This closes the largest supply chain attack surface. Without it, a compromised Docker registry could push malicious images.
2. **Fix postMessage wildcard origins** (`AppSession.vue`, `appLauncher.ts`). Replace `postMessage(msg, '*')` with targeted origins. Add `event.origin` validation on incoming Nostr signing requests. This prevents malicious iframes from harvesting signed events.
3. **Migrate password hashing to Argon2id** (`auth.rs`). Add a version field to the user JSON. On login, if the hash is bcrypt, verify with bcrypt, then re-hash with Argon2id and save. This unifies the crypto stack and provides better GPU resistance.
These three changes address the top three risks identified in this audit and are achievable without architectural changes.
docs/archive/three-mode-ui-design.md
+377
View File
@@ -0,0 +1,377 @@
# Three-Mode UI System: Easy / Pro / Chat
## Overview
Archipelago's UI will support three switchable modes, each targeting a different user experience level:
| Mode | Label in UI | Target User | What They See |
|------|-------------|-------------|---------------|
| **Pro** | Pro | Power users, developers, node operators | Current full interface — all services, configs, technical details |
| **Easy** | Easy | Complete beginners, non-technical users | Goal-based interface — "Open a Shop", "Store My Photos" |
| **Chat** | Chat | Everyone (future) | Conversational AI interface powered by AIUI |
### Key Principles
1. **Pro mode is preserved** — the current interface stays exactly as-is and continues to be improved
2. **Same URLs** — modes don't change route paths. `/dashboard` shows different content based on mode
3. **Cross-surfacing** — Easy mode goals are searchable from Spotlight (Cmd+K) and suggested in Pro mode
4. **Persistent preference** — mode choice saved to localStorage + backend UIData
---
## How Modes Work
### Architecture: Conditional Rendering
Rather than separate route trees (`/easy/home`, `/pro/home`), the mode controls **what renders within existing routes**:
```
Dashboard.vue (shared shell)
├── Sidebar → nav items change per mode
├── ModeSwitcher → always visible in sidebar
└── <RouterView>
└── Home.vue (dispatcher)
├── <GamerHome /> (Pro mode)
├── <EasyHome /> (Easy mode)
└── <ChatHome /> (Chat mode)
```
This means:
- Auth guards, WebSocket, stores — all shared
- URLs never change — bookmarks work regardless of mode
- Both modes use the same component library (glass-card, glass-button, etc.)
### Navigation Per Mode
**Pro Mode** (current, 7 items):
```
Home → My Apps → App Store → Cloud → Network → Web5 → Settings
```
**Easy Mode** (simplified, 3 items):
```
Home → My Services → Settings
```
**Chat Mode** (4 items):
```
Home → Chat → My Apps → Settings
```
---
## Easy Mode: Goal-Based Interface
### The Problem
Current interface says: "Here are 20+ services you can install. Figure out which ones you need, install them, configure them to talk to each other."
Easy mode says: **"What do you want to do?"**
### Goal Cards (Easy Mode Home)
When in Easy mode, the Home screen shows goal cards instead of the current 4 technical overview cards:
```
┌─────────────────────┐ ┌─────────────────────┐
│ 🏪 Open a Shop │ │ ⚡ Accept Payments │
│ │ │ │
│ Set up your own │ │ Receive Bitcoin & │
│ Bitcoin-powered │ │ Lightning payments │
│ online store │ │ │
│ │ │ ~30 min • Beginner │
│ ~45 min • Beginner │ │ ▸ Start │
│ ▸ Start │ └─────────────────────┘
└─────────────────────┘
┌─────────────────────┐ ┌─────────────────────┐
│ 📸 Store My Photos │ │ 📁 Store My Files │
│ │ │ │
│ Private photo │ │ Personal cloud │
│ backup & gallery │ │ storage & sync │
│ │ │ │
│ ~15 min • Beginner │ │ ~20 min • Beginner │
│ ▸ Start │ │ ▸ Start │
└─────────────────────┘ └─────────────────────┘
┌─────────────────────┐ ┌─────────────────────┐
│ ⚡ Lightning Node │ │ 🔑 Create Identity │
│ │ │ │
│ Run your own │ │ Sovereign DID & │
│ Lightning Network │ │ Nostr identity │
│ routing node │ │ │
│ │ │ ~5 min • Beginner │
│ ~40 min • Beginner │ │ ▸ Start │
│ ▸ Start │ └─────────────────────┘
└─────────────────────┘
┌─────────────────────┐
│ 💾 Back Up │
│ │
│ Encrypted backup │
│ of your entire │
│ node │
│ │
│ ~10 min • Beginner │
│ ▸ Start │
└─────────────────────┘
```
### Goal Workflow Wizard
Clicking a goal opens a **multi-step wizard** at `/dashboard/goals/:goalId`:
```
┌──────────────────────────────────────────────────────┐
│ ← Back to Goals │
│ │
│ Open a Shop │
│ Set up your own Bitcoin-powered online store │
│ │
│ Step 2 of 4 │
│ ═══════════════════════▓▓▓░░░░░░░░░░░░ │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ ✅ Step 1: Install Bitcoin Node │ │
│ │ Bitcoin Core is running and syncing │ │
│ └─────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ ⏳ Step 2: Install Lightning Network │ │
│ │ Installing LND... [45%] │ │
│ │ ████████████████████░░░░░░░░░░░░░ │ │
│ └─────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ ○ Step 3: Install BTCPay Server │ │
│ │ Waiting for Lightning to be ready │ │
│ └─────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ ○ Step 4: Set Up Your Store │ │
│ │ Configure your store name and settings │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ ℹ️ Bitcoin needs to sync before Lightning can │
│ start. This takes 2-3 days on first run. │
│ │
└──────────────────────────────────────────────────────┘
```
**Smart features:**
- Steps already satisfied (app running from a previous goal) are auto-completed
- Dependency resolution: Bitcoin must be running before LND can start
- Real-time progress from WebSocket data patches
- `configure` steps open the app in the iframe launcher for the user to complete
### Goal Definitions
| Goal | What It Provisions | Estimated Time |
|------|-------------------|----------------|
| **Open a Shop** | Bitcoin Knots + LND + BTCPay Server | ~45 min |
| **Accept Payments** | Bitcoin Knots + LND | ~30 min |
| **Store My Photos** | Immich (photo management) | ~15 min |
| **Store My Files** | Nextcloud (cloud storage) | ~20 min |
| **Run a Lightning Node** | Bitcoin Knots + LND + channel setup | ~40 min |
| **Create My Identity** | Built-in DID + Nostr keypair | ~5 min |
| **Back Up Everything** | Built-in encrypted backup | ~10 min |
---
## Mode Switcher UI
### Desktop Sidebar
A compact three-segment toggle sits below the logo, above navigation:
```
┌──────────────────────┐
│ 🏝️ Archipelago │
│ v0.1.0 │
│ │
│ ┌──────┬──────┬────┐ │
│ │ Easy │ Pro │Chat│ │ ← Mode switcher
│ └──────┴──────┴────┘ │
│ │
│ ○ Home │
│ ○ My Apps │ ← Nav items change
│ ○ App Store │ per mode
│ ○ ... │
│ │
│ ⚙ Settings │
│ ↪ Logout │
│ ● Online │
└──────────────────────┘
```
### Settings Page
Full-width selection cards in a new "Interface Mode" section:
```
┌──────────────────────────────────────────────────────┐
│ Interface Mode │
│ Choose how you want to interact with your node. │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ │ │ ██████ │ │ │ │
│ │ Easy Mode │ │ Pro Mode │ │ Chat Mode │ │
│ │ │ │ (Active) │ │ (Soon) │ │
│ │ Goal-based │ │ Full │ │ AI chat │ │
│ │ guided │ │ control │ │ interface │ │
│ │ setup │ │ of all │ │ │ │
│ │ │ │ services │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└──────────────────────────────────────────────────────┘
```
Uses the existing `.path-option-card` / `.path-option-card--selected` pattern from OnboardingPath.vue.
### Mobile
Mode switcher is in Settings only (bottom tab bar has limited space).
---
## Cross-Surfacing: Goals Everywhere
### Spotlight Search (Cmd+K)
Goals are added to the help tree and appear in search results regardless of mode:
```
┌──────────────────────────────────────┐
│ 🔍 shop │
│ │
│ Quick Start Goals │
│ 🚀 Open a Shop │
│ 🚀 Accept Payments │
│ │
│ Navigate │
│ → App Store │
│ │
│ Actions │
│ → Install an App │
└──────────────────────────────────────┘
```
### Pro Mode Home
A "Quick Start Goals" section appears at the bottom of Pro mode's Home, giving power users easy access to the guided workflows:
```
┌──────────────────────────────────────────────────────┐
│ Quick Start Goals │
│ Not sure where to start? Try a guided setup. │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Open a Shop │ │ Accept │ │ Store Photos │ │
│ │ │ │ Payments │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└──────────────────────────────────────────────────────┘
```
---
## Chat Mode (Placeholder)
For now, Chat mode shows a placeholder with a disabled input:
```
┌──────────────────────────────────────────────────────┐
│ │
│ 💬 AI Assistant │
│ │
│ Conversational interface coming soon. │
│ Talk to your node, ask questions, and │
│ manage everything through natural language. │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ What would you like to do? │ │
│ └──────────────────────────────────────┘ │
│ │
│ AIUI integration in development │
│ │
└──────────────────────────────────────────────────────┘
```
When AIUI is integrated, this becomes the conversational interface where users can say things like "Set up a Lightning node" and the system guides them through it via chat.
---
## Data Model
### UIMode Type
```typescript
type UIMode = 'gamer' | 'easy' | 'chat'
```
Stored in:
- `localStorage` as `archipelago-ui-mode` (immediate, works offline)
- `UIData.mode` on the backend (synced via WebSocket, persists across devices)
### Goal Types
```typescript
interface GoalDefinition {
id: string // 'open-a-shop'
title: string // 'Open a Shop'
subtitle: string // 'Accept Bitcoin payments with your own store'
icon: string // Icon identifier
category: string // 'commerce', 'payments', 'storage', etc.
requiredApps: string[] // ['bitcoin-core', 'lnd', 'btcpay-server']
steps: GoalStep[] // Sequential steps
estimatedTime: string // '~45 minutes'
difficulty: 'beginner' | 'intermediate'
}
interface GoalStep {
id: string
title: string // 'Install Bitcoin Node'
description: string
appId?: string // Which app this step provisions
action: 'install' | 'configure' | 'verify' | 'info'
isAutomatic: boolean // Can system do this without user input?
}
```
---
## Implementation Order
| Phase | What | Files Changed | Visible Effect |
|-------|------|--------------|----------------|
| 1 | Data layer | types, stores, data files | None (foundation) |
| 2 | Mode switching | Dashboard, Settings, Router | Mode toggle appears, nav changes |
| 3 | Easy mode views | Home refactor, EasyHome, GoalDetail | Easy mode is functional |
| 4 | Chat + polish | Chat placeholder, Spotlight goals, Pro goals section | Complete system |
Each phase deploys independently. Phase 1 is invisible. Phase 2 adds the switcher. Phase 3 makes Easy mode work. Phase 4 polishes everything.
---
## File Inventory
### New Files (10)
```
src/types/goals.ts — Goal type definitions
src/data/goals.ts — Goal catalog (7 goals)
src/stores/uiMode.ts — UI mode Pinia store
src/stores/goals.ts — Goal progress tracking
src/components/ModeSwitcher.vue — Mode toggle widget
src/components/GamerHome.vue — Extracted current Home content
src/components/EasyHome.vue — Easy mode goal cards
src/components/ChatHome.vue — Chat mode home wrapper
src/views/GoalDetail.vue — Goal workflow wizard
src/views/Chat.vue — Chat placeholder
```
### Modified Files (11)
```
src/types/api.ts — Add UIMode type + mode field to UIData
src/router/index.ts — Add goals/:goalId and chat routes
src/views/Dashboard.vue — Computed nav items, ModeSwitcher in sidebar
src/views/Home.vue — Mode dispatcher (GamerHome/EasyHome/ChatHome)
src/views/Settings.vue — Interface Mode selection section
src/data/helpTree.ts — Goals in Spotlight search
src/style.css — Mode switcher, goal card, wizard CSS
src/stores/app.ts — Sync mode from backend
src/api/rpc-client.ts — setUIMode() RPC method
src/components/SpotlightSearch.vue — Visual indicator for goal items
mock-backend.js — ui.set-mode handler
```
docs/bitcoin-multi-version-design.md
+300
View File
@@ -0,0 +1,300 @@
# Bitcoin Multi-Version Support — Design
<!-- ════════════════════════════════════════════════════════════════════
PROGRESS TRACKER / RESUME POINT (keep this current — update each session)
════════════════════════════════════════════════════════════════════
**Branch/worktree:** `bitcoin-multi-version` @ `/home/archipelago/Projects/archy-btcver`
(isolated — never touch `main` or the other agent's branch). All work UNCOMMITTED on
that branch as of last update.
**Last updated:** 2026-06-28 (session 2 — software end-to-end implemented)
**Motivation refresh:** BIP-110 signalling makes per-node version *choice* a real
requirement — runners must be able to pick / pin / switch Core & Knots versions.
**User direction this session:** finish the SOFTWARE end-to-end (Phase 1–3 + UI),
DEFER the Phase 0 image build pipeline. Downgrade policy = **warn + confirm + allow**.
### Status by phase
- [x] **Phase 1 — catalog schema** (`app_catalog.rs`): `CatalogVersion` struct +
`versions[]` + `catalog_versions()` / `catalog_default_version()` /
`catalog_image_for_version()` (same-repo guard) DONE. Pin suppresses update badge
in `available_update_for_app()` DONE. `versions[]` now EMITTED by
`scripts/generate-app-catalog.sh` (curated `VERSIONS` map) → `releases/app-catalog.json`
regenerated; bitcoin-core carries its one built version (28.4.0, default). **Knots
versions[] intentionally empty** (only floating `:latest` exists; design forbids
advertising floating). More versions light up automatically once Phase 0 builds
tagged images and they're appended to the `VERSIONS` map.
- [x] **Phase 2 — install-time selection**: `version_config.rs` (pin/auto-update
persistence + `is_downgrade()` + `auto_update_apps()`, unit-tested) DONE;
`install.rs` `persist_install_version_selection()` DONE; `prod_orchestrator.rs`
pinned-wins resolution DONE. **UI:** `MarketplaceAppDetails.vue` install panel shows
a version `<select>` (latest pre-selected) when the app offers ≥2 versions — passes
the choice to `package.install`. (Hidden today since only 1 version exists.)
- [x] **Phase 3 — in-app switch + auto-update toggle**:
- `package.versions` RPC (read) + `package.set-config` RPC (write, downgrade-gated)
→ new `api/rpc/package/set_config.rs`, wired in `mod.rs` + `dispatcher.rs`.
- Auto-update tick: `run_update_scheduler` now takes the orchestrator + calls
`apply_per_app_auto_updates()` hourly (opt-in, pin-respecting, catalog-driven).
- UI: "Version & Updates" card in `appDetails/AppSidebar.vue` (version switch +
auto-update toggle + downgrade warn/confirm); `rpc-client.ts` + types added.
- [x] **Phase 0 — image build pipeline**: `scripts/build-bitcoin-image.sh` —
downloads the OFFICIAL upstream tarball + SHA256SUMS(.asc), verifies SHA-256 **and**
the OpenPGP signature (fail-closed; pinned release-key fingerprints), builds a
minimal **rootless** image (debian-slim + verified `bitcoind`/`bitcoin-cli`),
smoke-tests `--version`, tags + pushes `:<version>`. Validated on Core 31.0
(pinned-GPG pass, smoke `v31.0.0`). **Published curated set** (registry
`lfg2025`): Core **31.0, 30.2, 29.3, 27.2, 26.2, 25.2** (28.4 already present —
kept, not overwritten) + Knots **29.3.knots20260508**. `VERSIONS` map in
`generate-app-catalog.sh` lists them; catalog regenerated. Adding a future release
= run the script for it, then prepend it to the map + regenerate.
### Verification status
- `cargo check -p archipelago` GREEN (backend). Frontend `npm run build` GREEN
(vue-tsc typecheck passes; new RPC strings confirmed in `web/dist`).
- Unit tests: `version_config` had a pre-existing parallel-test race (shared
process-global `ARCHIPELAGO_DATA_DIR`) — FIXED with an `ENV_LOCK` mutex + unique
per-test dirs. `set_config` `image_tag` test added.
- **Phase 0 images verified end-to-end**: SHA-256 + pinned-maintainer OpenPGP
signature (deterministic VALIDSIG check), built rootless, smoke-tested, **pushed
to the live registry** — confirmed remotely: `bitcoin` tags
{25.2,26.2,27.2,28.4,29.3,30.2,31.0} + `bitcoin-knots:29.3.knots20260508`.
- **NOT yet verified on `.228`** (CLAUDE.md invariant — do before any tag): install
bitcoin-core, open its page, switch/pin a version, confirm recreate. All code
UNCOMMITTED on the branch.
### Gotchas captured (for resume)
- `gpg --verify` exit code is unreliable on multi-sig `SHA256SUMS` — must parse
`--status-fd` VALIDSIG and require a pinned maintainer fpr (script does this).
- `podman push` needs the sandbox disabled (`/var/tmp` is RO under the harness
sandbox) and `--tls-verify=false` (registry serves HTTP). Persistent keyring
(`BITCOIN_KEYRING_DIR`) avoids flaky per-build keyserver fetches.
### Next action when resuming
1. Re-verify: `cd archy-btcver/core && CARGO_INCREMENTAL=0 cargo check -p archipelago`
and `cargo test -p archipelago -- version_config set_config`; `cd neode-ui && npm run build`.
2. Live-verify on `.228`: install bitcoin-core, open its detail page → "Version &
Updates" card; exercise `package.versions` / `package.set-config` via RPC.
3. Commit on the branch (checkpoint).
4. **Phase 0** when greenlit: build+push tagged Core/Knots images, then extend the
`VERSIONS` map in `scripts/generate-app-catalog.sh` and regenerate the catalog.
### Decisions still needed from user (see §6 open questions)
Curated version set + storage budget (defaulted to current+~3 majors); when to do
Phase 0 image pipeline; pruned-node downgrade policy refinement (currently warn+confirm
for all). Auto-update default = OFF (opt-in), as recommended.
════════════════════════════════════════════════════════════════════ -->
**Status:** design (2026-06-22)
**Goal:** let a user choose *which* version of Bitcoin Core / Bitcoin Knots to
install (latest pre-selected, older versions in a dropdown), and later switch
versions or opt into auto-update — all manifest/catalog-driven, all served from
**our signed registry**, rootless, with **zero data loss** across version
changes.
See also: [`docs/registry-manifest-design.md`](registry-manifest-design.md)
(catalog distribution + signing this builds on),
[`docs/PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md) (gate that must be
green first), `MEMORY → project_decoupled_app_updates`,
`MEMORY → project_manifest_driven_north_star`.
> **Scheduling:** this is net-new scope. It lands **after** the production test
> gate (`tests/lifecycle/run-20x.sh`) is green on `.228` + `.198`. The data-
> preservation invariant (downgrade vs. chainstate) is the highest risk here.
---
## 1. Where we are today
### Image source / build
| Thing | Today |
|-------|-------|
| `apps/bitcoin-core/Dockerfile` | `FROM bitcoin/bitcoin:24.0` — a **community** image, **stale** (manifest says 28.4), no project-official Docker image exists |
| `apps/bitcoin-knots/` | **no Dockerfile** — `:latest` is built/pushed by hand |
| Registry | `scripts/image-versions.sh` → `ARCHY_REGISTRY="146.59.87.168:3000/lfg2025"`; only `BITCOIN_KNOTS_IMAGE=…/bitcoin-knots:latest` pinned, no Core pin |
| Tags in registry | **one tag per image**. No historical versions. |
### Version pinning
- `apps/bitcoin-core/manifest.yml` → `…/bitcoin:28.4` (pinned).
- `apps/bitcoin-knots/manifest.yml` → `…/bitcoin-knots:latest` (**floating** — a
liability for reproducibility and for "switch back to the version I had").
- `core/archipelago/src/container/app_catalog.rs` + `app-catalog/catalog.json`:
signed, hourly-fetched, carries `version` (badge text) + `image`.
`catalog_image_override()` overrides the manifest image **only if same-repo**.
`available_update_for_app()` already ignores floating tags for update
detection.
### Install path
- `prod_orchestrator.rs::install_fresh()` resolves the image as
**manifest image → catalog override → pull**. There is **no per-install
version parameter** — `orchestrator.install(app_id)` takes only the id.
- RPC `package.install` (`api/rpc/package/install.rs`) *accepts* `dockerImage` /
`version` params but for orchestrator-managed apps (bitcoin-core / bitcoin-knots
are allowlisted) it **ignores them** and lets the orchestrator resolve.
- **Conflict guard** (`prod_orchestrator.rs` ~1306–1325): core and knots may not
run simultaneously. Must be preserved by everything below.
### UI
- Install is **one-click, no modal** (`MarketplaceAppDetails.vue::installApp()`).
- Update badge + "Update to X" already exist (`appDetails/AppHeroSection.vue`,
RPC `package.update`).
- **No** Bitcoin-specific settings panel; all apps share `AppSidebar.vue`.
- Per-app config persisted **only at install time** as `containerConfig` →
`/var/lib/archipelago/app-configs/<id>.json`. **No post-install set-config RPC.**
---
## 2. Source-of-truth decision: official upstream → our registry
We use the **official releases** as upstream provenance, but nodes only ever pull
from our registry. Nodes do **not** fetch bitcoin.org / GitHub at install time —
that would break rootless/offline installs and the signed-registry trust model,
and neither project publishes an official Docker image anyway.
**Official sources (verified):**
| Impl | Index | Per-version asset pattern |
|------|-------|---------------------------|
| Bitcoin Core | [bitcoincore.org/en/releases](https://bitcoincore.org/en/releases/) · [github bitcoin/bitcoin](https://github.com/bitcoin/bitcoin/releases) | `https://bitcoincore.org/bin/bitcoin-core-<ver>/bitcoin-<ver>-x86_64-linux-gnu.tar.gz` + `SHA256SUMS` + `SHA256SUMS.asc` |
| Bitcoin Knots | [github bitcoinknots/bitcoin](https://github.com/bitcoinknots/bitcoin/releases) · [bitcoinknots.org/files](https://bitcoinknots.org/) | `https://bitcoinknots.org/files/<maj>.x/<ver>/bitcoin-<ver>-x86_64-linux-gnu.tar.gz` (`<ver>` e.g. `29.3.knots20260508`) |
Both ship **signed binary tarballs** with multi-builder Guix attestations
(`SHA256SUMS.asc`). The build pipeline verifies these **once, at build**; our DHT
Phase 0 registry signature then carries provenance to the fleet.
> Knots version strings embed a build date (`29.3.knots20260508`). Treat the full
> string as the tag; surface a friendly `29.3` + date in the UI.
---
## 3. Design
### Phase 0 — Reproducible, verified image pipeline *(prerequisite)*
New `scripts/build-bitcoin-image.sh <impl> <version>` that, per version:
1. Downloads the official tarball + `SHA256SUMS(.asc)` (GitHub release assets are
an identical mirror → fallback).
2. Verifies SHA256 **and** the Guix/builder GPG signatures. **Fail closed.**
3. Builds a minimal **rootless** image: pin a small base, unpack
`bitcoind`/`bitcoin-cli`. Keep the existing entrypoint probe
(`command -v bitcoind || find /opt -path '*/bin/bitcoind'`) so per-version
layout differences don't break startup.
4. Tags + pushes `:<version>` **and** updates the default pin (`:latest` /
`:28.4`-style) to the registry.
**Curate, don't mirror everything.** Publish a bounded set (proposal: current +
last ~3 majors), e.g. Core `31.0, 30.0, 29.3, 28.4, 27.2` and Knots
`29.3.knots…, 28.1.knots…, 27.1.knots…`. **`log` / document dropped versions** —
silent truncation reads as "all versions supported" when it isn't.
Also fixes existing debt: replaces the stale community `FROM bitcoin/bitcoin:24.0`
and gives Knots a real Dockerfile + non-floating tags.
### Phase 1 — Version catalog (signed, registry-distributed)
Extend `AppCatalogEntry` (forward-compatible — no `deny_unknown_fields`, old nodes
ignore it):
```jsonc
"bitcoin-core": {
"version": "31.0", // default / latest (existing field)
"image": "…/bitcoin:31.0", // existing
"versions": [ // NEW
{ "version": "31.0", "image": "…/bitcoin:31.0", "default": true },
{ "version": "30.0", "image": "…/bitcoin:30.0" },
{ "version": "28.4", "image": "…/bitcoin:28.4", "deprecated": true, "eol": "2026-...." }
]
}
```
Published to `releases/app-catalog.json`, signed by the existing release-root
mechanism. This is the **single source of truth** the UI reads for "what can I
install / switch to," and third-party-registry apps inherit the capability for
free. `version`/`image` stay as the default for back-compat.
### Phase 2 — Install-time version selection
- **Orchestrator:** add `install_with_image(app_id, Option<image_tag>)` (or an
optional arg on `install`). When a tag is supplied, **validate same-repo**
against the manifest (reuse `image_without_registry_or_tag()`), then override in
`install_fresh()`. Default path unchanged. Preserve the core/knots conflict
guard.
- **RPC:** thread the selected version/image from `package.install` into the
orchestrator for the allowlisted apps (the param is already received — just not
forwarded).
- **UI:** the first **install modal** in the app — latest pre-selected, dropdown
of `versions[]`, deprecated/EOL badges on old entries. On confirm, pass the
chosen version to `package.install`.
### Phase 3 — In-app version switch + auto-update toggle
- **UI:** a Bitcoin **"Version & Updates"** card (conditional in `AppSidebar.vue`
for `bitcoin-core` / `bitcoin-knots`): current version, a switch dropdown, and
an **auto-update-to-latest** toggle.
- **Switch = controlled re-pull/recreate** reusing the `package.update`
machinery but targeting an arbitrary (incl. older) tag → effectively
`package.set-version`.
- **Persistence:** new `package.set-config` RPC writing the existing
`app-configs/<id>.json` (`{ pinnedVersion, autoUpdate }`).
- **Auto-update:** the existing hourly catalog check, when `autoUpdate:true`,
triggers `package.update` to the catalog default. A pinned version **suppresses
the update badge**.
---
## 4. Invariants & safety rails
- **Rootless only.** Pipeline images and run path stay rootless; no Docker-socket,
no privileged.
- **No data loss across version change.** Preserve `/var/lib/archipelago/bitcoin`,
secrets (`bitcoin-rpc-password`, `…-rpcauth`), ports, and the adoption container
name on every install / switch / update.
- **⚠️ Downgrade vs. chainstate (highest risk).** Bitcoin Core refuses to start on
a chainstate written by a *newer* version unless reindexed (expensive, or data
loss on a pruned node). The UI **must** warn loudly on downgrade; the
orchestrator should gate/confirm it and never silently wipe. Pruned nodes can't
simply `-reindex`.
- **Core ⇄ Knots switch** stays governed by the existing conflict guard; treat an
impl switch as distinct from a version switch.
- **Floating tags** (`latest`) are never advertised as a selectable "version" and
never counted as an available update (already handled by
`available_update_for_app`).
- **Verify on a real node** (`.228` then `.198`) and pass `run-20x` before any
tag.
---
## 5. Files / seams (no code yet)
| Concern | File |
|---------|------|
| Image build/push | new `scripts/build-bitcoin-image.sh`; `apps/bitcoin-core/Dockerfile`; new `apps/bitcoin-knots/Dockerfile`; `scripts/image-versions.sh` |
| Catalog schema | `core/archipelago/src/container/app_catalog.rs`; `releases/app-catalog.json` (+ `app-catalog/catalog.json`) |
| Install override | `core/archipelago/src/container/prod_orchestrator.rs` (`install` / `install_fresh`); `api/rpc/package/install.rs`; `api/rpc/dispatcher.rs` |
| Switch / set-config RPC | `api/rpc/package/update.rs`; new `package.set-config` handler; `app-configs/<id>.json` |
| Install modal | `neode-ui/src/views/MarketplaceAppDetails.vue`; new `…/marketplace/AppInstallModal.vue` |
| Version & Updates card | `neode-ui/src/views/appDetails/AppSidebar.vue`; `neode-ui/src/api/rpc-client.ts`; `neode-ui/src/types/api.ts` |
---
## 6. Open questions
1. **Curated version set** — how many majors back do we host, and storage budget
on the registry?
2. **Multi-arch** — fleet is x86_64 today; do any nodes need arm64 images?
3. **Pruned-node downgrade policy** — block outright, or allow with an explicit
"this will require re-sync / may lose pruned data" confirmation?
4. **Auto-update default** — off (opt-in) for a consensus-critical app like
Bitcoin? (Recommended: **off**, explicit opt-in.)
5. **Knots date-suffix UX** — how to display `29.3.knots20260508` cleanly.
---
## Sources
- [Bitcoin Core releases](https://bitcoincore.org/en/releases/)
- [bitcoin/bitcoin releases](https://github.com/bitcoin/bitcoin/releases)
- [bitcoinknots/bitcoin releases](https://github.com/bitcoinknots/bitcoin/releases)
- [Bitcoin Knots](https://bitcoinknots.org/)
- [bitcoin.org version history](https://bitcoin.org/en/version-history)
docs/bitcoin-rpc-relay.md
+291
View File
@@ -0,0 +1,291 @@
# Bitcoin RPC Relay for External Wallets
This note captures the pattern used to let an external wallet, such as Wasabi,
use an Archipelago Bitcoin node for transaction relay without exposing the
node's admin RPC credentials.
## Goal
Expose a public HTTPS JSON-RPC endpoint that can broadcast transactions and read
basic chain/mempool state, while preventing wallet and admin RPC access.
The endpoint should be fronted by nginx or another TLS reverse proxy:
```text
wallet client -> https://<subdomain>/ -> reverse proxy -> Archipelago node nginx -> bitcoind RPC
```
Do not expose Bitcoin RPC credentials with wallet/admin access to external
users.
## Restricted RPC User
Create a separate RPC user, currently named `txrelay`, with an `rpcauth` secret
and a Bitcoin RPC whitelist.
Allowed RPC methods:
```text
sendrawtransaction
submitpackage
testmempoolaccept
getmempoolinfo
getrawmempool
getmempoolentry
getnetworkinfo
getblockchaininfo
getblockcount
getblockhash
getblockheader
getrawtransaction
gettxout
decoderawtransaction
decodescript
estimatesmartfee
```
Wallet/admin access is denied by setting `-rpcwhitelistdefault=0` and giving the
`txrelay` user only the method whitelist above.
Secrets live under:
```text
/var/lib/archipelago/secrets/bitcoin-rpc-txrelay-password
/var/lib/archipelago/secrets/bitcoin-rpc-txrelay-rpcauth
/var/lib/archipelago/secrets/bitcoin-rpc-txrelay-client.env
```
Do not commit these files or paste them into docs.
## Archipelago UI/API Flow
The productized flow is managed from the Bitcoin Core/Knots custom UI in the
`Transaction Relay Sharing` panel.
Implemented RPC methods:
```text
bitcoin.relay-status
bitcoin.relay-update-settings
bitcoin.relay-request-peer
bitcoin.relay-approve-request
bitcoin.relay-reject-request
bitcoin.relay-create-tor-service
```
When peer sharing is enabled, `bitcoin.relay-update-settings` automatically
provisions the restricted `txrelay` password, `rpcauth`, and client env file if
they do not already exist. If those files were just generated, restart Bitcoin
Core/Knots so `bitcoind` reloads the `txrelay` `rpcauth` and whitelist flags.
The UI shows:
```text
HTTP / HTTPS / Tor relay endpoint settings
local sync status
restricted credential readiness, without printing the password
trusted peer dropdown, disabled until the local node is synchronized
incoming relay requests with approve/reject actions
outbound relay requests and approval status
```
Approving an incoming peer request sends the selected endpoint plus restricted
`txrelay` credentials through the existing encrypted peer-message path. On the
requesting node, approved peer credentials are stored in a per-peer secret env
file:
```text
/var/lib/archipelago/secrets/bitcoin-relay-peer-<peer-pubkey-prefix>.env
```
The UI returns the credential secret path and approved endpoint metadata, but it
does not display the raw password.
For dev review, the mock server exposes the Bitcoin UI at:
```text
http://localhost:8102/app/bitcoin-ui/
```
## Bitcoin Startup Flags
The Bitcoin Knots app should add the restricted user only when the secret exists:
```sh
RPC_TXRELAY_AUTH="$(printenv BITCOIN_RPC_TXRELAY_RPCAUTH || true)"
RPC_TXRELAY_FLAGS="-rpcwhitelistdefault=0"
if [ -n "$RPC_TXRELAY_AUTH" ]; then
RPC_TXRELAY_FLAGS="$RPC_TXRELAY_FLAGS -rpcauth=$RPC_TXRELAY_AUTH -rpcwhitelist=txrelay:sendrawtransaction,submitpackage,testmempoolaccept,getmempoolinfo,getrawmempool,getmempoolentry,getnetworkinfo,getblockchaininfo,getblockcount,getblockhash,getblock,getblockheader,getrawtransaction,gettxout,gettxspendingprevout,decoderawtransaction,decodescript,estimatesmartfee,uptime,ping,getconnectioncount,getpeerinfo,getindexinfo,getdeploymentinfo,getchaintips"
fi
```
Then include `$RPC_TXRELAY_FLAGS` in the `bitcoind` command. Keep the local
`archipelago` RPC user unrestricted for internal services by using
`-rpcwhitelistdefault=0` and only setting a whitelist for `txrelay`.
The current implementation touches:
```text
apps/bitcoin-knots/manifest.yml
scripts/container-specs.sh
```
## Node nginx
The Archipelago node can expose a host-based nginx vhost that proxies to local
Bitcoin RPC:
```nginx
limit_req_zone $binary_remote_addr zone=bitcoin_rpc_ext:10m rate=5r/s;
server {
listen 80;
server_name rpc.example.com;
client_max_body_size 2m;
location / {
limit_req zone=bitcoin_rpc_ext burst=20 nodelay;
limit_req_status 429;
proxy_pass http://127.0.0.1:8332;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s;
proxy_send_timeout 120s;
proxy_read_timeout 120s;
proxy_buffering off;
}
}
```
If another public reverse proxy terminates TLS, point it at:
```text
http://<archipelago-lan-ip>:80
```
For the tested node the LAN upstream was:
```text
http://192.168.1.116:80
```
The public proxy should serve a valid TLS certificate for the chosen subdomain.
## DNS and Routing
Use a subdomain that resolves to the public reverse proxy:
```text
Type: A
Host/Name: <subdomain-only>
Value: <public-ip>
```
For example, if the desired hostname is `rpc.example.com`, the DNS host/name
field is usually only `rpc`, not the full `rpc.example.com`. Entering the full
hostname in some DNS panels can accidentally create:
```text
rpc.example.com.example.com
```
The public proxy should forward:
```text
TCP 443 -> TLS reverse proxy for the subdomain
TCP 80 -> optional, needed for HTTP-01 certificate issuance or redirects
```
If the public proxy is separate from the Archipelago node, configure it with:
```text
server_name: <subdomain>
scheme: http
upstream host: <archipelago-lan-ip>
upstream port: 80
```
## Verification
Check authoritative DNS:
```sh
dig @<authoritative-dns-ip> <subdomain> A +noall +answer +authority
dig @1.1.1.1 +short <subdomain> A
```
Check TLS:
```sh
openssl s_client -connect <subdomain>:443 -servername <subdomain> </dev/null
```
Check the public RPC path:
```sh
. /var/lib/archipelago/secrets/bitcoin-rpc-txrelay-client.env
curl -sS --user "$BITCOIN_RPC_TXRELAY_USER:$BITCOIN_RPC_TXRELAY_PASSWORD" \
--data-binary '{"jsonrpc":"1.0","id":"check","method":"getblockchaininfo","params":[]}' \
"<relay-endpoint-url>"
```
Check that transaction broadcast reaches Bitcoin RPC, without needing a real
transaction:
```sh
curl -sS --user "$BITCOIN_RPC_TXRELAY_USER:$BITCOIN_RPC_TXRELAY_PASSWORD" \
--data-binary '{"jsonrpc":"1.0","id":"badtx","method":"sendrawtransaction","params":["00"]}' \
"<relay-endpoint-url>"
```
Expected result is a Bitcoin RPC validation error such as `TX decode failed`,
which confirms the request reached `sendrawtransaction`.
If a wallet verifies the connection but reports `RPC Forbidden` during
broadcast, the credentials authenticated but the broadcast method was outside
the loaded `txrelay` whitelist. Restart the active Bitcoin backend after
updating the whitelist, then test both `sendrawtransaction` and, for newer
package-relay clients, `submitpackage`. Also confirm the public reverse proxy
passes the wallet's `Authorization` header through to `127.0.0.1:8332`; do not
point public wallet traffic at the Bitcoin UI `/bitcoin-rpc/` helper, because
that helper injects the local dashboard credential.
Check that wallet/admin RPC is blocked:
```sh
curl -sS -o /tmp/txrelay-deny.json -w '%{http_code}\n' \
--user "$BITCOIN_RPC_TXRELAY_USER:$BITCOIN_RPC_TXRELAY_PASSWORD" \
--data-binary '{"jsonrpc":"1.0","id":"deny","method":"listwallets","params":[]}' \
"<relay-endpoint-url>"
```
Expected result:
```text
403
```
## Tested Outcome
The working endpoint used in this setup was:
```text
https://shard.tx1138.com/
```
It was verified with:
```text
DNS resolves
TLS certificate is valid
txrelay credentials authenticate
getblockchaininfo returns chain=main
sendrawtransaction reaches Bitcoin RPC
listwallets is blocked for txrelay
```
docs/bitcoin-version-bulletproof-rollout.md
+131
View File
@@ -0,0 +1,131 @@
# Bitcoin Multi-Version — Bulletproofing & Rollout (handoff)
> **Status 2026-06-29:** code + images + catalog + frontend DONE on branch
> `bitcoin-version-bulletproof` (base commit `095a76cd`, plus the catalog-generator
> + handoff follow-ups). **.228 is the test node**: binary + frontend + catalog are
> live there; its Knots chainstate is mid-**reindex recovery** (see §5). The fleet
> rollout (OTA binary+frontend, mirror catalog publish, `:latest` repoint) is the
> **coordinated step the other agent owns** — see §4. Pairs with
> `docs/bitcoin-multi-version-design.md` (the original design).
## 1. What was broken (root causes)
User report: "switched Knots to `v29.3.knots20260508`, version didn't update in the UI."
Three **stacked** bugs, plus a data-corruption hazard:
1. **Reconciler reverted the pin.** `prod_orchestrator::sync_quadlet_unit` re-rendered the
quadlet every reconcile tick using the manifest's `:latest`, ignoring the per-app
pinned version → any switch silently reverted within one tick.
2. **Entrypoint render bug.** The renderer folded the manifest `entrypoint: ["sh","-lc"]`
into `Exec=`. That only works when the image ENTRYPOINT is a passthrough shell wrapper.
The versioned images use `ENTRYPOINT ["bitcoind"]`, so `Exec=sh -lc …` became
`bitcoind sh -lc …` → `unexpected token 'sh'` → crash loop.
3. **Image USER divergence.** The versioned images were built `USER bitcoin` (uid 1000);
the legacy `:latest` ran as **root**. Chain data is owned by the `data_uid`
(host 100101 / container uid 102). Root reads it via `CAP_DAC_OVERRIDE` (granted in the
manifest); uid-1000 cannot → `Error initializing block database`.
4. **Data hazard (already hit on .228).** Repeated failed starts under mixed UIDs left
bitcoind's two LevelDBs (`blocks/index/` + `chainstate/`) truncated to KB stubs while
the raw `blocks/blk*.dat` (797 GB) stayed intact. Recovery = `bitcoind -reindex` from
local blocks (no re-download). The uniform-root image fix (below) removes the mixed-UID
cause going forward; the proper switch flow was already data-safe (600s stop grace,
clean stop→rm→recreate, conflict-stops the other impl — they share port 8332 + datadir
`/var/lib/archipelago/bitcoin`).
## 2. What was fixed (all on the branch)
- **Renderer** (`core/archipelago/src/container/`):
- `prod_orchestrator.rs`: factored `resolve_catalog_image()` (catalog/pinned-version →
image) and call it in BOTH `install_fresh` and `sync_quadlet_unit` — the pin now
survives reconcile.
- `quadlet.rs`: emit a real `Entrypoint=<first>` + `Exec=<rest+cmd>` instead of folding;
`exec_changed` now also diffs `Entrypoint=` so the recreate fires. Validated against
the live podman 5.4.2 quadlet generator.
- **Images** (`scripts/build-bitcoin-image.sh`, `apps/bitcoin-{knots,core}/Dockerfile`):
removed `USER bitcoin` → run as **container-root** like legacy (still 100% rootless:
container-root maps to the unprivileged host service user; `CAP_DAC_OVERRIDE` from the
manifest lets bitcoind read the `data_uid`-owned datadir). **All** images rebuilt root +
pushed to the mirror (`146.59.87.168:3000/lfg2025`):
- Knots: `29.3.knots20260508`, `29.3.knots20260507`, `29.3.knots20260210`, `29.2.knots20251110`
- Core: `25.2 26.2 27.2 28.4 29.2 29.3 30.2 31.0` + `latest` (→31.0)
- **Catalog** (`scripts/generate-app-catalog.sh` VERSIONS map + regenerated
`releases/app-catalog.json`): Knots & Core `versions[]` populated; the generator now
forces top-level `version` == the `default` entry's version (the `169ff2e2` invariant)
regardless of the manifest version. Knots `latest` entry points at the newest **dated**
image (`29.3.knots20260508`) so "Always use latest" = newest on fixed-binary nodes.
- **Frontend** (`neode-ui/`):
- `AppSidebar.vue`: rename the latest option to **"Always use the latest version"**
(no `v` prefix), fix right padding, and `pickSelection()` guarantees the bound value is
a real option (fixes the blank dropdown).
- New `components/InstallVersionModal.vue`: full-screen version chooser shown from the
App Store / Discover **card** install button for multi-version apps — app icon +
"Install <name>", latest pre-selected. Wired in `Discover.vue handleInstall`.
- i18n keys: `appDetails.alwaysUseLatestVersion`, `marketplace.installModalTitle/Hint`.
## 3. Current live state on .228 (test node)
- Binary with both renderer fixes: **deployed** (`/usr/local/bin/archipelago`).
- New frontend bundle: **deployed** to `/opt/archipelago/web-ui` (hard-refresh to see it).
- Updated catalog: placed at `/var/lib/archipelago/app-catalog.json` (local override —
will refresh from the mirror's OLDER copy at the next hourly fetch until §4 publishes it).
- Knots: `bitcoin-knots` service held **stopped** (`package.stop`, user_stopped);
a detached `bitcoin-knots-reindex` container is rebuilding the index+UTXO (§5).
## 4. Remaining — coordinated fleet rollout (OTHER AGENT)
Do this together with the other workstream's release, AFTER both are ready:
1. **Merge** branch `bitcoin-version-bulletproof` into the release line.
2. **Build + OTA** the binary + frontend (these carry the renderer fix + UI). The renderer
fix is a **hard prerequisite** for the new images everywhere — see fleet-safety below.
3. **Publish the catalog** to the mirror (push `releases/app-catalog.json` to gitea-vps2
`main`, the raw URL nodes fetch hourly). The current catalog is **fleet-safe even before
the binary lands**: unpinned/auto-update nodes resolve via the manifest's floating
`:latest` (still the legacy image); only explicit version selection (needs the new UI)
uses the new root images.
4. **Only AFTER the binary is fleet-wide:** optionally repoint the `bitcoin-knots:latest`
tag → `29.3.knots20260508` (root) and simplify the catalog `latest` entry back to the
`:latest` tag. **Do NOT repoint `:latest` before then** — old-binary nodes fold
`Exec=sh -lc …` and would crash on an `ENTRYPOINT ["bitcoind"]` image. (Core never
worked on old binaries — it always shipped `ENTRYPOINT ["bitcoind"]` — so Core has no
such constraint.)
5. **Verify the full switch matrix** on a healthy node (§6).
## 5. Finishing .228's reindex (OTHER AGENT owns this — not babysat by the original author)
The detached `bitcoin-knots-reindex` container runs the new **root** `29.3.knots20260508`
image with `-reindex -server=0` against `/var/lib/archipelago/bitcoin`. It holds the datadir
lock, so the managed service (held stopped) can't collide. When it has connected blocks up
to ~the prior tip (height ≥ ~955800) it's done; then:
```sh
# on .228 (SSH/sudo/UI pw all: ThisIsWeb54321@)
podman stop -t 600 bitcoin-knots-reindex && podman rm bitcoin-knots-reindex
# start the managed service via RPC (sets desired=running, clears user_stopped):
# package.start {id: bitcoin-knots} (POST https://127.0.0.1/rpc/v1, CSRF: echo csrf_token cookie as X-CSRF-Token)
# verify:
podman exec bitcoin-knots sh -lc '$(command -v bitcoind) --version | head -1' # → v29.3.knots20260508
# RPC up → the Bitcoin UI populates; it syncs the gap to tip.
```
The "Bitcoin RPC connection refused (127.0.0.1:8332)" the UI shows is EXPECTED until this
swap (reindex runs with RPC off).
## 6. Switch-matrix test plan (what "bulletproof" must prove)
On a healthy node, each step must end with bitcoind running + RPC answering + syncing, with
NO `Error initializing block database` and NO data loss:
- Knots: switch `latest` → `29.3.knots20260507` → `29.3.knots20260210` → back to `latest`.
- Core: install `latest`; switch `31.0` → `28.4.0`.
- **Knots ↔ Core** (shared datadir/port): Knots→Core upgrade path (Core ≥ data version) and
the reverse. **Cross-major DOWNGRADES** (e.g. 29.x data → Core 28.4) legitimately need a
reindex — the UI already surfaces a downgrade warning; confirm it does and that confirming
reindexes cleanly rather than crash-looping.
- Reboot survival after each switch.
## 7. Notes / assumptions
- **"29.2"** in the request doesn't exist as a Knots build (404 upstream); added as **Bitcoin
Core 29.2** (exists). Revisit if a Knots 29.2 was meant.
- Reindex is unavoidable ONLY because .228's index was already corrupted by the pre-fix
crash loop; a normal switch on the fixed binary does NOT reindex.
- Creds for .228: SSH/sudo + UI/RPC all `ThisIsWeb54321@`.
docs/bulletproof-containers.md
+314
View File
@@ -0,0 +1,314 @@
# Bulletproof Containers for Beta
**Status**: plan agreed 2026-04-22, implementation started.
**Target**: zero-manual-intervention container lifecycle for the beta launch. A user installs, uninstalls, reboots, updates, or loses power — every combination must leave the node in a known-good state without SSH.
**Project memory**: `~/.claude/projects/-home-archipelago-Projects-archy/memory/project_reconcile_architecture.md`
**Failure log**: `~/.claude/projects/-home-archipelago-Projects-archy/memory/feedback_container_lifecycle_failure_modes.md`
---
## Why we're doing this
The v1.7.38 and v1.7.39 rollouts on 2026-04-22 exposed a cluster of container-lifecycle failures that required manual SSH recovery on every affected node (.116, .198, .228, .253). If a user had been on those nodes, they'd have been stuck with "can't reach" or 500 errors and no path forward. We can't ship beta with this class of failure on the table.
The pattern under every failure: **the canonical source of truth had the right answer, but derived state drifted away from it and nothing noticed or fixed it.**
### The six failure modes
| # | Symptom | Root cause |
|---|---|---|
| FM1 | `archy-bitcoin-ui` + `archy-lnd-ui` disappeared from `podman ps -a` after a daemon restart | Archipelago owns container creation imperatively; no owner recreates companions after a crash mid-transition |
| FM2 | ElectrumX "Daemon connection problem" | `bitcoin.conf`'s `rpcauth` drifted from `/var/lib/archipelago/secrets/bitcoin-rpc-password` — config written once at install, never re-derived |
| FM3 | archipelago.service `status=226/NAMESPACE` crash-loop SIGKILL'd every child container | Containers were children of archipelago's cgroup; systemd teardown killed them. `KillMode=control-group` default |
| FM4 | `host.containers.internal` inside containers resolved to LAN gateway (192.168.1.254) | Known podman bug on bridge networks pre-5.3 ([#22644](https://github.com/containers/podman/issues/22644)) |
| FM5 | Nginx 500 fleet-wide after OTA | Tarball root dir was `drwx------` (700), extracted identically on every node. Fixed in v1.7.40 at build time; still need post-OTA auto-rollback |
| FM6 | Rootless podman's `libpod/bolt_state.db` vanished → whole registry node unreachable | No detection of corrupt state; required manual `rm -rf /run/user/$UID/libpod` + `podman system renumber` |
---
## Architecture decision
**Adopt balena-style, level-triggered, desired-state reconciler built on Quadlet + sdnotify.**
This is the one architecture that would have prevented all six failures, because each one is "reality drifted from the intended config and nothing noticed" — the exact problem reconcilers are designed for.
### Why not the alternatives
- **Keep imperative + patch per-failure** — we've been doing this. Five releases in a day. Doesn't scale.
- **Migrate to LXC (StartOS's path)** — 6-month project. Our investment in podman (`install.rs`, `docker_packages.rs`, `image_versions.rs`) is substantial. Quadlet gives us StartOS's isolation property without the migration.
- **Ship k3s / MicroShift** — 400-800 MB RAM baseline on top of bitcoind/electrs. Overkill for a home node OS.
- **Edge-triggered like Umbrel** — their `app.ts` has an explicit TODO admitting they don't handle failure events. We'd inherit the same bug class.
### The four patterns (from mature players)
1. **Desired-state-first, level-triggered reconcile.** balena-supervisor, Kubernetes operators, NixOS. A supervisor owns a manifest of *what should run*; on every tick it diffs against *what is running* and issues steps.
2. **Every container is its own systemd unit, not a child of the daemon.** Red Hat's Quadlet pattern: a `.container` file is parsed by a systemd *generator* into a normal `.service`. The daemon can crash without taking any containers with it.
3. **sdnotify readiness + HealthCmd + rollback.** Podman v3.4+ has real rollback: bad image fails health check, systemd considers service failed, Podman re-tags the previous image digest.
4. **Credentials and config derived from canonical secrets on every apply.** Not trusted across upgrades; re-rendered idempotently from single source of truth.
### Fix-per-failure
| Failure | Fix |
|---|---|
| FM1 | Move companions to Quadlet `.container` files in `/etc/containers/systemd/`. systemd (not archipelago) owns them |
| FM2 | `reconcile::derived::render_bitcoin_conf(secrets)` — pure function, runs every tick, atomic rewrite + HUP on drift |
| FM3 | `KillMode=mixed` in archipelago.service + containers in their own `archipelago-apps.slice`. Quadlet units already live outside archipelago's cgroup |
| FM4 | Ship `/etc/containers/containers.conf` with `host_containers_internal_ip = "10.89.0.1"` + `default_rootless_network_cmd = "pasta"`; also `--add-host=host.archipelago:10.89.0.1` in every unit |
| FM5 | Post-OTA `curl -k https://127.0.0.1/` health probe in new binary startup. If non-200 within 90s, rollback to `web-ui.bak` + binary-backup |
| FM6 | Startup probe: `podman info` with timeout. On "invalid internal status", clear `/run/user/$UID/{containers,libpod,podman}` + `podman system renumber` + reconcile tick rebuilds from Quadlet units |
---
## New code layout (lands in v1.7.48)
```
core/archipelago/src/reconcile/
mod.rs run_reconcile_loop, reconcile_once — called from main.rs
desired.rs DesiredState built from packages.json + catalog + secrets
current.rs snapshot via `systemctl list-units archy-*.service` + `podman ps -a --format json`
diff.rs pure: reconcile(desired, current) -> Vec<Step> (unit-testable without podman)
apply.rs step executor with timeouts, structured logs, backoff
quadlet.rs write `.container` / `.volume` / `.network` units atomically
derived.rs render_bitcoin_conf, render_containers_conf, render_nginx_app_routes
backoff.rs restart-history tracking (moved from health_monitor.rs)
```
### Step types (idempotent)
```rust
enum Step {
WriteQuadletUnit(path, content),
WriteDerivedFile(path, content),
WriteSecret(path, content),
DaemonReload,
EnsureStarted(unit),
StopUnit(unit),
RestartUnit(unit),
PullImage(ref),
}
```
### Triggers
- 30s interval tick
- install/uninstall RPC
- update-applied event
- explicit `/rpc/v1/reconcile.tick`
- podman event stream (if available)
Level-triggered + idempotent — every call considers full desired vs current diff. Missed ticks/events are irrelevant.
### Edits to existing code
- **`src/main.rs`**: replace `tokio::spawn(crash_recovery::start_stopped_containers)` with `tokio::spawn(reconcile::run_reconcile_loop(state))`. Keep self-heal perms + PID-marker crash detection.
- **`src/api/rpc/package/install.rs`**: stop calling `podman run` directly. Writes desired state + Quadlet unit + signals reconciler. Reconciler does pull + `systemctl start`.
- **`src/api/rpc/package/runtime.rs`** + `lifecycle.rs` + `stacks.rs`: same pattern — mutate desired state, reconciler applies.
- **`src/crash_recovery.rs`**: keep PID-marker + snapshot. Delete `start_stopped_containers` (reconciler handles cold boot). Keep `user-stopped.json` as `AppSpec.desired_state: Started | UserStopped | Uninstalled`.
- **`src/health_monitor.rs`**: strip restart logic. Keep memory-leak detection; push unhealthy events as `Trigger::ContainerUnhealthy(name)`.
- **`src/bitcoin_rpc.rs`**: add `pub fn derive_rpcauth_line(user, pass) -> String` (HMAC-SHA256 per Bitcoin Core's `rpcauth.py`).
- **`src/update.rs`**: post-swap health probe + auto-rollback (v1.7.41).
---
## Shipping order
Each release is independently deployable. Not a big-bang rewrite.
### v1.7.41 — Post-OTA health probe + auto-rollback (closes FM5)
- In `update.rs`: write `/var/lib/archipelago/update-pending-verify.json` just before service restart, with `applied_at`, `new_version`, `previous_version`, deadline.
- In `main.rs` startup: read marker, spawn verification task. Wait 15s for full startup, then `curl -k https://127.0.0.1/` with retries up to 90s.
- On 200: delete marker.
- On non-200 after window: call `rollback_update(data_dir)` (already exists), restart service to boot the old binary.
- Smallest diff, highest ROI.
### v1.7.42 — containers.conf + host.archipelago alias (closes FM4)
- Idempotent write of `/etc/containers/containers.conf` on startup (archipelago compares hash, rewrites only on drift).
- Add `--add-host=host.archipelago:10.89.0.1` to every generated container in `install.rs` / `docker_packages.rs`.
- ElectrumX `DAEMON_URL` migrates from `host.containers.internal` → `host.archipelago`.
### v1.7.43 — `reconcile::derived` for bitcoin.conf / lnd.conf (closes FM2)
- Pure function `render_bitcoin_conf(secrets) -> String`.
- Tick every 30s: read secret, derive `rpcauth`, compare to on-disk, atomic rewrite (via `tempfile::NamedTempFile::persist`) + `podman exec ... kill -HUP 1` on drift.
- Same pattern for `lnd.conf`.
- First user of the eventual `reconcile::` module — ships the `derived.rs` piece early.
### v1.7.44 — Podman state self-heal on startup (closes FM6)
- Startup probe: `podman info --format '{{.Host.OS}}'` with 10s timeout.
- On "invalid internal status" or similar:
- `systemctl --user stop podman.socket podman.service`
- `rm -rf /run/user/$UID/{containers,libpod,podman}`
- `podman system renumber`
- Trigger reconcile tick (will rebuild containers from their source of truth)
- Surface clear error on `/health` if recovery fails — don't silently serve 502.
### v1.7.45–47 — Quadlet migration per companion (closes FM1 + FM3)
One companion per release so regressions have a narrow blame window:
- **v1.7.45**: `archy-bitcoin-ui` → Quadlet `.container` unit
- **v1.7.46**: `archy-lnd-ui` → Quadlet
- **v1.7.47**: `archy-electrs-ui` → Quadlet
Each:
1. Write `.container` file to `/etc/containers/systemd/<name>.container`
2. `systemctl daemon-reload`
3. `systemctl enable --now <name>.service`
4. Remove the `podman run` path from `install.rs` for that name
5. Add Goss probe for the lifecycle test matrix
### v1.7.48+ — Full reconcile module
- `core/archipelago/src/reconcile/` replaces imperative `install.rs` container management.
- Main app containers (bitcoin-knots, bitcoin-core, lnd, electrumx, btcpay-server, mempool, fedimint) become Quadlet units.
- `install.rs` shrinks to ~300 lines of "write desired state, poke reconciler."
- Biggest diff, lands last.
---
## Test harness (parallel track)
### Stack
- **Outer runner**: `bats-core` — TAP-style bash testing, readable by anyone
- **Verifier**: `goss` — YAML assertions on ports, processes, HTTP endpoints, files. Reused by CI + live probe
- **Chaos layer**: Chaos Toolkit JSON experiments (steady-state-hypothesis → method → rollback → verify)
- **VM layer**: `vmtest` (Go) for reboot-survival + ISO-boot tests, or raw QEMU+SSH
- **Tor probe**: curl through archipelago's own tor SOCKS5 (`--socks5-hostname 127.0.0.1:9050`), 60-180s retry window
- **Live probe**: small Rust agent on every fleet node, ships same Goss YAMLs to Prometheus. Neither Umbrel nor StartOS has this — real differentiator.
- **Reproducibility**: btrfs subvolume snapshots primary (fast), QEMU qcow2 for ISO/kernel-level repro
### Directory layout
```
tests/lifecycle/
bats/
_helpers.bash # install_app, wait_healthy, assert_no_orphans
00_bootstrap.bats
10_install.bats # per-app install
20_ui_reachable.bats # direct port + HTTPS proxy + iframe
30_tor_reachable.bats # .onion probe
40_stop_start.bats
50_restart.bats
60_reboot.bats # vmtest-driven
70_reinstall.bats # idempotence + data preservation
80_uninstall.bats # leak check
90_soak.bats # 2-6h hold, periodic probe
goss/
bitcoin-knots.yaml
bitcoin-core.yaml
lnd.yaml
electrumx.yaml
btcpay-server.yaml
mempool.yaml
fedimint.yaml
chaos/
kill9_archipelago_mid_install.json
wipe_bolt_db.json
kill9_bitcoind.json
reboot_during_ota.json
corrupt_bitcoin_conf.json
systemctl_restart_mid_install.json
fill_disk_99_percent.json
kill_tor.json
delete_nginx_snippet.json
clock_jump_30min.json
vm/
iso_boot_smoke.go
reboot_survival.go
ci/
vm_runner.sh
collect_artifacts.sh
probe/archy-probe/ # Rust bin, reuses goss YAMLs, ships to fleet
Makefile # `make beta-matrix`, `make chaos`, `make soak`
```
### Minimum beta matrix
7 apps × 9 lifecycle events × 10 chaos scenarios. Pass = every MUST-ship cell green on fresh rootless-podman single-node CI.
| Case \ App | knots | core | lnd | electrumx | btcpay | mempool | fedimint |
|---|---|---|---|---|---|---|---|
| Fresh install | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| UI direct port | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| UI HTTPS proxy | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| UI iframe | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Tor .onion reachable | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ |
| Stop → ports released | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Restart → integrations | — | — | ✓↔btc | ✓↔btc | ✓↔btc,lnd | ✓↔electrs | — |
| Reboot survival | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Reinstall idempotent | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Uninstall no orphans | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| 6h soak | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
**Harness scaffold lands in v1.7.41.** First lifecycle tests blocking v1.7.45. Full matrix + chaos suite blocking beta tag.
### Chaos scenarios (10)
Ordered by likelihood × severity:
1. `kill -9 archipelagod` mid-install → systemd restart, in-flight install resumes or cleanly rolls back
2. `rm bolt_state.db` while service stopped → restart regenerates, no data loss in named volumes
3. `systemctl restart archipelago` mid-install → no orphans, no half-state
4. Reboot mid-OTA → old version intact OR new version active, never half
5. Corrupt `bitcoin.conf` → container restart-loops; UI surfaces banner; reconcile re-derives; other apps unaffected
6. Fill `/var` to 99% → graceful degradation, disk-pressure report
7. Revoke rootless-netns → self-heal within Tor descriptor window
8. `pkill -9 tor` → supervisor restarts; onions reachable within 3–5 min
9. Delete nginx conf snippet → reconciler rewrites or `archipelago doctor` flags drift
10. Clock jump +30min → daemons survive; Tor recovers
---
## Decision log
| Decision | Answer | Rationale |
|---|---|---|
| Scope | 6+ incremental releases, not big-bang rewrite | Each closes one failure class, narrow blame window |
| Quadlet migration | Yes | Isolation from daemon crashes, systemd-native recovery, free from Red Hat's production patterns. Minimum podman version becomes 4.4+ (fine for modern Debian) |
| Live probe to Prometheus | Yes, part of beta | Genuine differentiator — neither Umbrel nor StartOS has this. Adds Grafana dep |
| Test gating | Scaffold in v1.7.41, first tests blocking v1.7.45, full matrix blocking beta tag | Gradual rather than all-or-nothing |
---
## Key sources
### Architecture
- Umbrel [app.ts](https://raw.githubusercontent.com/getumbrel/umbrel/master/packages/umbreld/source/modules/apps/app.ts) — edge-triggered, TODO on failure handling
- StartOS [repo](https://github.com/Start9Labs/start-os), [v0.4 podman→LXC announce](https://community.start9.com/t/startos-v0-4-0-alpha-10-has-replaced-podman-new-commands-for-terminal/4062)
- balena-supervisor [repo](https://github.com/balena-os/balena-supervisor), [Supervisor API](https://docs.balena.io/reference/supervisor/supervisor-api)
- Quadlet: [Dan Walsh 2023 blog](https://www.redhat.com/en/blog/quadlet-podman), [podman-systemd.unit(5)](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html)
- Podman rollback: [auto-update blog](https://www.redhat.com/en/blog/podman-auto-updates-rollbacks), [podman-auto-update(1)](https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html)
- Kubernetes operator pattern: [Kubebuilder reconcile](https://deepwiki.com/kubernetes-sigs/kubebuilder/5.2-reconciliation-loop), [good practices](https://book.kubebuilder.io/reference/good-practices)
- NixOS containers: [wiki](https://wiki.nixos.org/wiki/NixOS_Containers)
### Known bugs & references
- `host.containers.internal` → LAN: [podman #22644](https://github.com/containers/podman/issues/22644), [#23782](https://github.com/containers/podman/issues/23782)
- `bolt_state.db` recovery: [podman #17730](https://github.com/containers/podman/issues/17730), [staticdir mismatch #20872](https://github.com/containers/podman/issues/20872)
- aardvark-dns flakiness: [#20396](https://github.com/containers/podman/issues/20396), [#22407](https://github.com/containers/podman/issues/22407)
- systemd 226/NAMESPACE: [Arch forum](https://bbs.archlinux.org/viewtopic.php?id=156963), [systemd #29526](https://github.com/systemd/systemd/issues/29526)
- [systemd CGROUP_DELEGATION](https://systemd.io/CGROUP_DELEGATION/), [systemd.kill(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html)
### Test harness prior art
- Umbrel [ci.yml](https://github.com/getumbrel/umbrel/blob/master/.github/workflows/ci.yml) — Vitest + qemu matrix fan-out
- [YunoHost package_check](https://github.com/YunoHost/package_check) — closest analog, scored per-app lifecycle harness on LXC
- [bats-core](https://github.com/bats-core/bats-core)
- [Goss](https://github.com/goss-org/goss), [dgoss](https://github.com/aelsabbahy/goss-docker)
- [Chaos Toolkit](https://chaostoolkit.org/)
- [vmtest (Go)](https://github.com/anatol/vmtest)
### Tor
- [rend-spec-v3](https://github.com/torproject/torspec/blob/main/rend-spec-v3.txt) — descriptor lifetime + republish cadence
- [stem](https://stem.torproject.org/) — Python Tor controller for `HS_DESC UPLOADED` waits
---
## To resume
1. Read project memory: `~/.claude/projects/-home-archipelago-Projects-archy/memory/project_reconcile_architecture.md`
2. Read failure-mode memory: `~/.claude/projects/-home-archipelago-Projects-archy/memory/feedback_container_lifecycle_failure_modes.md`
3. Check task list for current release (should start with v1.7.41)
4. Current state on fleet as of 2026-04-22:
- All 4 mirrors (tx1138, gitea-local, .160, .168) synced to v1.7.40-alpha
- .116, .198, .228, .253 healed manually via `systemd-run chmod 755 /opt/archipelago/web-ui`
- .228 still has stale `bitcoin.conf` rpcauth (regenerated during triage; will drift again until v1.7.43)
- .228 UI companions (archy-bitcoin-ui, archy-lnd-ui) keep vanishing (Quadlet migration in v1.7.45+ fixes)
- .160 Gitea required `podman system renumber` recovery (v1.7.44 automates this)
5. Implementation is in progress on `main` branch — next edit is `core/archipelago/src/update.rs` for v1.7.41.
docs/container-architecture.html
+4279
View File
File diff suppressed because it is too large Load Diff
docs/demo-build-info.md
+109
View File
@@ -0,0 +1,109 @@
# Archipelago Public Demo — build info & status
**Status:** implemented & deployable (2026-07-14)
**Branch:** `main` — the demo machinery was merged from the old `demo-build`
branch and now lives on main, pushed to
`gitea-vps2` = `http://146.59.87.168:3000/lfg2025/archy.git`.
A public, click-to-play demo of the Archipelago UI, 100% mock-data driven,
multi-visitor, deployed via Portainer. See also `docs/archive/demo-deployment-design.md`
(original design) and `demo-deploy/` (thin prebuilt-image stack).
---
## Deploy (Portainer)
Build-from-repo (works today, no registry needed):
| Field | Value |
|-------|-------|
| Repository URL | `http://146.59.87.168:3000/lfg2025/archy.git` |
| Reference | `refs/heads/main` |
| Compose path | `docker-compose.demo.yml` |
| Auth | user `lfg2025`, password = Gitea token |
| UI port | **2100** · Login password: **`entertoexit`** |
Redeploy after each push. `docker-compose.demo.yml` builds two images
(`neode-ui/Dockerfile.backend` = mock server, `neode-ui/Dockerfile.web` = nginx+UI).
The thin `demo-deploy/docker-compose.yml` pulls prebuilt `:demo` images instead
(needs the CI image pipeline / registry wired — `.github/workflows/demo-images.yml`).
### Flags / env
- Backend: `DEMO=1` (compose sets it) → multi-session sandbox, no real runtime.
- Web build: `VITE_DEMO=1` (Dockerfile.web ARG, default 1) → inlined demo UI behaviour.
- Optional: `ANTHROPIC_API_KEY` (NOT needed — AIUI chat is canned in demo),
`DEMO_SESSION_TTL_MS` (45m), `DEMO_MAX_SESSIONS` (500), `DEMO_FILE_QUOTA_BYTES` (50MB).
---
## Architecture
Everything is gated behind `DEMO` (off = classic single-user dev mock, unchanged).
- **`neode-ui/mock-backend.js`** — the entire fake backend (Node/Express, ~95+ RPCs).
- **Per-session isolation:** `AsyncLocalStorage` + Proxy. Globals (`mockData`,
`walletState`, `userState`, `mockState`, `bitcoinRelayMockState`) are Proxies
that resolve to the current request's store, keyed by a `demo_sid` cookie.
Deep-cloned from `SEED_*` on first hit; idle-reaped; per-session WS fan-out.
- **Files:** per-session in-memory store + curated disk files (see below).
- Forces simulation mode in DEMO (`docker=null`).
- **`neode-ui/src/composables/useDemoIntro.ts`** — the frontend demo switch
(`IS_DEMO`), per-day intro gate, `DEMO_PASSWORD`, app demoability + launch URLs.
- **`neode-ui/docker/nginx-demo.conf`** — routes `/rpc`, `/ws`, `/app/*`,
`/electrs-status`, `/proxy/`, `/lnd-connect-info`, the IndeeHub/Mempool
reverse-proxies, and the SPA.
- **`docker/{bitcoin-ui,electrs-ui,lnd-ui,fedimint-ui}/`** — the REAL registry app
UIs, served statically under `/app/<id>/` with mocked data endpoints.
- **`demo/aiui/`** — prebuilt AIUI dist (chat is canned; `?mockArchy&seed`).
- **`demo/files/`** — curated cloud files drop-in (see below).
## Demo features (all implemented)
Per-session sandbox · per-session file upload (Range streaming) · testnet/signet
flavor · per-day intro replay · `entertoexit` login (prefilled + hint) · version
`<real>-demo` · onboarding wizard skipped (intro kept) · "No demo" install gating ·
real app UIs (Bitcoin Core vs Knots by subversion, ElectrumX, LND, Fedimint;
Mempool/IndeeHub iframed) · 12 federation nodes / 5 peers · FIPS active · interactive
buy flow (testnet addresses, bolt11, 2s QR) · real testnet tx links (mempool.space) ·
networking profits 5,231,978 sats + labelled wallet txs · VPN · Nostr relays ·
node-visibility toggle · dummy Cashu mints + Fedimint federations · AIUI canned
reply + `?mockArchy` mock data + `?seed` pre-loaded "Content Showcase" chat.
---
## Curated cloud files (`demo/files/`)
Drop real files into `demo/files/<Folder>/<file>` and commit — they become the
cloud content for every visitor (read-only; git access = the "private login").
Loader **merges per top-level folder**: adding `Music/` swaps only Music and keeps
the sample Documents/Photos/Videos. Empty → built-in seeds. Text inlined; binaries
streamed from disk with HTTP Range (seek). Backend reads `/demo/files` —
**Dockerfile.backend COPYs it; `.dockerignore` must allow it.**
---
## Gotchas (READ before editing)
- **Sibling dirs need both the Dockerfile COPY and a `.dockerignore` allow.**
`docker/bitcoin-ui`, `docker/electrs-ui`, `docker/lnd-ui`, `docker/fedimint-ui`,
`demo/files` are outside `neode-ui/`; they're copied into the backend image and
un-ignored in `.dockerignore` (`* ` + `!docker/` + `docker/*` + `!docker/<ui>/`).
Forgetting either → Portainer build "not found" or runtime 500/404.
- **Real app UIs assume root-serving** — served via `express.static('/app/<id>')`
+ `/app/<id>/assets/*` → `/assets/*` redirect + per-path data endpoints
(`bitcoin-status`, `rpc/v1`, `bitcoin-rpc/`, `/proxy/lnd/*`, `/electrs-status`).
- **Uploaded-via-UI files are ephemeral** (per-session, lost on redeploy/reap).
Only `demo/files/` persists.
- **Mempool iframe is best-effort** (third-party CSP/websockets). **IndeeHub** is
reverse-proxied with header-strip + `sub_filter` asset rewrite; if still black,
it's indee's own `X-Frame-Options` (fix on that server).
- **AIUI `?seed` bootstrap hardcodes the current AIUI bundle hash**
(`/aiui/assets/seedPrompts-CLWaUv28.js`) — re-paste if AIUI is rebuilt. Tiny
first-load IndexedDB race (one refresh shows the chat).
- **Running mock-backend.js locally in the sandbox is flaky:** start backgrounded,
`sleep 5+`, then curl; NEVER `pkill -f mock-backend` (it matches & kills the
shell) — use `pkill -x node`.
- **Delete-405** seen pre-redeploy was nginx/stale; backend DELETE returns 200.
---
## Commit trail (demo-build, newest last)
`2715f2d8` sandbox → … → `7efebb4a` media merge + AIUI seed. ~14 commits, all
`feat(demo)/fix(demo)`.
docs/developer-guide.md
+313
View File
@@ -0,0 +1,313 @@
# Archipelago Developer Guide
## Project Structure
```
archy/
├── core/ # Rust backend
│ └── archipelago/
│ ├── src/
│ │ ├── main.rs # Entry point, module declarations
│ │ ├── api/rpc/ # RPC endpoint handlers
│ │ │ ├── dispatcher.rs # Route dispatcher (~380 method arms)
│ │ │ ├── auth.rs # Login, session, TOTP
│ │ │ ├── container.rs # Container lifecycle
│ │ │ ├── package/ # Package install/lifecycle/stacks
│ │ │ ├── interfaces.rs # Network interfaces, WiFi, DNS
│ │ │ ├── federation/ # Federation management
│ │ │ ├── marketplace.rs # Community marketplace
│ │ │ └── ... # Other endpoint groups (mesh/, identity/, lnd/, tor/, system/)
│ │ ├── auth.rs # Password hashing, sessions
│ │ ├── config.rs # Configuration loading
│ │ ├── server.rs # HTTP/WS server (axum)
│ │ ├── container/ # Podman integration
│ │ ├── network/ # Network management
│ │ │ ├── dns.rs # DNS configuration
│ │ │ ├── router.rs # UPnP, diagnostics
│ │ │ └── dwn_*.rs # DWN protocol
│ │ ├── federation/ # Federation protocol
│ │ ├── marketplace.rs # Marketplace discovery
│ │ ├── identity.rs # DID key management
│ │ ├── vpn.rs # VPN (Tailscale/WireGuard)
│ │ ├── mesh/ # Tri-protocol mesh (Meshtastic/MeshCore/Reticulum)
│ │ └── ...
│ ├── Cargo.toml
│ └── tests/ # Integration tests
├── neode-ui/ # Vue 3 frontend
│ ├── src/
│ │ ├── api/ # RPC client, WebSocket, container client
│ │ │ └── rpc-client.ts # Central RPC client (all backend calls)
│ │ ├── views/ # Page components
│ │ │ ├── Home.vue # Dashboard with system stats
│ │ │ ├── Marketplace.vue # App store (curated + community)
│ │ │ ├── Server.vue # Network, VPN, DNS management
│ │ │ ├── Federation.vue # Federation dashboard
│ │ │ ├── Settings.vue # User settings
│ │ │ ├── Web5.vue # DID, DWN, Nostr
│ │ │ └── ...
│ │ ├── stores/ # Pinia state management
│ │ ├── components/ # Reusable UI components
│ │ ├── composables/ # Vue composables
│ │ ├── router/ # Vue Router with guards
│ │ ├── types/ # TypeScript type definitions
│ │ └── style.css # Global styles + Tailwind utilities
│ ├── vite.config.ts
│ └── package.json
├── scripts/ # Deployment and utility scripts
│ ├── deploy-to-target.sh # Main deploy script
│ ├── first-boot-containers.sh # ISO first-boot setup
│ └── run-tests.sh # CI test runner
├── image-recipe/ # ISO build configuration
│ ├── build-auto-installer-iso.sh
│ └── configs/ # Nginx, systemd configs
├── docs/ # Documentation
│ ├── architecture.md
│ ├── app-manifest-spec.md
│ ├── marketplace-protocol.md
│ └── multi-node-architecture.md
├── apps/ # App manifests (YAML)
├── CLAUDE.md # AI development instructions
└── docs/ROADMAP.md # Project roadmap
```
## Development Setup
### Prerequisites
- **macOS** (development machine): Node.js 20+, npm
- **Linux server** (`192.168.1.228`): Rust toolchain, Podman, Nginx, Debian 13
- SSH key: `~/.ssh/archipelago-deploy`
### Local Frontend Development
```bash
cd neode-ui
npm install
npm start # Vite dev server on :8100, mock backend on :5959
```
The dev server at `http://localhost:8100` uses a mock backend. Login with `password123`.
### Deploying Changes
**Never build Rust on macOS.** The deploy script rsyncs source to the Linux server and builds there.
```bash
# Deploy to live server (builds backend + frontend, restarts services)
./scripts/deploy-to-target.sh --live
# Deploy to both servers
./scripts/deploy-to-target.sh --both
```
The deploy script:
1. Rsyncs source to the server
2. Builds Rust backend on the server (`cargo build --release`)
3. Builds Vue frontend (`npm run build`)
4. Copies artifacts to production paths
5. Restarts the `archipelago` systemd service
6. Runs a health check
### Running Tests
```bash
# Frontend tests
cd neode-ui && npm test
# Backend tests (on dev server via SSH)
ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228 \
"cd ~/archy/core && cargo test --all-features"
# Both
./scripts/run-tests.sh
```
## Adding a New RPC Endpoint
### 1. Create the Handler
Add a handler method in the appropriate file under `core/archipelago/src/api/rpc/`. If no existing file fits, create a new one.
```rust
// core/archipelago/src/api/rpc/mymodule.rs
use super::RpcHandler;
use anyhow::Result;
impl RpcHandler {
/// mymodule.action — description of what it does.
pub(super) async fn handle_mymodule_action(
&self,
params: Option<serde_json::Value>,
) -> Result<serde_json::Value> {
let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("Missing required parameter: name"))?;
// Your logic here
let result = do_something(name).await?;
Ok(serde_json::json!({ "ok": true, "result": result }))
}
}
```
**Key patterns:**
- Handlers are `pub(super)` — visible only to the RPC router
- Accept `Option<serde_json::Value>` for params (omit for parameterless endpoints)
- Return `Result<serde_json::Value>`
- Use `self.config.data_dir` for data persistence
- Use `anyhow::bail!()` for error responses
### 2. Register the Route
Add the module declaration in `core/archipelago/src/api/rpc/mod.rs`, then add
the route arm to the `dispatch()` match in
`core/archipelago/src/api/rpc/dispatcher.rs`:
```rust
// api/rpc/mod.rs, at the top:
mod mymodule;
// api/rpc/dispatcher.rs, in the dispatch() match statement:
"mymodule.action" => self.handle_mymodule_action(params).await,
```
### 3. Add Module (if new)
If your logic warrants a separate module:
```rust
// core/archipelago/src/main.rs
mod mymodule; // Add to module declarations
```
### 4. Frontend Client
Add a convenience method to `neode-ui/src/api/rpc-client.ts`:
```typescript
async myAction(params: { name: string }): Promise<{ ok: boolean; result: string }> {
return this.call({
method: 'mymodule.action',
params,
})
}
```
### 5. Deploy and Test
```bash
./scripts/deploy-to-target.sh --live
curl -X POST http://192.168.1.228/rpc/v1 \
-H "Content-Type: application/json" \
-b "archipelago_session=YOUR_SESSION" \
-d '{"method":"mymodule.action","params":{"name":"test"}}'
```
## Adding a New Vue Page
### 1. Create the Component
```vue
<!-- neode-ui/src/views/MyPage.vue -->
<template>
<div>
<h1 class="text-4xl font-bold text-white mb-2">My Page</h1>
<div class="glass-card p-6">
<!-- Content here -->
</div>
</div>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue'
import { rpcClient } from '@/api/rpc-client'
// State and logic
</script>
```
### 2. Add the Route
In `neode-ui/src/router/index.ts`, add inside the dashboard children:
```typescript
{
path: 'my-page',
name: 'my-page',
component: () => import('@/views/MyPage.vue'),
},
```
### 3. Standards
- Always use `<script setup lang="ts">` — never Options API
- Use `glass-card` for containers, `bg-white/5 rounded-lg` for sub-rows
- Create global CSS classes in `src/style.css` instead of inline Tailwind
- Use `rpcClient` from `@/api/rpc-client.ts` for all backend calls
- Handle loading states and errors for all async operations
## Writing Tests
### Frontend (Vitest)
```typescript
// neode-ui/src/api/__tests__/my-test.test.ts
import { describe, it, expect, vi, beforeEach } from 'vitest'
describe('MyFeature', () => {
beforeEach(() => {
vi.restoreAllMocks()
})
it('should do something', async () => {
vi.stubGlobal('fetch', vi.fn().mockResolvedValue({
ok: true,
json: () => Promise.resolve({ result: 'ok' }),
}))
// Test your logic
expect(true).toBe(true)
})
})
```
### Backend (Rust)
```rust
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
#[tokio::test]
async fn test_my_function() {
let dir = tempdir().unwrap();
let result = my_function(dir.path()).await.unwrap();
assert_eq!(result, expected);
}
}
```
## Code Quality Checklist
- [ ] TypeScript strict mode: no `any`, use `unknown` or proper types
- [ ] No `unwrap()` or `expect()` in production Rust code — use `?`
- [ ] No `console.log` — wrap in `if (import.meta.env.DEV)`
- [ ] No empty catch blocks — log or handle errors
- [ ] Functions under 50 lines
- [ ] `cargo clippy` and `cargo fmt` pass
- [ ] `npx vue-tsc --noEmit` passes
- [ ] Security: validate all inputs, no command injection
- [ ] Container security: readonly_root, no_new_privileges, non-root user
## Contributing
1. Create a feature branch: `git checkout -b feature/my-feature`
2. Make changes following the standards above
3. Test locally: `cd neode-ui && npm test`
4. Deploy to dev server: `./scripts/deploy-to-target.sh --live`
5. Verify at `http://192.168.1.228`
6. Commit with conventional format: `feat: add my feature`
docs/dht-distribution-design.md
+185
View File
@@ -0,0 +1,185 @@
# DHT / Peer-Distributed Content Design
**Status:** Design (no code yet) · **Date:** 2026-06-16 · **Author:** archipelago + Claude
## 1. Purpose
Make Archipelago's large-file movement **peer-distributed**: a node should be able to
fetch content (OTA updates, app/OCI images, IndeeHub films) from *any other node that
already has it*, falling back to the central origin only when no peer can serve it.
This document covers three use-cases that are **the same problem** —
"fetch content-addressed bytes from whatever node already has them, verify, fall back to
origin":
1. **OTA releases** — node binaries + frontend tarballs.
2. **App installs** — container/OCI images.
3. **IndeeHub streaming** — films created in "backstage" on one node, streamable from any
node that has them stored or cached.
### Guiding principle (decided 2026-06-16)
> **Swarm-assist, origin always wins.** The peer swarm is an *optimization*. The central
> origin (OVH HTTP release assets / MinIO) remains the **guaranteed fallback** and the
> source of truth for reliability. We never bet correctness or availability on the P2P
> layer. This is what keeps the system bulletproof while the P2P stack matures.
## 2. Current state (verified 2026-06-16)
### OTA (`core/archipelago/src/update.rs`)
- Manifest at `DEFAULT_UPDATE_MANIFEST_URL` (`update.rs:67`) = vps2 OVH
(`146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json`).
- `check_for_updates()` (`:565`) walks an operator mirror list (`default_mirrors()` `:105`,
`load_mirrors()` `:123`), origin-rewrites component URLs to the chosen mirror
(`rewrite_manifest_origins()` `:227`).
- `download_component_resumable()` (`:821`) — resumable HTTP Range download, 6 retries,
exponential backoff.
- **Integrity: SHA-256 only** (`:984`), compared against `ComponentUpdate.sha256`.
- **No authenticity:** manifests are *unsigned*. A compromised mirror can serve a malicious
but hash-consistent binary. Post-apply health probe + auto-rollback exist
(`verify_pending_update()` `:389`, `rollback_update()` `:1423`) but that is not a
substitute for signature verification.
- Manifest schema: `{version, release_date, changelog[], components[{name, current_version,
new_version, download_url, sha256, size_bytes}]}`.
### App installs (`core/archipelago/src/api/rpc/package/install.rs`)
- `handle_package_install()` (`:195`) → `do_pull_image()` (`:1062`) tries each registry from
`container/registry.rs` in priority order (OVH primary), `rewrite_image()` rewrites the
origin, `podman pull`. Same centralized-mirror shape as OTA.
### Transport & identity (already P2P-capable)
- `transport/mod.rs` — `NodeTransport` trait (`:74`), `TransportRouter` (`:336`), priority
stack Mesh→LAN→FIPS→Tor. `PeerRegistry` (`:199`) tracks per-peer addresses
(mesh id, LAN ip:port, `fips_npub`, onion).
- Seed-derived identity (`seed.rs`): node Ed25519 (`archipelago/node/ed25519/v1`), node
Nostr secp256k1 (`archipelago/nostr-node/secp256k1/v1`), FIPS secp256k1
(`archipelago/fips/secp256k1/v1`). DID + npub per node.
- **Already content-addressed:** `blobs.rs` stores `blobs/<cid>` keyed by **SHA-256** hex,
with HMAC-SHA256 capability tokens (`BlobMeta`, 64 MiB cap). `transport/chunking.rs` does
Reed-Solomon chunking for LoRa.
### Trust scaffolding — **NOT built yet**
- No `core/src/trust/`, no `ROOT_PUBKEY`, no `derive_release_root_*`, no
`archipelago/release/root/*` HKDF strings, no JCS/canonical JSON, no signing ceremony
scripts, no `manifest-v2.json`. The "Phase 0 signed manifest" design exists only as notes.
### IndeeHub (the streaming target)
- Original platform (not a fork). Working source: `~/Projects/Indeedhub Prototype/`
(Vue 3 + NestJS). Submodule `146.59.87.168:3000/lfg2025/indeehub.git` (repointed off the retired host —
needs a live remote). In `archy`: image-only, `apps/indeedhub/manifest.yml` pulls
`146.59.87.168:3000/lfg2025/indeedhub:1.0.0` (+ `-api`, `-ffmpeg`, postgres, redis,
minio, nostr-rs-relay).
- Streaming today: FFmpeg → **HLS (.m3u8 + AES-128 .ts segments)** in **MinIO**
(`indeedhub-private`/`-public`), metadata in Postgres, transcode queue in Redis,
auth via Nostr (NIP-98). Glue: `install.rs:68` `patch_indeedhub_nostr_provider()`
injects the NIP-07 provider into the nginx-wrapped frontend.
- **No "backstage" code yet** — it's the creator/upload side we're introducing.
## 3. Protocol evaluation (verified maintenance status, 2026-06-16)
| Option | Verdict | Why |
| --- | --- | --- |
| **Web5 / TBD / DWN** | ❌ Reject | Block **wound TBD down**, handed components to DIF (`TBD54566975`→`decentralized-identity`). `web5-js` latest release **0.12.0, Oct 2024** (~20 mo stale). DWN spec still **Draft**. DWNs are DID-scoped *record stores*, not a blob-streaming swarm. Fails the "well-maintained + bulletproof" bar. |
| **iroh / iroh-blobs** | ✅ Swarm engine | **v1.0.0 shipped 2026-06-15.** Rust (matches core), **BLAKE3 verified streaming** over **QUIC + hole-punching + relays**, content-addressed, KB→TB, **native byte-range** support (ideal for HLS). n0 team, production relays. |
| **Nostr Blossom** | ✅ Index/catalog layer | SHA-256-addressed blobs over HTTP, modular BUD specs (BUD-01/02/04/05/06/08), actively developed, **already aligned** (Nostr identity everywhere; `blobs.rs` already SHA-256). Server-centric (not a peer swarm) → use as discovery + IndeeHub catalog + HTTP fallback, not the distribution engine. |
| **libp2p-kad (hand-rolled DHT)** | ⚠️ De-prioritize | Was the old "Phase 4 build a Kademlia" plan. iroh 1.0 supersedes the need to hand-roll discovery + swarm. Revisit only if iroh proves unworkable. |
**Note vs. prior plan:** the saved DHT design said "no iroh as a Phase 0–5 dep (revisit
post-Phase 3)." iroh hitting 1.0 removes the main reason for that deferral — **this design
reverses that non-choice** and adopts iroh as the swarm layer, collapsing the from-scratch
Kademlia work.
## 4. Recommended architecture — three layers, one engine
Build **one** peer-distribution layer; use it for all three use-cases.
```
┌─────────────────────────────────────────────┐
Authenticity │ Signed Nostr events (per-node npub) + │ "who published this,
& Discovery │ seed-derived RELEASE ROOT key for OTA + │ who has it"
│ Blossom BUD catalog for IndeeHub │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
Integrity & │ BLAKE3 content addressing (iroh-native, │ "name bytes by hash,
Addressing │ range-verifiable). SHA-256 kept in manifest │ verify on arrival"
│ during migration window. │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
Transport & │ iroh-blobs swarm (peers that already have │ "move the bytes"
Swarm │ it) ─── fallback ───▶ OVH HTTP / MinIO │
│ origin (ALWAYS wins) │
└─────────────────────────────────────────────┘
```
- **Integrity/addressing — BLAKE3.** iroh-native, supports verified *range* streaming
(essential for HLS + resumable). Keep SHA-256 in the manifest for back-compat through the
migration window; add a `blake3` field alongside.
- **Discovery/authenticity — signed Nostr events + release root key.**
- OTA: the **Phase 0 seed-derived release root key** signs the manifest (BLAKE3 root hash
+ version). Integrity ≠ authenticity — content addressing proves *bytes are intact*, the
signature proves *we authorized them*. Both are required.
- "Who has blob X" advertised via signed Nostr events `{content-hash, provider-npub, ts}`,
so nodes find seeds without a central tracker.
- IndeeHub: Blossom BUDs for the film catalog + provider/mirror lists.
- **Transport/swarm — iroh-blobs, origin fallback.** Node asks the swarm for a hash; peers
that have it serve range-verified BLAKE3 streams; if the swarm yields nothing, fall back to
the existing resumable HTTP path (`update.rs:821`) against OVH/MinIO. **A node that
finishes a download automatically becomes a seed.**
### Bulletproof posture
The swarm sits *above* a proven HTTP path, never in place of it. Worst case (every peer
offline, iroh bug, NAT failure) the node downloads exactly as it does today. iroh 1.0 is new;
this containment is deliberate.
## 5. Use-case flows
### OTA / app installs
1. Node reads the **signed** manifest (via signed Nostr event or HTTP), gets BLAKE3 root hash
+ release-root signature; verify signature → reject on failure.
2. Query swarm (signed provider events) for peers holding that hash.
3. Download range-verified BLAKE3 stream from peers; verify full BLAKE3 (+ SHA-256 during
migration).
4. No peers / failure → resumable HTTP from OVH (current path).
5. Apply + health-probe + auto-rollback (unchanged). Updated node **becomes a seed**.
6. OCI images: content-address image layers the same way; OVH registry stays the origin.
### IndeeHub streaming ("backstage → any node")
1. Creator publishes a film in **backstage** → FFmpeg → HLS; **each .ts segment is a
content-addressed (BLAKE3) blob**, immutable and small → ideal swarm objects.
2. Publish a **signed Nostr event** advertising title + segment hashes (Blossom catalog).
3. Any node running IndeeHub resolves the content address and **streams from the nearest
node(s) that have it stored/cached** via iroh range streaming; MinIO/OVH is origin.
4. AES-128 key delivery + NIP-98 auth unchanged (keys gate decryption; swarm only moves
encrypted segments — so untrusted seeds can cache without seeing plaintext).
## 6. Phasing (folds into the existing Phase 0–6 plan)
0. **Signed manifests (required first, unbuilt).** `derive_release_root_ed25519` /
`derive_release_root_nostr` in `seed.rs` (HKDF `archipelago/release/root/ed25519/v1`,
`.../secp256k1/v1`); `core/src/trust/` (anchor/bundle/manifest/timestamp/nostr); JCS
canonical JSON; ceremony scripts; `manifest-v2.json` with signature. Gives *authenticity*,
which content-addressing does not.
1. **BLAKE3 alongside SHA-256** in the manifest + `blobs.rs`.
2. **iroh-blobs PoC** behind a feature flag: serve OTA blobs from the swarm with HTTP
fallback; measure on a scratch/test node, then the fleet.
3. **Signed Nostr advertisement events** for releases (publisher identity + provider lists).
4. **IndeeHub on the same blob layer** (Blossom catalog + iroh swarm; MinIO origin).
This collapses the old "Phase 4: build S/Kademlia from scratch" into "adopt iroh," a large
de-risking.
## 7. Open decisions
- **BLAKE3 migration scope:** dual-hash window length; whether to re-hash historical
releases or only BLAKE3 going forward.
- **iroh ↔ existing transports:** iroh brings its own QUIC + hole-punching + relays; decide
how it coexists with FIPS/Tor (run iroh standalone first; integrate with `TransportRouter`
later if useful).
- **Seed retention policy:** how long nodes keep blobs to seed others (disk pressure on small
nodes); pinning rules for IndeeHub films vs. transient OTA blobs.
- **Privacy:** iroh dial-by-key vs. Tor's anonymity; default transport per content type.
## References
- iroh: https://github.com/n0-computer/iroh · iroh-blobs: https://github.com/n0-computer/iroh-blobs · docs: https://docs.iroh.computer/protocols/blobs
- Blossom: https://github.com/hzrd149/blossom · NIP-B7: https://nips.nostr.com/B7 · nostr-blossom (Rust): https://docs.rs/nostr-blossom
- Web5/DWN (rejected): https://github.com/decentralized-identity/web5-js · https://identity.foundation/decentralized-web-node/spec/ · https://block.xyz/inside/block-contributes-digital-identity-components-to-the-decentralized-identity-foundation
docs/dual-ecash-design.md
+135
View File
@@ -0,0 +1,135 @@
# Dual-ecash: Cashu + Fedimint, seamlessly
Status: **in progress** (2026-06-17). FE scaffolding + Fedimint HTTP bridge landed and
compile-checked; live federation round-trip and networking-sats routing are not yet validated.
## Why
Today the node's wallet (`core/archipelago/src/wallet/ecash.rs`, `mint_client.rs`, `cashu.rs`)
speaks **only** the Cashu NUT HTTP protocol (BDHKE, `cashuA…` tokens). There is **no** Fedimint
*client* — `apps/fedimint` is only the guardian server, and the "local Fedimint" default mint at
`127.0.0.1:8175` is just the guardian UI nginx, which does not expose the Cashu NUT API. So:
- The node can hold/spend generic Cashu tokens, but cannot hold Fedimint ecash or join federations.
- "Networking sats" (streaming/seeding revenue) is hardcoded to the Cashu wallet.
Goal: support **both** ecash protocols seamlessly — hold balances in either, join arbitrary
federations, and let networking-sats be paid/received over whichever protocol the peer accepts.
## Architecture decision
**Containerized `fedimint-clientd` + thin HTTP bridge** (chosen over linking the native
`fedimint-client` Rust SDK into the binary, and over a Lightning-only bridge).
```
archipelago binary
├─ CashuMintClient ──HTTP (NUT /v1/*)──▶ cashu mint
└─ FedimintClient ──REST (/v2/*)─────▶ fedimint-clientd container ──▶ federation guardians
```
Rationale: keeps the heavy, fast-moving Fedimint SDK **out** of the main binary (no compile-time
coupling, no rebuild bloat, OTA-friendly), and fits the existing app/container architecture
(`apps/fedimint`, `apps/fedimint-gateway`). The Rust side is just a `reqwest` client, mirroring
`MintClient`.
### fedimint-clientd REST surface (v0.3.x)
- Auth: `Authorization: Bearer <password>`. Default port 8080 (we map it to host **8178** because
8080 is LND REST). Base path `/v2/...`.
- `GET /v2/admin/info` — per-federation balances (`totalAmountMsat`, denominations, meta).
- `POST /v2/admin/join` — `{ "inviteCode": "fed1…", "useManualSecret": false }` → joins / returns `federationId`.
- `POST /v2/mint/spend` — `{ federationId, amountMsat }` → serialized notes (ecash to send).
- `POST /v2/mint/reissue` — `{ federationId, notes }` → redeem received notes; returns reissued amount.
- `POST /v2/ln/invoice` / `POST /v2/ln/pay` — Lightning in/out (used for cross-protocol swaps).
- `GET /health`.
Multi-federation: requests carry a `federationId`; clientd's `multimint` manages many clients.
**Exact JSON field names must be pinned to the clientd image tag we vendor** — code defensively.
## Components
### 1. Container app — `apps/fedimint-clientd/manifest.yml`
Sidecar running `fedimint-clientd`, mirroring `apps/fedimint-gateway`. Host port 8178 → container
8080. Password from node secret `fedimint-clientd-password`. State volume at
`/var/lib/archipelago/fedimint-clientd`. Added to `RESERVED_PORTS` (port_allocator.rs) and
`fallback_package_port()` (server.rs).
### 2. Rust bridge — `core/archipelago/src/wallet/fedimint_client.rs`
Thin `reqwest` client: `info()`, `join()`, `spend()`, `reissue()`, `ln_invoice()`, `ln_pay()`.
`from_node(data_dir)` resolves base URL + password (env `FEDIMINT_CLIENTD_URL` /
`FEDIMINT_CLIENTD_PASSWORD`, else defaults + secret file). Tor-proxy support via `with_client`,
mirroring `MintClient`.
### 3. RPCs — `core/archipelago/src/api/rpc/fedimint.rs`
- `wallet.fedimint-list` → joined federations + balances (`{federation_id, name, balance_sats}[]`).
- `wallet.fedimint-join` `{invite_code}` → joins via clientd, persists to
`wallet/fedimint_federations.json`, returns `{federation_id}`.
- `wallet.fedimint-leave` `{federation_id}` → untracks locally.
- `wallet.fedimint-balance` → total sats across federations (from clientd `info`).
Local registry `wallet/fedimint_federations.json` = `{ federations: [{federation_id, name}] }` so
the list survives clientd being temporarily down; balances are live from clientd.
### 4. Frontend — `WalletSettingsModal.vue`
Tabbed: **Cashu Mints** (live: `streaming.list-mints` / `streaming.configure-mints`) and
**Fedimint Federations** (`wallet.fedimint-list` / `-join` / `-leave`). Gear icon on
`HomeWalletCard`. `fedimintBackendReady` flips to `true` once the RPCs ship; join degrades
gracefully with a clear error if the clientd app isn't installed.
### 4b. Default federation (zero-touch)
clientd auto-joins a default federation at boot via `FEDIMINT_CLIENTD_INVITE_CODE` (manifest), and
the Rust bridge `ensure_default_federation()` idempotently joins + tracks it (called from
`wallet.fedimint-list`) so already-running nodes pick it up too. Constant
`DEFAULT_FEDERATION_INVITE` in `fedimint_client.rs` is the single source on the Rust side; keep it
in sync with the manifest env. clientd is a **client, not the guardian** — it needs no local
`fedimintd`, so it bundles standalone.
### 4c. Bundling on every node
- **Bundled ISO:** add image to `scripts/image-versions.sh`, the ISO bundle `.tar` list, and a
core-create block in `scripts/first-boot-containers.sh`; mark `tier: "core"` in
`app-catalog/catalog.json`.
- **Unbundled ISO:** `first-boot-containers.sh` exits after FileBrowser only — add clientd to that
early-exit block so unbundled nodes also get it out of the box.
- **CAVEAT:** confirm the *current* ISO assembler before editing the bundle list — the one found is
under `image-recipe/_archived/` (likely stale); `first-boot-containers.sh`/`image-versions.sh`
are current.
- Image: build from source (no official image; `flake.nix` only) → push to vps2
`146.59.87.168:3000/lfg2025/fedimint-clientd:v0.4.0`.
### 5. Unified balance
`HomeWalletCard` ecash row = Cashu `wallet.ecash-balance` + Fedimint `wallet.fedimint-balance`.
(Home already calls `wallet.ecash-balance`; add fedimint and sum.)
## Networking sats — dual protocol (phase 6, NOT yet wired)
The economic layer (`streaming.rs`, `streaming/gate.rs`, sessions, pricing, metering) is already
protocol-agnostic — it just calls into the wallet. The injection points are:
1. **Protocol-tag accepted mints.** `accepted_mints: Vec<String>` → carry protocol, e.g.
`cashu:https://mint…` / `fedimint:<federation_id>`. Migrate `wallet/accepted_mints.json` with a
back-compat reader (bare URL ⇒ `cashu:`).
2. **`MintClient::new()` is the bottleneck** (~10 call sites). Introduce a `MintBackend` trait with
`CashuBackend` (wraps current code) and `FedimintBackend` (calls `FedimintClient`).
3. **`Token` enum** `Cashu(CashuToken) | Fedimint(notes)`; serialize/verify by variant.
4. `build_payment_token()` picks a `(backend, id)` the peer accepts; `verify_and_receive_payment()`
auto-detects the token variant and reissues/swaps on the right backend.
5. Cross-protocol settlement (Cashu↔Fedimint) bridges over Lightning (BOLT11) — both sides already
have mint/melt (Cashu) and `ln/invoice`+`ln/pay` (Fedimint).
6. `Web5NetworkingProfitsSettings.vue`: per-service payout protocol/mint selector.
## Phases
- [x] **P0** FE tabbed Wallet Settings modal + gear (Cashu live, Fedimint tab structured).
- [x] **P1** `fedimint-clientd` container manifest + ports.
- [x] **P2** `FedimintClient` HTTP bridge + `wallet.fedimint-*` RPCs (compiles).
- [ ] **P3** Validate join / balance / spend / reissue against a live clientd + real federation on a scratch node.
- [ ] **P4** Unified ecash balance in the wallet card (Cashu + Fedimint).
- [ ] **P5** Flip FE fully live; surface "install Fedimint client app" when clientd unreachable.
- [ ] **P6** Networking-sats dual-protocol routing (the `MintBackend`/`Token` refactor above).
## Validation (per project testing discipline)
clientd image + a real federation are required; cannot be validated from the dev tree. Validate on
a scratch node: install Fedimint app + clientd, join a known test federation, confirm
`wallet.fedimint-balance`, then a spend→reissue round-trip between two nodes, then networking-sats
payment over Fedimint. Heavy/iterative work belongs in a worktree (see CLAUDE.md memory).
docs/hardware-signer-design.md
+171
View File
@@ -0,0 +1,171 @@
# Archipelago Hardware Signer — Design Notes (PSBT + Nostr)
> Status: **exploratory / spec stub** (2026-06-24). No code yet. This captures the
> hardware-selection reasoning and architecture for a small, air-gapped, super-secure
> signing device built around the Tropic Square **TROPIC01** secure element, intended
> to integrate with Archipelago as an external signer.
## 1. Goal
A small, super-secure, air-gapped handheld device that:
- Signs **Bitcoin PSBTs** for the Archipelago wallet.
- (Stretch / dual-function) Signs **Nostr events** for the node's sovereign identity.
- Communicates **only via QR** (camera in, screen out) — no USB data path, no radio in
use. Pure air-gap, same threat model as SeedSigner but with a real audited secure element.
- Anchors key-at-rest security and RNG in the **TROPIC01** open-source secure element.
## 2. The critical curve caveat
**TROPIC01's signing engine supports P-256 (ECDSA) and Ed25519 (EdDSA) — NOT secp256k1.**
Bitcoin and Nostr both require secp256k1. Therefore:
- The secure element is the **vault + RNG + attestation**, not the signer.
- The seed lives encrypted inside TROPIC01 (tamper mesh, pairing, secure channel).
- The host MCU does the actual **secp256k1 ECDSA (Bitcoin)** and **Schnorr / BIP-340
(Taproot + Nostr)** signing in software.
- TODO before committing: re-check whether a firmware revision adds secp256k1 — it's
open RISC-V silicon and has been a community ask. If/when it lands, this design gets
materially stronger (signing in-silicon).
## 3. Architecture (two chips)
```
[ QR in ] --> Camera (OV2640)
|
Host MCU (ESP32-S3) <--SPI--> TROPIC01 (Mini Board)
| (seed vault, RNG,
Touch screen secure channel, attest)
|
[ QR out ] <-- Display (signed PSBT / signed event)
```
- **Host MCU** drives camera, touch screen, QR parse/render, PSBT + Nostr logic, and
the secp256k1/Schnorr signing.
- **TROPIC01** protects the seed at rest and supplies the TRNG + secure boot/attestation
over an authenticated+encrypted SPI channel.
## 4. Hardware selection
### 4.1 MCU — the camera-ease vs radio-purity fork
| | **ESP32-S3** (recommended) | **RP2350** |
|---|---|---|
| Camera | Native DVP interface; huge QR-scan code ecosystem | No camera peripheral — bit-bang over PIO (harder) |
| Radios on die | WiFi + BLE present (con for air-gap purists) | **None** |
| Security | Secure boot, flash encryption | Cortex-M33 + TrustZone, signed boot, OTP |
| secp256k1 in SW | Fine (240 MHz dual-core) | Fine (150 MHz dual-core M33) |
| Price (chip / board) | ~$3 / ~$6 | ~$1.20 / ~$5 |
**Pick: ESP32-S3 (N16R8 — 16MB flash / 8MB PSRAM).** The camera is the hard part of the
build and the S3 is the only cheap MCU with a native camera interface. PSRAM matters for
holding camera frames during QR decode. The on-die radio is the one downside — acceptable
because trust is anchored in the TROPIC01, not the MCU. If radio-on-die is a hard no,
switch to RP2350 and accept harder camera bring-up. (SeedSigner deliberately chose a
no-WiFi Pi Zero 1.3 for exactly this reason — the concern is legitimate.)
### 4.2 Camera
- **OV2640** 2MP module — standard ESP32-cam sensor, code everywhere. ~$2–4.
### 4.3 Thin touch screen
Pick by review legibility (the whole security value is the human verifying address +
amount before tap-to-approve):
- **2.0" IPS ST7789 capacitive, 240×320 — recommended.** Easiest to read a full Bitcoin
address/amount. ~$8–12.
- 1.69" rounded-rect IPS ST7789 + CST816 cap touch — best size/compactness balance.
~$7–10.
- 1.28" round (GC9A01 + CST816) — smallest/thinnest but **too cramped** for address
verification; skip for a signer.
**Do not go below ~1.69".** Use capacitive (not resistive) touch for a thin glass-front
tap-to-confirm feel.
### 4.4 TROPIC01 board (from the Tropic Square order form)
All options speak SPI (wires to the S3 the same way). Two-board plan:
- **Development: TROPIC01 USB DevKit (€50)** — STM32 + USB-to-SPI stick. Bring up the
secure-element stack (pairing, key gen, secure channel) on a PC first, independent of
the camera/screen work.
- **Final device: TROPIC01 Mini Board (€9.50)** — small easy-to-solder module exposing
SPI; solder straight to the S3's SPI bus inside the enclosure.
- Skip: Standalone Sample (€5, bare QFN — needs hot-air), Raspberry Pi / Arduino Shields
(wrong host form factor), MIKROE Click (€20, only if you have a mikroBUS rig).
### 4.5 Rough BOM
| Item | ~Cost |
|---|---|
| ESP32-S3 N16R8 board | $6–8 |
| OV2640 camera | $2–4 |
| 2.0" cap-touch IPS | $8–12 |
| TROPIC01 Mini Board | €9.50 |
| (Dev only) TROPIC01 USB DevKit | €50 |
**Core device BOM ≈ $20–30** + TROPIC01 Mini Board, before enclosure/battery.
## 5. Dual-function: Nostr signer
Genuinely viable and a natural fit — **Nostr signs with Schnorr/BIP-340 over secp256k1,
the same scheme as Bitcoin Taproot.** So Nostr signing reuses the secp256k1+Schnorr code
already needed for Bitcoin — near-zero marginal firmware cost.
### 5.1 One seed → two separated keys
From the single seed in the TROPIC01:
- **Bitcoin:** BIP-32/39/84 HD derivation.
- **Nostr:** **NIP-06** deterministic derivation (`m/44'/1237'/…`) → `nsec`/`npub`.
One backup, two independent identities, no cross-contamination.
### 5.2 Cold vs hot tension
| | Bitcoin | Nostr |
|---|---|---|
| Frequency | Rare, high-value | Frequent, often interactive |
| Natural transport | QR / PSBT — air-gap perfect | Apps want real-time signing |
| Air-gap comfort | Excellent | Fine for occasional events, painful for chat |
Two possible modes:
1. **Air-gapped QR Nostr signer (recommended):** app shows unsigned-event QR → camera
scan → touch approve → signed-event QR back. Great for high-value/infrequent events
(root identity, profile/metadata, key rotation, announcements). Keeps 100% air-gap.
2. **Connected NIP-46 "bunker" over USB/serial:** enables interactive real-time signing
but **breaks the air-gap** and reintroduces the USB/radio attack surface. Not
recommended for this device.
### 5.3 Recommendation
Keep it **cold for both roles.** The device guards the Bitcoin spending key *and* the
high-value Nostr **identity** key — neither ever touches a network. Day-to-day Nostr
chatter uses a separate hot software key; the hardware device protects only the
identity-defining key you can't afford to leak. Avoids putting a hot key next to cold
Bitcoin funds.
## 6. Archipelago integration
- Slots in as an **external signer** path alongside the existing wallet flow — does not
touch the orchestrator. Archipelago builds PSBT → renders QR (animated QR for large
txs) → device scans → touch review → returns signed-PSBT QR → Archipelago broadcasts.
- Especially apt given Archipelago's Nostr/Blossom catalog + node-identity direction
(see `dht-distribution-design.md`): the device becomes the **hardware root of trust**
for both halves of a node's identity — its `npub`/DID and its Bitcoin keys — aligning
with the sovereign/secure/rootless north star.
## 7. Open items / next steps
- [ ] **Pin budget:** confirm the S3 GPIO/SPI budget fits camera DVP + display SPI +
TROPIC01 SPI simultaneously. (Biggest unknown before buying.)
- [ ] Confirm current TROPIC01 firmware secp256k1 status (could remove the §2 caveat).
- [ ] Define QR payload formats for both roles (PSBT vs unsigned Nostr-event JSON) so a
single scan→approve→return firmware loop handles either transparently.
- [ ] Animated/multi-part QR strategy for large PSBTs.
- [ ] Seed provisioning ceremony into the TROPIC01 (gen on-device via its TRNG; never
import in clear).
- [ ] Enclosure + power (battery vs USB-power-only-while-airgapped).
- [ ] Decide: ESP32-S3 (radio present) vs RP2350 (no radio, harder camera) — final call.
docs/hotfix-process.md
+52
View File
@@ -0,0 +1,52 @@
# Hotfix Process
For critical bugs discovered after a tagged release.
## Severity Classification
| Level | Response Time | Examples |
|-------|--------------|---------|
| P0 — Critical | < 4 hours | Data loss, security vulnerability, node bricked |
| P1 — High | < 24 hours | App won't start, auth broken, major UI failure |
| P2 — Medium | < 72 hours | Non-critical feature broken, performance regression |
| P3 — Low | Next release | Cosmetic, minor UX, edge cases |
## Hotfix Workflow
### 1. Triage
- Reproduce the issue on dev server (192.168.1.228)
- Classify severity (P0-P3)
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
### 2. Fix
- Create branch: `hotfix/vX.Y.Z-description`
- Fix the issue with minimal code changes
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
- Verify fix on live server
### 3. Release
- Merge hotfix branch to `main`
- Tag: `vX.Y.Z` (increment patch version)
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
`releases/manifest.json` and signs it)
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
### 4. Communicate
- Update RELEASE-NOTES with hotfix details
- Note in CHANGELOG.md
## Monitoring Dashboards
- **Uptime monitor**: `/var/lib/archipelago/uptime-monitor/summary.json`
- **Soak test**: `/tmp/stability-test-*.log` on dev server
- **Health endpoint**: `http://192.168.1.228/health`
## Rollback
If a hotfix causes regressions:
1. The updater self-verifies after applying (health check on restart) and rolls the
binary back automatically if the new one fails to come up
2. Point `releases/manifest.json` back at the last-known-good version and push
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
docs/manifest-hooks-design.md
+114
View File
@@ -0,0 +1,114 @@
# Manifest Lifecycle Hooks — Design
**Status:** implemented through Phase 4 (see §6; updated 2026-07-08) — only declarative `pre_start` remains · originally Task #20
(indeedhub, netbird) off legacy Rust installers.
See `docs/PRODUCTION-MASTER-PLAN.md`, `docs/APP-PACKAGING-MIGRATION-PLAN.md`
("controlled hooks").
---
## 1. Problem
Some apps need a step the static manifest can't express: a **post-start container
mutation**. The motivating case is indeedhub's `patch_indeedhub_nostr_provider()`:
1. `podman exec indeedhub sed -i '/X-Frame-Options/d' /etc/nginx/conf.d/default.conf`
(strip the header so the app loads in our iframe)
2. `podman cp /opt/archipelago/web-ui/nostr-provider.js indeedhub:/usr/share/nginx/html/`
3. patch nginx conf to inject `<script src="/nostr-provider.js">` and reload
A manifest `files:` entry writes files on the **host** before create; it cannot
patch a **running** container or copy a host file into it. Without a hook,
migrating indeedhub to the orchestrator ships a broken UI.
## 2. Non-goals / security posture
Per the packaging plan: **NOT arbitrary host scripts.** Hooks are declarative,
allowlisted operations, run against the app's **own** (already manifest-sandboxed)
container. This preserves "no arbitrary privileged execution" while giving a
reviewed escape hatch.
- **No host execution.** `exec` runs *inside the container* (`podman exec`), never
on the host.
- **No arbitrary host reads.** `copy_from_host.src` is **relative to an allowlist
root** (`<data_dir>` and `/opt/archipelago/web-ui`), resolved + canonicalised;
any `..` escape or absolute path outside the allowlist is rejected at validate().
- **Same privileges as the container.** `exec` inherits the container's caps
(already dropped per `security:`), so a hook can't exceed the app's own sandbox.
- **Best-effort + idempotent.** Hooks must be safe to re-run (guard with
`grep -q … || …`). A hook failure is logged, not fatal — matching the legacy
best-effort patch, so a transient hook error never bricks an install.
## 3. Schema (`AppDefinition.hooks`)
```yaml
app:
id: indeedhub
hooks:
post_install: # after the container is created + running, on install
- exec: ["sed", "-i", "/X-Frame-Options/d", "/etc/nginx/conf.d/default.conf"]
- copy_from_host:
src: "web-ui/nostr-provider.js" # relative to allowlist root
dest: "/usr/share/nginx/html/nostr-provider.js"
- exec: ["sh", "-c", "grep -q nostr-provider /etc/nginx/conf.d/default.conf || sed -i 's#</head>#<script src=\"/nostr-provider.js\"></script></head>#' /etc/nginx/conf.d/default.conf"]
- exec: ["nginx", "-s", "reload"]
pre_start: [] # (future) run before each start — repair/ownership
```
Types (in `archipelago-container`):
```rust
pub enum HookStep {
Exec { exec: Vec<String> },
CopyFromHost { copy_from_host: HostCopy },
}
pub struct HostCopy { pub src: String, pub dest: String }
pub struct LifecycleHooks {
#[serde(default)] pub post_install: Vec<HookStep>,
#[serde(default)] pub pre_start: Vec<HookStep>,
}
```
`hooks` is `#[serde(default)]` + forward-compatible (absent = no hooks).
## 4. Execution
`container::hooks::run_post_install(manifest, container_name, data_dir)`:
- Resolve container name via `compute_container_name`.
- For each step in order:
- `Exec` → `podman exec <container> <args…>` (timeout-bounded).
- `CopyFromHost` → canonicalise `src` against the allowlist roots; reject on
escape; `podman cp <abs-src> <container>:<dest>`.
- Log each step; on error, `warn!` and continue (best-effort).
Called from the orchestrator's install path **after** the container is up
(post-create/health), and gated so it runs on install (not every reconcile).
Validation (`AppManifest::validate`): every `copy_from_host.src` must resolve
inside an allowlist root and contain no `..`; `exec` must be non-empty.
## 5. indeedhub migration (the payoff)
With hooks, indeedhub becomes fully manifest-driven: 7 member manifests
(postgres/redis/minio/relay/api/ffmpeg/frontend) + the frontend manifest carries
the `post_install` hook above. `install_indeedhub_stack` becomes orchestrator-first
(like btcpay), legacy as fallback. Same pattern unblocks netbird's setup steps.
## 6. Phases
1. ✅ **Schema + validation + unit tests** — `LifecycleHooks`/`HookStep`/`HostCopy`
in `archipelago-container::manifest`, allowlist-enforced at `validate()`.
(commit `4c1a4e59`)
2. ✅ **Executor + wire into orchestrator install** — `container::hooks::run_post_install`
(`exec` + `copy_from_host`, canonicalise + symlink-escape prefix check, best-effort);
called from `install_fresh` after the container is up, fresh-container-only.
(commit `955c54b7`)
3. ✅ **indeedhub**: member manifests + frontend `post_install` hooks shipped
(`apps/indeedhub/manifest.yml` declares the nostr-provider copy + nginx
reload; `install_indeedhub_stack` is orchestrator-first via
`install_stack_via_orchestrator`).
4. ✅ **netbird** (resolved differently): installs via the stack orchestrator,
but its setup is handled by `generated_secrets`/`generated_certs` + the
per-app Rust `run_pre_start_hooks` path rather than manifest hooks — no
`hooks:` block in its manifest.
5. ⏳ `pre_start` hooks (repair/ownership) — type exists; executor not yet
wired. Note: `prod_orchestrator.rs::run_pre_start_hooks` is a hardcoded
per-app Rust match today, NOT this declarative path.
docs/marketplace-protocol.md
+341
View File
@@ -0,0 +1,341 @@
# Decentralized App Marketplace Protocol
**Status:** implemented (updated 2026-07-08). This started as a protocol
proposal; the described subsystem is now shipped end-to-end —
`core/archipelago/src/marketplace.rs` (discover/publish/trust scoring),
the `marketplace.*` RPC namespace, and `Marketplace.vue`. Beyond this doc,
the code also adds `marketplace.create-invoice` (Lightning BOLT11 app
purchases). What remains is maturation: publishing tooling and trust UX
(see `ROADMAP.md`). Note: the manifest schema below is the marketplace's
own flatter format, **not** the runtime `apps/*/manifest.yml` schema
(`app-manifest-spec.md`).
## Overview
Archipelago's community marketplace enables developers to publish app manifests to Nostr relays, where nodes discover and install them without a central app store. Trust is established through DID-signed manifests and community reputation.
## Architecture
```
Developer Node Nostr Relays User Node
│ │ │
│── Publish signed manifest ──► │ │
│ (NIP-78, kind 30078) │ │
│ │ ◄── Query app manifests ── │
│ │ (filter by d-tag) │
│ │ │
│ │── Return signed manifests ──► │
│ │ │
│ │ [Verify DID signature] │
│ │ [Check trust score] │
│ │ [Display in marketplace] │
│ │ │
│ │ [User clicks Install] │
│ │ [Pull container image] │
│ │ [Start container] │
```
## Manifest Schema
App manifests published to Nostr relays follow the existing `apps/{app-id}/manifest.yml` schema (see `docs/app-manifest-spec.md`), serialized as JSON within a Nostr event.
### Marketplace Manifest Fields
```json
{
"app_id": "my-bitcoin-tool",
"name": "My Bitcoin Tool",
"version": "1.2.0",
"description": {
"short": "A useful Bitcoin utility",
"long": "Detailed description of what this app does..."
},
"author": {
"name": "Developer Name",
"did": "did:key:z6Mkh...",
"nostr_pubkey": "npub1..."
},
"container": {
"image": "docker.io/developer/my-bitcoin-tool:1.2.0",
"ports": [{ "container": 8080, "host": 8180, "protocol": "tcp" }],
"volumes": [{ "name": "data", "path": "/data" }],
"env": {
"NETWORK": "mainnet"
},
"capabilities": [],
"readonly_root": true,
"no_new_privileges": true,
"run_as_user": 1000
},
"category": "money",
"icon_url": "https://example.com/icon.png",
"repo_url": "https://github.com/developer/my-bitcoin-tool",
"license": "MIT",
"min_archipelago_version": "0.1.0",
"dependencies": [],
"signatures": {
"manifest_hash": "sha256:abc123...",
"did_signature": "base64-encoded-signature"
}
}
```
### Required Fields
| Field | Type | Description |
|-------|------|-------------|
| `app_id` | string | Unique identifier, lowercase kebab-case |
| `name` | string | Human-readable display name |
| `version` | string | Semantic version (major.minor.patch) |
| `description.short` | string | One-line description (max 120 chars) |
| `author.did` | string | Developer's DID (did:key method) |
| `container.image` | string | Full container image reference with tag (never `latest`) |
| `category` | string | One of: money, commerce, data, networking, home, community, other |
### Security-Required Fields
| Field | Default | Description |
|-------|---------|-------------|
| `container.readonly_root` | true | Container root filesystem is read-only |
| `container.no_new_privileges` | true | Prevent privilege escalation |
| `container.run_as_user` | 1000 | UID to run as (must be > 1000) |
| `container.capabilities` | [] | Required Linux capabilities (drop all, add only needed) |
## Nostr Event Format
### Event Kind
App manifests use **NIP-78 application-specific data** with event kind **30078** (replaceable parameterized). This matches the existing node discovery pattern in `nostr_discovery.rs`.
### Event Structure
```json
{
"kind": 30078,
"tags": [
["d", "archipelago-app:<app_id>"],
["t", "archipelago-marketplace"],
["t", "category:<category>"],
["version", "<semver>"],
["image", "<container_image>"],
["L", "archipelago"],
["l", "app-manifest", "archipelago"]
],
"content": "<JSON-serialized manifest>",
"created_at": 1710000000,
"pubkey": "<developer's secp256k1 pubkey hex>",
"sig": "<schnorr signature>"
}
```
### Tag Semantics
| Tag | Purpose |
|-----|---------|
| `d` | Unique identifier for NIP-33 replaceable events. Format: `archipelago-app:<app_id>` |
| `t` | Searchable topic tags for relay filtering |
| `version` | Allows version-specific queries |
| `image` | Container image for quick display without parsing content |
| `L`/`l` | NIP-32 labeling namespace for structured queries |
### Publishing a Manifest
1. Developer creates/updates their app manifest
2. Serialize manifest as JSON
3. Compute SHA-256 hash of the serialized manifest
4. Sign the hash with the developer's DID key
5. Embed manifest + signature in Nostr event content
6. Sign the Nostr event with the node's secp256k1 key
7. Publish to all configured Nostr relays
### Discovering Manifests
1. Node queries configured relays with filter:
```json
{
"kinds": [30078],
"limit": 100,
"#t": ["archipelago-marketplace"]
}
```
2. For each returned event:
a. Verify Nostr event signature (standard NIP-01)
b. Parse manifest JSON from content
c. Verify DID signature on manifest hash
d. Check manifest against security requirements
e. Calculate trust score
3. Return manifests sorted by trust score
## Trust Model
### Trust Score Calculation
Each discovered app receives a trust score (0-100) based on:
| Factor | Weight | Description |
|--------|--------|-------------|
| **DID Verification** | 30 | Manifest is signed by a valid DID key |
| **Relay Consensus** | 20 | Manifest found on multiple independent relays |
| **Federation Trust** | 20 | Developer's DID is in the user's federation network |
| **Version History** | 15 | App has multiple published versions (shows maintenance) |
| **Security Compliance** | 15 | Manifest follows all security requirements |
### Trust Tiers
| Score | Tier | UI Treatment |
|-------|------|--------------|
| 80-100 | Verified | Green badge, install with one click |
| 50-79 | Community | Yellow badge, install with confirmation |
| 20-49 | Unverified | Orange badge, install with warning dialog |
| 0-19 | Untrusted | Red badge, requires explicit security override |
### Federation-Based Trust
When a developer's DID appears in the user's federation network (trusted peer), the app automatically receives +20 trust points. This creates organic trust propagation: if you trust a node operator, you're more likely to trust their published apps.
### ADR: Nostr Relays over Centralized Registry
**Decision**: Use Nostr relays as the app discovery layer instead of a centralized registry.
**Context**: A centralized app store contradicts Archipelago's sovereignty principles. Nostr relays provide censorship-resistant, decentralized event distribution.
**Consequences**:
- (+) No single point of failure for app discovery
- (+) Developers publish without permission or review gates
- (+) Multiple relay sources increase availability
- (+) Leverages existing Nostr infrastructure and key management
- (-) No global content moderation (each node decides trust locally)
- (-) Spam is possible (mitigated by DID verification and trust scoring)
- (-) Relay availability varies (mitigated by querying multiple relays)
## Signing Protocol
### Manifest Signing (DID Layer)
```
1. Serialize manifest to canonical JSON (sorted keys, no whitespace)
2. Compute: manifest_hash = SHA-256(canonical_json)
3. Sign: did_signature = Ed25519_Sign(did_private_key, manifest_hash)
4. Attach to manifest:
{
"signatures": {
"manifest_hash": "sha256:<hex>",
"did_signature": "<base64>"
}
}
```
### Event Signing (Nostr Layer)
Standard NIP-01 Schnorr signature over the event ID (hash of serialized event fields). This is handled by the Nostr client library.
### Verification Flow
```
Receiving Node:
1. Verify Nostr event signature (NIP-01) → Proves event authenticity
2. Extract manifest JSON from event content
3. Compute SHA-256 of manifest content
4. Compare with manifest.signatures.manifest_hash → Proves content integrity
5. Resolve DID document for manifest.author.did
6. Verify did_signature with DID public key → Proves developer identity
7. Check container.image tag is pinned (not :latest)
8. Validate security fields meet minimums
```
## RPC Endpoints
### Marketplace Discovery
| Method | Description | Auth |
|--------|-------------|------|
| `marketplace.discover` | Query relays for app manifests, verify, score, return sorted | Local |
| `marketplace.publish` | Publish an app manifest to configured relays | Local |
| `marketplace.get-manifest` | Get full manifest for a specific app by ID | Local |
| `marketplace.verify` | Verify a manifest's signatures and security compliance | Local |
### Manifest Management
| Method | Description | Auth |
|--------|-------------|------|
| `marketplace.list-published` | List manifests published by this node | Local |
| `marketplace.unpublish` | Remove a published manifest from relays | Local |
## Security Requirements
### Container Security Enforcement
Before installing a community app, the node validates:
1. **No `latest` tag**: Image must use a specific version tag
2. **Read-only root**: `readonly_root` must be true (or explicitly overridden by user)
3. **No root**: `run_as_user` must be > 1000
4. **No new privileges**: `no_new_privileges` must be true
5. **Minimal capabilities**: Only allowed capabilities are accepted (CHOWN, NET_BIND_SERVICE, etc.)
6. **No host networking**: Apps cannot use `--network host`
7. **Volume restrictions**: Apps cannot mount system paths (/, /etc, /var, /usr)
### Image Verification
- Container images are pulled from registries, never transferred between nodes
- Future: Cosign signature verification for container images (leverages `core/security/`)
- Image digest pinning recommended for production apps
## UI: Community Marketplace Tab
### Route
Extends existing `/dashboard/marketplace` page.
### Layout
Two tabs at the top of Marketplace.vue:
1. **Curated** (existing): Built-in apps maintained by Archipelago team
2. **Community** (new): Apps discovered from Nostr relays
### Community Tab Components
1. **App Grid**: Same card layout as curated tab, with trust score badge
2. **Search & Filter**: Category filter + text search across community apps
3. **Trust Indicators**: Color-coded badges (Verified/Community/Unverified/Untrusted)
4. **App Detail**: Shows full manifest, developer DID, relay sources, version history
5. **Install Flow**: Trust-level-dependent confirmation (one-click for Verified, warning for Untrusted)
### Publishing UI
Accessible from Settings or a "Developer" section:
1. Select a local app container to publish
2. Fill in manifest metadata (description, category, icon)
3. Review security compliance
4. Sign and publish to relays
5. View published manifests and their discovery status
## Data Storage
```
/var/lib/archipelago/marketplace/
├── cache/
│ ├── manifests.json # Cached discovered manifests
│ └── trust-scores.json # Cached trust scores
├── published/
│ └── <app-id>.json # Manifests published by this node
└── config.json # Marketplace preferences (auto-refresh interval, etc.)
```
## Implementation Notes
### Relay Query Strategy
1. Query all enabled relays in parallel (from `nostr_relays.rs` config)
2. Deduplicate manifests by `app_id` + `version`
3. If same manifest found on multiple relays, boost trust score
4. Cache results with 15-minute TTL
5. Background refresh every 30 minutes
### Version Comparison
- Use semantic versioning for all version comparisons
- When multiple versions exist for the same `app_id`, show the latest
- Keep version history available in app detail view
- Flag apps with versions older than 6 months as potentially unmaintained
docs/meshroller-integration-design.md
+176
View File
@@ -0,0 +1,176 @@
# Meshroller → Rust-native mesh assistant (issue #50)
**Decision (2026-06-17): seam (a) — lift Meshroller's *behaviors* into our Rust
mesh stack as typed message kinds.** We do NOT package the Python/Meshtastic
daemon. Meshroller rides Meshtastic-serial + a local Ollama; our radio is
**meshcore** (Heltec V3) and the `meshtastic` Python module cannot drive it. So
we reimplement its four behaviors natively against `core/archipelago/src/mesh/`,
drop the Python + Meshtastic dependency, and reuse our existing event/transport
seams.
Meshroller's behaviors (from the Phase-0 review of `meshroller.py`):
1. **LLM bridge** — relay an inbound mesh message to a local LLM, send the reply
back on the mesh.
2. **Trusted-node auth** — only trusted senders may invoke commands.
3. **Scheduled / queued messaging** — send messages at a future time; queue for
peers that are currently offline.
4. **On-channel command parser** — recognise commands in channel traffic.
---
## Where this plugs in (verified seam map)
| Concern | File / type | Anchor |
|---|---|---|
| Wire message kinds | `mesh/message_types.rs` `MeshMessageType` (`#[repr(u8)]`) | 28–73 |
| Envelope (CBOR, `0x02` marker, `seq`, `sig`) | `mesh/message_types.rs` `TypedEnvelope` | 183–197 |
| Inbound dispatch match | `mesh/listener/dispatch.rs` `handle_typed_envelope_direct()` | 80–691 |
| Outbound send | `mesh/mod.rs` `send_typed_wire()` / `send_channel_typed_wire()` | 848 / 1152 |
| Radio I/O command channel | `mesh/listener/mod.rs` `MeshCommand` (`SendText`/`BroadcastChannel`) | 55–73 |
| Frame chunking (≤160 B/frame, transparent) | `mesh/listener/session.rs` `send_dm_via_channel()` | — |
| UI push | `mesh/types.rs` `MeshEvent` (broadcast on `state.event_tx`, cap 64) | 125–164 |
| Trust gate | `federation/types.rs` `TrustLevel::Trusted` on `FederatedNode`; `federation::load_nodes()` | 5–52 |
| Block on user-blocklist | `mesh/listener/mod.rs` `ContactEntry.blocked` (`state.contacts`) | 110 |
| Local model | Ollama container, port **11434** (`port_allocator.rs:11`); call via `reqwest` (already a dep) | — |
No in-Rust LLM exists yet; we call the **local Ollama HTTP API** (the same model
Meshroller used) so nothing new is baked into the binary.
---
## Phase 1 — the assistant on the wire
### 1.1 New typed message kinds (`message_types.rs`)
Add two variants (next free tag = 24):
```rust
AssistQuery = 24, // "ask the node's AI" — prompt + optional model
AssistResponse = 25, // reply — request_id + text + done flag
```
Wire the four spots the enum requires (`from_u8` 76–104, `from_label` 109–137,
`label()` 139–166, plus the variant) — mirror the `Invoice` variant exactly.
Payloads (CBOR via `encode_payload`/`decode_payload`):
```rust
pub struct AssistQueryPayload { pub req_id: u64, pub prompt: String, pub model: Option<String> }
pub struct AssistResponsePayload { pub req_id: u64, pub text: String, pub seq: u16, pub done: bool }
```
`seq`/`done` let a long reply span multiple `AssistResponse` messages without
relying solely on frame reassembly (radio airtime is scarce — see §1.4 cap).
### 1.2 Inbound handler (`listener/dispatch.rs`)
Add a match arm for `AssistQuery`, mirroring the **`TxRelay`** arm (169–207):
validate → **gate** → spawn background work (never block the radio loop).
```rust
Some(MeshMessageType::AssistQuery) => {
let payload = decode_payload::<AssistQueryPayload>(&envelope.v)?;
if !assistant_enabled(state) { return; } // kill switch (config)
if !sender_is_allowed(state, sender_contact_id).await { warn!(..); return; }
if !rate_limit_ok(state, sender_contact_id).await { return; } // 1 in-flight / sender
let _ = state.event_tx.send(MeshEvent::AssistQueryReceived { from_contact_id, prompt });
let st = Arc::clone(state);
tokio::spawn(async move { run_assist(&st, sender_contact_id, payload).await; });
}
```
`run_assist`: POST `http://localhost:11434/api/generate`
(`{model, prompt, stream:false}`), cap + chunk the response (§1.4), and emit each
chunk back to the sender via `send_typed_wire(contact_id, …, "assist_response", …)`.
Also store via the existing `store_typed_message` path so it lands in history,
and emit `MeshEvent::AssistResponseReady`.
### 1.3 Trust gate (`sender_is_allowed`)
Reuse the federation trust list — no new store:
```rust
let nodes = federation::load_nodes(&data_dir).await.unwrap_or_default();
let peer = state.peers.read().await.get(&sender_contact_id).cloned();
let trusted = peer.and_then(|p| nodes.iter().find(|n|
Some(&n.pubkey) == p.pubkey_hex.as_ref() || Some(&n.did) == p.did.as_ref())
.map(|n| n.trust_level == TrustLevel::Trusted)).unwrap_or(false);
```
Plus honour `ContactEntry.blocked`. Config picks the policy:
**trusted-only** (default) | **specific contacts** | **anyone on channel** (opt-in).
### 1.4 Airtime discipline (meshcore reality)
Frames are ≤160 B and reassembly is automatic, but bandwidth is tiny. So:
- **Cap** the reply (default ~480 chars / ≤3 `AssistResponse` chunks); append
`…(truncated — reply '!more')` and keep the tail server-side for a `!more`.
- **Rate-limit**: one in-flight query per sender; drop/deny extras.
- **Timeout** the Ollama call (e.g. 60 s) and reply with a short error on failure
(`MeshEvent::AssistResponseReady { error }`).
### 1.5 Channel command parser
The killer entry point is a plain channel message, not a typed one. In the
inbound **`Text`** path, when a channel-0/1 message starts with the trigger
(default `!ai ` / `!ask `), synthesise an `AssistQuery` from the remainder and
run the same gated `run_assist`. This means **any meshcore client** (even a bare
Meshtastic-style sender) can ask, while typed `AssistQuery` is the rich path our
own UI uses. Trigger + enable are config.
A sibling command, **`!archy`**, answers node-status questions from the local
status caches with no model in the loop (`mesh/listener/node_cmd.rs`). It reuses
this design's trust gate (`is_sender_allowed`) and reply routing verbatim, but
deliberately does *not* require `assistant_enabled` — turning the LLM off should
not take node status with it. See [COMMANDS.md](COMMANDS.md) for the full
user-facing command surface.
### 1.6 UI events (`types.rs`)
```rust
AssistQueryReceived { from_contact_id: u32, prompt: String },
AssistResponseReady { req_id: u64, to_contact_id: u32, error: Option<String> },
ScheduledMessageFired { message_id: u64 }, // for Phase 1.7
```
Subscribers already flow through the single `event_tx` broadcast — no extra
wiring.
### 1.7 Scheduled / queued messaging
A small `AssistScheduler` owned by `MeshService` (sits beside `relay_tracker` /
`dead_man_switch` in `mod.rs`):
- Persisted queue `{ id, contact_id|channel, wire, fire_at, attempts }` under
`data_dir/mesh/scheduled.json`.
- A tokio task wakes at the earliest `fire_at`, sends via the normal
`send_typed_wire` / `MeshCommand::SendText` path, emits `ScheduledMessageFired`.
- **Offline queue**: on send failure (peer unreachable) keep the item and retry
when a `PeerDiscovered` / `PeerUpdated` event names that peer.
- RPC: `mesh.schedule-message { contact_id|channel, body, fire_at }`,
`mesh.list-scheduled`, `mesh.cancel-scheduled`.
---
## Phase 2 — killer Mesh-tab UX (ties into `project_mesh_telegram_plan`)
**Onboarding (one screen, three steps):**
1. *Model* — detect Ollama on :11434. If absent, a single "Install AI (Ollama)"
button deep-links to the App Store entry; if present, pick the model
(default the one already pulled).
2. *Who can ask* — Trusted nodes only (default) · Pick contacts · Anyone on the
mesh channel (with a clear "uses your node's compute / airtime" warning).
3. *Trigger word* — default `!ai`; toggle the whole feature on.
**Usage (Mesh tab):**
- An **Assistant** card: on/off, model, policy, trigger; live feed driven by
`AssistQueryReceived` / `AssistResponseReady`.
- Composer gains two actions: **Ask the mesh AI** (sends a typed `AssistQuery`)
and **Send later** (date/time → `mesh.schedule-message`), with a "Scheduled"
list (`mesh.list-scheduled`, cancel).
The 1–2 killer actions: *ask the island's AI from any radio*, and *queue a
message that sends itself when a peer comes back in range.*
---
## Verification
Needs **2 radios** (the .116 meshcore + a second) + Ollama running on the
answering node:
1. From radio B send `!ai what's the block height?` → node A (trusted) answers on
the channel; untrusted B is silently denied.
2. Typed `AssistQuery` from our UI → chunked `AssistResponse` renders in the feed.
3. Long reply → truncation + `!more` continues.
4. Schedule a message to an out-of-range peer → it fires when the peer reappears.
## Effort & order
Multi-day. Land in this order so each step is testable alone:
1.1 enum + payloads → 1.2/1.3/1.4 gated bridge → 1.5 channel trigger →
1.6 events → 1.7 scheduler → Phase 2 UI. Phases 1.1–1.4 are the minimum
demoable slice (ask over the mesh, get an answer).
docs/multi-node-architecture.md
+188
View File
@@ -0,0 +1,188 @@
# Multi-Node Architecture
## Overview
Archipelago supports federation — multiple nodes can form a trusted cluster to share status, deploy apps remotely, and coordinate services. This document describes the architecture for multi-node orchestration.
## Discovery & Trust Model
### Node Discovery
Nodes discover each other through two complementary channels:
1. **Nostr Relay Discovery**: Each node publishes its identity (DID, onion address, pubkey) to configured Nostr relays as a NIP-78 application-specific event. Other nodes query relays to find peers.
2. **Direct Invite**: A node generates an invite code containing its DID, onion address, and a one-time authentication token. The recipient node uses this code to establish a direct connection.
3. **Tor Hidden Services**: All inter-node communication uses Tor hidden services (.onion addresses) for privacy and NAT traversal.
### Trust Establishment
Federation uses a mutual DID verification model:
```
Node A Node B
│ │
│── federation.invite (generates invite code) ──► │
│ │
│ ◄── federation.join (presents invite + DID) ── │
│ │
│── Verify Node B's DID Document over Tor ──────► │
│ ◄── Verify Node A's DID Document over Tor ── │
│ │
│── Exchange signed challenge/response ─────────► │
│ ◄── Exchange signed challenge/response ────── │
│ │
│ [Mutual trust established] │
│ [Both nodes add each other to federation] │
```
**Trust Levels**:
- `trusted`: Full federation — can deploy apps, sync state, see all container statuses
- `observer`: Read-only — can see status but cannot deploy or modify
- `untrusted`: Discovered but not yet verified — pending invite acceptance
### ADR: Decentralized Trust over Centralized Authority
**Decision**: Use DID-based mutual verification instead of a central authority or PKI.
**Context**: Archipelago nodes are sovereign — no central server should control trust. Each node maintains its own trust list.
**Consequences**:
- (+) No single point of failure for trust
- (+) Nodes can federate without internet (direct Tor connection)
- (+) Consistent with the DID identity model already in use
- (-) No global revocation mechanism (each node manages its own trust)
- (-) Trust is bilateral — A trusting B doesn't imply C trusts B
## Shared State Protocol
### State Sync
Federated nodes periodically sync their state. Each node exposes a state summary via its RPC endpoint, accessible only to trusted federation peers.
**Synced data**:
- Container/app statuses (installed, running, stopped, version)
- Node health (CPU, memory, disk, uptime)
- Available storage capacity
- Tor hidden service status
- Lightning Network status (channels, capacity)
**Not synced** (privacy):
- Credentials and secrets
- Private keys
- Session data
- User passwords
### Sync Protocol
```
Every 5 minutes (configurable):
For each federated node:
1. POST to peer's /rpc/ endpoint: federation.get-state
2. Authenticate with signed challenge (DID key)
3. Receive state snapshot
4. Store in local federation cache
5. Broadcast changes via WebSocket to local UI
```
### State Storage
```
/var/lib/archipelago/federation/
├── nodes.json # List of federated nodes with trust levels
├── state-cache/
│ ├── <node-did>.json # Latest state snapshot from each peer
│ └── ...
└── invites/
├── pending.json # Outgoing invites awaiting acceptance
└── received.json # Incoming invites awaiting approval
```
## RPC Endpoints
### Federation Management
| Method | Description | Auth |
|--------|-------------|------|
| `federation.invite` | Generate invite code for a new peer | Local |
| `federation.join` | Accept an invite and establish federation | Local |
| `federation.list-nodes` | List all federated nodes with status | Local |
| `federation.remove-node` | Remove a node from federation | Local |
| `federation.set-trust` | Change trust level for a federated node | Local |
### Federation Data Exchange
| Method | Description | Auth |
|--------|-------------|------|
| `federation.get-state` | Return node's state snapshot | Federation peer |
| `federation.deploy-app` | Request remote app installation | Trusted peer |
| `federation.sync-state` | Trigger manual state sync | Local |
### Authentication for Inter-Node RPC
Federation RPC calls between nodes use DID-based authentication:
1. Caller includes `X-Federation-DID` header with their DID
2. Caller includes `X-Federation-Sig` header with a signed timestamp
3. Receiver verifies the DID is in their trusted federation list
4. Receiver verifies the signature using the DID's public key
5. Timestamp must be within 5 minutes to prevent replay attacks
## Federated App Deployment
### Flow
```
Local Node Remote Node
│ │
│── federation.deploy-app ──────► │
│ {app_id, version, config} │
│ │
│ [Remote verifies trust level] │
│ [Remote checks if app exists] │
│ [Remote pulls container image] │
│ [Remote starts container] │
│ │
│ ◄── Status update via sync ── │
│ {app_id: "running"} │
```
### Constraints
- Only `trusted` peers can deploy apps to each other
- Remote node can reject deployment (insufficient resources, policy)
- Container images are pulled from registry, not transferred between nodes
- App configuration is sent with the deploy command
- Remote node applies its own security policies (AppArmor, capabilities)
## UI: Federation Dashboard
**Route**: `/dashboard/server/federation`
**Components**:
1. **Node List**: Table of federated nodes showing:
- Node name (DID-derived or custom alias)
- Status: online/offline (based on last successful sync)
- Trust level badge (trusted/observer)
- App count, resource usage summary
- Last seen timestamp
2. **Add Node**: Form with invite code input or QR code scanner
3. **Node Detail Modal**: Clicking a node shows:
- Full DID and onion address
- Container/app list with statuses
- Resource usage (CPU, memory, disk)
- Deploy app button (if trusted)
- Change trust level / remove node
## Security Considerations
1. **All federation traffic over Tor**: Prevents IP address leakage between nodes
2. **DID-based auth**: No shared secrets; each node proves identity with its key
3. **Replay protection**: Signed timestamps prevent replay attacks
4. **Trust is bilateral**: Both nodes must agree to federate
5. **App deployment is opt-in**: Remote node can refuse deployment requests
6. **State snapshots are read-only**: A compromised peer cannot modify another node's state
7. **Invite codes are single-use**: Once accepted, the invite token is invalidated
docs/multinode-testing-plan.md
+69
View File
@@ -0,0 +1,69 @@
# Multinode / Fleet Testing Plan (separate from the single-node gate)
> **Scope split (2026-06-22):** the production test gate (`docs/PRODUCTION-MASTER-PLAN.md` §5,
> `tests/lifecycle/TESTING.md`) is now a **single-node criterion on .228**. Verifying the same
> lifecycle matrix across the rest of the fleet (.198 and the other testers) lives HERE and is run
> **after** the .228 single-node gate is green. This is intentionally NOT a blocker on the .228 gate.
## Why split it out
The lifecycle gate must be **run ON the node under test** — its bitcoin/companion/orphan/endpoint
checks use local `podman`/`systemctl`/`bitcoin-cli`/`curl`, not RPC to a remote host. Running it from
one host against another silently tests the *runner*. So "multinode" isn't "point the harness at N
hosts" — it's "run the on-node gate on each host," plus the genuinely cross-node concerns (federation,
mesh, transport, sync) that a single node can't exercise.
## How to run the gate on another node
Bats + jq usually aren't installed on ISO nodes. Bootstrap (one-time per node):
```
# from a host that has them (e.g. .116):
dpkg -L bats | grep -E '^/usr/(bin|lib|libexec)' | tar czf /tmp/bats.tgz -P -T - $(which jq)
tar czf /tmp/tests.tgz -C <repo> tests/lifecycle
scp /tmp/bats.tgz /tmp/tests.tgz <node>:/tmp/
# on the node:
sudo tar xzf /tmp/bats.tgz -P -C / # bats (jq here is dynamically linked — may need libs)
sudo curl -fsSL -o /usr/local/bin/jq \
https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 && sudo chmod +x /usr/local/bin/jq
mkdir -p /tmp/lifecycle-run && tar xzf /tmp/tests.tgz -C /tmp/lifecycle-run
cd /tmp/lifecycle-run/tests/lifecycle
ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=https ARCHY_PASSWORD=<node pw> \
ARCHY_ALLOW_DESTRUCTIVE=1 ARCHY_ITERATIONS=5 nohup ./run-gate.sh > /tmp/gate.log 2>&1 &
```
## Per-node preconditions (learned on .228)
- **Bitcoin must be fully synced + archival** (`initialblockdownload:false`, `pruned:false`).
test 83 reads the *real* `getblockchaininfo`, not the UI's headers-height. A node mid-IBD will
cascade-fail electrumx/lnd/btcpay/mempool even though the apps run.
- **Backends should be proper installs** (in `manifest_ids`), not adopted plain-podman left over
from ad-hoc `package.start`/cascade churn — otherwise companion self-heal and quadlet checks skew.
- **No stale per-app nginx proxy targets.** e.g. `/app/lnd/` must point at the lnd-ui port (18083),
not a stale `8081`. Repo code is correct; old node configs may be stale — re-check + regenerate.
- **No orphan quadlet units** (e.g. a `home-assistant.container` whose ContainerName ≠ the real
`homeassistant` container) — these wedge `systemctl --user` "activating" and fail the quadlet checks.
## Node roster (carry-over)
| Node | Role | Notes |
|------|------|-------|
| .228 | **single-node gate** (primary) | 14-app resilience node; bitcoin synced archival; gate GREEN. |
| .198 | fleet verify | was weak/loaded (load ~3–5) + **bitcoin mid-IBD** at split time → must finish syncing first; sshd wedges under concurrent SSH (use ONE session; gate uses HTTPS RPC so fine). |
| .5 / .120 | x250 testers (Tailscale) | flaky cellular; SSH via `tailscale nc` ProxyCommand. |
| .116 | dev/validation | local repo; its own bitcoin may be mid-IBD — do NOT treat as a gate target unless synced. |
## Cross-node concerns (only a multinode setup can test)
- Federation sync (Tor/FIPS transports), DID/contact federation, peer file fetch.
- Mesh (Meshtastic/MeshCore) + mesh-AI gating.
- Dual-ecash federation validation + networking-sats routing.
- DHT / iroh swarm distribution (origin-always-wins) once that dep lands.
## Sequence
1. Get the **.228 single-node gate green 5×** (master plan §5/§6) — DONE/in progress.
2. THEN: bring each fleet node to the preconditions above; run the on-node gate 5× per node.
3. THEN: the cross-node suites (federation/mesh/transport), tracked here.
This plan does not gate the v1.7.x single-node criterion; it is the next layer.
docs/operations-runbook.md
+366
View File
@@ -0,0 +1,366 @@
# Archipelago Operations Runbook
Quick reference for common operational tasks on Archipelago nodes.
**Primary node**: `192.168.1.228` (Arch 1)
**Secondary node**: `192.168.1.198` (Arch 2)
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
---
## 1. Check Node Health
```bash
# Quick health check (from any machine)
curl http://192.168.1.228/health # Should return "OK"
curl http://192.168.1.198/health
# Detailed system stats via RPC
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"system.stats"}' \
http://192.168.1.228:5678/rpc/v1
# Check services
ssh archipelago@192.168.1.228
sudo systemctl status archipelago # Backend service
sudo systemctl status nginx # Web server
sudo systemctl status tor # Tor hidden services
```
## 2. Check Container Status
```bash
# List all containers
podman ps -a
# Running count
podman ps --format '{{.Names}}' | wc -l
# Find exited/crashed containers
podman ps -a --filter status=exited
# Container logs
podman logs {container-name} --tail 50
# Container resource usage
podman stats --no-stream
```
## 3. Fix Crashed Containers
```bash
# Restart a specific container
podman restart {container-name}
# If container won't start, check logs first
podman logs {container-name} --tail 100
# Remove and recreate (last resort)
podman rm -f {container-name}
# Then redeploy with: ./scripts/deploy-to-target.sh --live
# The health monitor auto-restarts containers every 60s
# Check its status:
sudo journalctl -u archipelago --grep="health_monitor" --no-pager -n 20
```
## 4. Add/Remove Federation Peers
```bash
# Generate invite code (on inviting node)
# Via UI: Federation page > Generate Invite
# Via RPC:
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.invite"}' \
http://localhost:5678/rpc/v1
# Join federation (on joining node)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.join","params":{"invite_code":"{code}"}}' \
http://localhost:5678/rpc/v1
# List peers
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"federation.list-nodes"}' \
http://localhost:5678/rpc/v1
# Remove a peer
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.remove-node","params":{"did":"{peer-did}"}}' \
http://localhost:5678/rpc/v1
```
## 5. Rotate Tor Address
```bash
# Delete current hidden service keys
sudo rm -rf /var/lib/tor/hidden_service/
sudo systemctl restart tor
# Wait for new hostname
sleep 15
sudo cat /var/lib/tor/hidden_service/hostname
# The backend picks up the new address automatically (30s refresh)
# Federation peers need to re-discover via sync
```
## 6. Create/Restore Backups
```bash
# Create encrypted backup (via RPC)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.create","params":{"passphrase":"your-passphrase","description":"manual backup"}}' \
http://localhost:5678/rpc/v1
# List backups
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.list"}' \
http://localhost:5678/rpc/v1
# Verify backup integrity
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.verify","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Restore (warning: overwrites current identity/data)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.restore","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Backup files stored at: /var/lib/archipelago/backups/
```
## 7. Update the Node
```bash
# From development machine:
./scripts/deploy-to-target.sh --live # Deploy to .228
./scripts/deploy-to-target.sh --both # Deploy to both nodes
./scripts/deploy-to-target.sh --dry-run --live # Preview changes
# The deploy script:
# 1. Syncs code to target
# 2. Builds frontend (vue-tsc + vite)
# 3. Builds backend (cargo build --release)
# 4. Deploys binary, frontend, configs
# 5. Restarts services
# 6. Verifies health
```
## 8. Diagnose High CPU
```bash
# Check system load
uptime
# Find CPU-heavy processes
top -b -n 1 | head -15
# Check container CPU usage
podman stats --no-stream --format '{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
# Common causes:
# - Bitcoin IBD (initial block download): normal, takes days
# - Container crash loops: check `podman ps -a --filter status=exited`
# - mempool-electrs indexing: normal after Bitcoin sync
```
## 9. Diagnose High Memory
```bash
# Check memory
free -h
# Check swap usage
swapon --show
# Per-container memory
podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
# Check for OOM kills
dmesg --level=err,crit | grep -i oom
# Add swap if missing
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
## 10. Diagnose Disk Space
```bash
# Disk usage overview
df -h /
# Find large directories
sudo du -h --max-depth=2 /var/lib/archipelago/ | sort -rh | head -20
# Container image sizes
podman images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}'
# Clean unused images
podman image prune -a
# Clean old journal logs
sudo journalctl --vacuum-size=500M
```
## 11. Check Tor Connectivity
```bash
# Tor service status
sudo systemctl status tor
# Get onion address
sudo cat /var/lib/tor/hidden_service/hostname
# Test self-connection via Tor
curl --socks5-hostname 127.0.0.1:9050 http://$(sudo cat /var/lib/tor/hidden_service/hostname)/health
# Test cross-node Tor
curl --socks5-hostname 127.0.0.1:9050 http://{peer-onion}/health
```
## 12. Check DWN Sync
```bash
# DWN status (via RPC, needs auth)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.status"}' \
http://localhost:5678/rpc/v1
# Trigger manual sync
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.sync"}' \
http://localhost:5678/rpc/v1
# Check message count
ls /var/lib/archipelago/dwn/messages/ | wc -l
```
## 13. Restart Services
```bash
# Restart backend only
sudo systemctl restart archipelago
# Restart nginx
sudo systemctl restart nginx
# Restart Tor
sudo systemctl restart tor
# Full service restart (backend + nginx)
sudo systemctl restart archipelago nginx
# Reboot (containers auto-recover via restart policy + health monitor)
sudo reboot
```
## 14. View Logs
```bash
# Backend logs
sudo journalctl -u archipelago --no-pager -n 100
# Follow logs in real time
sudo journalctl -u archipelago -f
# Nginx access log
sudo tail -f /var/log/nginx/access.log
# Nginx error log
sudo tail -f /var/log/nginx/error.log
# Container logs
podman logs {container-name} --tail 50 -f
```
## 15. Network Diagnostics
```bash
# Check listening ports
sudo ss -tlnp
# Check firewall rules
sudo ufw status verbose
# Required ports:
# 22 - SSH
# 80 - HTTP (nginx)
# 443 - HTTPS (nginx)
# 5678 - Backend API (localhost only, proxied by nginx)
# 8332 - Bitcoin RPC (container network only)
# 9050 - Tor SOCKS proxy (localhost only)
# If ports are blocked after reboot, re-add UFW rules:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.88.0.0/16 # Podman container subnet
sudo ufw allow from 10.89.0.0/16 # Podman container subnet
```
## 16. Emergency: Node Won't Boot
If a node responds to ping but SSH/HTTP are down:
1. **Check UFW**: After reboot, UFW may block all ports
```bash
# If you have console access:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload
```
2. **Check services**: SSH or nginx may not have started
```bash
sudo systemctl start ssh
sudo systemctl start nginx
sudo systemctl start archipelago
```
3. **Check disk**: If root filesystem is full, services won't start
```bash
df -h /
sudo journalctl --vacuum-size=200M
podman image prune -a
```
## 17. Run Tests
```bash
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
# Cross-node suites (federation/mesh):
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
# E2E / post-install:
./scripts/run-e2e-tests.sh
./scripts/run-post-install-tests.sh
```
docs/phase4-streaming-ecash-plan.md
+380
View File
@@ -0,0 +1,380 @@
# Phase 4+ — Paid swarm streaming & the IndeeHub "Archipelago" source
**Status:** PLAN / design (2026-06-17) · **Branch:** `agent-trust-wip` · not implemented
**Builds on:** `docs/dht-distribution-design.md` (Phases 0–3, swarm + Blossom), the
Phase 3 swarm work just landed (`swarm/`, `content_hash.rs`, `trust/`).
This plans three things the user asked for, in one coherent architecture:
1. **Pay sats (ecash) for transport** of streaming film data between nodes.
2. **Networking *through* nodes** — relaying/routing a stream via intermediate peers.
3. An **"Archipelago" content source in IndeeHub** that shows every film uploaded
to *backstage*, on every node running the IndeeHub app.
> ## Headline finding
> **Most of the primitives already exist.** This is ~80% integration glue, not
> greenfield. A full Cashu/ecash wallet, a metered streaming payment gate, a
> 4-tier transport layer, the iroh-blobs swarm (just added), signed Nostr
> advertisements, and the Ed25519 trust module are all already in the tree. The
> genuinely new code is: (a) a paid-serving hook on the iroh side, (b) a relay
> protocol, and (c) the IndeeHub film catalog + Archipelago-local API.
---
## 0. Inventory — what we can build on (all already in `core/archipelago/src`)
| Capability | Where | State |
| --- | --- | --- |
| **Cashu ecash wallet** (mint/melt/send/receive, BDHKE) | `wallet/ecash.rs`, `wallet/cashu.rs`, `wallet/mint_client.rs`, `wallet/bdhke.rs` | ✅ implemented |
| **Local mint** (Fedimint) backing the wallet | `apps/fedimint` (`http://127.0.0.1:8175`) | ✅ deployed |
| **Lightning** (invoices, pay, channels) for mint/melt | `api/rpc/lnd/*`, `container/lnd.rs`, `apps/lnd` | ✅ implemented |
| **Streaming payment gate** (accepts `cashuA` tokens, opens metered session) | `streaming/gate.rs` | ✅ implemented |
| **Metering & pricing** (sats per byte / ms / request; e.g. content-download = 1 sat/MB) | `streaming/meter.rs`, `streaming/pricing.rs`, `streaming/session.rs` | ✅ implemented |
| **Revenue/profit accounting** (incl. `StreamingRevenue` tx type) | `wallet/profits.rs` | ✅ implemented |
| **Paid-service discovery** on Nostr (kind 10021, TollGate TIP-01 shape) | `streaming/advertisement.rs` | ✅ implemented |
| **Content server** that verifies+receives payment before serving | `content_server.rs` (`verify_and_receive_payment()`) | ✅ implemented |
| **iroh-blobs swarm** (fetch content-addressed blobs from peers, verify, seed) | `swarm/` (`iroh-swarm` feature) | ✅ just added |
| **Signed seed adverts** (NIP-33 kind 30081, blake3→endpoint) | `swarm/seed_advert.rs` | ✅ just added |
| **BLAKE3 content addressing** | `content_hash.rs` | ✅ implemented |
| **Ed25519 trust / `did:key` / detached signatures** | `trust/` | ✅ implemented (anchor ceremony pending) |
| **4-tier transport** (Mesh > LAN > FIPS > Tor) + `last_transport` | `transport/*`, `fips/dial.rs` | ✅ implemented |
| **Node discovery + federation trust** (Trusted/Observer) | `nostr_handshake.rs`, `federation/*` | ✅ implemented |
What is **NOT** present and must be built:
- **A paid-serving hook on the iroh-blobs provider.** Today the swarm seeds to
anyone (`BlobsProtocol::new(&store, None)` — no authorization). To charge for
swarm bandwidth we need a per-request gate that consults `streaming/gate.rs`.
- **A relay protocol.** No "peer A asks peer B to forward traffic to peer C".
Transport is point-to-point; there is no multi-hop routing, TTL, or relay
accounting.
- **IndeeHub Archipelago catalog.** The shipped IndeeHub points at the external
`staging-api.indeehub.studio` + AWS S3/CloudFront. Nothing makes a film
uploaded on node A visible on node B. No *backstage* code exists yet.
---
## 1. Pay sats (ecash) for transport of streaming films
### Goal
When node B streams a film blob (an HLS `.ts` segment) *from* node A's swarm,
A earns sats for the bytes it serves — using the ecash gate that already meters
`content-download`.
### What exists vs. what's new
- ✅ The economic machinery is done: `streaming/pricing.rs` already ships a
`content-download` service priced per MB; `streaming/gate.rs` turns a `cashuA`
token into a metered session; `meter.rs` deducts bytes; `profits.rs` records
`StreamingRevenue`.
- ❌ The swarm serving path doesn't consult any of it. `IrohProvider::new`
spins up `BlobsProtocol` that answers every blob request unconditionally.
### Design — "paid swarm" as a gated blob protocol
The clean seam is the iroh-blobs **accept** side. Two viable shapes:
**(A) In-band gate via a custom ALPN (preferred).** Keep iroh-blobs for the raw
byte transfer but front it with a tiny request/grant exchange on a second ALPN
(`archy/paid-blobs/1`):
1. B wants `blake3:H`. It dials A's endpoint and sends `{want: H, token?: cashuA}`.
2. A calls `streaming::gate::check_gate("content-download", peer=B, bytes≈len(H), token)`.
- `PaymentRequired` → A replies with price + its accepted mints
(`streaming.list-mints`) and the sat amount; B mints/sends a `cashuA` and retries.
- `PaidAndAllowed` / `Allowed` (within existing session allotment) → A authorizes
the blob hash for this connection and hands off to iroh-blobs to stream it.
3. A meters served bytes via `meter::record_and_check` and records revenue.
**(B) Pre-paid session, then open serving.** B opens a metered session up front
(buys N MB of `content-download` allotment with one token), and A's blob protocol
checks "does this peer have remaining allotment?" before each blob. Simpler, fewer
round-trips, slightly looser accounting. Good first cut.
Recommend **(B) for v1** (least new protocol surface — reuses sessions verbatim),
graduating to **(A)** when we want per-blob price discovery.
### Free vs. paid policy (important)
- **OTA + app-catalog blobs stay FREE.** Charging for security updates is hostile
and breaks the "origin always wins" guarantee. Gating applies **only** to the
IndeeHub film scope (a per-blob or per-advert "monetized" flag).
- Trusted federation peers (`TrustLevel::Trusted`) can be configured to serve each
other free; payment is for untrusted/public swarm peers.
### Integration points
- **DONE (2026-06-17):** `swarm/paid.rs` — the accept-side gate. Builds the
iroh-blobs `EventSender` (intercept connect + GET, hard-disable `push`) and
authorizes each request through `streaming::gate::check_gate("content-download",
peer_endpoint, blob_size, None)`. Free when the service is disabled (default);
denies unpaid peers when enabled; fails OPEN on internal error. Wired into
`IrohProvider::new`; unit-tested. The Settings toggle the user just got drives it.
- Reuse: `streaming/gate.rs`, `meter.rs`, `session.rs`, `wallet/ecash.rs`,
`streaming/advertisement.rs` (advertise the node as a paid blob seeder).
- TODO (fetch side): `swarm::fetch_content_addressed` gains an optional
"willing-to-pay budget + token source" so a downloading node can auto-pay from
its ecash wallet up to a cap (opening a session via `streaming.pay`), then fall
back to origin if too expensive. This is where **cross-mint settlement (§2a)**
plugs in — the payer may need to swap into the seeder's accepted mint first.
---
## 2. Networking *through* nodes (relayed / routed streaming)
This is the largest genuinely-new piece. Two distinct meanings — both useful:
### 2a. iroh-native relays (cheap, already mostly free)
iroh 1.0 already hole-punches and falls back to **relay servers** for connectivity
when a direct QUIC path can't be established. So "streaming through a node that
can reach the seed when I can't" partly exists at the iroh layer. Action: run/seed
our **own** iroh relay(s) on the OVH/hub infrastructure and pin them in config, so
the swarm doesn't depend on n0's public relays. Low effort, high resilience.
### 2b. Application-level paid relay (the real gap)
"Node B pays node A to fetch a film from origin/swarm on B's behalf and forward it"
— useful when B is behind a censored/expensive link and A has good connectivity
(the beta-cellular-node scenario from memory). This needs a real protocol:
- **`relay.offer` advert** (Nostr kind 10021 with a `relay` tag + price/MB) — reuse
`streaming/advertisement.rs`; add a `relay-bandwidth` service to `pricing.rs`.
- **`relay.fetch` request** over the existing transport (`PeerRequest` in
`fips/dial.rs`): `{content: blake3:H | url, pay: cashuA}`. The relay runs the
normal `swarm::fetch_content_addressed` (swarm-assist, origin fallback), meters
the bytes through `streaming/gate`, and streams them back to the requester.
- **Accounting:** add a `RelayBytes` metric to `streaming/meter.rs` distinct from
origin `content-download`, so "relay provided" is tracked separately in
`profits.rs` (the doc already separates `routing_fees` from `streaming_revenue`).
- **Safety rails:** single-hop only for v1 (no A→B→C→D); TTL + loop guard before
any multi-hop; cap per-session bytes; only relay the **public film scope**, never
private user blobs or arbitrary URLs (prevent open-proxy abuse).
### Phasing for §2
1. Pin our own iroh relays (config only). — *days*
2. Single-hop paid `relay.fetch` for film blobs, gated by ecash. — *the core build*
3. Multi-hop routing + path discovery. — *deferred; only if single-hop proves out*
---
## 2a. Cross-mint ecash settlement — paying across *different* mints
**Problem (user, 2026-06-17):** payment must work when the payer and the seeder
use **different** mints — not only two nodes on the same Fedimint. A node holding
tokens on mint **A** must be able to pay a seeder that only accepts mint **B**,
automatically.
### Why this is mostly a generalization, not new crypto
The wallet already tracks proofs **per-mint**: `WalletData::balance_for_mint(url)`,
`select_proofs(url, amount)`, `add_proofs(url, proofs)` are all mint-scoped, and
`MintClient::new(url)` targets any mint. What's hardcoded is convenience: `mint_quote`
/ `melt_quote` / `mint_tokens` / `melt_tokens` always use the single home
`wallet.mint_url`. So the data model is multi-mint already; we add the *swap* and
parameterize the helpers by target mint.
### The swap primitive (Cashu/Fedimint settle over Lightning)
To move value **A → B**, both mints expose BOLT11 mint+melt quotes (already in
`mint_client.rs`), and Lightning bridges them:
1. `MintClient::new(B).mint_quote(amount)` → a BOLT11 invoice `inv_B` (pay it to get B tokens).
2. `MintClient::new(A).melt_quote(inv_B)` → cost in A tokens (`amount + fee_reserve`).
3. Select A proofs and `melt` them on A to pay `inv_B` over Lightning.
4. When `inv_B` settles, `MintClient::new(B).mint_tokens(quote_B)` → claim B tokens;
`wallet.add_proofs(B, …)`.
Net: value lands on B minus (A melt fee + LN routing + B mint fee). The node's LND
isn't strictly required — the mints' own LN gateways settle — but a healthy local
node/route improves success. Implementation = three thin `*_at(mint_url, …)`
variants of the existing helpers + one composer:
`swap_between_mints(data_dir, from, to, amount, max_fee_sats) -> Result<u64>`.
### Where the swap happens — two models
- **Payer-side swap (recommended default).** Before paying seeder S (whose
`accepted_mints` are advertised via `streaming.advertise` / the gate's
`PaymentRequired.pricing.accepted_mints`), the payer picks the cheapest path:
pay directly if it already holds a token on one of S's mints; otherwise
`swap_between_mints(A → S_mint)` then send a token denominated in S's mint. **S
never has to trust mint A** — it only ever receives its own mint's tokens. Clean.
- **Payee-side auto-consolidation (optional, more liberal).** S widens
`accepted_mints` to any mint it's willing to melt-swap from, accepts an A token,
then swaps A → home-mint in the background. Broader acceptance, but S briefly
carries mint-A counterparty risk.
A node can do both: advertise a broad accept list *and* have payers prefer
direct/cheap mints.
### Guardrails (these are the real design decisions)
- **Mint trust list.** Mints can be insolvent or rug. Only swap *into* / accept
mints on a configured allow-list (default: home mint + a small curated set, with
the local Fedimint always trusted). Surface this in the Settings UI alongside the
per-service pricing.
- **Fee/slippage cap.** Every swap costs sats. `max_fee_sats` (or a max %) refuses a
swap that would cost more than the content is worth; the payer then declines and
uses origin. Show the all-in cost (price + swap fee) before auto-paying.
- **Origin always wins.** If the LN swap fails (no route, mint offline, over
budget), fall back to the HTTP origin with no payment. A mint problem must never
block content.
- **Idempotency / crash-safety.** Persist in-flight swaps (`melt` quote id + `mint`
quote id) so a crash between "paid `inv_B`" and "claimed B tokens" resumes the
claim instead of double-paying. Reuse the wallet's tx log.
- **Liquidity.** Swaps need the mints to have inbound/outbound LN liquidity; cache
recent swap success per mint-pair and prefer routes that have worked.
### Phasing for §2a
1. `*_at(mint_url, …)` helpers + `swap_between_mints` + mint trust list + fee cap. — *the core*
2. Payer-side auto-swap in the payment builder (pick cheapest accepted mint). — *wires §1/§2 to it*
3. Idempotent resume + per-pair liquidity cache. — *hardening*
4. (Optional) payee-side auto-consolidation.
This keeps the headline promise intact: **pay anyone, on any trusted mint,
automatically — or fall back to free origin.**
---
## 3. IndeeHub "Archipelago" content source
### Goal
A new source tab inside the IndeeHub app, **"Archipelago"**, listing every film
uploaded to *backstage*, streamable on any node — independent of the external
`indeehub.studio` API.
### Today (from the research)
- IndeeHub frontend (Next.js) is built against `NEXT_PUBLIC_API_URL =
staging-api.indeehub.studio` and pulls media from AWS S3/CloudFront. It is
**not Archipelago-aware**. nginx proxies it at `/app/indeedhub/` and injects a
NIP-07 Nostr provider.
- A MinIO stack exists (`indeedhub-public` / `indeedhub-private` buckets); FFmpeg
produces HLS there. **No backstage upload UI/code exists yet.**
- The design doc's Phase 4 already describes the target: backstage → FFmpeg → HLS
→ each `.ts` is a BLAKE3 blob → signed Nostr "Blossom" catalog event → any node
resolves the content address and streams from the nearest holder; MinIO origin.
### Architecture — four pieces
**(i) Backstage upload + transcode (origin side).**
Minimal creator flow on a publisher node: upload master → FFmpeg → HLS
(`.m3u8` + `.ts`) into MinIO (reuse the existing `indeedhub-ffmpeg`/MinIO stack).
For each segment compute `blake3_hex` (`content_hash::blake3_hex`) and import it
into the iroh seed store (`IrohProvider::seed_and_advertise`, generalized beyond
releases). The playlist references segments by content hash.
**(ii) Signed film catalog on Nostr (the "Archipelago" source).**
Define a new addressable event — **kind 30082, `archy-film`** (sibling of the
30081 seed advert) — published by the publisher node, **signed via `trust/`**:
```jsonc
{
"title": "...", "creator_did": "did:key:z...", "duration_s": 5400,
"poster": "blake3:...", // poster image blob
"playlist": "blake3:...", // the .m3u8 (itself a blob)
"segments": ["blake3:...", ...], // ordered .ts segment hashes
"enc": { "scheme": "aes-128", "key_ref": "nip98" }, // see (iv)
"monetized": { "service": "content-download", "sats_per_mb": 1 } // optional
}
```
The signature uses `trust::sign_detached`; consumers verify with
`trust::verify_detached`. **Publisher trust:** films show in the Archipelago tab
only from publishers on the node's trusted/federation set (or a pinned
"Archipelago film-root" key, mirroring the release-root anchor concept). This is
the key that stops the shared catalog from being a spam vector.
**(iii) Archipelago-local film API (makes it appear on every node).**
New RPC + HTTP endpoints in `api/`:
- `film.catalog` / `GET /api/film-catalog` — query Nostr relays for kind-30082
events from trusted publishers, verify signatures, dedupe, return merged JSON.
Cache like `app_catalog.rs` does (mtime/TTL, atomic write).
- `GET /api/film/:blake3` — serve a segment: `swarm::fetch_content_addressed`
(swarm-assist → MinIO/OVH origin), BLAKE3-verified, with HTTP range support so
the player can seek. This is where §1 (paid serving) and §2 (relay) plug in.
- The IndeeHub frontend gets an **"Archipelago" source** that points at
`/api/film-catalog` instead of `indeehub.studio`. Cleanest: a small build/runtime
flag or an injected config (same nginx `sub_filter` mechanism already used to
inject the NIP-07 provider) that registers the Archipelago source alongside the
existing studio source — additive, not a replacement.
**(iv) Encryption / access (private films).**
Public films: plaintext segments, freely cacheable, swarm-distributable. Private
films: keep AES-128 HLS; **untrusted seeds cache only ciphertext** (they never see
plaintext), and the decryption key is delivered per-viewer via NIP-98 auth (the
mechanism IndeeHub already uses) or NIP-44 DM. Payment (§1) gates *bytes*; the
*key* gates *plaintext* — two independent locks. This lets us pay strangers to
seed encrypted blobs without leaking content.
### "On every node" — propagation
Propagation is **pull**, not push: every node's `film.catalog` periodically queries
the same Nostr relays (already configured for discovery) for trusted-publisher film
events. A film uploaded on node A is therefore visible on node B as soon as B
refreshes its catalog — exactly how `app_catalog.rs` already distributes app
updates fleet-wide. No central server; the relays carry only signed metadata, the
blobs flow peer-to-peer with MinIO/OVH as origin.
---
## 4. Suggested end-to-end phasing
| Step | Deliverable | Risk | Reuses |
| --- | --- | --- | --- |
| **A** | Generalize `seed_and_advertise` beyond releases → arbitrary public blob scope (films) | low | swarm/ |
| **B** | `film.catalog` RPC + signed kind-30082 events + trusted-publisher gating | low–med | trust/, app_catalog.rs pattern |
| **C** | `GET /api/film/:blake3` range-streaming via swarm-assist + MinIO origin | med | swarm/, content_server.rs |
| **D** | IndeeHub "Archipelago" source wired to the local API (additive) | med (frontend, external repo) | nginx sub_filter |
| **E** | Backstage: upload → FFmpeg → HLS → blob import + catalog publish | med | MinIO/ffmpeg stack |
| **F** | **DONE** — paid swarm serving (`swarm/paid.rs` gates the blob protocol via `streaming/gate`); free by default | med | streaming/* |
| **F2** | Cross-mint settlement (§2a): `swap_between_mints` + payer-side auto-swap + mint trust list + fee cap | med–high | wallet/ecash, mint_client, lnd |
| **G** | Pin our own iroh relays (config) | low | iroh |
| **H** | Single-hop paid `relay.fetch` for film blobs | high | transport/, streaming/* |
| **I** | Multi-hop routing | high / deferred | — |
A→E delivers "films on every node" with free volunteer seeding (the design-doc
vision). F→H layer the sats economy on top. I is genuinely future work.
> **Shipping directive (user, 2026-06-17):** the IndeeHub "Archipelago" change
> ships — after testing — as a **decoupled app-catalog update**, NOT a binary
> OTA. Publish the new IndeeHub image + bump `releases/app-catalog.json` so every
> node gets the per-app "Update" badge (the mechanism in
> `container/app_catalog.rs` / `package.check-updates`). Node-side API changes
> (steps B/C) that need the binary go through the normal OTA; the *app* (step D,
> the IndeeHub frontend image) goes through the app catalog. See memories
> `project_decoupled_app_updates` + `reference_indeehub_canonical_source`.
---
## 5. Open questions / decisions needed
1. **iroh-blobs authorization granularity.** ✅ **RESOLVED (2026-06-17 spike).**
iroh-blobs 0.103 exposes exactly the hook we need: `BlobsProtocol::new(&store,
Some(EventSender))`. With an `EventMask` set to intercept, the provider asks our
handler to authorize each request and we return `EventResult = Result<(),
AbortReason>`:
- `RequestMode::Intercept` / `InterceptLog` — per-blob-request allow/deny
(`Err(AbortReason::Permission)` denies, `Err(AbortReason::RateLimited)` defers).
- `ConnectMode::Intercept` — reject at the connection handshake (cheap pre-filter).
- `ThrottleMode::Intercept` — per-request throttle/meter hook for byte accounting.
- `RequestMode::Disabled` — hard-reject a whole request kind (e.g. disable `Push`
so peers can never write into our store).
→ **§1 shape (A) is the recommended path** (native, no fork): the accept-side
handler calls `streaming::gate::check_gate("content-download", peer_endpoint,
bytes, token)` and maps `PaymentRequired`/`InsufficientPayment` →
`Err(Permission)`, `Allowed`/`PaidAndAllowed` → `Ok(())`. Peer identity comes
from the `Connection`'s remote endpoint id. (See `iroh_blobs::provider::events`.)
2. **Film-publisher trust anchor.** One global "Archipelago film-root" key (curated
store, like release-root) vs. per-node trusted-publisher sets vs. both. Affects
spam resistance and who can publish to *everyone's* Archipelago tab.
3. **MinIO as origin across the fleet** — single canonical MinIO on the hub vs.
per-node MinIO with cross-seeding. The swarm makes per-node origin viable but
the *first* upload needs a home.
4. **IndeeHub frontend is an external repo** (`~/Projects/indeehub-frontend`,
built into `apps/indeedhub`). Adding an "Archipelago" source needs changes
there; scope whether it's a build-time source registration or a runtime-injected
config (preferred — keeps the node OS in control).
5. **Pricing defaults & free tier.** What's free (OTA, trusted peers, first N MB?)
vs. paid, and the default sats/MB. `pricing.json` already supports this; needs a
policy.
6. **Payment UX / auto-pay caps.** A downloading node auto-paying from its ecash
wallet needs a user-set ceiling and a "prefer free origin if peer wants > X"
rule, so streaming never silently drains the wallet.
---
## 6. Why this is tractable
The hard, slow-to-build substrate — an ecash wallet, a metered payment gate,
content addressing, a verifying swarm, signed discovery, a trust module, a
multi-transport stack — is **already in the tree and (for the swarm) just tested**.
The remaining work is wiring those together along the three axes above, with the
two new protocols (paid blob serving, single-hop relay) being the only substantial
net-new surface. Everything stays behind feature flags / opt-in config and obeys
the project's north star: **swarm-assist, origin always wins** — and now,
**free updates, optional paid films.**
docs/registry-manifest-design.md
+148
View File
@@ -0,0 +1,148 @@
# Registry-Distributed App Manifests — Design
**Status:** implemented — Phases 1–3 shipped (schema + catalog-wins overlay,
signed publisher generator with embedded manifests for all apps, immich
end-to-end via `install_stack_via_orchestrator`); Phases 4–5 (build-context
apps content-addressed, drop `apps/` from OTA) remain open. Updated 2026-07-08.
**Goal (north-star):** every app installs from a manifest distributed via the
signed app-catalog on the registry — **no OS-level code reliance, no
OTA-shipped disk manifest required**. Rootless, signed, robust, reboot-survivable.
See also: [`docs/dht-distribution-design.md`](dht-distribution-design.md) (this is
its "discovery/authenticity" layer), `MEMORY → project_manifest_driven_north_star`.
---
## 1. Where we are today
Two distinct mechanisms, only one of which is registry-distributed:
| Thing | Source | Reaches node via | Carries |
|-------|--------|------------------|---------|
| `apps/*/manifest.yml` (48) | repo working tree | **OTA**: `self-update.sh` rsyncs `apps/ → /opt/archipelago/apps/` | full manifest (the orchestrator's real source of truth) |
| `app-catalog.json` (28) | `releases/app-catalog.json` | **registry HTTP fetch**, hourly, **signed** (`app_catalog::refresh_catalog`) | version + image override only |
- Orchestrator registry = in-memory `state.manifests: HashMap<app_id, LoadedManifest>`,
populated by `ProdContainerOrchestrator::load_manifests()` walking the disk dir.
`install(app_id)` → `loaded(app_id)` → "unknown app_id" if absent.
- `app_catalog.rs` is already: signed (release-root, `trust::verify_detached` over
the raw JSON), mirror-derived URLs, atomic cache at `<data_dir>/app-catalog.json`,
**forward-compatible** (no `deny_unknown_fields` — adding fields never breaks old nodes).
**Gap:** the manifest itself is never registry-distributed. Every app — btcpay,
grafana, immich — depends on an OTA-shipped disk file. That is the OS-level
reliance to eliminate.
## 2. Target
The signed catalog entry carries the **full manifest**. The orchestrator loads
manifests from the catalog cache (origin), falling back to disk only during the
migration window. Publishing an app = editing the catalog + signing + push — no
binary OTA, no disk manifest.
```
publisher: apps/*/manifest.yml ──generate──▶ releases/app-catalog.json (embeds + signs)
node: refresh_catalog() ──fetch+verify──▶ <data_dir>/app-catalog.json
load_manifests() ──merge──▶ state.manifests (catalog wins; disk = fallback)
install(app_id) ──▶ render Quadlet unit (rootless, systemd-managed)
```
## 3. Schema change (`app_catalog::AppCatalogEntry`)
Add one optional, forward-compatible field:
```rust
/// Full app manifest, embedded so the app installs from the registry alone
/// (no OTA-shipped disk file). Carried as the raw value the publisher signed;
/// deserialized into `AppManifest` at load time. Absent during migration =>
/// the node uses the disk manifest fallback.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub manifest: Option<serde_json::Value>,
```
Why `serde_json::Value`, not `AppManifest`:
- keeps the **signed preimage** intact (we verify over the raw JSON bytes; a typed
round-trip could drop/reorder unknown fields and break the signature),
- decouples catalog schema from manifest schema churn,
- deserialize + `validate()` happens at orchestrator load, exactly like `from_file`.
Authenticity is **free**: `fetch_one` already verifies the release-root signature
over the whole document, so an embedded manifest is covered by the same signature.
A present-but-bad signature is already a hard reject.
## 4. Orchestrator load path (`load_manifests`)
Extend (not replace) the disk walk:
1. Load disk manifests as today → `disk: HashMap<app_id, LoadedManifest>`.
2. Load catalog manifests from the cache: for each entry with `manifest: Some(v)`,
`serde_json::from_value::<AppManifest>(v)` then `validate()`; on success build a
`LoadedManifest { manifest, manifest_dir }`.
3. **Merge, catalog-wins**: a catalog manifest overrides the disk one for the same
`app_id`. Disk remains the fallback for apps the catalog doesn't cover (migration).
- Rationale: the registry is the authoritative origin; disk is the legacy
transport we're retiring. This matches `app_catalog`'s "catalog verdict is
authoritative when it covers the app" posture.
4. A catalog manifest that fails parse/validate is logged and skipped → disk
fallback used (one bad entry never blocks the fleet, same as the disk walk).
### `manifest_dir` for registry manifests — IMPLEMENTED
`LoadedManifest.manifest_dir` is used **only** in the `ResolvedSource::Build` branch
(relative `container.build.context` resolution — two call sites). Image-only apps
(`ResolvedSource::Pull`) never read it.
**Decision (phase 1, shipped):** keep `manifest_dir: PathBuf` (no `Option` ripple
through the codebase). A catalog manifest with a **build source is skipped** so its
disk manifest stays in effect — build contexts aren't registry-distributed until a
later phase (content-addressed, per the DHT plan). For an accepted (image-only)
catalog manifest, `manifest_dir` = the disk app dir if the app also exists on disk,
else a sentinel `<manifests_dir>/<app_id>` (never read for image-only apps).
This is enforced by `catalog_manifest_to_overlay(app_id, value) -> Option<AppManifest>`
in `prod_orchestrator.rs`, which returns `None` (→ disk fallback) for: unparseable
value, embedded-id ≠ catalog-key, failed `validate()`, or a build source.
## 5. Publishing (publish-side generator)
Add a generator (extend `create-release.sh` / a small `scripts/gen-app-catalog`):
- walk `apps/*/manifest.yml`, parse, embed each as the entry's `manifest` (JSON),
- keep `version`/`image`/`images` derived from the manifest for the badge path,
- write `releases/app-catalog.json`, then **sign** with the existing release-root
ceremony (`archipelago ceremony` / Phase 0 seed). Unsigned still accepted in the
migration window.
## 6. Migration & rollback
- **Backward compatible**: old nodes ignore the new `manifest` field (no
`deny_unknown_fields`) and keep using disk manifests.
- **Forward**: new nodes prefer catalog manifests, disk as fallback. Once the
catalog covers every app and is verified live, drop `apps/` from the OTA rsync.
- **Rollback**: delete `<data_dir>/app-catalog.json` (or revert the published
catalog) → nodes fall back to disk manifests. No data touched.
## 7. Phases
1. ✅ **Schema + load merge** (this design): `manifest` field, `load_manifests`
catalog-wins merge, unit tests (catalog overrides disk; bad catalog
manifest → disk fallback; absent → disk); `manifest_dir` stayed a plain
`PathBuf` (see §4). Image-only apps.
2. ✅ **Publisher generator + signing**: `releases/app-catalog.json` embeds a
full `manifest` block per app (all disk manifests covered) and is verified
via the release-root detached signature.
3. ✅ **First real app end-to-end**: immich installs via
`install_stack_via_orchestrator` with `generated_secrets`; the
`install_immich_stack` name survives only as an orchestrator-first wrapper.
4. ⏳ **Build-context apps**: content-addressed build contexts in the catalog (DHT
swarm fetch) so companions stop needing disk too.
5. ⏳ **Drop `apps/` from OTA** once coverage + live verification complete.
## 8. Open questions
- Do we embed manifests inline or reference them by content hash (BLAKE3) with a
separate signed blob? Inline is simplest for Phase 1; hashing aligns with the
DHT image-by-digest plan and keeps the catalog small. Lean inline now, revisit
at Phase 4 when build contexts (large) need addressing anyway.
- `generated_files` with inline content (vs. source-dir) — already supported in the
manifest schema? If so, registry manifests can carry small rendered files inline,
removing another disk dependency.
docs/troubleshooting.md
+506
View File
@@ -0,0 +1,506 @@
# Archipelago Troubleshooting Guide
This guide covers the 20 most common issues you may encounter with Archipelago, along with diagnostic commands and solutions.
## Connection & Access
### 1. Can't connect to the web UI
**Symptoms**: Browser shows "connection refused" or spins forever when accessing `http://<your-server-ip>`
**Diagnosis**:
```bash
# Check if the server is reachable on the network
ping <server-ip>
# SSH in and check Nginx
ssh archipelago@<server-ip>
sudo systemctl status nginx
sudo nginx -t
# Check if the backend is running
sudo systemctl status archipelago
curl -s http://localhost:5678/health
```
**Solutions**:
- Ensure you're on the same network (LAN) as the server
- If Nginx is down: `sudo systemctl restart nginx`
- If backend is down: `sudo systemctl restart archipelago`
- Check firewall: `sudo ufw status` — port 80 (HTTP) and 443 (HTTPS) must be allowed
- If the server IP changed, check your router's DHCP lease table or run `ip addr show` on the server
### 2. Login page loads but login fails
**Symptoms**: You see the login screen but entering the correct password shows an error
**Diagnosis**:
```bash
# Check backend logs
sudo journalctl -u archipelago --since "5 minutes ago" --no-pager
# Test the RPC endpoint directly
curl -s -X POST http://localhost:5678/rpc/v1 \
-H 'Content-Type: application/json' \
-d '{"method":"server.echo","params":{"message":"test"}}' | head -100
```
**Solutions**:
- Default password is `password123` — change it after first login
- Clear browser cookies and try again (stale session cookie)
- Restart the backend: `sudo systemctl restart archipelago`
- Check if the database is accessible: `ls -la /var/lib/archipelago/`
### 3. Web UI loads but shows blank white page
**Symptoms**: Browser loads but nothing renders, or you see a white screen
**Diagnosis**:
```bash
# Check if frontend files exist
ls -la /opt/archipelago/web-ui/index.html
ls -la /opt/archipelago/web-ui/assets/
# Check browser console (F12 > Console) for JavaScript errors
# Check Nginx error log
sudo tail -20 /var/log/nginx/error.log
```
**Solutions**:
- Redeploy the frontend: run the deploy script from the development machine
- Check if files exist in `/opt/archipelago/web-ui/` — if missing, the deploy didn't complete
- Clear browser cache (Ctrl+Shift+R or Cmd+Shift+R)
- Try a different browser or incognito mode
### 4. HTTPS certificate warning
**Symptoms**: Browser shows "Your connection is not private" or certificate error
**Solutions**:
- Archipelago uses a self-signed certificate by default — this is expected on first visit
- Click "Advanced" > "Proceed to site" (Chrome) or "Accept the Risk" (Firefox)
- For permanent fix, configure a domain name and use Let's Encrypt
- On kiosk mode, the certificate is auto-accepted
---
## App Issues
### 5. App won't start (container fails to launch)
**Symptoms**: Clicking "Start" on an app shows an error, or the app stays in "stopped" state
**Diagnosis**:
```bash
# Check container status
podman ps -a --filter "name=<app-id>"
# Check container logs
podman logs <app-id> --tail 50
# Check if the image exists
podman images | grep <app-id>
# Check available disk space
df -h /var/lib/archipelago
```
**Solutions**:
- If the image is missing: reinstall the app from the Marketplace
- If disk is full: run disk cleanup from Settings, or manually `podman system prune`
- If the container exits immediately: check logs for the root cause (usually missing config or permissions)
- Restart podman: `sudo systemctl restart podman`
### 6. App shows "unhealthy" status
**Symptoms**: App is running but shows a yellow or red health indicator
**Diagnosis**:
```bash
# Check container health
podman healthcheck run <app-id>
# Check container resource usage
podman stats <app-id> --no-stream
# Check container logs for errors
podman logs <app-id> --tail 100 | grep -i error
```
**Solutions**:
- Some apps take time to become healthy after starting (especially Bitcoin which needs to sync)
- Check if the app has enough resources (RAM, CPU)
- Restart the specific app from the UI or: `podman restart <app-id>`
- Check if dependent services are running (e.g., LND requires Bitcoin)
### 7. Bitcoin not syncing / stuck at a block height
**Symptoms**: Bitcoin node shows the same block height for an extended period
**Diagnosis**:
```bash
# Check Bitcoin logs
podman logs bitcoin-knots --tail 50
# Check if Bitcoin is connected to peers
podman exec bitcoin-knots bitcoin-cli -datadir=/data getpeerinfo | grep -c '"addr"'
# Check sync progress
podman exec bitcoin-knots bitcoin-cli -datadir=/data getblockchaininfo | grep -E "blocks|headers|verificationprogress"
```
**Solutions**:
- Initial sync takes 1-7 days depending on hardware — be patient
- Ensure the server has a stable internet connection
- Check disk space: Bitcoin requires 600GB+ for full chain
- If stuck: restart the container `podman restart bitcoin-knots`
- If peers = 0: check firewall allows port 8333 outbound
- Add manual peers: edit bitcoin.conf to add `addnode=` entries
### 8. LND won't connect to Bitcoin
**Symptoms**: LND shows errors about Bitcoin connection, or channels aren't working
**Diagnosis**:
```bash
# Check LND logs
podman logs lnd --tail 50
# Check if Bitcoin RPC is accessible from LND
podman exec lnd wget -qO- http://bitcoin-knots:8332/ 2>&1 | head -5
# Check LND status
podman exec lnd lncli getinfo 2>&1 | head -20
```
**Solutions**:
- Ensure Bitcoin is fully synced before starting LND
- Both containers must be on the same Podman network (`archy-net`)
- Check Bitcoin RPC credentials match what LND expects
- Restart both containers in order: Bitcoin first, then LND
---
## Backup & Recovery
### 9. Backup fails to create
**Symptoms**: Backup button shows an error, or backup file is empty
**Diagnosis**:
```bash
# Check disk space
df -h /var/lib/archipelago
# Check backup directory permissions
ls -la /var/lib/archipelago/backups/
# Check backend logs for backup errors
sudo journalctl -u archipelago --since "10 minutes ago" | grep -i backup
```
**Solutions**:
- Ensure sufficient disk space (backups can be large)
- Check permissions: backup directory should be owned by `archipelago` user
- Try creating a smaller backup (exclude app data)
- Restart the backend service and try again
### 10. Can't restore from backup
**Symptoms**: Restore process fails or data doesn't appear after restore
**Diagnosis**:
```bash
# Verify backup file integrity
file /path/to/backup.archipelago
ls -la /path/to/backup.archipelago
# Check backend logs during restore
sudo journalctl -u archipelago -f
```
**Solutions**:
- Ensure the backup file is not corrupted (check file size is reasonable)
- Passphrase must match what was used during backup creation
- Stop all running apps before restoring
- After restore, restart the backend: `sudo systemctl restart archipelago`
---
## System Updates
### 11. System update fails
**Symptoms**: Update button shows an error, or update process hangs
**Diagnosis**:
```bash
# Check internet connectivity
curl -s https://debian.org > /dev/null && echo "Internet OK" || echo "No internet"
# Check backend logs
sudo journalctl -u archipelago --since "15 minutes ago" | grep -i update
# Check disk space (updates need temporary space)
df -h /
```
**Solutions**:
- Ensure stable internet connection during updates
- Ensure at least 2GB free disk space
- If update hangs: wait 10 minutes, then restart the backend
- Do NOT power off during an update — this can corrupt the system
- If system is in a bad state after failed update: boot from the USB installer and select "Repair"
### 12. Server won't boot after update
**Symptoms**: Server doesn't respond after a system update
**Solutions**:
- Wait 5 minutes — the first boot after update may take longer
- If still unresponsive: connect a monitor/keyboard to check boot messages
- Try the recovery mode: boot from USB installer and select "Repair"
- As a last resort: reflash the USB and restore from backup
---
## Kiosk Mode
### 13. Kiosk display shows black screen
**Symptoms**: Connected monitor shows black screen instead of the Archipelago UI
**Diagnosis**:
```bash
# SSH in and check kiosk service
sudo systemctl status archipelago-kiosk
# Check if X11/Wayland is running
ps aux | grep -E "(Xorg|weston|chromium|firefox)"
# Check display output
ls /dev/dri/
xrandr --query 2>/dev/null || echo "No display server"
```
**Solutions**:
- Restart the kiosk service: `sudo systemctl restart archipelago-kiosk`
- Check HDMI cable is securely connected
- Try a different HDMI port or cable
- Check if the display is set to the correct input source
- Review kiosk logs: `sudo journalctl -u archipelago-kiosk --since "5 minutes ago"`
### 14. Kiosk display is stuck or frozen
**Symptoms**: Kiosk shows the UI but it's unresponsive to touch/mouse
**Solutions**:
- The watchdog service should auto-restart frozen kiosk — wait 30 seconds
- SSH in and restart: `sudo systemctl restart archipelago-kiosk`
- Check if the backend is responsive: `curl -s http://localhost:5678/health`
- If backend is down too, restart everything: `sudo systemctl restart archipelago archipelago-kiosk`
---
## Network & Connectivity
### 15. Tor address not available
**Symptoms**: Settings shows "Tor: Not configured" or the .onion address is missing
**Diagnosis**:
```bash
# Check Tor container
podman ps --filter "name=tor"
podman logs tor --tail 20
# Check if Tor hostname file exists
cat /var/lib/archipelago/tor/hidden_service/hostname 2>/dev/null
```
**Solutions**:
- Tor takes 30-60 seconds to bootstrap — wait and refresh
- If Tor container is stopped: start it from the Apps page
- Check that the Tor data directory exists and has correct permissions
- Restart Tor: `podman restart tor`
### 16. Peers can't reach my node
**Symptoms**: Federation peers show "unreachable" status
**Diagnosis**:
```bash
# Check if Tor is running (needed for peer connectivity)
podman ps --filter "name=tor"
# Check your Tor address
cat /var/lib/archipelago/tor/hidden_service/hostname
# Test connectivity from the server side
curl -s http://localhost:5678/rpc/v1 \
-H 'Content-Type: application/json' \
-d '{"method":"node.tor-address","params":{}}' | head -50
```
**Solutions**:
- Ensure Tor is running (required for peer-to-peer communication)
- Tor circuits can be slow — connections may take 30+ seconds
- Share your correct .onion address with peers
- Both nodes must have Tor running and be on the same federation
### 17. DNS resolution issues
**Symptoms**: Apps can't reach external services, container downloads fail
**Diagnosis**:
```bash
# Test DNS from the server
nslookup google.com
dig google.com
# Check DNS configuration
cat /etc/resolv.conf
# Test from within a container
podman exec bitcoin-knots nslookup seed.bitcoin.sipa.be
```
**Solutions**:
- Configure DNS from Settings > Network: try Cloudflare (1.1.1.1) or Google (8.8.8.8)
- If using custom DNS, verify the server addresses are correct
- Restart networking: `sudo systemctl restart systemd-resolved`
---
## Performance & Resources
### 18. Server is very slow / high CPU usage
**Symptoms**: Web UI is slow to respond, apps are laggy
**Diagnosis**:
```bash
# Check CPU and memory usage
top -bn1 | head -15
# Check per-container resource usage
podman stats --no-stream
# Check disk I/O
iostat -x 1 3
```
**Solutions**:
- Bitcoin initial sync uses heavy CPU — this is normal and temporary
- Check which container is using the most resources with `podman stats`
- Stop apps you don't need
- If RAM is full: add swap space or upgrade hardware
- Consider using an SSD if running on HDD (massive I/O improvement)
### 19. Disk full
**Symptoms**: Apps fail, UI shows disk warning, new installs fail
**Diagnosis**:
```bash
# Check disk usage
df -h /var/lib/archipelago
# Find largest directories
du -sh /var/lib/archipelago/*/ | sort -rh | head -10
# Check Podman image/container sizes
podman system df
```
**Solutions**:
- Run disk cleanup from Settings
- Remove unused app data: `podman system prune -a` (WARNING: removes all stopped containers and unused images)
- Move Bitcoin data to external drive if chain data is too large
- Check for large log files: `du -sh /var/log/*/ | sort -rh`
- Consider upgrading to a larger disk
### 20. WebSocket disconnections / "Reconnecting..." banner
**Symptoms**: UI shows a reconnecting indicator, real-time updates stop
**Diagnosis**:
```bash
# Check backend health
curl -s http://localhost:5678/health
# Check backend logs for WebSocket errors
sudo journalctl -u archipelago --since "5 minutes ago" | grep -i websocket
# Check system resources (WebSocket can drop under load)
free -h
```
**Solutions**:
- Brief disconnections are normal during backend restarts — the UI auto-reconnects
- If persistent: check if the backend is overloaded (high CPU/RAM)
- Restart the backend: `sudo systemctl restart archipelago`
- Check Nginx WebSocket proxy config: `/etc/nginx/sites-available/archipelago` must include `proxy_set_header Upgrade $http_upgrade`
- If on WiFi, try wired Ethernet for more stable connectivity
---
## General Maintenance
### Quick Health Check Commands
```bash
# Overall system status
sudo systemctl status archipelago nginx
# All containers
podman ps -a
# Disk usage
df -h /var/lib/archipelago
# Memory usage
free -h
# Recent errors
sudo journalctl -u archipelago --since "1 hour ago" -p err
# Backend health endpoint
curl -s http://localhost:5678/health
```
### Emergency Recovery
If the system is completely unresponsive:
1. **Power cycle**: Hold power button for 10 seconds, then turn back on
2. **Wait 5 minutes**: Services take time to start, especially if containers need to recover
3. **SSH in**: If web UI is down but SSH works, restart services manually
4. **USB recovery**: Boot from the Archipelago USB installer and select "Repair"
5. **Clean install + restore**: As last resort, do a fresh install and restore from backup
### Collecting Diagnostic Information
If you need to report an issue, collect this information:
```bash
# System info
uname -a
cat /etc/os-release
# Service status
sudo systemctl status archipelago nginx
# Recent logs (last 100 lines)
sudo journalctl -u archipelago --no-pager -n 100
# Container status
podman ps -a
# Disk and memory
df -h
free -h
# Network
ip addr show
```
docs/user-walkthrough.md
+402
View File
@@ -0,0 +1,402 @@
# Archipelago User Walkthrough
A complete guide to setting up and using Archipelago, from hardware to daily use. Each section describes what the user sees and does, serving as the basis for video tutorials.
---
## Part 1: Hardware & Preparation
### What You Need
- **Hardware**: Any x86_64 PC (Intel NUC, mini PC, old desktop) or Raspberry Pi 5
- Minimum: 4GB RAM, 32GB SSD
- Recommended: 8GB+ RAM, 1TB+ NVMe SSD (for Bitcoin full node)
- **USB drive**: 8GB+ for the installer
- **Network**: Ethernet cable (recommended) or WiFi
- **Monitor + keyboard**: For initial setup (optional if using headless mode)
- **Another computer**: To flash the USB and access the web UI
### Step 1: Download the ISO
> **Screenshot**: Browser showing the Archipelago releases page with download buttons for x86_64 and ARM64 ISOs.
1. Go to the Archipelago releases page
2. Download the latest `archipelago-auto-installer-*.iso` for your architecture
3. Verify the checksum matches the published hash
### Step 2: Flash the USB Drive
> **Screenshot**: Balena Etcher with the ISO selected and a USB drive ready to flash.
1. Download [Balena Etcher](https://etcher.io) (free, cross-platform)
2. Insert your USB drive
3. Open Etcher, select the downloaded ISO
4. Select your USB drive
5. Click "Flash!" — wait for completion (2-5 minutes)
### Step 3: Boot from USB
> **Screenshot**: BIOS boot menu showing USB drive as an option.
1. Insert the flashed USB into your target hardware
2. Power on and enter BIOS/boot menu (usually F2, F12, or Del during boot)
3. Select the USB drive as the boot device
4. The installer will start automatically
---
## Part 2: Installation
### Step 4: Auto-Installer Runs
> **Screenshot**: Terminal showing the auto-installer progress — partitioning, copying files, setting up the system.
The auto-installer handles everything:
- Partitions the target disk (erases existing data)
- Copies the Archipelago system
- Installs the bootloader
- Pre-loads container images for offline app installation
**Duration**: 5-15 minutes depending on hardware speed.
### Step 5: First Boot
> **Screenshot**: Console showing systemd services starting — archipelago.service, nginx, podman.
1. Remove the USB drive when prompted
2. The system reboots into Archipelago
3. Services start automatically (takes 30-60 seconds)
4. If a monitor is connected, the kiosk mode launches showing the web UI
---
## Part 3: First-Time Setup (Onboarding)
### Step 6: Connect to the Web UI
> **Screenshot**: Browser address bar showing `http://192.168.1.x` with the Archipelago splash screen loading.
1. Find your server's IP address:
- Check your router's DHCP client list
- Or connect a monitor — the IP is shown on the kiosk display
2. Open a browser on any device on the same network
3. Navigate to `http://<server-ip>`
4. The splash screen plays the Archipelago intro animation
### Step 7: Tap to Start
> **Screenshot**: The splash screen with "Tap anywhere to begin" text and cosmic background animation.
1. The intro screen shows the Archipelago logo with atmospheric music
2. Tap or click anywhere to proceed
3. A typing animation welcomes you: "Welcome, Noderunner"
### Step 8: Login Screen
> **Screenshot**: The login screen with a password field and glass-morphism design.
1. Enter the default password: `password123`
2. Click "Login"
3. You'll be prompted to change this password immediately
### Step 9: Choose Your Path (Onboarding)
> **Screenshot**: The onboarding path selection screen showing three options: Bitcoin Node, Home Server, Full Sovereignty.
The onboarding wizard guides you through setup:
1. **Choose your path**:
- **Bitcoin Node**: Bitcoin Knots + LND + Mempool (focused)
- **Home Server**: Bitcoin + Home Assistant + File Manager (balanced)
- **Full Sovereignty**: Everything — Bitcoin, Lightning, Nostr, VPN, Cloud (maximum)
2. **Create your identity**:
> **Screenshot**: DID creation screen showing the generated decentralized identifier.
- A DID (Decentralized Identifier) is generated for your node
- This is your sovereign digital identity — no third party needed
3. **Backup your seed**:
> **Screenshot**: Seed phrase display with 12 words and a "I've saved this" checkbox.
- Write down or save your backup passphrase
- This is the only way to recover your node if hardware fails
- Store it securely offline
4. **Verify your backup**:
> **Screenshot**: Verification screen asking to confirm specific words from the backup.
- Confirm you've saved your backup by entering requested words
5. **Setup complete**:
> **Screenshot**: Completion screen with confetti animation and "Enter your node" button.
- Click to enter the dashboard
---
## Part 4: The Dashboard (Daily Use)
### Step 10: Home Screen
> **Screenshot**: The Archipelago dashboard with glass-card layout — system status, Bitcoin sync progress, quick actions.
The home screen shows:
- **System status**: CPU, RAM, disk usage, uptime
- **Bitcoin sync progress**: Block height, peer count, sync percentage
- **Quick actions**: Start/stop apps, check notifications
- **Node identity**: Your DID and Nostr public key
### Step 11: My Apps
> **Screenshot**: The Apps page showing installed containers as glass cards — Bitcoin Knots (running), LND (running), Mempool (stopped).
- View all installed applications
- **Green dot**: Running
- **Red dot**: Stopped
- Click an app to see details, logs, and actions
- Start/stop apps with one click
### Step 12: App Details
> **Screenshot**: Bitcoin Knots detail page showing sync status, peer count, block height, and action buttons.
Each app detail page shows:
- Container status and health
- Live logs (scrollable)
- Start / Stop / Restart buttons
- Launch button (opens the app's own UI in a new tab)
- Resource usage
### Step 13: Marketplace
> **Screenshot**: The Marketplace page with curated app cards — each showing name, description, and install button.
- Browse available applications
- Install with one click
- Apps are verified containers with security hardening
- Categories: Bitcoin, Lightning, Home, Nostr, Other
### Step 14: Cloud (File Manager)
> **Screenshot**: The Cloud page showing folders (Documents, Photos, Music) with breadcrumb navigation.
- Browse files stored on your node
- Upload and download files
- Organized with breadcrumb navigation
- Files are stored locally — not in the cloud
### Step 15: Server Status
> **Screenshot**: The Server page showing CPU/RAM/Disk gauges, Tor status, and network information.
- Real-time system metrics
- Tor connectivity status and .onion address
- DNS configuration
- VPN status
- Federation peers (if configured)
### Step 16: Web5 Identity
> **Screenshot**: The Web5 page showing DID document, Nostr public key, and credential management.
- View your decentralized identity (DID)
- Manage verifiable credentials
- Publish your identity to Nostr relays
- Create and verify presentations
### Step 17: Settings
> **Screenshot**: The Settings page with sections for password, appearance, system update, and shutdown.
- Change password
- Configure TOTP two-factor authentication
- Check for system updates
- Restart or shutdown the server
- Reset onboarding (for testing)
---
## Part 5: Advanced Operations
### Accessing via Tor
> **Screenshot**: Browser showing the .onion address in the Tor Browser URL bar.
1. Install [Tor Browser](https://torproject.org)
2. Find your .onion address in Settings > Server
3. Access your node from anywhere in the world via Tor
### Federation (Multi-Node)
> **Screenshot**: Federation page showing connected peer nodes with status indicators.
1. Generate an invite code from Server > Federation
2. Share the code with a trusted peer
3. They join using the code on their node
4. Monitor peer status and deploy apps remotely
### Hardware Wallet Integration
> **Screenshot**: PSBT signing flow — creating a transaction, scanning QR with hardware wallet.
1. Create a PSBT (Partially Signed Bitcoin Transaction) from Web5
2. Transfer to your hardware wallet (QR code or file)
3. Sign on the hardware wallet
4. Import the signed PSBT back to finalize and broadcast
### Controller / Gamepad Navigation
Archipelago supports Xbox-style controller navigation throughout the UI.
#### Global Controls
| Button | Action |
|--------|--------|
| D-pad Up/Down | Navigate between elements |
| D-pad Left/Right | Move between zones (sidebar ↔ content) |
| A / Enter | Select / activate / enter container |
| B / Escape | Go back / exit container / return to sidebar |
#### Navigation Zones
**Sidebar** (left column — always visible on desktop):
- Up/Down = move between items (wraps), auto-navigates page links
- Right = enter main content (first container, or first button on container-free pages)
- Left = nothing
**Nav Bar** (mode-switcher tabs at top of content — e.g. My Apps / App Store / Services):
- Left/Right = move between tabs
- Down = jump to first card/container below (remembers tab for Up return)
- Up = nothing (Escape to sidebar)
- Left from leftmost = sidebar
**Container Grid** (card tiles — Apps, Discover, Network, Home):
- Arrows = spatial navigation between cards
- Enter = primary action (Install, Launch, or enter inner controls)
- Escape = sidebar
- Left from leftmost card = sidebar
- Up from top row = return to remembered nav bar tab
**Inside Container** (after Enter on a card — inner buttons/controls):
- Arrows = move between inner controls
- Escape = exit back to the card
- Cannot leave via arrows — must Escape first
**Text Inputs** (search bars, form fields):
- Up/Down = exit field, navigate to nearest element
- Enter = submit (clicks the next button)
- Left/Right = cursor movement (exits field at edges)
#### Per-Page Mapping
**Home** (`/dashboard`)
- Right from sidebar → first status card
- D-pad navigates between status cards spatially
- Enter on card → navigates to that section
**My Apps** (`/dashboard/apps`)
- Right from sidebar → first app card
- D-pad navigates app card grid spatially
- Enter on card → app details page
- Enter on focused card with Launch button → launches app
**App Store / Discover** (`/dashboard/discover`)
- Right from sidebar → first featured card
- D-pad navigates card grid (Sovereignty Stack + All Applications)
- Down from nav tabs → first card below
- Up from top card → returns to last-focused tab
- Enter on card → app detail / install
- Cards lift on hover/focus (same as My Apps)
**Network** (`/dashboard/server`)
- Right from sidebar → Quick Actions card
- D-pad navigates between cards: Quick Actions → Local Network / Web3 → Network Interfaces / Tor Services
- Enter on Quick Actions → enters inner buttons (Restart, Check Tor, View Logs)
- Escape from inner buttons → back to card
- All cards lift on hover/focus
**Settings** (`/dashboard/settings`) — **Linear navigation, no containers**
- Right from sidebar → first button (server name row)
- D-pad Up/Down steps through ALL buttons/controls top-to-bottom:
1. Server Name / What's New
2. Copy DID
3. Copy Onion Address
4. Change Password
5. Enable/Disable 2FA
6. Logout
7. Choose Language
8. Login with Claude
9. AI Data Access toggles (each enable/disable row)
10. Manage Updates
11. Webhook URL input
12. Webhook Secret input
13. Container Crash / Update Available toggles
14. Disk Space Warning / Backup Complete toggles
15. Save Configuration / Send Test Webhook
16. Enable Beta Telemetry
17. Create Backup
18. Export Channel Backup
19. Network Diagnostics
20. Reboot
21. Factory Reset
- Enter = activates the focused button/toggle
- Escape / Left = sidebar
**Mesh** (`/dashboard/mesh`)
- Right from sidebar → Device status card (left column)
- D-pad navigates between left-column containers (Device, Actions, Peers)
- Enter on peer → opens chat, auto-focuses message input
- Type message + Enter = send
- Escape = close chat / back to sidebar
**Cloud** (`/dashboard/cloud`)
- Right from sidebar → first folder/file card
- D-pad navigates file grid spatially
- Enter = open folder / file details
**Detail Pages** (app details, marketplace app details):
- Escape / B = go back to previous page
---
## Part 6: Maintenance
### Regular Tasks
| Task | Frequency | How |
|------|-----------|-----|
| Check for updates | Weekly | Settings > System Update |
| Review app health | Daily (glance) | Home screen status cards |
| Backup | Monthly | Settings > Backup |
| Check disk space | Monthly | Server status page |
### Updating Archipelago
1. Go to Settings > System Update
2. Click "Check for Updates"
3. If available, click "Install Update"
4. The system restarts automatically — do not power off during update
### Creating a Backup
1. Go to Settings > Backup
2. Enter a passphrase (remember this!)
3. Click "Create Backup"
4. Download the backup file and store it safely offline
---
## Quick Reference
| Action | Where |
|--------|-------|
| Start/stop an app | My Apps > App Card > Start/Stop |
| Install new app | Marketplace > Find App > Install |
| Check system health | Home or Server page |
| Change password | Settings > Security |
| Enable 2FA | Settings > Security > TOTP |
| View logs | My Apps > App > Logs |
| Access via Tor | Settings > Server > Tor Address |
| Restart server | Settings > System > Restart |
| Create backup | Settings > Backup |
docs/workstream-b-signing-runbook.md
+74
View File
@@ -0,0 +1,74 @@
# Workstream B — Signed app-catalog: completion runbook
**Status (2026-06-28):** The registry-distributed manifest pipeline is live — nodes fetch
`releases/app-catalog.json` from the OTA mirror and embed manifests (origin-wins, disk
fallback). What remains for Workstream B is **authenticity**: pin the release-root anchor and
ship a *signed* catalog so nodes can cryptographically verify the publisher.
Today the catalog is **accepted unsigned** ("migration window") and the anchor is **unpinned**
(`core/archipelago/src/trust/anchor.rs:21` → `RELEASE_ROOT_PUBKEY_HEX = None`). Completing B is
a coordinated ceremony that **only the publisher can run** — it needs the offline
`RELEASE_MASTER_MNEMONIC`, which is not (and must not be) stored on any node or build host.
## Why this is gated on you (not automatable)
- The signing key is an **offline mnemonic** you hold (`archipelago ceremony gen` output, backed
up offline / via `seed.reveal`). It is intentionally absent from the repo and all hosts.
- Order matters: once a binary **pins** the anchor, a catalog carrying a signature from the
*wrong* key is **hard-rejected fleet-wide** (`trust/signed_doc.rs:79`). Unsigned and
correctly-signed catalogs are both accepted; only a *mismatched* signature breaks nodes.
- So the pinned pubkey and the signature MUST come from the same key, shipped consistently.
## The ceremony (run from `core/`, with your mnemonic)
```bash
# 0. (only if you don't already have a release-root key) generate one and back the
# mnemonic up OFFLINE. Prints the pubkey hex + signer did:key.
cargo run --release -p archipelago -- ceremony gen
# 1. Print the release-root pubkey hex for the anchor (idempotent; same mnemonic → same key)
RELEASE_MASTER_MNEMONIC="word1 word2 …" cargo run --release -p archipelago -- ceremony pubkey
# → copy the 64-char hex.
# 2. Pin it in code:
# core/archipelago/src/trust/anchor.rs:21
# - pub const RELEASE_ROOT_PUBKEY_HEX: Option<&str> = None;
# + pub const RELEASE_ROOT_PUBKEY_HEX: Option<&str> = Some("<64-char-hex-from-step-1>");
# 3. Sign the published catalog in place (inserts `signature` + `signed_by` over the
# canonical JSON — re-run after ANY catalog regen, since signing covers the exact bytes):
RELEASE_MASTER_MNEMONIC="word1 word2 …" \
cargo run --release -p archipelago -- ceremony sign releases/app-catalog.json
# 4. Verify locally before shipping (optional sanity): a node build with the pinned anchor
# should log "app-catalog: release-root signature verified (<did>)" rather than
# "self-consistent but anchor not pinned".
```
## Ship order (backward-compatible)
1. Commit the **signed** `releases/app-catalog.json` + the `anchor.rs` change together.
2. Push the signed catalog to the OTA mirror (gitea-vps2 `main`) — old binaries (no pinned
anchor) still accept it (verified-but-unconfirmed); nothing breaks.
3. Build + OTA the binary with the pinned anchor. New nodes now **verify** the catalog against
the anchor. (This is the normal release path — gate the tag per the ship-ritual.)
4. **Later / optional hardening:** once the whole fleet is on the pinned-anchor binary, flip
the policy from "accept unsigned (migration window)" to "reject unsigned" in
`container/app_catalog.rs` (the `SignatureStatus::Unsigned` arm). Do this LAST — while any
node still runs an unsigned catalog it must keep being accepted.
## Env-override escape hatch (no rebuild)
For staging/canary you can pin the anchor without editing code via
`ARCHY_RELEASE_ROOT_PUBKEY=<hex>` (`trust/anchor.rs:23`) on a single node, then sign the catalog
and confirm that node verifies it before baking the constant in.
## What's already done (so this is the only remaining step)
- Catalog distribution + manifest embedding: live (this session's `169ff2e2` published the
corrected catalog to the mirror).
- `ceremony gen|pubkey|sign` tooling: shipped (`core/archipelago/src/ceremony.rs`).
- Verify path: `trust::verify_detached` accepts unsigned, verifies signed against the anchor,
hard-rejects mismatches (`trust/signed_doc.rs`).
- Detached-signature schema fields (`signature`/`signed_by`) already part of the signed
preimage (`container/app_catalog.rs`).