security: untrack operations docs; scrub infra identifiers from public docs
Operations docs move out of git entirely rather than being sanitized. They
stay on disk for local use and are gitignored, so the Phase 6 export (which
takes HEAD) can never carry them. 15 files: the fleet runbook, hotfix
process, node inventories, internal trackers, session handoffs, the key
rotation/signing-posture records, and the open-source plan itself.
For the docs that remain public, infra identifiers are replaced with things
that are better documentation rather than placeholders: curl examples now
use `archipelago.local`, the product's own mDNS name, so a reader can run
them as-is instead of substituting an address that was never theirs.
Deliberately NOT scrubbed, both verified as functional rather than leaked:
- `tx1138.com` is the shipped default block explorer (DEFAULT_TX_EXPLORER in
useTxExplorer.ts, surfaced in WalletSettingsModal). Product behavior.
- `git.tx1138.com` in core/container/{image_policy,registry}.rs is a retired-
registry constant the code matches on to strip stale entries from legacy
node configs. Removing it would break migration for older nodes.
- `192.168.1.254` in bulletproof-containers.md is the LAN gateway in a podman
bug description, and `192.168.1.x` in user-walkthrough.md is already generic.
Whether a personal domain should be the shipped explorer default in a public
product is a separate product question, not a security one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
39eb6b0553
commit
fda7feda60
@@ -1,151 +0,0 @@
|
||||
# Handover — fresh-ISO feedback bug-bash (2026-07-02)
|
||||
|
||||
**For: the agent building the next ISO + fleet deploy.** All fixes below are
|
||||
**merged and pushed: gitea-ai main = `f5d24796`** (merge of `c375ecc4`,
|
||||
65 files; branch `iso-feedback-fixes-2026-07-02` also pushed). Source
|
||||
feedback: user's fresh ISO install on a Framework (11th-gen Tiger Lake)
|
||||
machine, node `192.168.1.81` (SSH `archipelago` / `archipelago`).
|
||||
Diagnostic bundle: `/home/archipelago/incoming-logs/node-logs-192.168.1.81/`.
|
||||
|
||||
**⚠️ Known-red tests on main (NOT from this work):** `trust::anchor::
|
||||
unset_constant_is_none` + 2 `trust::signed_doc` tests fail because a prior
|
||||
commit pinned `RELEASE_ROOT_PUBKEY_HEX` without updating them. The signing/
|
||||
audit agent's uncommitted changes in the shared tree fix exactly these —
|
||||
coordinate with them; don't "fix" it independently or you'll collide. This
|
||||
bug-bash branch alone was 898/898 green; merged with main it's 894/898 with
|
||||
only those three.
|
||||
|
||||
## ⚠️ Outstanding user request for the deploy
|
||||
|
||||
- **Change .81's web-UI password to `<FLEET_PW>`** — the user forgot the
|
||||
current one. Node was unreachable from .116 during this session (flaky WiFi
|
||||
AP, IP flapped .68↔.81). Do this during deploy (SSH works from the user's
|
||||
machine; `archipelago`/`archipelago`).
|
||||
|
||||
## What changed (by file)
|
||||
|
||||
### Backend (core/archipelago/src) — builds clean, targeted tests pass
|
||||
- `api/handler/websocket.rs` — **subscribe BEFORE initial snapshot** (the
|
||||
"everything needs ctrl-r" root cause: broadcasts in the snapshot→subscribe
|
||||
gap were silently lost; a stale client never learned containers-scanned).
|
||||
- `main.rs` — crash check now runs BEFORE writing the PID marker (**crash
|
||||
recovery had never run on any node** — it always saw its own PID and
|
||||
skipped); tracing default demoted debug→info (journal volume).
|
||||
- `crash_recovery.rs` — PID-reuse guard (`process_is_archipelago`); new
|
||||
**pending-boot-starts registry** (names queued for recovery/reconcile) with
|
||||
writers in `recover_containers` + stack recovery.
|
||||
- `server.rs` — scanner overlays Stopped/Exited → **Restarting** for
|
||||
pending-boot-start ids (user ask: "status should be restarting if they are
|
||||
being restarted"); `SCANNER_RESTARTING` ownership set so scanner-authored
|
||||
Restarting resolves immediately instead of wedging in the 20-min
|
||||
transitional-preserve.
|
||||
- `container/prod_orchestrator.rs` — reconcile pass + `adopt_existing`
|
||||
register/deregister pending boot-starts; LND pre-start hook passes detected
|
||||
`bitcoin_host()` (Knots vs Core) into `lnd::ensure_config`; new
|
||||
`fedimint-clientd` pre-start hook (mkdir + chown 1000:1000 of
|
||||
`/var/lib/archipelago/fmcd` — self-heals the crash-loop).
|
||||
- `container/lnd.rs` — `ensure_config(paths, rpc_pass, bitcoin_host)`;
|
||||
bitcoind.rpchost no longer hardcoded `bitcoin-knots`; drift check rewrites
|
||||
host changes; +unit test `ensure_config_repairs_bitcoin_host_drift`.
|
||||
- `api/rpc/package/dependencies.rs` — bounded **dependency wait**
|
||||
(`wait_for_install_deps`, 36×5s): installed-but-starting deps wait with
|
||||
"Waiting for Bitcoin to start…" on the card; not-installed deps fail fast
|
||||
with `DependencyGateError` marker; +5 unit tests.
|
||||
- `api/rpc/package/install.rs`, `stacks.rs` — call sites wired to
|
||||
`gate_install_deps` (lnd/electrumx/mempool/btcpay).
|
||||
- `api/rpc/package/async_lifecycle.rs` — `DependencyGateError` removes the
|
||||
optimistic entry (**no more phantom "Stopped" LND tile**) + pushes an Error
|
||||
notification with the reason.
|
||||
- `api/rpc/package/progress.rs` — `set_install_message` helper.
|
||||
- `api/rpc/seed_rpc.rs` — `save_pending_seed_encrypted`; seed.restore also
|
||||
stashes the mnemonic; `auth.rs` — **auth.setup persists the encrypted seed
|
||||
backup** (recovery-phrase reveal previously failed on EVERY node because
|
||||
nothing ever wrote `master_seed.enc`).
|
||||
- `api/rpc/middleware.rs` — sanitizer allowlist extended (seed/2FA/auth
|
||||
errors reach the user instead of "Check server logs"); +2 tests.
|
||||
- `bitcoin_status.rs` — friendly status for "connection reset" (bitcoind
|
||||
starting); raw URL/os-error chains no longer shown; +3 tests.
|
||||
- `bootstrap.rs` — journald drop-in self-heal (OTA nodes get log caps);
|
||||
bitcoin.conf printtoconsole heal. (Log-spam agent's work; verified.)
|
||||
- `api/rpc/package/config.rs` — bitcoin args `-printtoconsole=0`.
|
||||
|
||||
### Manifests / scripts / configs
|
||||
- `apps/lnd/manifest.yml` — BITCOIND_HOST now `derived_env {{BITCOIN_HOST}}`.
|
||||
- `apps/bitcoin-knots/manifest.yml`, `apps/bitcoin-core/manifest.yml` —
|
||||
`-printtoconsole=0` (90.6% of the journal was IBD UpdateTip spam;
|
||||
debug.log in the datadir keeps full logs).
|
||||
- `scripts/first-boot-containers.sh` — chown 1000:1000 of
|
||||
`/var/lib/archipelago/fmcd` in BOTH fmcd blocks (root-owned dir was the
|
||||
fedimint-clientd "Permission denied os error 13" crash-loop);
|
||||
printtoconsole=0.
|
||||
- `scripts/container-doctor.sh`, `scripts/reconcile-containers.sh` —
|
||||
printtoconsole=0.
|
||||
- `image-recipe/configs/journald-archipelago.conf` (NEW) — SystemMaxUse=500M,
|
||||
rate limits; baked by ISO builder + bootstrap self-heal.
|
||||
- `image-recipe/configs/nginx-archipelago.conf` — `/assets/` 404s no longer
|
||||
cacheable (the `always` immutable header could pin a missing background for
|
||||
a YEAR); HTTPS block gained the missing `/assets/` location (was silently
|
||||
serving index.html as images).
|
||||
- `image-recipe/configs/archipelago-kiosk.service` — MemoryMax 1500→2800M,
|
||||
MemoryHigh 1200→2200M (kiosk was riding reclaim-throttle = the lag).
|
||||
- `image-recipe/_archived/build-auto-installer-iso.sh` — kiosk launcher/service
|
||||
now spliced from `image-recipe/configs/` at build time (was a stale inline
|
||||
heredoc that force-disabled GPU); **+ `firmware-intel-graphics` +
|
||||
`firmware-amd-graphics`** (Debian trixie split the i915 DMC blobs out of
|
||||
firmware-misc-nonfree; the .81 kernel logged tgl_dmc missing).
|
||||
|
||||
### Frontend (neode-ui) — vue-tsc clean, vitest green
|
||||
- `views/Login.vue` — Enter in field 1 → focus confirm; Enter in confirm →
|
||||
submit; submit button always clickable (shows inline mismatch/length error
|
||||
instead of being silently disabled); errors clear on input; **Restart
|
||||
Onboarding needs a confirming second click** (5s window) — this button is
|
||||
the likely cause of the "onboarding restarted after mismatch" report.
|
||||
+`login.restartConfirm` key in en/es locales.
|
||||
- `stores/sync.ts` — 30s staleness reconciliation (server.get-state) while
|
||||
connected; already-connected fast path now refetches too.
|
||||
- `composables/useContainersScanTimeout.ts` (NEW, +tests) — 20s escape hatch;
|
||||
wired into `Apps.vue` / `Discover.vue` / `Marketplace.vue`; fresh empty node
|
||||
reaches the real "no apps yet" empty state; "Checking…" can never persist.
|
||||
- Backgrounds: 10 heaviest bg JPEGs → **WebP q90** (9.4MB→6.6MB; refs updated
|
||||
in OnboardingWrapper/Dashboard/useRouteTransitions); 7 remaining images
|
||||
stayed JPEG (WebP came out LARGER on those — noisy sources; deliberate).
|
||||
- `public/assets/video/video-intro.mp4` — re-encoded CRF20 (SSIM 0.988) with
|
||||
**+faststart** (moov was at EOF → browser had to download all 15MB before
|
||||
playing = the intro lag). 12.7MB now, streams immediately.
|
||||
- LND icon: stale dist artifact; any fresh `npm run build` ships
|
||||
`app-icons/lnd.png` correctly.
|
||||
|
||||
## Verification done here
|
||||
- `cargo build -p archipelago` + `cargo check` clean; targeted tests
|
||||
(bitcoin_status, middleware sanitize, dep_wait, lnd, crash_recovery,
|
||||
boot_reconciler, bitcoin_host, prod_orchestrator lnd hooks): **52 passed,
|
||||
0 failed**. Full suite: **898 passed, 0 failed, 1 ignored** (22s).
|
||||
- `npm run build` green; dist verified: 10 bg-*.webp present, `lnd.png`
|
||||
icon present, `restartConfirm` string in bundle, optimized faststart
|
||||
video (12,740,782 bytes) in place. Note: main had a latent build breaker
|
||||
(unused template ref in `Web5ConnectedNodes.vue` from commit 8256fde1,
|
||||
vue-tsc TS6133) — fixed here by removing the dead ref/binding; without
|
||||
this fix `npm run build` fails on current main.
|
||||
- vitest: new composable tests + related suites pass.
|
||||
- `bash -n` clean on all touched scripts; nginx conf live-verified by agent
|
||||
(200/404/cache headers on both HTTP+HTTPS blocks).
|
||||
- ISO kiosk splice byte-verified against configs/ by agent simulation.
|
||||
|
||||
## NOT done / left for you
|
||||
1. **Full test-suite run + gate**: run the complete `cargo test` and (after
|
||||
deploy) `tests/lifecycle/run-gate.sh` ON .228 per CLAUDE.md before any tag.
|
||||
2. **Frontend bundle grep before shipping** (per memory/feedback): verify new
|
||||
strings (e.g. `restartConfirm`, `bg-home.webp`) in the built tarball.
|
||||
3. **Diagnostics collector** (`data-dir-listing.txt` = 15MB of podman overlay
|
||||
internals; dmidecode empty) — collector script wasn't found in this repo
|
||||
(likely lives on-node or in the user's collection script); fix when found.
|
||||
4. **podman healthcheck cgroup EPERM spam** (1,250 journal errors, healthchecks
|
||||
unreliable fleet-wide) — real open bug, Quadlet-phase territory, NOT fixed.
|
||||
5. **DP link-training failures on .81** (display corruption) — likely
|
||||
cable/dock/port hardware; firmware fix may help; tell user to try another
|
||||
cable/port if corruption recurs.
|
||||
6. **LoRa/RNode onboarding surface** — never scoped; user may want it as a
|
||||
feature (mesh device-found modal exists only on Mesh page post-login).
|
||||
7. The concurrent audit agent's files (`docs/1.8.0-RELEASE-HARDENING-PLAN.md`,
|
||||
`core/.../trust/*`, parts of `bootstrap.rs`) are ALSO uncommitted here —
|
||||
coordinate before committing; don't mix attribution.
|
||||
@@ -92,9 +92,9 @@ Constraints: bash TUI only (no ncurses). ANSI colors available:
|
||||
|
||||
After reboot, open the Web UI from any device:
|
||||
|
||||
http://192.168.1.198
|
||||
http://archipelago.local
|
||||
|
||||
SSH: ssh archipelago@192.168.1.198
|
||||
SSH: ssh archipelago@archipelago.local
|
||||
Password: archipelago
|
||||
Web Login: password123
|
||||
|
||||
|
||||
@@ -1,344 +0,0 @@
|
||||
# 1.8.0 OTA Session Progress
|
||||
|
||||
Updated: 2026-06-30
|
||||
|
||||
> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc is kept
|
||||
> as the historical session-by-session log; open items were consolidated into the
|
||||
> unified tracker on 2026-07-01 (several turned out already shipped — see that doc for
|
||||
> current status instead of re-deriving it from the log below).
|
||||
|
||||
---
|
||||
|
||||
## ▶️▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (evening) — #17 deployed + verified on .198/.228
|
||||
|
||||
**#17 (3ccc / stock-peer E2E pill) is now built, deployed, and live-verified** on `.198` and
|
||||
`.228` only (`.116` skipped per the hardware notice below — its radio is mid-reflash to RNode).
|
||||
|
||||
- Built release binary **sha `b1d695fc626a7382`** from the working tree (`cargo check` +
|
||||
`cargo test -p archipelago mesh::` both green, 99 passed/0 failed/1 ignored, right before
|
||||
building — tree was settled, no collision with the Reticulum agent's concurrent edits).
|
||||
- Deployed via stop/swap/start to `.198` (192.168.1.198) and `.228` (192.168.1.228), sha256
|
||||
confirmed matching on both, `systemctl is-active` = `active` on both (`.228` took its usual
|
||||
~couple-minute convergence — heavy resilience node, unrelated bitcoind/fedimint container
|
||||
startup noise in the logs during that window, no mesh errors).
|
||||
- **Live-verified the actual fix**, not just deploy: on `.198`, `mesh.peers` shows
|
||||
`"advert_name":"Meshtastic 3ccc", "pkc_capable":true`, and `mesh.send` to 3ccc
|
||||
(`contact_id:1128152268`) now returns **`"encrypted":true`** — confirms the
|
||||
`archy || peer_pkc_capable(contact_id)` TX fix is live, not just compiled.
|
||||
- `.228`'s RPC password in memory (`password123`) was stale — user confirmed the correct
|
||||
password is `<FLEET_PW>` (same as `.198`/`.116`, i.e. fully unified now). Re-verified via
|
||||
RPC: `mesh.peers` shows 3ccc `pkc_capable:true`, and `mesh.send` to 3ccc returns
|
||||
`"encrypted":true` — #17 confirmed live on `.228` too, not just `.198`.
|
||||
|
||||
**NOT yet done:** push commit to gitea-vps2 (still uncommitted in the working tree, by design —
|
||||
shares the tree with the Reticulum agent's uncommitted work); user on-device confirmation that
|
||||
the E2E pill actually renders in the Mesh UI for 3ccc.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ HARDWARE NOTICE 2026-06-30 (~16:30) — .116's Heltec V3 is being repurposed
|
||||
|
||||
**The Reticulum agent is reflashing .116's Heltec V3 (the board on `/dev/ttyUSB0`, currently
|
||||
.116's live Meshtastic radio) to RNode firmware**, with explicit user approval, to unblock the
|
||||
Reticulum Phase-0 hardware gates (real RNode needed; see `docs/RETICULUM-TRANSPORT-PROGRESS.md`).
|
||||
This was user-confirmed specifically because it takes .116 offline as a Meshtastic radio.
|
||||
|
||||
**Effect on this workstream: do all on-device Meshtastic testing on .198 and .228 only — .116 no
|
||||
longer has a Meshtastic-firmware radio attached once this lands.** `cargo check`/`cargo test
|
||||
-p archipelago` were both confirmed clean (99/99 mesh tests) right before the reflash started, so
|
||||
the earlier "wait for their edit to settle" blocker above is cleared — software-side it's safe to
|
||||
build/test/deploy; only .116's *physical radio role* changed.
|
||||
|
||||
---
|
||||
|
||||
## ▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (later PM, ~15:50) — READ THIS FIRST IF RESUMING
|
||||
|
||||
**#17 (3ccc / stock-peer E2E pill) is CODE-COMPLETE in the working tree**, isolated
|
||||
to `meshtastic.rs`/`protocol.rs`/`types.rs`/`mod.rs` as planned (no `session.rs`
|
||||
transport-plumbing changes from this side):
|
||||
- `ParsedContact.pkc_capable` (`protocol.rs`) + `MeshPeer.pkc_capable` (`types.rs`),
|
||||
both `#[serde(default)]`/defaulted `false` at every construction site.
|
||||
- `MeshtasticDevice::get_contacts()` now stamps `pkc_capable` per contact from the
|
||||
existing `peer_is_pkc_capable(node_num)` seam (de-`allow(dead_code)`'d).
|
||||
- `listener/session.rs::refresh_contacts` ORs the new value into `MeshPeer.pkc_capable`
|
||||
(capability only grows, never cleared by a transient refresh) — this IS a touch of
|
||||
session.rs, but additive/non-colliding with the Reticulum device-enum match arms
|
||||
already there; did not touch transport plumbing/routing.
|
||||
- `mod.rs::MeshService::send_message` now does `archy || self.peer_pkc_capable(contact_id)`
|
||||
for the Sent-row `encrypted` flag (was `archy`-only before).
|
||||
- Verified via `cargo check -p archipelago --bin archipelago` (clean, exit 0) **before**
|
||||
the other agent's latest edit landed.
|
||||
|
||||
**NOT YET DONE:** rebuild release binary → redeploy 5 nodes → push → user on-device test
|
||||
(same as #16, both still pending live verification).
|
||||
|
||||
**⚠️ BLOCKED right now — do not build/deploy/push until this clears:** the Reticulum
|
||||
agent is actively mid-edit in the *same* working tree. A `cargo test` run right after
|
||||
the clean `cargo check` above failed with a real (but transient, not mine) signature
|
||||
mismatch: `session.rs::auto_detect_and_open` / `run_mesh_session` were observed with a
|
||||
new `device_kind: Option<DeviceType>` param that `listener/mod.rs`'s call site didn't
|
||||
have yet — a normal in-flight snapshot of their work, not a regression to fix here.
|
||||
**Action on resume: re-run `cargo check` first; if it's clean, the other agent's edit
|
||||
has settled and it's safe to proceed to build/test/deploy. If still broken, wait —
|
||||
do not stash, revert, or patch their in-progress session.rs/listener/mod.rs changes**
|
||||
(see memory `feedback_concurrent_agent_tree.md`). Also: building/deploying right now
|
||||
would bundle their not-yet-finished `reticulum.rs` wiring into the binary — confirm
|
||||
with the user before shipping a combined build, since only the meshtastic `#17` piece
|
||||
has been asked for/owned by this session.
|
||||
|
||||
---
|
||||
|
||||
## ▶️▶️ LIVE CHECKPOINT 2026-06-30 (late PM) — READ THIS FIRST
|
||||
|
||||
**Fleet state:** all **5 test nodes** on binary **`38c456b0bacec3c4`** + frontend
|
||||
**`Mesh-CAkPgvLo.js`**, `archipelago` active on each:
|
||||
`.116`, `.198`, `.228` (LAN, archipelago@ + `~/.ssh/archipelago-deploy`),
|
||||
`100.72.136.5`, `100.89.209.89` (Tailscale, same key — installed this session;
|
||||
SSH user `archipelago` / pw `<FLEET_PW>`; NOPASSWD sudo on all 5).
|
||||
|
||||
**Shipped this session (commit `12e7990b` on `main`, pushed to gitea-vps2):**
|
||||
- ✅ **#16 public-channel routing** — inbound Meshtastic text to `BROADCAST_NUM`
|
||||
now files under the **public channel thread** (contact_id `u32::MAX - idx`),
|
||||
attributed to its real sender, instead of polluting per-sender DM threads.
|
||||
Directed text (`to == our node`) still routes to the DM thread (regression test
|
||||
`packet_to_inbound_frame_directed_dm_stays_a_contact_message`). `send_channel_text`
|
||||
now sets `MeshPacket.channel` so archy TX's on channel 0 (public).
|
||||
Code: `meshtastic.rs` (`packet_to_inbound_frame`, `parse_mesh_packet` to/channel,
|
||||
`send_channel_text`), `protocol.rs` (`RESP_MESHTASTIC_CHANNEL_TEXT = 0x70`),
|
||||
`listener/frames.rs` (handler + sender attribution), `Mesh.vue` (`senderLabelFor`).
|
||||
Tests green (95 mesh tests). **Pending: user on-device test with the radios.**
|
||||
|
||||
**Push access:** `main` is a PROTECTED branch on gitea-vps2. Direct push uses the
|
||||
dedicated **`ai`** account via remote **`gitea-ai`** (`git push gitea-ai main`).
|
||||
See memory `reference_gitea_ai_push_account.md`.
|
||||
|
||||
**Coordination:** another agent owns **Reticulum** (`reticulum-daemon/` + Rust
|
||||
transport wiring). DO NOT touch `mesh/listener/session.rs` transport plumbing or
|
||||
`mod.rs` routing in ways that collide. Keep #17 work isolated to `meshtastic.rs`
|
||||
RX/TX + (if needed) the sent-row encrypted flag.
|
||||
|
||||
### ✅ CODE-COMPLETE (not yet deployed/tested live) — #17 (3ccc / stock-peer E2E pill)
|
||||
Goal: DMs **to and from** a PKC-capable stock peer (3ccc, NodeInfo public_key
|
||||
key_len=32 confirmed) must show the E2E pill.
|
||||
- **RX side is already correct:** `parse_mesh_packet` reads `public_key` (field 16)
|
||||
+ `pki_encrypted` (field 17) per the MeshPacket proto; the directed-DM RX path
|
||||
promotes to `RESP_CONTACT_MSG_V3_E2E` when `pki_encrypted`. (Verify live.)
|
||||
- **TX bug (root cause) — FIXED:** `mod.rs::send_message` now records the Sent row
|
||||
with `encrypted = archy || peer_pkc_capable(contact_id)`. `peer_is_pkc_capable`
|
||||
(meshtastic.rs) is wired out via `get_contacts()` → `ParsedContact.pkc_capable` →
|
||||
`refresh_contacts` (session.rs) → `MeshPeer.pkc_capable` → `MeshService::peer_pkc_capable`.
|
||||
See the LIVE CHECKPOINT at the top of this file for the exact touch points.
|
||||
- NEXT STEP when resuming: confirm `cargo check` is clean (the other agent's
|
||||
Reticulum work shares this tree and may be mid-edit — see top checkpoint), then
|
||||
rebuild → redeploy 5 nodes → push → user test (same pending step as #16).
|
||||
|
||||
**Remaining open after #17:** #12 (provisioning robustness — HOLD, session.rs churn
|
||||
risks reticulum collision), #8 (Device-tab settings panel + reboot button — RPC
|
||||
`mesh.reboot-radio` already exists), #6 (onboarding modal), #7 (.116 re-verify),
|
||||
#14 (RSSI/SNR per-contact indicator), #15 (peer-location map, POSITION_APP portnum=3).
|
||||
|
||||
---
|
||||
|
||||
## ▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST
|
||||
|
||||
**Goal:** archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill,
|
||||
identical in off-grid and normal mode. Test bed = `.116` / `.198` / `.228` (all EU_868).
|
||||
Don't touch the federation/FIPS path.
|
||||
|
||||
### ✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)
|
||||
VERIFIED: `.198→.228` directed DM → `.228` row `RECEIVED enc=True peer="Arch Optiplex"`.
|
||||
All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary
|
||||
**`737b16c3235b`** active on all three. Fix source **COMMITTED as `a57ae388`** on `main`
|
||||
(not yet pushed to gitea-vps2/origin).
|
||||
|
||||
**THE fix (receive stream):** archy ignored `FromRadio.rebooted` (field 8). Every config
|
||||
write reboots the radio → firmware PhoneAPI resets to `STATE_SEND_NOTHING` and stops
|
||||
streaming received packets until the client re-sends `want_config`. archy never did →
|
||||
went deaf to inbound (that's why old messages only arrived after a full restart = fresh
|
||||
want_config). Fix: handle `FROM_RADIO_REBOOTED` → set `pending_reinit` → re-send
|
||||
want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and
|
||||
a pinned `modem_preset=LONG_FAST` so all radios share frequency. Combined with the earlier
|
||||
E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.
|
||||
|
||||
**Open follow-ups:** #A surface received msgs under archy identity in all UI views; #6
|
||||
device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation;
|
||||
#12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog;
|
||||
#14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr);
|
||||
#15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3
|
||||
lat/lon). See the resume memory `project_session_resume_2026_06_30_lora.md` for the full
|
||||
task list.
|
||||
|
||||
### (historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix
|
||||
The **archy software is correct and deployed.** The blocker was at the
|
||||
**radio/RF layer: the three radios are not hearing each other over the air at all.** No
|
||||
amount of archy code change will fix that until the radios actually RF-link. **Resume by
|
||||
testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS
|
||||
FIRST AT HOME" below.** ← this turned out to be the want_config resubscribe bug above.
|
||||
|
||||
### What is DONE and deployed (commit pending — see below)
|
||||
- **E2E send fix** (`core/archipelago/src/mesh/mod.rs` `send_message`, ~L1542): archy↔archy
|
||||
plain chat text is now sent as a **native `TEXT_MESSAGE_APP` DM** (firmware PKC-encrypts
|
||||
it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked
|
||||
`encrypted=true` so the pill shows. Rich typed msgs still use `send_typed_wire`. This was
|
||||
the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
|
||||
- **NEW: software radio-reboot** end-to-end, so a wedged/RX-deaf radio can be rebooted
|
||||
without physical access (and for the Device-tab settings panel the user requested):
|
||||
- `meshtastic.rs`: `reboot(seconds)` driver method + `ADMIN_REBOOT_SECONDS_FIELD = 97`
|
||||
(verified vs meshtastic/protobufs admin.proto — `set_owner=32/set_channel=33/set_config=34`
|
||||
matched our existing constants, confirming the proto read).
|
||||
- `listener/mod.rs`: `MeshCommand::RebootRadio { seconds }`.
|
||||
- `listener/session.rs`: device-enum `reboot()` dispatch (Meshtastic only) + handler arm.
|
||||
- `mesh/mod.rs`: `MeshService::reboot_radio(seconds)`.
|
||||
- `api/rpc/mesh/messaging.rs`: `handle_mesh_reboot_radio` → RPC **`mesh.reboot-radio`**
|
||||
`{seconds?}` (default 2); dispatcher arm in `api/rpc/dispatcher.rs`.
|
||||
- `cargo check` passes. Built release **sha `ba4aed590027690d`** and DEPLOYED + active on
|
||||
`.116/.198/.228`. The RPC works (`{"reboot":true,"seconds":2}`).
|
||||
- ⚠️ **Caveat:** when called, archy logged "Sent Meshtastic radio reboot" but the radio did
|
||||
**not** visibly reboot afterward (no config re-stream). Either field 97 is still off, or
|
||||
newer firmware requires an admin session passkey even over local serial, or the USB serial
|
||||
stayed open through the 2s reboot so no reconnect was logged. **Needs on-device verification.**
|
||||
|
||||
### The hard evidence (why "nothing works")
|
||||
- Directed DM tests `.198→.228` AND `.116→.228` (neither path reflashed): sender logs
|
||||
`Sent plain native DM dest=30d258436d65 part=1 total=1` and RPC returns `sent:true,
|
||||
encrypted:true`, but `.228` logs **nothing** — packet never reaches archy from the radio.
|
||||
- A raw broadcast from `.198` (`mesh.broadcast`) was accepted by its radio but **not heard**
|
||||
by `.228`/`.116`.
|
||||
- In an 8-minute window, **all three nodes received 0 inbound OTA packets from any other node.**
|
||||
Each only logs its OWN once-a-minute `Broadcast Meshtastic NodeInfo advert` + local TX
|
||||
`field=11` queue-status. `.228 mesh.status` = `messages_received:1` total.
|
||||
- `.198`'s radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that
|
||||
**reception is broken on the receivers.** A radio cannot drop a broadcast AND a unicast to
|
||||
its own node number while config matches, unless it simply isn't on the same airwaves.
|
||||
- archy provisioning is correct & identical across nodes (read back from device): PRIMARY =
|
||||
public LongFast (`name="" psk_len=1`), SECONDARY = `archipelago`, region=3 (EU_868). Admin
|
||||
field constants verified. The send path hands the radio a correct unicast MeshPacket
|
||||
(`to`=node, want_ack, hop_limit=3, plaintext `decoded` for the firmware to PKC-encrypt).
|
||||
|
||||
### PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch
|
||||
archy only ever writes `region` + `use_preset` and **never explicitly pins `modem_preset`**
|
||||
(it parses region but not preset; `set_lora_region` relies on the LongFast default). If ANY
|
||||
radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic
|
||||
app, or a different factory default after the `.198` reflash), the radios are on **different
|
||||
airwaves despite identical channel name + region**, and archy would never correct it.
|
||||
|
||||
### DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)
|
||||
Open the **Meshtastic phone app over Bluetooth** (works alongside archy's USB serial) on each
|
||||
of `.116/.198/.228` and check:
|
||||
1. Do the 3 nodes **see each other** in the node list (recent "heard")? → if NO, they're not
|
||||
RF-reaching (preset/freq/antenna/range).
|
||||
2. Do all 3 show the **same** Modem preset (LongFast), Region (EU_868), Frequency slot, and
|
||||
the same PRIMARY channel? → any difference = the cause.
|
||||
This single test separates "archy misconfigures the radios" from "radios physically can't
|
||||
reach each other."
|
||||
|
||||
### THEN — the archy fix to apply (if preset/config differs)
|
||||
Make archy **authoritatively write the full LoRaConfig** and force re-provision so all radios
|
||||
converge: in `core/archipelago/src/mesh/meshtastic.rs::set_lora_region` (and its
|
||||
caller/guard `ensure_lora_region` ~L304), explicitly set `modem_preset = LONG_FAST (0)` as a
|
||||
field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision
|
||||
path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new
|
||||
`mesh.reboot-radio`). Also verify the `mesh.reboot-radio` actually reboots the radio
|
||||
on-device (the caveat above).
|
||||
|
||||
### TEST RECIPE (works on each node)
|
||||
- RPC helper used this session: a node-side `rpc.sh` that logs in (password
|
||||
`<FLEET_PW>`), grabs the `csrf_token` cookie, echoes it as `X-CSRF-Token`, and POSTs to
|
||||
`http://127.0.0.1:5678/rpc/v1`. Recreate it or run archy's RPC directly. Methods:
|
||||
`mesh.peers`, `mesh.status`, `mesh.messages`, `mesh.send {contact_id,message}`,
|
||||
`mesh.broadcast`, `mesh.reboot-radio {seconds}`.
|
||||
- **LoRa contact ids:** `.116=1135977788` (prefix `3ca5b543`), `.198=3677050140` (`db2b551c`),
|
||||
`.228=1129894448` (prefix `30d25843`), stock `3ccc=1128152268`.
|
||||
- **Link health check (run on each node):** look for inbound `from=Some("!...")` lines in
|
||||
`journalctl -u archipelago` that are NOT the node's own `Broadcast ... NodeInfo advert`. If
|
||||
zero across all nodes → RF link is down (the current state).
|
||||
- **E2E success criteria:** send `.198→.228`, the marker appears in `.228` `mesh.messages` as
|
||||
an inbound row with `encrypted:true` / `transport:"lora"`, AND `.116↔.228` likewise.
|
||||
|
||||
### DEPLOY / BUILD RECIPE
|
||||
- Build: from `core/`, `CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo
|
||||
build --release -p archipelago --bin archipelago`. (If `rust-lld: undefined hidden symbol`,
|
||||
it's incremental cache — `CARGO_INCREMENTAL=0` fixes it.)
|
||||
- SSH key `~/.ssh/archipelago-deploy` is authorized on `.116/.198/.228`. SSH/UI/RPC password
|
||||
`<FLEET_PW>`. Per node: scp the binary, `sudo systemctl stop archipelago` →
|
||||
`kill -9 $(pgrep -x archipelago)` → `install -m0755` to `/usr/local/bin/archipelago` →
|
||||
`systemctl start archipelago`. Verify by `sha256sum` match + `systemctl is-active`.
|
||||
- **Current deployed sha on all 3 = `ba4aed590027690d`** (the reboot-enabled build).
|
||||
|
||||
### Fleet state (as of 2026-06-30 PM)
|
||||
- All 3 nodes on binary `ba4aed59`, active. Off-grid mode currently OFF (`mesh_only:false`).
|
||||
- `.198` radio was reflashed to factory `firmware-heltec-v3-2.7.26` (recovered from corrupt
|
||||
NVS); region EU_868 persists. Its archy identity is NOT re-bound on `.228` (`.228` shows
|
||||
`.198` as raw radio "Meshtastic 551c", `arch_pubkey_hex` absent) because `.228` hasn't heard
|
||||
`.198`'s identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
|
||||
- The radios are powered & each transmitting; they are simply not hearing each other.
|
||||
|
||||
### Deferred UI (after LoRa works)
|
||||
- Device-tab **settings panel** (gear/desktop) — host the "Reboot radio" button there; calls
|
||||
`mesh.reboot-radio`. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid
|
||||
buttons) + a `rebootRadio()` method in `neode-ui/src/stores/mesh.ts`. See `Mesh.vue` ~L1484
|
||||
actions row and `mesh.ts` ~L373 `broadcastIdentity()` pattern.
|
||||
- Device-onboarding modal (detect plugged-in radio).
|
||||
|
||||
---
|
||||
|
||||
Current scope:
|
||||
- Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
|
||||
- Take over the `3ccc` stock Meshtastic peer bug: LoRa text from `3ccc` to Archipelago `.116` does not surface in `mesh.messages`.
|
||||
- Keep release-gate fixes already made in this session.
|
||||
|
||||
Local gate status so far:
|
||||
- `cargo test -p archipelago --bin archipelago`: green, 849/849 after Meshtastic fixes.
|
||||
- `python3 scripts/check-app-catalog-drift.py --release --strict`: green.
|
||||
- `npm run type-check`: green.
|
||||
|
||||
Key changes made so far:
|
||||
- Added cascade uninstall progress truthfulness assertion to `tests/lifecycle/bats/cascade-uninstall.bats`.
|
||||
- Fixed release catalog drift filters and regenerated catalog metadata.
|
||||
- Fixed invalid `apps/fedimint-clientd/manifest.yml` `cpu_limit` schema value.
|
||||
- Updated stale/tight Rust tests without changing production behavior.
|
||||
|
||||
Remaining non-automatable / operational gates:
|
||||
- Workstream B signing is blocked on the offline `RELEASE_MASTER_MNEMONIC`; code + runbook exist, but the publisher must pin/sign the release-root catalog.
|
||||
- Phase-3 Quadlet backend rollout is implemented behind `use_quadlet_backends` and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
|
||||
- `.116` read-only `use-quadlet-backends-install.bats`: 6/6 skip-clean; no backend `.container` units, so Phase-3 is not active on that node.
|
||||
- Release metadata still says `1.7.99-alpha` in `releases/manifest.json`; changelog top is `v1.8.00-alpha`. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.
|
||||
|
||||
Do not discard:
|
||||
- `core/archipelago/src/mesh/listener/decode.rs`
|
||||
- `core/archipelago/src/mesh/listener/session.rs`
|
||||
- `core/archipelago/src/mesh/meshtastic.rs`
|
||||
|
||||
3ccc bug current hypothesis:
|
||||
- The prior attempted Meshtastic fix added a hard stale-packet filter using `rx_time`.
|
||||
- Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
|
||||
- That would make live `3ccc` packets look older than 10 minutes and get dropped before `mesh.messages`.
|
||||
- Current patch treats implausibly early `rx_time` values as unknown rather than stale.
|
||||
|
||||
.116 live validation after 2026-06-30 hotfix:
|
||||
- `.116` reachable by SSH; `archipelago` active; `/dev/mesh-radio -> ttyUSB0` attached.
|
||||
- Current canary deploy is commit `b4531bb4`; backend sha
|
||||
`4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf`; frontend bundle
|
||||
`index-YOAeJF7w.js` / `Mesh-BSAo88jN.js`.
|
||||
- `main` pushed to `gitea-vps2`.
|
||||
- RPC on `.116`:
|
||||
- `transport.status` currently reports `mesh_only:false` (off-grid mode is not enabled unless
|
||||
the user toggles it).
|
||||
- `mesh.status` reports Meshtastic connected: `device_type:"meshtastic"`,
|
||||
`self_node_id:1135977788`, `peer_count:13`.
|
||||
- Recent `.116` -> `3ccc` sent rows are stored with real 2026 timestamps and `transport:"lora"`.
|
||||
- UI/backend fixes included in `b4531bb4`:
|
||||
- `transportLabel("lora")` displays **LoRa**.
|
||||
- mesh sends refetch messages after send so transport pills settle without browser refresh.
|
||||
- off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends;
|
||||
banner text is `Tor/FIPS disabled - LoRa only`.
|
||||
- empty mesh-chat placeholder opacity reduced.
|
||||
- Meshtastic diagnostics now identify the remaining blocker:
|
||||
- 3ccc NodeInfo is discovered:
|
||||
`Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32`.
|
||||
- Bytes from stock Meshtastic text reach `.116`, but the custom parser rejects the packet:
|
||||
`Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed`.
|
||||
- Non-text packets decode and are ignored with port numbers (`portnum=3/4/5`), so the serial
|
||||
read path is alive. Resume inside `core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet`.
|
||||
- LoRa is therefore **not fully fixed** yet: stock `3ccc` -> `.116` text does not surface in
|
||||
`mesh.messages`, and `.116` -> `3ccc` still needs user-visible confirmation in the Meshtastic app.
|
||||
Reference in New Issue
Block a user