diff --git a/docs/OPEN-SOURCE-READINESS-PLAN.md b/docs/OPEN-SOURCE-READINESS-PLAN.md new file mode 100644 index 00000000..607f86f1 --- /dev/null +++ b/docs/OPEN-SOURCE-READINESS-PLAN.md @@ -0,0 +1,219 @@ +# Open-Source Readiness Plan — Archipelago public launch + +> Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup. +> A second agent is working the same goal concurrently — before executing any phase, +> diff against `git log` since `7e8d3314` and skip/merge what's already done. +> (Session plan file: `~/.claude/plans/resilient-moseying-reef.md`.) + +## Context + +The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep +exploration passes (docs/structure, code health, secrets sweep) found the repo is +fundamentally strong — README, `apps/` manifest examples, ADRs, the bats lifecycle gate, +1,104 Rust tests — but has hard blockers: **two live Anthropic API keys committed in +tracked files**, node passwords in 7 tracked files, no LICENSE (README links a 404), +5.5 GB `.git` (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea +registry `146.59.87.168:3000` that make every app image unpullable for outsiders, and +~28 internal AI-session/tracker docs mixed into `docs/`. + +**Decisions made by the user:** +1. **Fresh-history publish** — new public repo with a clean initial commit; private repo keeps full history. +2. **Registry: domain + parameterize** — real domain in front of the existing registry; host configurable everywhere. +3. **Deep code cleanup** — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly). +4. **Internal docs: sanitize and keep public** — scrub creds/IPs/hostnames but publish plans/trackers for transparency. + +**Invariant throughout:** the single-node production gate (`tests/lifecycle/run-gate.sh`) +is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E +especially). All cargo verification uses `--all-features` to match CI. Stage by explicit +path, never `git add -A` (shared tree). + +--- + +## Phase 0 — Credential rotation (immediate, independent of the repo) + +Treat all of these as already compromised; rotate even though we're doing fresh-history: + +- **Anthropic API key #1**: `image-recipe/_archived/build-auto-installer-iso.sh:2837` (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (`ISO_ANTHROPIC_API_KEY`), keep the alpha-baking behavior if desired but never the literal in git. +- **Anthropic API key #2**: `scripts/setup-aiui-server.sh:28` — a *different* live key, not covered by the documented alpha exception. Revoke; parameterize the script. +- **The shared node SSH/sudo/UI password** (two variants) — in 7 tracked files + 24+ commits. Rotate fleet-wide (user task). +- **Gitea `ai` account password + 2 Gitea tokens** — embedded in `.git/config` remote URLs (not tracked, but leaks in any directory copy/tarball). Rotate; switch remotes to credential-helper storage instead of URL-embedded creds. + +## Phase 1 — Secrets & sanitization of tracked files + +1. Strip the password/credential lines from the 7 files: + `docs/PRODUCTION-MASTER-PLAN.md` (lines ~428–429, 454–457, 483, 521–528, 886 — the fleet cred table), + `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`, `docs/archive/HANDOVER-2026-07-02-iso-feedback.md`, + `docs/bitcoin-version-bulletproof-rollout.md`, `tests/production-quality/TRACKER.md`, + `tests/multinode/meshtastic.sh:26`, `neode-ui/test-openwrt.mjs:4` (→ env var). +2. `.gitea/workflows/post-install-tests.yml` — remove `sshpass -p '…'` + default target IP; use secrets/vars. +3. Sanitize infra identifiers repo-wide (in the *sanitize-and-keep* docs and scripts): + replace Tailscale IPs (17 unique, 14 files), LAN IPs (`192.168.1.x`, 93 files), hostnames + (`tx1138`, `shorty-s`, `archy-x250`, `archy-dev-pa`) with placeholders like `` / + `NODE_IP`. Key script targets: `scripts/deploy-config-defaults.sh`, `scripts/deploy-tailscale.sh`, + `docs/operations-runbook.md` (opens with real node IPs), `docs/developer-guide.md`, `docs/api-reference.md`, `docs/hotfix-process.md`. +4. Fix the audit tool that let this happen: `scripts/audit-secrets.sh:28` — remove `\.md$` and + bare `test` from ALLOW_PATTERNS; add `sk-ant-` and password-table patterns; scan all + tracked files not just `*.env`. Run it clean as a Phase-1 exit check. +5. `.gitignore` additions: `.claude/`, `*.key`, `*.pem`, `id_rsa*`, `*.sqlite`, `*.db` + (`.claude/settings.local.json` with creds is currently only ignored by a machine-global rule). +6. Product-security note to raise (not fix now): `password123` is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced. + +## Phase 2 — Repo restructure: deletions, binaries, layout + +Delete (each its own commit): +- `loop/` (AI overnight harness w/ node SSH lines), `.agents/`, `.codex`, `.githooks/pre-push` + (the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history). +- `indeedhub/` submodule + `.gitmodules` entry (points at private HTTP Gitea, breaks `--recursive` + clones); `indeedhub-demo/` (single Dockerfile — merge or drop). +- `RELEASE-NOTES-v1.0.0.md` (superseded by CHANGELOG), `neode-ui/docs/GAMEPAD-NAV-MAP.md` (duplicate of `docs/GAMEPAD-NAV.md`). +- Stray generated HTML: `docs/container-architecture.html` (311 KB), `docs/archive/architecture-review.html`, `docs/archive/lora-functionality.html`. +- `Android/local.properties` from tracking (local absolute path); remove `Android/app/debug.keystore` (standard practice). + +Move out of git (→ release assets on the Releases page, referenced by URL): +- `neode-ui/public/packages/archipelago-companion.apk` (27 MB), `wireguard.apk` (17 MB), `atob.s9pk` (23 MB). +- `Android/archipelago-0.3.0-debug.apk.zip` (16 MB, stale). +- `demo/content/music/*` + heavy `demo/aiui/assets` (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size). +- `neode-ui/dev-dist/` (generated Workbox output) → gitignore. + +Rename/fix the naming lie: `image-recipe/_archived/` contains the *production* ISO builder +(`build-auto-installer-iso.sh`, referenced by `.gitea/workflows/build-iso.yml`). Move live +files up into `image-recipe/`, delete the genuinely archived rest. + +## Phase 3 — Registry domain + parameterization (functional blocker) + +Infra (user assists: DNS + TLS): +- Put a domain (e.g. `registry.archipelago-os.org` / `git.archipelago-os.org`) with HTTPS in + front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS. + +Repo changes: +- Introduce a single source of truth for the registry host (e.g. `REGISTRY_HOST` in + `scripts/lib/` + a default in the orchestrator config). Replace `146.59.87.168:3000` in: + all 56 `apps/*/manifest.yml`, `app-catalog/catalog.json`, `releases/manifest.json`, + `release-manifest.json`, the 11 scripts (`self-update.sh`, `create-release.sh`, + `generate-app-catalog.sh`, `validate-app-manifest.sh`, `first-boot-containers.sh`, …), + both `demo-images.yml` workflows, `demo-deploy/.env.example`, and the Android sources + (`FipsPreferences.kt`, `PartyScreen.kt`). +- Because the catalog is signed: regenerate + re-sign + republish the app catalog after the + manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise). + Signing needs the user's mnemonic → schedule one ceremony after manifests are final. +- Verify: fresh machine with no LAN/tailnet access can `podman pull` one app image via the + domain and the gate node still installs apps after the re-signed catalog lands. + +## Phase 4 — Documentation overhaul + +### 4a. Community/legal files (missing today) +- `LICENSE` — MIT (matches existing README badge). Add `[workspace.package] license` + + `license.workspace = true` in the 5 member Cargo.tomls (also see Phase A4). +- `SECURITY.md` — disclosure address, PGP key, supported-versions; cite the March 2026 audit (`docs/archive/security-code-audit-2026-03.md`). +- `CODE_OF_CONDUCT.md` — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today). +- `CONTRIBUTING.md` edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb + the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model, + non-destructive migrations), versioning policy note for the `-alpha` scheme. +- `CLAUDE.md` — rewrite: keep invariants/build-verify (public-worthy), remove status banner, + node numbers, `gitea-ai` push mechanics, MEMORY references (those move to private notes). + +### 4b. New developer docs (the three real gaps for app developers) +1. **`docs/quadlet-compilation.md`** — how a manifest becomes a Quadlet/systemd unit: naming, + `systemctl --user` lifecycle, where units land, how to inspect/debug one. (Source: + `core/archipelago/src/container/quadlet*.rs`, prod_orchestrator.) +2. **`docs/container-lifecycle.md`** — the 30 s level-triggered reconciler, install/adopt/ + restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped + `docs/bulletproof-containers.md` as the current description; salvage its content.) +3. **`docs/secrets.md`** — `generated_secrets` declaration → materialisation by + `container::secrets` (0600, rootless) → injection; what developers must never do. +- Also: make every example in `docs/app-developer-guide.md` + `apps/*/manifest.yml` copy-paste + work against the new public registry host; add an end-to-end "write your first app" walkthrough + that a stranger can follow with only the public repo + an Archipelago node. + +### 4c. Sanitize-and-keep internal docs (user's transparency choice) +- Keep, after Phase-1 scrubbing: `docs/PRODUCTION-MASTER-PLAN.md`, `docs/UNIFIED-TASK-TRACKER.md`, + `docs/1.8.0-RELEASE-HARDENING-PLAN.md`, `docs/RETICULUM-TRANSPORT-PROGRESS.md`, HANDOFF-*, test + plans, `docs/archive/*` — but **move all session/handoff/tracker material under + `docs/history/`** (extending the existing honest `docs/archive/README.md` pattern) so the + top-level `docs/` reads as current reference only. Add a banner to each: "historical working + document, sanitized; not maintained." +- Remove dangling agent-memory references in tracked docs (`docs/bulletproof-containers.md`, + `docs/RETICULUM-TRANSPORT-PROGRESS.md`, `docs/registry-manifest-design.md`, + `docs/bitcoin-multi-version-design.md` progress block). +- De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status + field; e.g. `docs/APP-PACKAGING-MIGRATION-PLAN.md` → public app-platform design doc). +- Extract North-Star narrative from PRODUCTION-MASTER-PLAN into `docs/ROADMAP.md`; extract + the "run the gate ON the node" philosophy from `docs/multinode-testing-plan.md` into + `tests/lifecycle/TESTING.md`. +- Add `docs/README.md` index (bitcoin/bitcoin `doc/` style): Getting started / Architecture / + App development / Operations / Design docs (ADRs) / History. +- README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized + docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at + CONTRIBUTING.md only. + +## Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit) + +### A. Zero-risk deletions & metadata (S each, own commits) +- **A1** Delete orphan non-compiling StartOS crates: `core/models`, `core/helpers`, + `core/js-engine` (incl. 2 committed `JS_SNAPSHOT.*.bin`), `core/container-init` (~4,100 LOC, + zero references). Verify: `cargo build --workspace && cargo test --all-features`. +- **A2** Delete unreferenced Vue components: `neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue`. Verify: `npm run type-check && npm run build`. +- **A3** Fix `.gitignore` lockfile lines (7: `Cargo.lock`, 15: `package-lock.json`) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles. +- **A4** LICENSE + Cargo license fields (see 4a). Verify with `cargo metadata`. +- **A5** `core/rust-toolchain.toml` pinning `1.95.0`; align `.github/workflows/ci.yml` (remove explicit `toolchain: stable` input so the file wins). Upgrades become deliberate PRs. +- **A6** `core/rustfmt.toml` codifying **defaults only** (`edition = "2021"` + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify `cargo fmt --all -- --check` yields no diff. + +### B. CI guards (zero runtime risk) +- **B1** Enable vitest in CI: run `cd neode-ui && npm run test` locally; fix trivial failures, `.skip`+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration). +- **B2** Raw podman/systemctl **ratchet, not migration**: the 132 raw `Command::new("podman"/"systemctl")` sites use subcommands the `core/container/src/podman_client.rs` wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical `install.rs`, and the prod path intentionally uses Quadlet+systemctl. Add `scripts/ci/raw-podman-ratchet.sh` (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design. + +### C. Clippy suppression trim (`core/archipelago/src/main.rs:8-18`, per-lint commits) +- Remove cheaply: `assertions_on_constants`, `drop_non_drop`, `wildcard_in_or_patterns`, `doc_lazy_continuation`, `enum_variant_names` (targeted allows on serde enums — never rename wire variants). +- Own careful commit: `unused_io_amount` — a **correctness** lint; fix sites with `read_exact`/`write_all` or documented targeted allows (`mesh/serial.rs:456,496` has raw partial reads; serial framing may be intentional). Full test suite + gate after. +- Keep crate-wide with justifying comment: `too_many_arguments`, `type_complexity`; attempt `ptr_arg` (`&Vec`→`&[T]`, mechanical) if time allows — first to cut. +- Verify each: `cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features`. + +### D. dead_code lift — Tiers 1–2 pre-launch, Tier 3 → commented allows + issues +Per-module procedure (one file per commit): remove `#![allow(dead_code)]` → `cargo check +--all-targets --all-features` → triage each warning: (a) genuinely dead → delete; +(b) future-feature/protocol-mandated → targeted `#[allow(dead_code)] // TODO(#NNN): …`; +(c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) → +clippy `-D warnings` + tests → commit. +- **Tier 1 (small/leaf, S each):** `swarm/seed_advert.rs`, `transport/{mesh_transport,lan,chunking,delta}.rs`, `mesh/{crypto,alerts,types,outbox}.rs`, `streaming/mod.rs`, `wallet/mod.rs`. +- **Tier 2 (M each):** `fips/{mod,iface,dial}.rs` (41 external refs → little residual deadness), `mesh/{x3dh,ratchet,steganography,message_types}.rs` — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why). +- **Tier 3 (defer, riskiest):** `mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs`, `transport/mod.rs` — change each blanket allow to `#![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN`. +- Optional S/M win: move `prod_orchestrator.rs`'s 5,034-line `#[cfg(test)]` module to a sibling file via `#[path]` (pure move, halves the 6,291-line file). + +### E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated) +Legacy installers for immich/btcpay/mempool/indeedhub (`core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498`, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging `INSTALL ORCH SKIP` (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920). +1. Run the full gate on the node; grep install logs for `INSTALL ORCH SKIP`. +2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + `adopt_stack_if_exists`; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue. +3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers". + +### Explicitly deferred → public tracked issues at launch +PodmanClient API extension + call-site migration; god-module splits (`install.rs`, `update.rs`, `mesh/mod.rs`); Playwright in CI; Tier-3 dead_code triage; `password123` default hardening. + +## Phase 6 — Fresh-history publish + +1. Freeze: all phases merged on internal `main`, gate green, catalog re-signed. +2. Build the public tree: `git archive`-style export of HEAD (never copy `.git/` — it holds + credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"), + optionally preserving CHANGELOG.md as the human-readable history. +3. Pre-publish gate on the export: `scripts/audit-secrets.sh` (fixed version) clean; grep-zero for + `sk-ant-`, rotated-password strings, `146.59.87.168`, tailnet `100.` IPs, `192.168.1.`, + internal hostnames; `du -sh .git` sanity (< ~100 MB); fresh `git clone` + `cd core && cargo build` + + `cd neode-ui && npm ci && npm run build` on a clean machine/container; one app image pull + from the public domain. +4. Publish to GitHub; enable issue templates (already present in `.github/`); file the deferred-work + issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points. +5. Internal repo remains the private full-history remote; decide sync direction post-launch + (recommend: public repo becomes canonical, private keeps only ops/infra notes). + +## Verification (end-to-end) + +- `tests/lifecycle/run-gate.sh` green on the node after Phases 3 + 5E (and after any lifecycle-touching commit). +- CI green on every phase commit: `cargo fmt --check`, `clippy -D warnings`, `cargo test --all-features`, frontend type-check + build + (new) vitest. +- Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer. +- Docs acceptance: a reader following `docs/app-developer-guide.md` + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra. + +## Sequencing / cut-line + +Order: 0 → 1 → 2 → (3 ∥ 4) → 5 (A→E) → 6. Phases 0–2 are non-negotiable security; Phase 3 is the +functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any +commit (minimum viable: A1–A6, B1–B2, unused_io_amount fix); Phase 6 last. If the timeline +compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds.