lfg2025/archy
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects Releases 61 Wiki Activity
archy/docs/OPEN-SOURCE-READINESS-PLAN.md
archipelago c5eeb4392f docs: open-source readiness plan — phases 0-6 for public launch 
Consolidated plan from the deep repo review: credential rotation, secret
scrub, repo restructure, registry parameterization, docs overhaul, deep
code cleanup, and fresh-history publish mechanics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:38:01 -04:00

18 KiB
Raw Blame History

Open-Source Readiness Plan — Archipelago public launch

Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup. A second agent is working the same goal concurrently — before executing any phase, diff against git log since 7e8d3314 and skip/merge what's already done. (Session plan file: ~/.claude/plans/resilient-moseying-reef.md.)

Context

The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep exploration passes (docs/structure, code health, secrets sweep) found the repo is fundamentally strong — README, apps/ manifest examples, ADRs, the bats lifecycle gate, 1,104 Rust tests — but has hard blockers: two live Anthropic API keys committed in tracked files, node passwords in 7 tracked files, no LICENSE (README links a 404), 5.5 GB .git (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea registry 146.59.87.168:3000 that make every app image unpullable for outsiders, and ~28 internal AI-session/tracker docs mixed into docs/.

Decisions made by the user:

  1. Fresh-history publish — new public repo with a clean initial commit; private repo keeps full history.
  2. Registry: domain + parameterize — real domain in front of the existing registry; host configurable everywhere.
  3. Deep code cleanup — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly).
  4. Internal docs: sanitize and keep public — scrub creds/IPs/hostnames but publish plans/trackers for transparency.

Invariant throughout: the single-node production gate (tests/lifecycle/run-gate.sh) is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E especially). All cargo verification uses --all-features to match CI. Stage by explicit path, never git add -A (shared tree).

Phase 0 — Credential rotation (immediate, independent of the repo)

Treat all of these as already compromised; rotate even though we're doing fresh-history:

  • Anthropic API key #1: image-recipe/_archived/build-auto-installer-iso.sh:2837 (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (ISO_ANTHROPIC_API_KEY), keep the alpha-baking behavior if desired but never the literal in git.
  • Anthropic API key #2: scripts/setup-aiui-server.sh:28 — a different live key, not covered by the documented alpha exception. Revoke; parameterize the script.
  • The shared node SSH/sudo/UI password (two variants) — in 7 tracked files + 24+ commits. Rotate fleet-wide (user task).
  • Gitea ai account password + 2 Gitea tokens — embedded in .git/config remote URLs (not tracked, but leaks in any directory copy/tarball). Rotate; switch remotes to credential-helper storage instead of URL-embedded creds.

Phase 1 — Secrets & sanitization of tracked files

  1. Strip the password/credential lines from the 7 files: docs/PRODUCTION-MASTER-PLAN.md (lines ~428–429, 454–457, 483, 521–528, 886 — the fleet cred table), docs/archive/SESSION-1.8.0-OTA-PROGRESS.md, docs/archive/HANDOVER-2026-07-02-iso-feedback.md, docs/bitcoin-version-bulletproof-rollout.md, tests/production-quality/TRACKER.md, tests/multinode/meshtastic.sh:26, neode-ui/test-openwrt.mjs:4 (→ env var).
  2. .gitea/workflows/post-install-tests.yml — remove sshpass -p '…' + default target IP; use secrets/vars.
  3. Sanitize infra identifiers repo-wide (in the sanitize-and-keep docs and scripts): replace Tailscale IPs (17 unique, 14 files), LAN IPs (192.168.1.x, 93 files), hostnames (tx1138, shorty-s, archy-x250, archy-dev-pa) with placeholders like <node-a> / NODE_IP. Key script targets: scripts/deploy-config-defaults.sh, scripts/deploy-tailscale.sh, docs/operations-runbook.md (opens with real node IPs), docs/developer-guide.md, docs/api-reference.md, docs/hotfix-process.md.
  4. Fix the audit tool that let this happen: scripts/audit-secrets.sh:28 — remove \.md$ and bare test from ALLOW_PATTERNS; add sk-ant- and password-table patterns; scan all tracked files not just *.env. Run it clean as a Phase-1 exit check.
  5. .gitignore additions: .claude/, *.key, *.pem, id_rsa*, *.sqlite, *.db (.claude/settings.local.json with creds is currently only ignored by a machine-global rule).
  6. Product-security note to raise (not fix now): password123 is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced.

Phase 2 — Repo restructure: deletions, binaries, layout

Delete (each its own commit):

  • loop/ (AI overnight harness w/ node SSH lines), .agents/, .codex, .githooks/pre-push (the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history).
  • indeedhub/ submodule + .gitmodules entry (points at private HTTP Gitea, breaks --recursive clones); indeedhub-demo/ (single Dockerfile — merge or drop).
  • RELEASE-NOTES-v1.0.0.md (superseded by CHANGELOG), neode-ui/docs/GAMEPAD-NAV-MAP.md (duplicate of docs/GAMEPAD-NAV.md).
  • Stray generated HTML: docs/container-architecture.html (311 KB), docs/archive/architecture-review.html, docs/archive/lora-functionality.html.
  • Android/local.properties from tracking (local absolute path); remove Android/app/debug.keystore (standard practice).

Move out of git (→ release assets on the Releases page, referenced by URL):

  • neode-ui/public/packages/archipelago-companion.apk (27 MB), wireguard.apk (17 MB), atob.s9pk (23 MB).
  • Android/archipelago-0.3.0-debug.apk.zip (16 MB, stale).
  • demo/content/music/* + heavy demo/aiui/assets (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size).
  • neode-ui/dev-dist/ (generated Workbox output) → gitignore.

Rename/fix the naming lie: image-recipe/_archived/ contains the production ISO builder (build-auto-installer-iso.sh, referenced by .gitea/workflows/build-iso.yml). Move live files up into image-recipe/, delete the genuinely archived rest.

Phase 3 — Registry domain + parameterization (functional blocker)

Infra (user assists: DNS + TLS):

  • Put a domain (e.g. registry.archipelago-os.org / git.archipelago-os.org) with HTTPS in front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS.

Repo changes:

  • Introduce a single source of truth for the registry host (e.g. REGISTRY_HOST in scripts/lib/ + a default in the orchestrator config). Replace 146.59.87.168:3000 in: all 56 apps/*/manifest.yml, app-catalog/catalog.json, releases/manifest.json, release-manifest.json, the 11 scripts (self-update.sh, create-release.sh, generate-app-catalog.sh, validate-app-manifest.sh, first-boot-containers.sh, …), both demo-images.yml workflows, demo-deploy/.env.example, and the Android sources (FipsPreferences.kt, PartyScreen.kt).
  • Because the catalog is signed: regenerate + re-sign + republish the app catalog after the manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise). Signing needs the user's mnemonic → schedule one ceremony after manifests are final.
  • Verify: fresh machine with no LAN/tailnet access can podman pull one app image via the domain and the gate node still installs apps after the re-signed catalog lands.

Phase 4 — Documentation overhaul

4a. Community/legal files (missing today)

  • LICENSE — MIT (matches existing README badge). Add [workspace.package] license + license.workspace = true in the 5 member Cargo.tomls (also see Phase A4).
  • SECURITY.md — disclosure address, PGP key, supported-versions; cite the March 2026 audit (docs/archive/security-code-audit-2026-03.md).
  • CODE_OF_CONDUCT.md — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today).
  • CONTRIBUTING.md edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model, non-destructive migrations), versioning policy note for the -alpha scheme.
  • CLAUDE.md — rewrite: keep invariants/build-verify (public-worthy), remove status banner, node numbers, gitea-ai push mechanics, MEMORY references (those move to private notes).

4b. New developer docs (the three real gaps for app developers)

  1. docs/quadlet-compilation.md — how a manifest becomes a Quadlet/systemd unit: naming, systemctl --user lifecycle, where units land, how to inspect/debug one. (Source: core/archipelago/src/container/quadlet*.rs, prod_orchestrator.)
  2. docs/container-lifecycle.md — the 30 s level-triggered reconciler, install/adopt/ restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped docs/bulletproof-containers.md as the current description; salvage its content.)
  3. docs/secrets.md — generated_secrets declaration → materialisation by container::secrets (0600, rootless) → injection; what developers must never do.
  • Also: make every example in docs/app-developer-guide.md + apps/*/manifest.yml copy-paste work against the new public registry host; add an end-to-end "write your first app" walkthrough that a stranger can follow with only the public repo + an Archipelago node.

4c. Sanitize-and-keep internal docs (user's transparency choice)

  • Keep, after Phase-1 scrubbing: docs/PRODUCTION-MASTER-PLAN.md, docs/UNIFIED-TASK-TRACKER.md, docs/1.8.0-RELEASE-HARDENING-PLAN.md, docs/RETICULUM-TRANSPORT-PROGRESS.md, HANDOFF-*, test plans, docs/archive/* — but move all session/handoff/tracker material under docs/history/ (extending the existing honest docs/archive/README.md pattern) so the top-level docs/ reads as current reference only. Add a banner to each: "historical working document, sanitized; not maintained."
  • Remove dangling agent-memory references in tracked docs (docs/bulletproof-containers.md, docs/RETICULUM-TRANSPORT-PROGRESS.md, docs/registry-manifest-design.md, docs/bitcoin-multi-version-design.md progress block).
  • De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status field; e.g. docs/APP-PACKAGING-MIGRATION-PLAN.md → public app-platform design doc).
  • Extract North-Star narrative from PRODUCTION-MASTER-PLAN into docs/ROADMAP.md; extract the "run the gate ON the node" philosophy from docs/multinode-testing-plan.md into tests/lifecycle/TESTING.md.
  • Add docs/README.md index (bitcoin/bitcoin doc/ style): Getting started / Architecture / App development / Operations / Design docs (ADRs) / History.
  • README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at CONTRIBUTING.md only.

Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit)

A. Zero-risk deletions & metadata (S each, own commits)

  • A1 Delete orphan non-compiling StartOS crates: core/models, core/helpers, core/js-engine (incl. 2 committed JS_SNAPSHOT.*.bin), core/container-init (~4,100 LOC, zero references). Verify: cargo build --workspace && cargo test --all-features.
  • A2 Delete unreferenced Vue components: neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue. Verify: npm run type-check && npm run build.
  • A3 Fix .gitignore lockfile lines (7: Cargo.lock, 15: package-lock.json) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles.
  • A4 LICENSE + Cargo license fields (see 4a). Verify with cargo metadata.
  • A5 core/rust-toolchain.toml pinning 1.95.0; align .github/workflows/ci.yml (remove explicit toolchain: stable input so the file wins). Upgrades become deliberate PRs.
  • A6 core/rustfmt.toml codifying defaults only (edition = "2021" + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify cargo fmt --all -- --check yields no diff.

B. CI guards (zero runtime risk)

  • B1 Enable vitest in CI: run cd neode-ui && npm run test locally; fix trivial failures, .skip+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration).
  • B2 Raw podman/systemctl ratchet, not migration: the 132 raw Command::new("podman"/"systemctl") sites use subcommands the core/container/src/podman_client.rs wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical install.rs, and the prod path intentionally uses Quadlet+systemctl. Add scripts/ci/raw-podman-ratchet.sh (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design.

C. Clippy suppression trim (core/archipelago/src/main.rs:8-18, per-lint commits)

  • Remove cheaply: assertions_on_constants, drop_non_drop, wildcard_in_or_patterns, doc_lazy_continuation, enum_variant_names (targeted allows on serde enums — never rename wire variants).
  • Own careful commit: unused_io_amount — a correctness lint; fix sites with read_exact/write_all or documented targeted allows (mesh/serial.rs:456,496 has raw partial reads; serial framing may be intentional). Full test suite + gate after.
  • Keep crate-wide with justifying comment: too_many_arguments, type_complexity; attempt ptr_arg (&Vec<T>→&[T], mechanical) if time allows — first to cut.
  • Verify each: cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features.

D. dead_code lift — Tiers 1–2 pre-launch, Tier 3 → commented allows + issues

Per-module procedure (one file per commit): remove #![allow(dead_code)] → cargo check --all-targets --all-features → triage each warning: (a) genuinely dead → delete; (b) future-feature/protocol-mandated → targeted #[allow(dead_code)] // TODO(#NNN): …; (c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) → clippy -D warnings + tests → commit.

  • Tier 1 (small/leaf, S each): swarm/seed_advert.rs, transport/{mesh_transport,lan,chunking,delta}.rs, mesh/{crypto,alerts,types,outbox}.rs, streaming/mod.rs, wallet/mod.rs.
  • Tier 2 (M each): fips/{mod,iface,dial}.rs (41 external refs → little residual deadness), mesh/{x3dh,ratchet,steganography,message_types}.rs — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why).
  • Tier 3 (defer, riskiest): mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs, transport/mod.rs — change each blanket allow to #![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN.
  • Optional S/M win: move prod_orchestrator.rs's 5,034-line #[cfg(test)] module to a sibling file via #[path] (pure move, halves the 6,291-line file).

E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated)

Legacy installers for immich/btcpay/mempool/indeedhub (core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging INSTALL ORCH SKIP (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920).

  1. Run the full gate on the node; grep install logs for INSTALL ORCH SKIP.
  2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + adopt_stack_if_exists; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue.
  3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers".

Explicitly deferred → public tracked issues at launch

PodmanClient API extension + call-site migration; god-module splits (install.rs, update.rs, mesh/mod.rs); Playwright in CI; Tier-3 dead_code triage; password123 default hardening.

Phase 6 — Fresh-history publish

  1. Freeze: all phases merged on internal main, gate green, catalog re-signed.
  2. Build the public tree: git archive-style export of HEAD (never copy .git/ — it holds credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"), optionally preserving CHANGELOG.md as the human-readable history.
  3. Pre-publish gate on the export: scripts/audit-secrets.sh (fixed version) clean; grep-zero for sk-ant-, rotated-password strings, 146.59.87.168, tailnet 100. IPs, 192.168.1., internal hostnames; du -sh .git sanity (< ~100 MB); fresh git clone + cd core && cargo build
    • cd neode-ui && npm ci && npm run build on a clean machine/container; one app image pull from the public domain.
  4. Publish to GitHub; enable issue templates (already present in .github/); file the deferred-work issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points.
  5. Internal repo remains the private full-history remote; decide sync direction post-launch (recommend: public repo becomes canonical, private keeps only ops/infra notes).

Verification (end-to-end)

  • tests/lifecycle/run-gate.sh green on the node after Phases 3 + 5E (and after any lifecycle-touching commit).
  • CI green on every phase commit: cargo fmt --check, clippy -D warnings, cargo test --all-features, frontend type-check + build + (new) vitest.
  • Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer.
  • Docs acceptance: a reader following docs/app-developer-guide.md + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra.

Sequencing / cut-line

Order: 0 → 1 → 2 → (3 ∥ 4) → 5 (A→E) → 6. Phases 0–2 are non-negotiable security; Phase 3 is the functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any commit (minimum viable: A1–A6, B1–B2, unused_io_amount fix); Phase 6 last. If the timeline compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds.