docs(ops): fix runbook/hotfix/troubleshooting against reality + redact committed sudo password

- operations-runbook: redact the plaintext sudo password (rotate it — it
  stays in git history); replace the nonexistent test-cross-node.sh /
  test-reboot-survival.sh with the real entry points (lifecycle gate,
  multinode smoke, e2e/post-install scripts)
- hotfix-process: de-hardcode the v1.0.x framing, point releases at
  create-release.sh + primary Gitea, correct the rollback backup paths
  (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin)
- troubleshooting: connectivity probe start9.com -> debian.org
- INSTALL.sh: drop removed endurain app, point at developer-guide.md
  (development-setup.md never existed)
- tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session
  resume block with a resolved note; release-gate item 8 reflects the
  2026-07-08 version decision (1.8.0-alpha)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-08 18:46:51 -04:00
co-authored by Claude Fable 5
parent 9ce8a0df05
commit 1394994e0c
5 changed files with 34 additions and 78 deletions
+9 -58
View File
@@ -50,64 +50,15 @@ separate pass → `docs/multinode-testing-plan.md`.)
`lnd` on .228 — they are synced and healthy; destructive cycles there would
cost hours of resync.
### Session work log
### Session work log (resolved)
| Date | App | Change | State |
|---|---|---|---|
| 2026-06-21 | fedimint-gateway / -clientd | **Generated-secrets system** (Pillar 4+5). New `generated_secrets:` manifest field (`hex16`/`hex32`/`bcrypt`); materialised generically at the `resolve_dynamic_env` chokepoint — atomic `0600`, rootless-owned, idempotent, and **self-healing** (recreates a wrongly `root:root`-owned secret via the service-owned dir, no chown/privilege). Removed per-app `ensure_fmcd_password` (30 LoC). Fixes gateway never starting (`resolving secret_env` → missing/unreadable `fedimint-gateway-hash`). | ◐ code complete, `cargo check` + 3 unit tests green; **not yet deployed/validated on .228** |
| 2026-06-21 | fedimint-gateway | Icon placeholder | ○ investigating: marketplace catalog has title+icon (fedimint.png, shared); `BUNDLED_APPS` frontend list omits fedimint → installed view falls back to 📦 |
### ⏯ RESUME POINT (2026-06-21, mid-session)
**Done (working tree, NOT git-committed):**
- Generated-secrets system — all files below written, `cargo check` clean, 3 unit tests green.
- Manifests declare `generated_secrets` (fmcd-password hex16; fedimint-gateway-hash bcrypt).
- Tracker refreshed with 5 pillars + this log.
**In flight:**
- Local release build RUNNING (`cd core && cargo build --release -p archipelago`,
log `/tmp/archy-local-build.log`, output `core/target/release/archipelago`).
⚠️ **.228 has NO cargo and NO rsync** — build LOCALLY on .116, ship binary + files
via **tar-over-ssh** (`tar -cf - … | ssh … 'tar -xf -'`).
**Next steps (in order):**
1. Wait for local build → `Finished`. scp/tar `core/target/release/archipelago` → .228.
2. Ship updated manifests to **`/opt/archipelago/apps/fedimint-{gateway,clientd}/`** (canonical runtime dir; cwd-relative `apps` doesn't resolve — WorkingDirectory is empty).
3. **Binary swap is SAFE for protected backends:** `archipelago.service` is
`KillMode=control-group` BUT bitcoin-knots/electrumx/lnd conmons live under
`user.slice/.../libpod-*.scope`, NOT the service cgroup. Only fedimint-clientd +
immich conmons are in-cgroup (non-protected, reconciled back). `systemctl stop
archipelago``cp` binary → `start`.
4. Validate: install fedimint-gateway → assert `fedimint-gateway-hash` (0600,
archipelago-owned) + `.pw` generated → container starts healthy.
5. Run `tests/lifecycle/run-gate.sh` for the gateway (do NOT touch knots/electrumx/lnd).
6. Frontend fixes (separate from binary): see icon/rename below; rebuild neode-ui,
ship `dist + catalog.json + assets` to `/opt/archipelago/web-ui` (chown 1000:1000).
**Icon / naming (frontend, user-confirmed):**
- Gateway icon = **reuse fedimint.png** (user choice). Static catalogs already map all 3
→ fedimint.png; deployed `/catalog.json` on .228 also correct; `/api/app-catalog`
(decoupled, dict form) returns no fedimint → frontend falls through to `/catalog.json`.
Placeholder is therefore a **stale deployed bundle** and/or the **hardcoded fallback gap**:
`getCuratedAppList()` in `neode-ui/src/views/discover/curatedApps.ts` omits
fedimint-gateway + fedimint-clientd entirely — add both (icon fedimint.png).
- Base **`fedimint` → display "Fedimint Guardian"** (user ask). Edit name/title in:
`apps/fedimint/manifest.yml`, `app-catalog/catalog.json`,
`neode-ui/public/catalog.json`, `web/dist/neode-ui/catalog.json`,
`curatedApps.ts:101`. (`INSTALLED_ALIASES.fedimint = ['fedimint-gateway']` in curatedApps.ts.)
**.228 access:** `sshpass -p archipelago ssh archipelago@192.168.1.228`; UI/RPC pw
`password123` (https). Binary `/usr/local/bin/archipelago` (v1.7.99-alpha).
### Generated-secrets — files touched
- `core/container/src/manifest.rs``GeneratedSecret` + `SecretGenKind` types, `ContainerConfig.generated_secrets`, validation (bare-filename, unique target files).
- `core/container/src/lib.rs` — re-export the new types.
- `core/archipelago/src/container/secrets.rs`**new** generator module (atomic write, idempotent, self-heal) + 3 unit tests.
- `core/archipelago/src/container/mod.rs` — register module.
- `core/archipelago/src/container/prod_orchestrator.rs` — call `ensure_generated_secrets` in `resolve_dynamic_env`; drop fmcd special-case.
- `core/archipelago/src/wallet/fedimint_client.rs` — delete orphaned `ensure_fmcd_password` (reader keeps `FMCD_PASSWORD_SECRET`).
- `apps/fedimint-clientd/manifest.yml`, `apps/fedimint-gateway/manifest.yml` — declare `generated_secrets`.
The 2026-06-21 mid-session resume block that lived here (generated-secrets
rollout for fedimint-gateway/-clientd, icon/naming fixes) is **done and
shipped**: the generated-secrets system is a platform primitive
(`container.generated_secrets`, see `docs/app-manifest-spec.md`), the
manifests declare it, and the single-node gate went green on .228 on
2026-06-23. Day-to-day open items live in `docs/UNIFIED-TASK-TRACKER.md`
don't add session logs here.
---
@@ -302,7 +253,7 @@ priority-ordered list of what's still open across these):
7. ☑ Layman-readable changelog — `CHANGELOG.md` backfilled through v1.8.00-alpha
(per `feedback_changelog_layman.md`)
8. ☐ Tag pushed to origin + gitea-local + gitea-vps2 (per `feedback_ship_ritual.md`) —
blocked on the version-naming decision, see unified tracker Tier 3
version decided 2026-07-08 (`1.8.0-alpha`); tag once the pre-tag items above close
## How to update this document