docs(ops): fix runbook/hotfix/troubleshooting against reality + redact committed sudo password

- operations-runbook: redact the plaintext sudo password (rotate it — it
  stays in git history); replace the nonexistent test-cross-node.sh /
  test-reboot-survival.sh with the real entry points (lifecycle gate,
  multinode smoke, e2e/post-install scripts)
- hotfix-process: de-hardcode the v1.0.x framing, point releases at
  create-release.sh + primary Gitea, correct the rollback backup paths
  (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin)
- troubleshooting: connectivity probe start9.com -> debian.org
- INSTALL.sh: drop removed endurain app, point at developer-guide.md
  (development-setup.md never existed)
- tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session
  resume block with a resolved note; release-gate item 8 reflects the
  2026-07-08 version decision (1.8.0-alpha)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-07-08 18:46:51 -04:00
co-authored by Claude Fable 5
parent 9ce8a0df05
commit 1394994e0c
5 changed files with 34 additions and 78 deletions
+12 -9
View File
@@ -1,6 +1,6 @@
# Hotfix Process
For critical bugs discovered after v1.0.0 release.
For critical bugs discovered after a tagged release.
## Severity Classification
@@ -16,10 +16,10 @@ For critical bugs discovered after v1.0.0 release.
### 1. Triage
- Reproduce the issue on dev server (192.168.1.228)
- Classify severity (P0-P3)
- P0/P1: proceed immediately. P2/P3: add to v1.1 roadmap.
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
### 2. Fix
- Create branch: `hotfix/v1.0.1-description`
- Create branch: `hotfix/vX.Y.Z-description`
- Fix the issue with minimal code changes
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
@@ -27,9 +27,10 @@ For critical bugs discovered after v1.0.0 release.
### 3. Release
- Merge hotfix branch to `main`
- Tag: `v1.0.1` (increment patch version)
- Update release manifest for OTA updates (`releases/manifest.json`)
- Push to both Gitea mirrors so nodes can pull via `self-update.sh`
- Tag: `vX.Y.Z` (increment patch version)
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
`releases/manifest.json` and signs it)
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
### 4. Communicate
- Update RELEASE-NOTES with hotfix details
@@ -44,6 +45,8 @@ For critical bugs discovered after v1.0.0 release.
## Rollback
If a hotfix causes regressions:
1. OTA system supports rollback to previous version (`scripts/self-update.sh --rollback`)
2. Point `releases/manifest.json` back at the last-known-good version and push to mirrors
3. Backend binary backup at `/usr/local/bin/archipelago.bak`
1. The updater self-verifies after applying (health check on restart) and rolls the
binary back automatically if the new one fails to come up
2. Point `releases/manifest.json` back at the last-known-good version and push
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
+10 -8
View File
@@ -5,7 +5,7 @@ Quick reference for common operational tasks on Archipelago nodes.
**Primary node**: `192.168.1.228` (Arch 1)
**Secondary node**: `192.168.1.198` (Arch 2)
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
**Sudo**: `echo 'EwPDR8q45l0Upx@' | sudo -S {command}`
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
---
@@ -350,15 +350,17 @@ If a node responds to ping but SSH/HTTP are down:
podman image prune -a
```
## 17. Run Cross-Node Tests
## 17. Run Tests
```bash
# Full test suite (all features, 10 iterations)
./scripts/test-cross-node.sh --iterations 10
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
# Skip reboot tests
./scripts/test-cross-node.sh --iterations 10 --skip-reboot
# Cross-node suites (federation/mesh):
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
# Reboot survival test (single node)
./scripts/test-reboot-survival.sh --node 192.168.1.228 --iterations 3
# E2E / post-install:
./scripts/run-e2e-tests.sh
./scripts/run-post-install-tests.sh
```
+1 -1
View File
@@ -236,7 +236,7 @@ sudo journalctl -u archipelago -f
**Diagnosis**:
```bash
# Check internet connectivity
curl -s https://start9.com > /dev/null && echo "Internet OK" || echo "No internet"
curl -s https://debian.org > /dev/null && echo "Internet OK" || echo "No internet"
# Check backend logs
sudo journalctl -u archipelago --since "15 minutes ago" | grep -i update