docs(ops): fix runbook/hotfix/troubleshooting against reality + redact committed sudo password
- operations-runbook: redact the plaintext sudo password (rotate it — it stays in git history); replace the nonexistent test-cross-node.sh / test-reboot-survival.sh with the real entry points (lifecycle gate, multinode smoke, e2e/post-install scripts) - hotfix-process: de-hardcode the v1.0.x framing, point releases at create-release.sh + primary Gitea, correct the rollback backup paths (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin) - troubleshooting: connectivity probe start9.com -> debian.org - INSTALL.sh: drop removed endurain app, point at developer-guide.md (development-setup.md never existed) - tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session resume block with a resolved note; release-gate item 8 reflects the 2026-07-08 version decision (1.8.0-alpha) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
9ce8a0df05
commit
1394994e0c
+12
-9
@@ -1,6 +1,6 @@
|
||||
# Hotfix Process
|
||||
|
||||
For critical bugs discovered after v1.0.0 release.
|
||||
For critical bugs discovered after a tagged release.
|
||||
|
||||
## Severity Classification
|
||||
|
||||
@@ -16,10 +16,10 @@ For critical bugs discovered after v1.0.0 release.
|
||||
### 1. Triage
|
||||
- Reproduce the issue on dev server (192.168.1.228)
|
||||
- Classify severity (P0-P3)
|
||||
- P0/P1: proceed immediately. P2/P3: add to v1.1 roadmap.
|
||||
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
|
||||
|
||||
### 2. Fix
|
||||
- Create branch: `hotfix/v1.0.1-description`
|
||||
- Create branch: `hotfix/vX.Y.Z-description`
|
||||
- Fix the issue with minimal code changes
|
||||
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
|
||||
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
|
||||
@@ -27,9 +27,10 @@ For critical bugs discovered after v1.0.0 release.
|
||||
|
||||
### 3. Release
|
||||
- Merge hotfix branch to `main`
|
||||
- Tag: `v1.0.1` (increment patch version)
|
||||
- Update release manifest for OTA updates (`releases/manifest.json`)
|
||||
- Push to both Gitea mirrors so nodes can pull via `self-update.sh`
|
||||
- Tag: `vX.Y.Z` (increment patch version)
|
||||
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
|
||||
`releases/manifest.json` and signs it)
|
||||
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
|
||||
|
||||
### 4. Communicate
|
||||
- Update RELEASE-NOTES with hotfix details
|
||||
@@ -44,6 +45,8 @@ For critical bugs discovered after v1.0.0 release.
|
||||
## Rollback
|
||||
|
||||
If a hotfix causes regressions:
|
||||
1. OTA system supports rollback to previous version (`scripts/self-update.sh --rollback`)
|
||||
2. Point `releases/manifest.json` back at the last-known-good version and push to mirrors
|
||||
3. Backend binary backup at `/usr/local/bin/archipelago.bak`
|
||||
1. The updater self-verifies after applying (health check on restart) and rolls the
|
||||
binary back automatically if the new one fails to come up
|
||||
2. Point `releases/manifest.json` back at the last-known-good version and push
|
||||
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
|
||||
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
|
||||
|
||||
@@ -5,7 +5,7 @@ Quick reference for common operational tasks on Archipelago nodes.
|
||||
**Primary node**: `192.168.1.228` (Arch 1)
|
||||
**Secondary node**: `192.168.1.198` (Arch 2)
|
||||
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
|
||||
**Sudo**: `echo 'EwPDR8q45l0Upx@' | sudo -S {command}`
|
||||
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
|
||||
|
||||
---
|
||||
|
||||
@@ -350,15 +350,17 @@ If a node responds to ping but SSH/HTTP are down:
|
||||
podman image prune -a
|
||||
```
|
||||
|
||||
## 17. Run Cross-Node Tests
|
||||
## 17. Run Tests
|
||||
|
||||
```bash
|
||||
# Full test suite (all features, 10 iterations)
|
||||
./scripts/test-cross-node.sh --iterations 10
|
||||
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
|
||||
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
|
||||
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
|
||||
|
||||
# Skip reboot tests
|
||||
./scripts/test-cross-node.sh --iterations 10 --skip-reboot
|
||||
# Cross-node suites (federation/mesh):
|
||||
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
|
||||
|
||||
# Reboot survival test (single node)
|
||||
./scripts/test-reboot-survival.sh --node 192.168.1.228 --iterations 3
|
||||
# E2E / post-install:
|
||||
./scripts/run-e2e-tests.sh
|
||||
./scripts/run-post-install-tests.sh
|
||||
```
|
||||
|
||||
@@ -236,7 +236,7 @@ sudo journalctl -u archipelago -f
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check internet connectivity
|
||||
curl -s https://start9.com > /dev/null && echo "Internet OK" || echo "No internet"
|
||||
curl -s https://debian.org > /dev/null && echo "Internet OK" || echo "No internet"
|
||||
|
||||
# Check backend logs
|
||||
sudo journalctl -u archipelago --since "15 minutes ago" | grep -i update
|
||||
|
||||
Reference in New Issue
Block a user