diff --git a/.planning/phases/09-botfights-platform-upgrade/09-04-SUMMARY.md b/.planning/phases/09-botfights-platform-upgrade/09-04-SUMMARY.md new file mode 100644 index 00000000..34bcf5e2 --- /dev/null +++ b/.planning/phases/09-botfights-platform-upgrade/09-04-SUMMARY.md @@ -0,0 +1,177 @@ +--- +phase: 09-botfights-platform-upgrade +plan: 04 +subsystem: infra +tags: [docker, vps2, botfights, sqlite, jwt, gitea-registry] + +requires: [] +provides: + - "Running canonical BotFights arena container on VPS2 (146.59.87.168:9100)" + - "docker-compose.arena.yml — reproducible arena compose definition, registry image only" + - "docs/arena-deployment.md — full runbook (secret handling, seed method, TLS upgrade path, teardown)" + - "Canonical arena URL for downstream plans: http://146.59.87.168:9100" +affects: [09-05, 09-06, 09-07] + +tech-stack: + added: [] + patterns: + - "Read-only sqlite VACUUM INTO export for live-DB seeding without touching the source" + - "docker-compose healthcheck override for images built before the Dockerfile HEALTHCHECK existed" + +key-files: + created: + - /home/archipelago/Projects/botfight/docker-compose.arena.yml + - /home/archipelago/Projects/botfight/docs/arena-deployment.md + modified: [] + +key-decisions: + - "Canonical arena URL is http://146.59.87.168:9100 — plain HTTP, no DNS/TLS (user decision 2026-07-30, resolved before this plan ran)" + - "Data seed: full copy of archi-dev-box's botfights.db (115 bots / 102,440 fights) — user decision, option-c" + - "Image ref uses localhost:3000/lfg2025/botfights (not the public 146.59.87.168:3000) — Docker trusts loopback registries without insecure-registry config, verified working on VPS2" + - "Added an explicit docker-compose healthcheck override since the running 1.1.0 image tag predates the Dockerfile's HEALTHCHECK directive" + +requirements-completed: [BOT-03] + +coverage: + - id: D1 + description: "Canonical BotFights arena container running and healthy on VPS2 at http://146.59.87.168:9100" + requirement: "BOT-03" + verification: + - kind: other + ref: "ssh debian@146.59.87.168 'curl -fsS http://127.0.0.1:9100/api/health' — {\"status\":\"ok\",\"name\":\"botfights\"}" + status: pass + - kind: other + ref: "curl -fsS --max-time 10 http://146.59.87.168:9100/api/health (off-host, from archi-dev-box)" + status: pass + - kind: other + ref: "docker ps --filter name=botfights-arena --format '{{.Status}}' -> Up ... (healthy)" + status: pass + human_judgment: false + - id: D2 + description: "docker-compose.arena.yml committed — registry image only, no build, no payment env, no TRUSTED_PROXY" + requirement: "BOT-03" + verification: + - kind: other + ref: "grep -c 'BOTFIGHTS_WALLET_ENCRYPTION_KEY|BOTFIGHTS_NWC_URL|BOTFIGHTS_CASHU_MINT_URL' docker-compose.arena.yml — only commented lines" + status: pass + - kind: other + ref: "docker inspect botfights-arena env — no ARENA_UPSTREAM_URL, no TRUSTED_PROXY entries" + status: pass + human_judgment: false + - id: D3 + description: "docs/arena-deployment.md runbook committed with host/port/secret-handling/seed-decision/NPM-TLS-upgrade-path/teardown, including the arena-as-relay architecture section" + requirement: "BOT-03" + verification: + - kind: other + ref: "test -f docs/arena-deployment.md; grep -c 146.59.87.168 docs/arena-deployment.md -> 15 matches" + status: pass + human_judgment: false + - id: D4 + description: "Full-copy data seed live on the arena (115 bots via read-only export from archi-dev-box); source database provably untouched" + requirement: "BOT-03" + verification: + - kind: other + ref: "curl http://146.59.87.168:9100/api/bots -> 100 rows (default filter excludes botType=classic); GET /api/bots?type=classic -> 15 rows; 100+15=115 matches source" + status: pass + - kind: other + ref: "stat mtime/size of /var/lib/archipelago/botfights/botfights.db before and after export — 1782916151 / 367144960 bytes, byte-identical" + status: pass + human_judgment: false + +duration: ~65min +completed: 2026-07-31 +status: complete +--- + +# Phase 9 Plan 4: Canonical BotFights Arena on VPS2 Summary + +**One canonical BotFights arena deployed and verified on VPS2 at `http://146.59.87.168:9100` (plain HTTP, no DNS/TLS by user decision), seeded with a full read-only copy of archi-dev-box's 351 MB production database (115 bots, 102,440 fights) — the shared endpoint every node's BotFights instance will proxy to.** + +## Performance + +- **Duration:** ~65 min +- **Completed:** 2026-07-31T01:25Z +- **Tasks:** 2 of 3 executed (Task 1 was a decision checkpoint already resolved before this run — see plan frontmatter/resume-signal) +- **Files modified:** 2 (both new) + +## Accomplishments + +- Deployed `botfights-arena` container on VPS2 running the published `localhost:3000/lfg2025/botfights:1.1.0` registry image (no local build), bound to `9100:9100`, with a host-generated `JWT_SECRET` (0600 `.env`, never committed), no `TRUSTED_PROXY` (direct exposure, rate-limits on real socket peer IP), and no payment env vars. +- Exported archi-dev-box's live `botfights.db` (351 MB, 115 bots, 102,440 fights) via a read-only SQLite URI connection (`mode=ro`) + `VACUUM INTO`, confirmed the source file's mtime/size were byte-identical before and after (`1782916151` / `367144960`), and seeded the arena's volume with the full copy. +- Verified the arena healthy both on-host (`127.0.0.1:9100/api/health`) and off-host (`146.59.87.168:9100/api/health` from archi-dev-box) — internet-reachable with no firewall changes needed (host firewall already permitted the port; `ufw` isn't even installed on this host). +- Committed `docker-compose.arena.yml` and `docs/arena-deployment.md` to the `botfight` repo (staged by explicit path — the working tree is shared with another concurrently-active agent) and pushed to `origin/main`. + +## Task Commits + +1. **Task 2/3 combined (deploy + prove internet-reachable)** — botfight repo `e4b82fd`: `feat(09-04): deploy canonical BotFights arena on VPS2` + +**Plan metadata:** this SUMMARY + STATE/ROADMAP updates, committed to `archy` (`git push gitea-ai main`) + +_Note: Task 1 (the hostname/data-seed decision) was already resolved by the user before this execution run began, per the plan's `resolved="2026-07-30"` frontmatter — no checkpoint was re-raised._ + +## Files Created/Modified + +- `/home/archipelago/Projects/botfight/docker-compose.arena.yml` — canonical-arena compose definition (registry image only, `TRUSTED_PROXY` deliberately omitted, payment env vars commented out, explicit `healthcheck:` override) +- `/home/archipelago/Projects/botfight/docs/arena-deployment.md` — full runbook: arena-as-relay architecture explanation, current instance table, plain-HTTP rationale + later TLS upgrade path, secret handling/rotation, data-seed export method, verification commands, image-roll procedure, teardown + +## Decisions Made + +- **Image reference uses `localhost:3000/...`, not the public IP.** VPS2's `daemon.json` only lists `146.59.87.168:3000` as an insecure registry, but Docker auto-trusts loopback (`127.0.0.0/8`) registries without any config — verified by a direct `docker pull localhost:3000/lfg2025/botfights:1.1.0` test on the host before committing to this pattern in the compose file. This matches the plan's own suggestion ("pull from localhost:3000... on the VPS itself"). +- **Added an explicit `healthcheck:` block to the compose file** (not in the original plan text) because the currently-published `1.1.0` image tag predates the Dockerfile's `HEALTHCHECK` directive — without it, `docker ps` showed no health status at all, and the plan's acceptance criteria explicitly requires "Up and healthy". This is forward-compatible: plan 09-05's 1.2.0 image will have its own baked-in healthcheck, and the override just re-states the same check. +- **Ownership of the seeded DB file uses the container's actual runtime uid (999, the Dockerfile's `useradd --system` botfights user)**, not the host's `debian` (uid 1000) or my own local uid — discovered by inspecting the volume mountpoint's existing file ownership (created by the container's first fresh-start run) before overwriting it, rather than guessing. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 2 - missing critical functionality] Added explicit `healthcheck:` override to `docker-compose.arena.yml`** +- **Found during:** Task 2, verifying `docker ps` shows "Up ... healthy" +- **Issue:** The published `1.1.0` image tag was built before the Dockerfile's `HEALTHCHECK` directive existed, so `docker inspect botfights-arena --format '{{json .State.Health}}'` returned `null` and `docker ps` showed no health suffix at all — the acceptance criteria explicitly requires "shows Up and healthy". +- **Fix:** Added a `healthcheck:` block to the compose service mirroring the Dockerfile's own check (`node -e "fetch('http://localhost:9100/api/health')..."`, 30s interval, 5s timeout, 10s start period, 3 retries), redeployed with `docker compose up -d` (recreated the container; the named volume and its seeded data were untouched). +- **Files modified:** `docker-compose.arena.yml` +- **Verification:** `docker ps --filter name=botfights-arena --format '{{.Status}}'` now shows `Up ... (healthy)`; data still present post-recreate (`GET /api/bots` still returned the same 100-row count). +- **Committed in:** `e4b82fd` (botfight repo) + +**2. [Rule 3 - blocking issue, self-corrected] Accidentally printed the JWT_SECRET value to the transcript, rotated immediately** +- **Found during:** Task 2, a verification step (`docker inspect botfights-arena --format '{{json .Config.Env}}' | python3 -m json.tool`) printed the FULL container env list, including the literal `JWT_SECRET` value — directly violating the explicit instruction "Never print the JWT_SECRET value anywhere." +- **Issue:** The env-inspection command was not scoped to exclude/mask the secret before I ran it. +- **Fix:** Immediately generated a fresh `JWT_SECRET` on the VPS2 host (`openssl rand -hex 32`, 0600 `.env`), ran `docker compose down && docker compose up -d` to invalidate the old secret and pick up the new one, and switched all subsequent verification to `grep -c` presence/absence checks that never print values (e.g. counting `TRUSTED_PROXY`/`ARENA_UPSTREAM_URL` matches, never dumping the full env). This happened before any external client had authenticated against the arena (it was seconds after first boot, with no bots/users registered against JWT auth yet), so no live session was compromised by the rotation. +- **Files modified:** none in git — host-only `.env` rotation +- **Verification:** re-ran health + env-safety checks post-rotation; all passed with masked output only. +- **Committed in:** N/A (secret rotation is not a git-tracked change; documented in `docs/arena-deployment.md`'s Secret Handling section, which now explicitly names this incident and the rotation procedure) + +**3. [Rule 3 - blocking issue] VPS2 root filesystem was 100% full, blocking the `git push origin main`** +- **Found during:** the plan's final step — `git push origin main` (botfight repo, hosted on the same VPS2 host at `source.archipelago-foundation.org`) failed with `error: remote unpack failed: unable to create temporary object directory`. +- **Issue:** `df -h /` on VPS2 showed `74G 72G 0 100% /` — Gitea could not create the temp directory it needs to unpack an incoming push. Root cause investigation found `/opt/gitea/data/gitea/packages` (the Gitea container-registry package storage — the same registry this task pulls the `botfights` image from) at **43 GB**, the overwhelming majority of the used space. +- **Fix (minimal/reversible only):** Ran `docker builder prune -f` and `docker image prune -f` (both explicitly safe — build cache and untagged/dangling images only, reclaimed 0B in this case since there was nothing dangling) and removed the 362 MB scratch seed-DB temp file left on the VPS from Task 2's data-seed step (`/tmp/botfights-seed.db`, already consumed). Between that cleanup and earlier container-recreate operations freeing old writable layers, available space rose to 1.5 GB, enough for the push to succeed. +- **Files modified:** none — no compose/manifest changes, host cleanup only. +- **Verification:** `git push origin main` succeeded (`fb35075..e4b82fd main -> main`) after the cleanup. +- **NOT auto-fixed (deferred, flagged for the user):** The 43 GB `/opt/gitea/data/gitea/packages` directory itself was left untouched. Pruning old/duplicate container image tags in a shared production registry is a destructive, judgment-requiring operation (Rule 4 territory — it affects every app on the VPS2 fleet that pulls from this registry, not just `botfights`), well outside this task's file scope (`docker-compose.arena.yml`, `docs/arena-deployment.md`). **VPS2's root disk is at 99% (1.5 GB free) as of this plan's completion — the next multi-GB push or image pull to/from this host will likely fail the same way.** This needs a deliberate registry-retention decision from the user (e.g. `gitea` admin package-version pruning, or moving package storage to a larger volume), not an automated cleanup. + +--- + +**Total deviations:** 3 auto-fixed (1 Rule 2, 2 Rule 3) +**Impact on plan:** All three were necessary to meet the plan's own acceptance criteria and explicit secret-handling instruction; no scope creep beyond the deployed arena and its runbook. One (#3) surfaces an unresolved infra-capacity risk for the user's attention — see "Next Phase Readiness" below. + +## Issues Encountered + +- No `sqlite3` CLI binary was available locally to do the `.backup`/read-only export the plan suggested as one option — used Python's built-in `sqlite3` module instead (`mode=ro` URI + `VACUUM INTO`), which is functionally equivalent (read-only source handle, self-consistent snapshot including any checkpointed WAL data) and required no new package installs. +- `GET /api/bots` on the seeded arena initially appeared to under-report (100 vs. the expected 115) — investigated and confirmed this is pre-existing, correct API behavior: the endpoint filters out `botType === 'classic'` bots by default (15 of the 115), retrievable via `?type=classic`. Not a data-loss bug from the seed. + +## User Setup Required + +None for this plan's own scope. However, see Deviation #3 above: **VPS2's root disk is at 99% full (1.5 GB free)** because of a 43 GB Gitea package-registry directory. This is a pre-existing condition (not introduced by this plan) that this plan's own `git push` tripped over and had to work around minimally. Recommend a deliberate cleanup/retention decision before the next multi-GB operation on this host (image build+push for plan 09-05's 1.2.0 image will add to registry storage, not reduce it). + +## Next Phase Readiness + +- **Plan 09-05** (image build/push to 1.2.0) can proceed — the registry is reachable and the arena is running, but should budget for the low-disk-space risk above; consider checking `df -h /` on VPS2 before that plan's push step, or the user may want to prune old package versions first. +- **Plan 09-05's** precondition ("`curl -fsS http://146.59.87.168:9100/api/health` returns 200 from off-host") is already satisfied by this plan. +- **Plans 09-06/09-07** should read the canonical arena URL as `http://146.59.87.168:9100` (plain HTTP) from this SUMMARY / `docs/arena-deployment.md` when writing `ARENA_UPSTREAM_URL` into `apps/botfights/manifest.yml` — do NOT use an `https://arena.archipelago-foundation.org` placeholder; that subdomain does not exist yet (see the TLS-upgrade section of the runbook for how to add it later, env-only, no code change). + +--- +*Phase: 09-botfights-platform-upgrade* +*Completed: 2026-07-31* + +## Self-Check: PASSED + +All created files verified present; commit e4b82fd verified in botfight repo git log.