From cf240df4b6a54fc04e945a582bdfc6948ce58cbb Mon Sep 17 00:00:00 2001 From: ssmithx Date: Thu, 3 Sep 2026 08:20:52 +0000 Subject: [PATCH] =?UTF-8?q?fix(cuprate):=20enable=20fast=5Fsync=20and=20ra?= =?UTF-8?q?ise=20DB=20cache=20=E2=80=94=20sustained=2045%=20CPU?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The default manifest baked in the exact broken config found on an affected fleet node: no fast_sync (defaults false, forcing full ring-sig/ RandomX verification on every block) and target_max_memory capped at ~2.8GiB, which starved cuprated's DB cache into constant eviction/flush (595GB/24h of block I/O on a node just appending ~2MB blocks every 2 minutes). A reference node with fast_sync = true and an 8GiB cache ran at 2.8% CPU at the same chain height and block rate. Set fast_sync = true and target_max_memory = 8GiB to match the healthy reference config, and raise resources.memory_limit from 4Gi to 10Gi so the container still has headroom above the larger cache. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01RR7jRaicvqsJaqQQ92jpPQ --- apps/cuprate/manifest.yml | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/apps/cuprate/manifest.yml b/apps/cuprate/manifest.yml index 3ddbd0a6..823c44f3 100644 --- a/apps/cuprate/manifest.yml +++ b/apps/cuprate/manifest.yml @@ -45,7 +45,12 @@ app: resources: cpu_limit: 0 - memory_limit: 4Gi + # Raised from 4Gi alongside target_max_memory below (see files[] comment) + # — 2026-09-03 incident: a 4Gi/3GB-cache config starved + # cuprated's DB cache into constant eviction/flush, driving 45% sustained + # CPU and ~595GB/24h of block I/O on a fully-synced node. 10Gi leaves + # headroom above the 8GiB cache for the process itself. + memory_limit: 10Gi disk_limit: 300Gi security: @@ -103,11 +108,23 @@ app: # Settings that need to differ from cuprated's own documented defaults # (verified against `cuprated --generate-config` and `--dry-run` locally, # 2026-08-21): + # - fast_sync: cuprated's own default is false, which performs full + # cryptographic verification (ring signatures + RandomX PoW) on every + # incoming block instead of trusting checkpointed history. Root-caused + # 2026-09-03 as the dominant cause of a sustained 45% CPU node, + # vs. 2.8% on a reference node with fast_sync = true — same chain height, same + # block rate. Set explicitly rather than relying on the binary + # default so fresh deploys don't silently regress into full-verify. # - target_max_memory: cuprated's own default auto-detects total *host* # RAM via sysinfo, which inside a memory-limited container would let # it size caches far past what resources.memory_limit above actually # grants — same class of problem bitcoin-knots' -dbcache sizing - # comment addresses. Set explicitly, comfortably under the 4Gi limit. + # comment addresses. Set explicitly, comfortably under the 10Gi limit. + # Previously 3000000000 (~2.8GiB); that starved the DB cache and + # forced constant eviction/flush (595GB/24h block I/O on a node just + # appending ~2MB blocks every 2 minutes) — raised to 8GiB, matching + # the healthy reference node, and + # resources.memory_limit above raised in step to keep headroom above it. # - rpc.restricted.enable: cuprated ships this off by default; flip on # so the auth:none host port above actually serves something instead # of refusing every connection. port stays at its documented default @@ -128,21 +145,21 @@ app: # - tracing.stdout.level / tracing.file.{level,max_log_files}: an # operator reading Cuprated.toml on disk should be able to see and # tune the log level directly instead of the file silently omitting - # the whole [tracing] table (verified live on amishparadise + # the whole [tracing] table (verified live on the affected node # 2026-09-01: the deployed file had no [tracing] section at all, and # the level was only discoverable by running `cuprated # --generate-config` and diffing). file.level is set to "info", NOT # cuprated's own raw default of "debug" — matches the reference dev - # config this app was built and tested against - # (ssmithx@archy-dev-pa:/home/ssmithx/cuprate/Cuprated.toml, - # verified 2026-09-01), which deliberately runs file logging quieter + # config this app was built and tested against (verified 2026-09-01), + # which deliberately runs file logging quieter # than the binary default. max_log_files similarly follows that # reference (14, not the binary default of 7). files: - path: /var/lib/archipelago/cuprate/Cuprated.toml content: | network = "Mainnet" - target_max_memory = 3000000000 + fast_sync = true + target_max_memory = 8589934592 [rpc.restricted] enable = true