Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 archipelago
|
aee7ecaac1 | ||
|
archipelago
|
e51ceaa250 | ||
|
archipelago
|
7c9559aa57 |
@@ -1,7 +1,7 @@
|
||||
app:
|
||||
id: archy-nbxplorer
|
||||
name: NBXplorer
|
||||
version: 2.6.11
|
||||
version: 2.6.0
|
||||
# Where this app comes from, so scripts/check-upstream-releases.py can
|
||||
# tell us when the pin below has fallen behind. Without it nothing can:
|
||||
# container.image names our mirror, not the project it was mirrored from.
|
||||
@@ -11,7 +11,7 @@ app:
|
||||
description: BTCPay blockchain indexer service.
|
||||
|
||||
container:
|
||||
image: source.archipelago-foundation.org/lfg2025/nbxplorer:2.6.11
|
||||
image: source.archipelago-foundation.org/lfg2025/nbxplorer:2.6.0
|
||||
pull_policy: if-not-present
|
||||
network: archy-net
|
||||
secret_env:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
app:
|
||||
id: homeassistant
|
||||
name: Home Assistant
|
||||
version: 2026.8.3
|
||||
version: 2026.7.3
|
||||
# Where this app comes from, so scripts/check-upstream-releases.py can
|
||||
# tell us when the pin below has fallen behind. Without it nothing can:
|
||||
# container.image names our mirror, not the project it was mirrored from.
|
||||
@@ -11,7 +11,7 @@ app:
|
||||
description: Open source home automation platform. Control and monitor your smart home devices.
|
||||
|
||||
container:
|
||||
image: source.archipelago-foundation.org/lfg2025/home-assistant:2026.8.3
|
||||
image: source.archipelago-foundation.org/lfg2025/home-assistant:2026.8.2
|
||||
pull_policy: if-not-present
|
||||
network: pasta
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
app:
|
||||
id: vaultwarden
|
||||
name: Vaultwarden
|
||||
version: 1.37.2
|
||||
version: 1.30.0
|
||||
# Where this app comes from, so scripts/check-upstream-releases.py can
|
||||
# tell us when the pin below has fallen behind. Without it nothing can:
|
||||
# container.image names our mirror, not the project it was mirrored from.
|
||||
@@ -11,7 +11,7 @@ app:
|
||||
description: Self-hosted password vault with zero-knowledge encryption.
|
||||
|
||||
container:
|
||||
image: source.archipelago-foundation.org/lfg2025/vaultwarden:1.37.2-alpine
|
||||
image: source.archipelago-foundation.org/lfg2025/vaultwarden:1.37.1-alpine
|
||||
pull_policy: if-not-present
|
||||
network: pasta
|
||||
|
||||
|
||||
@@ -54,6 +54,7 @@ step-by-step guides, and some predate the current implementation.
|
||||
- [Dual Ecash](dual-ecash-design.md)
|
||||
- [Hardware Signer](hardware-signer-design.md)
|
||||
- [Manifest Hooks](manifest-hooks-design.md)
|
||||
- [kdump + rasdaemon Troubleshooting](kdump-rasdaemon-design.md) — post-mortem and hardware-error capture on nodes (#144)
|
||||
- [Meshroller Integration](meshroller-integration-design.md)
|
||||
- [Nostr Git Source Hosting](nostr-git-source-hosting.md)
|
||||
- [Nostr Identity Import](nostr-identity-import-plan.md) · [Nostr Signer Login (research)](nostr-signer-login-research.md)
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# kdump + rasdaemon — post-mortem and hardware-error capture (#144)
|
||||
|
||||
Status: DRAFT for review
|
||||
Owner: node image (image-recipe) + lifecycle gate
|
||||
Issue: #144 — "Configure kdump and rasdaemon for troubleshooting"
|
||||
|
||||
## The problem
|
||||
|
||||
When a fleet node hard-locks or a memory stick starts failing, today we get
|
||||
nothing: a frozen kiosk is power-cycled and the evidence is gone; a DIMM
|
||||
throwing correctable ECC errors for weeks is invisible until it starts
|
||||
corrupting things. Two standard kernel mechanisms capture this evidence:
|
||||
|
||||
- **kdump** — reserves a small crash kernel at boot; on a kernel panic (or,
|
||||
configured so, a hang) the running kernel hands the machine over to the
|
||||
crash kernel, which writes a compressed dump of memory to disk and
|
||||
reboots. The node comes back by itself *and* leaves a post-mortem.
|
||||
- **rasdaemon** — a userspace daemon that records hardware error events
|
||||
(correctable/uncorrectable ECC per DIMM, PCIe AER) from EDAC/sysfs into a
|
||||
sqlite database: persistent evidence of degrading hardware with no crash
|
||||
required.
|
||||
|
||||
## Facts the design rests on
|
||||
|
||||
- Installed-disk layout (auto-install.sh): BIOS boot 1MiB · EFI 512MiB ·
|
||||
**root ext4 30GiB, unencrypted** · data (rest, LUKS).
|
||||
- The data partition is LUKS and unlocked late by the node itself — the
|
||||
crash kernel must never be asked to handle key material.
|
||||
- The installed system's kernel command line is written by
|
||||
auto-install.sh:1810 (`GRUB_CMDLINE_LINUX_DEFAULT="quiet splash …"`).
|
||||
- Packages land via `Dockerfile.rootfs` (trixie) with `systemctl enable`
|
||||
in the same RUN block (nginx/tor/avahi pattern).
|
||||
- Kernel cmdline cannot be changed by OTA — it lives in GRUB. Existing
|
||||
nodes need a backfill step (bootstrap) plus a deliberate reboot.
|
||||
|
||||
## Design
|
||||
|
||||
### kdump
|
||||
|
||||
- **Packages:** `kdump-tools kexec-tools` added to Dockerfile.rootfs.
|
||||
- **Command line:** append `crashkernel=256M` to
|
||||
`GRUB_CMDLINE_LINUX_DEFAULT` in auto-install.sh. 256M covers the capture
|
||||
kernel plus makedumpfile on the fleet's 16–64GB amd64 machines (~1–2% of
|
||||
RAM reserved, permanently). The arm image (RPi, config.txt boot) is out
|
||||
of scope for phase 1.
|
||||
- **Dump target:** `local filesystem /var/crash` — on the unencrypted 30GiB
|
||||
root, deliberately *not* the encrypted data partition. No key handling
|
||||
in the crash initramfs, no dependency on the node's own unlock logic.
|
||||
- **Core collector:** `makedumpfile -l --message-level 1 -d 31`
|
||||
(compressed, zero/free pages excluded) — a dump lands at roughly 5–15%
|
||||
of RAM, i.e. ~1–2 GiB on a 16 GiB machine.
|
||||
- **Retention:** keep the **2 newest** dumps only. A small systemd timer
|
||||
(or kdump-tools' `KDUMP_POST_SCRIPT`) prunes older vmcores; a full root
|
||||
partition is already caught by disk_monitor's usage tracking. Two dumps
|
||||
≈ 4 GiB worst case on 30 GiB root — safe.
|
||||
- **When to dump — the deliberate trade-off (decision needed):**
|
||||
- Baseline: dump on real panics (`kernel.panic` path) — no behavioral
|
||||
change to a wedged node.
|
||||
- Recommended for this fleet: also enable hang capture
|
||||
(`kernel.hung_task_panic=1`, hardlockup via NMI watchdog). A kiosk
|
||||
that hard-locks is useless until power-cycled anyway; converting the
|
||||
hang into "dump + automatic reboot" turns every freeze into evidence
|
||||
*and* self-heals the node. Cost: a genuinely-busy-but-alive machine
|
||||
that trips the watchdog reboots — the threshold is kernel-default
|
||||
conservative (40s), so this should be rare.
|
||||
|
||||
### rasdaemon
|
||||
|
||||
- **Packages:** `rasdaemon`; `systemctl enable rasdaemon` in the
|
||||
Dockerfile.rootfs enable block (same pattern as nginx).
|
||||
- **Storage:** its default sqlite DB at
|
||||
`/var/lib/rasdaemon/ras-mc_event.db` on the unencrypted root.
|
||||
- **Human access today:** `ras-mc-ctl --summary` / `--errors` over SSH.
|
||||
No UI in phase 1.
|
||||
|
||||
### Surfacing (phase 2 — separate follow-up, not in this cut)
|
||||
|
||||
A small read-only `system.diagnostics` surface: last-crash timestamp and
|
||||
vmcore sizes from `/var/crash`, plus ECC error totals per DIMM from the
|
||||
rasdaemon DB — shown in Settings → System. Deliberately deferred: capture
|
||||
first, UI once there is something to show and a node in the fleet has
|
||||
actually produced a dump.
|
||||
|
||||
### Existing nodes (phase 1.5 backfill)
|
||||
|
||||
The OTA cannot change the bootloader. Bootstrap (which already delivers
|
||||
fixes to existing nodes) appends `crashkernel=256M` (and the chosen
|
||||
panic/hang params) to `/etc/default/grub` on machines that don't have it,
|
||||
and enables `rasdaemon` via the node's package install path. **Takes
|
||||
effect on the next reboot** — the operator reboots nodes when applying the
|
||||
release; no special ceremony needed beyond that.
|
||||
|
||||
## Testing
|
||||
|
||||
- Image: the new packages appear in the ISO; QEMU boot smoke
|
||||
(build-iso-release.sh stage 5) still green.
|
||||
- Lifecycle gate additions (bats, archi-dev-box first): `kdump-config show`
|
||||
reports a loaded crash kernel reservation; `systemctl is-active
|
||||
rasdaemon`; `/etc/default/grub` carries `crashkernel=`.
|
||||
- Live drill (once, on archi-dev-box, not in the gate): trigger
|
||||
`sysrq c` → vmcore appears in `/var/crash`, node reboots itself,
|
||||
second boot is clean. Keep this manual — it reboots the box.
|
||||
|
||||
## Implementation touchpoints
|
||||
|
||||
1. `image-recipe/build/auto-installer/Dockerfile.rootfs` — packages +
|
||||
`systemctl enable rasdaemon`.
|
||||
2. `image-recipe/build/auto-installer/installer-iso/archipelago/auto-install.sh:1810`
|
||||
— append `crashkernel=256M` (+ hang params if approved) to
|
||||
`GRUB_CMDLINE_LINUX_DEFAULT`.
|
||||
3. `kdump-tools` config: `/etc/default/kdump-tools` (dump target
|
||||
`/var/crash`, core_collector line, `KDUMP_POST_SCRIPT` or timer for
|
||||
retention).
|
||||
4. Bootstrap backfill for existing nodes.
|
||||
5. `tests/lifecycle` — presence assertions (crash kernel reserved,
|
||||
rasdaemon active).
|
||||
|
||||
## Decisions needed before implementation
|
||||
|
||||
1. **Hang capture on or off?** Recommended ON (`hung_task_panic=1` +
|
||||
NMI watchdog): every hard lockup becomes a dump + self-reboot. OFF
|
||||
means dumps only on true panics; wedged nodes still need the button.
|
||||
2. **crashkernel=256M vs 320M** — 256M is the common default for
|
||||
16–64GB machines; 320M if we expect large io-heavy kernels.
|
||||
3. **Backfill now or new-installs-only?** Recommended: ship the backfill
|
||||
with the next release so the whole fleet gains capture on reboot.
|
||||
4. Phase-2 UI surfacing scope — confirm "later" so phase 1 stays small.
|
||||
+123
-13
@@ -505,6 +505,10 @@
|
||||
"network_policy": "bridge",
|
||||
"readonly_root": true
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "gitlab",
|
||||
"repo": "ark-bitcoin/bark"
|
||||
},
|
||||
"version": "0.3.0",
|
||||
"volumes": [
|
||||
{
|
||||
@@ -978,13 +982,13 @@
|
||||
"version": "1.2.11"
|
||||
},
|
||||
"btcpay": {
|
||||
"image": "docker.io/btcpayserver/btcpayserver:2.4.2",
|
||||
"image": "docker.io/btcpayserver/btcpayserver:2.4.3",
|
||||
"images": {
|
||||
"archy-btcpay-db": "source.archipelago-foundation.org/lfg2025/postgres:15.17",
|
||||
"archy-nbxplorer": "source.archipelago-foundation.org/lfg2025/nbxplorer:2.6.0",
|
||||
"btcpay-server": "docker.io/btcpayserver/btcpayserver:2.4.2"
|
||||
"btcpay-server": "docker.io/btcpayserver/btcpayserver:2.4.3"
|
||||
},
|
||||
"version": "2.4.2"
|
||||
"version": "2.4.3"
|
||||
},
|
||||
"btcpay-server": {
|
||||
"manifest": {
|
||||
@@ -1000,7 +1004,7 @@
|
||||
"template": "{{HOST_IP}}:23000"
|
||||
}
|
||||
],
|
||||
"image": "docker.io/btcpayserver/btcpayserver:2.4.2",
|
||||
"image": "docker.io/btcpayserver/btcpayserver:2.4.3",
|
||||
"network": "archy-net",
|
||||
"pull_policy": "if-not-present",
|
||||
"secret_env": [
|
||||
@@ -1097,7 +1101,7 @@
|
||||
"kind": "github",
|
||||
"repo": "btcpayserver/btcpayserver"
|
||||
},
|
||||
"version": "2.4.2",
|
||||
"version": "2.4.3",
|
||||
"volumes": [
|
||||
{
|
||||
"options": [
|
||||
@@ -1110,7 +1114,7 @@
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": "2.4.2"
|
||||
"version": "2.4.3"
|
||||
},
|
||||
"core-lightning": {
|
||||
"manifest": {
|
||||
@@ -1205,6 +1209,96 @@
|
||||
"image": "source.archipelago-foundation.org/lfg2025/cryptpad:2024.12.0",
|
||||
"version": "2024.12.0"
|
||||
},
|
||||
"cuprate": {
|
||||
"manifest": {
|
||||
"app": {
|
||||
"category": "money",
|
||||
"container": {
|
||||
"custom_args": [
|
||||
"--config-file",
|
||||
"/home/cuprate/Cuprated.toml"
|
||||
],
|
||||
"data_uid": "1000:1000",
|
||||
"image": "source.archipelago-foundation.org/lfg2025/cuprate:0.1.0-preview-18-g618ff14",
|
||||
"network": "archy-net",
|
||||
"pull_policy": "if-not-present"
|
||||
},
|
||||
"dependencies": [
|
||||
{
|
||||
"storage": "300Gi"
|
||||
}
|
||||
],
|
||||
"description": "Alternative Monero node implementation in Rust. Independently validates Monero consensus rules, providing a layer of security and redundancy for the network.",
|
||||
"files": [
|
||||
{
|
||||
"content": "network = \"Mainnet\"\ntarget_max_memory = 3000000000\n\n[rpc.restricted]\nenable = true\n",
|
||||
"overwrite": false,
|
||||
"path": "/var/lib/archipelago/cuprate/Cuprated.toml"
|
||||
}
|
||||
],
|
||||
"health_check": {
|
||||
"endpoint": "localhost:18090",
|
||||
"interval": "30s",
|
||||
"retries": 3,
|
||||
"start_period": "5m",
|
||||
"timeout": "5s",
|
||||
"type": "tcp"
|
||||
},
|
||||
"id": "cuprate",
|
||||
"metadata": {
|
||||
"author": "Cuprate",
|
||||
"category": "money",
|
||||
"icon": "/assets/img/app-icons/cuprate.svg",
|
||||
"repo": "https://github.com/Cuprate/cuprate",
|
||||
"tier": "optional"
|
||||
},
|
||||
"name": "Cuprate",
|
||||
"ports": [
|
||||
{
|
||||
"auth": "none",
|
||||
"auth_rationale": "Monero p2p gossip. Peers are anonymous by design and speak the Monero wire protocol, not HTTP.",
|
||||
"container": 18080,
|
||||
"host": 18183,
|
||||
"protocol": "tcp"
|
||||
},
|
||||
{
|
||||
"auth": "none",
|
||||
"auth_rationale": "Monero restricted RPC — the subset upstream considers safe for public/remote-node use. Wallets (Feather, monero-wallet-rpc, GUI) connect directly over plain HTTP JSON-RPC and cannot hold a dashboard session cookie.",
|
||||
"container": 18089,
|
||||
"host": 18090,
|
||||
"protocol": "tcp"
|
||||
}
|
||||
],
|
||||
"resources": {
|
||||
"cpu_limit": 0,
|
||||
"disk_limit": "300Gi",
|
||||
"memory_limit": "4Gi"
|
||||
},
|
||||
"security": {
|
||||
"capabilities": [],
|
||||
"network_policy": "isolated",
|
||||
"no_new_privileges": true,
|
||||
"readonly_root": true
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "github",
|
||||
"repo": "Cuprate/cuprate"
|
||||
},
|
||||
"version": "0.1.0-preview",
|
||||
"volumes": [
|
||||
{
|
||||
"options": [
|
||||
"rw"
|
||||
],
|
||||
"source": "/var/lib/archipelago/cuprate",
|
||||
"target": "/home/cuprate",
|
||||
"type": "bind"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": "0.1.0-preview"
|
||||
},
|
||||
"did-wallet": {
|
||||
"manifest": {
|
||||
"app": {
|
||||
@@ -2375,6 +2469,10 @@
|
||||
"network_policy": "isolated",
|
||||
"readonly_root": false
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "ghcr",
|
||||
"repo": "immich-app/postgres"
|
||||
},
|
||||
"version": "14-vectorchord0.4.3-pgvectors0.2.0",
|
||||
"volumes": [
|
||||
{
|
||||
@@ -2788,6 +2886,10 @@
|
||||
"network_policy": "isolated",
|
||||
"readonly_root": false
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "github",
|
||||
"repo": "minio/minio"
|
||||
},
|
||||
"version": "RELEASE.2024-11-07T00-52-20Z",
|
||||
"volumes": [
|
||||
{
|
||||
@@ -3175,6 +3277,10 @@
|
||||
"seccomp_profile": "default",
|
||||
"user": 1000
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "manual",
|
||||
"url": "no public listing for lightninglabs/lightning-stack — verify by hand"
|
||||
},
|
||||
"version": "0.12.0",
|
||||
"volumes": [
|
||||
{
|
||||
@@ -3625,7 +3731,7 @@
|
||||
"key": "/var/lib/archipelago/netbird/tls.key"
|
||||
}
|
||||
],
|
||||
"image": "docker.io/library/nginx:1.31.3-alpine",
|
||||
"image": "docker.io/library/nginx:1.31.4-alpine",
|
||||
"network": "netbird-net",
|
||||
"pull_policy": "if-not-present"
|
||||
},
|
||||
@@ -4315,7 +4421,7 @@
|
||||
"key": "/var/lib/archipelago/pine/tls.key"
|
||||
}
|
||||
],
|
||||
"image": "docker.io/library/nginx:1.31.3-alpine",
|
||||
"image": "docker.io/library/nginx:1.31.4-alpine",
|
||||
"network": "archy-net",
|
||||
"network_aliases": [
|
||||
"pine"
|
||||
@@ -4701,6 +4807,10 @@
|
||||
"no_new_privileges": true,
|
||||
"readonly_root": false
|
||||
},
|
||||
"upstream": {
|
||||
"kind": "dockerhub",
|
||||
"repo": "rhasspy/wyoming-whisper"
|
||||
},
|
||||
"version": "3.4.2",
|
||||
"volumes": [
|
||||
{
|
||||
@@ -4998,7 +5108,7 @@
|
||||
"manifest": {
|
||||
"app": {
|
||||
"container": {
|
||||
"image": "dockurr/strfry:1.1.1",
|
||||
"image": "dockurr/strfry:1.1.2",
|
||||
"image_signature": "cosign://...",
|
||||
"pull_policy": "verify-signature"
|
||||
},
|
||||
@@ -5055,7 +5165,7 @@
|
||||
"kind": "github",
|
||||
"repo": "hoytech/strfry"
|
||||
},
|
||||
"version": "1.1.1",
|
||||
"version": "1.1.2",
|
||||
"volumes": [
|
||||
{
|
||||
"options": [
|
||||
@@ -5076,7 +5186,7 @@
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": "1.1.1"
|
||||
"version": "1.1.2"
|
||||
},
|
||||
"tailscale": {
|
||||
"image": "source.archipelago-foundation.org/lfg2025/tailscale:stable",
|
||||
@@ -5256,7 +5366,7 @@
|
||||
}
|
||||
},
|
||||
"schema": 1,
|
||||
"signature": "97628de24e3ffa17f639c663e19881cf6dea8c79aab272fe9c5442a4e951b3f0d257fee21aa9ce6158e3824b56a337acb71805f6fd34245e48345b86b46ec007",
|
||||
"signature": "da5b6b183ac46c062945c27abdc06affb558e805e1ccf67ac0ee17e5e3dd85cc05a0656dd83bdacb1e1d237445145d00995f55e77209e1cbb2b6d8ce47084e0a",
|
||||
"signed_by": "did:key:z6Mkfu5LT8d4DjETtrkATvHh9Dvcbnr7zBCUwfau8Sw7DLWT",
|
||||
"updated": "2026-08-19"
|
||||
"updated": "2026-08-30"
|
||||
}
|
||||
|
||||
@@ -38,18 +38,18 @@ MARIADB_IMAGE="$ARCHY_REGISTRY/mariadb:11.4.10"
|
||||
|
||||
# BTCPay
|
||||
BTCPAY_IMAGE="docker.io/btcpayserver/btcpayserver:2.4.3"
|
||||
NBXPLORER_IMAGE="$ARCHY_REGISTRY/nbxplorer:2.6.11"
|
||||
NBXPLORER_IMAGE="$ARCHY_REGISTRY/nbxplorer:2.6.0"
|
||||
POSTGRES_IMAGE="$ARCHY_REGISTRY/postgres:15.17"
|
||||
BTCPAY_POSTGRES_IMAGE="$ARCHY_REGISTRY/postgres:15.17"
|
||||
|
||||
# Apps
|
||||
HOMEASSISTANT_IMAGE="$ARCHY_REGISTRY/home-assistant:2026.8.3"
|
||||
HOMEASSISTANT_IMAGE="$ARCHY_REGISTRY/home-assistant:2026.8.2"
|
||||
GRAFANA_IMAGE="$ARCHY_REGISTRY/grafana:10.2.0"
|
||||
UPTIME_KUMA_IMAGE="$ARCHY_REGISTRY/uptime-kuma:1"
|
||||
JELLYFIN_IMAGE="$ARCHY_REGISTRY/jellyfin:10.11.11"
|
||||
PHOTOPRISM_IMAGE="$ARCHY_REGISTRY/photoprism:240915"
|
||||
OLLAMA_IMAGE="$ARCHY_REGISTRY/ollama:latest"
|
||||
VAULTWARDEN_IMAGE="$ARCHY_REGISTRY/vaultwarden:1.37.2-alpine"
|
||||
VAULTWARDEN_IMAGE="$ARCHY_REGISTRY/vaultwarden:1.37.1-alpine"
|
||||
NEXTCLOUD_IMAGE="$ARCHY_REGISTRY/nextcloud:29"
|
||||
SEARXNG_IMAGE="$ARCHY_REGISTRY/searxng:latest"
|
||||
# OnlyOffice removed — incompatible with rootless Podman (internal postgres/rabbitmq fail)
|
||||
|
||||
Reference in New Issue
Block a user