# Hard Lockup & Crash Diagnostics Setup (Debian/Ubuntu)

Purpose: when a machine hard-freezes (all CPUs lock up, no logs, no response), this setup
converts lockups into kernel panics, captures a vmcore, and auto-reboots — so the next
incident is diagnosable instead of silent. Also enables MCE/PCIe-error logging.

Tested on: Debian 13 (trixie), x86_64, systemd. Also applies to other Debian/Ubuntu versions.

## 1. Install packages

```bash
sudo apt-get update
DEBIAN_FRONTEND=noninteractive sudo apt-get install -y kdump-tools rasdaemon
```

- `kdump-tools` — loads a small crash kernel at boot (reserves RAM via `crashkernel=`),
  captures vmcores to `/var/crash` on panic/lockup.
- `rasdaemon` — logs machine-check (MCE) and PCIe AER errors to a queryable DB.

The kdump-tools postinst regenerates grub and adds `crashkernel=512M-:192M` automatically.

## 2. Sysctl settings

Create `/etc/sysctl.d/99-lockup-panic.conf`:

```ini
kernel.softlockup_panic = 1
kernel.hardlockup_panic = 1
kernel.panic = 10
```

Apply:

```bash
sudo sysctl -p /etc/sysctl.d/99-lockup-panic.conf
```

- `softlockup_panic=1` / `hardlockup_panic=1` — a CPU lockup becomes a panic instead of
  hanging forever (the NMI watchdog detects it).
- `panic=10` — auto-reboot 10 s after any panic, so kdump has already captured the vmcore
  and the machine recovers on its own.

## 3. Reboot

Required — the crash kernel and `crashkernel=` param only take effect on next boot.

```bash
sudo reboot
```

## 4. Verify (after reboot)

```bash
# 1. crashkernel reserved:
cat /proc/cmdline | grep -o 'crashkernel=[^ ]*'
# expect: crashkernel=512M-:192M

# 2. kdump armed:
sudo kdump-config show | grep 'current state'
# expect: ready to kdump

# 3. sysctl persisted:
sudo sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic

# 4. rasdaemon running, baseline error check:
systemctl is-active rasdaemon
sudo ras-mc-ctl --errors
```

## 5. Using the output after an incident

```bash
# vmcores from crashes:
ls -lh /var/crash/
# analyze with the crash utility (installed with kdump-tools):
sudo crash /var/lib/systemd/kdump/vmlinuz /var/crash/<date>/vmcore

# MCE / PCIe AER errors:
sudo ras-mc-ctl --errors
sudo ras-mc-ctl --record --errors | head
```

Notes:
- A full machine lockup (all CPUs frozen at once) may still not be catchable; kdump
  requires at least the panic path to run. It catches the common cases: single-CPU
  lockups, BUG()/panic, OOM panic, and most driver-induced faults.
- If `kdump-config show` says "Not ready" after reboot, check
  `journalctl -u kdump-tools` and that enough RAM is available for the reserved size.
- `kernel.panic=10` means the box reboots itself ~10 s after a panic. Remove that line if
  you prefer to investigate live (you lose the auto-recovery).