[Development miscellaneous][Debug]debug into kernel

problem

• a blocked process on server
• gdb get few infomation
• log get few infomation

top/ps check pid, check cpu utilization

strace check syscall

pstrack print userspace call stack

WCHAN, syscall

man ps | grep WCHANps -flp pidps -flp pidcat /proc/pid/wchan # check wait statuscat /proc/pid/syscall # check syscall262 .....

262 ⇒ newfstatat

man newfstatat

fstatat

print kernel space call stack

cat /proc/PID/stack

ps

ps fauwwwxUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 896 576 ? Sl 15:58 0:00 /initroot 239 0.0 0.0 896 84 ? Ss 16:01 0:00 /initroot 240 0.0 0.0 0 0 ? Z 16:01 0:00 \_ [init] <defunct>root 242 0.0 0.0 0 0 ? Z 16:01 0:00 \_ [init] <defunct>root 244 0.0 0.0 896 84 ? S 16:01 0:00 \_ /initroot 245 0.0 0.4 1242416 27228 pts/2 Ssl+ 16:01 0:01 \_ /mnt/wsl/docker-desktop/docker-desktop-proxy --distro-name Ubuntu-20.04 --docker-desktop-root /mnt/wsl/docker-desktop --show-kube-system-containers=true

list open file

 sudo lsof -p 1COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEinit 1 root cwd DIR 8,16 4096 2/init 1 root rtd DIR 8,16 4096 2/init 1 root txt REG 0,18 632048 281474977205323 /initinit 1 root 0u CHR 1,3 0t0 10 /dev/nullinit 1 root 1u CHR 1,3 0t0 10 /dev/nullinit 1 root 2u CHR 1,3 0t0 10 /dev/nullinit 1 root 3w CHR 1,11 0t0 15 /dev/kmsginit 1 root 5r REG 0,4 0 4026532184 mntinit 1 root 6r REG 0,4 0 4026532200 mntinit 1 root 8u DIR 8,16 4096 2/init 1 root 9u sock 0,8 0t0 15429 protocol: AF_VSOCKinit 1 root 10u a_inode 0,12 0 14787 [eventpoll]init 1 root 100u sock 0,8 0t0 15424 protocol: AF_VSOCK

kernel core dump

The ipmitool tool sends an NMI signal
to make the system generate vmcore (a kernel dump file generated when the Linux kernel crashes)
for subsequent analysis