Record the problem location experience when an application is suddenly killed

The problem background ： Start the local application at one time , Automatically exit after two minutes , No abnormal conditions were found in the log , Inexplicable applications are automatically killed ;

Solutions ：

1、linux adopt top see java Application memory and cpu It's not high , It just disappeared after a while ;

2、 No abnormality is found in the application log , The code has been checked several times ;

3、 adopt dmesg | grep java View kernel log information , Found the problem , as follows ：

[16949523.941194] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=991
[16949523.942914] java cpuset=73a35980233979bb67f20700c76d77805de6ced7cfd18de836238a7bdae7c1dd mems_allowed=0
[16949523.942918] CPU: 4 PID: 310033 Comm: java Tainted: G     U     O 3.10.107-1-tlinux2_kvm_guest-0052 #1
[16949523.971234] Memory cgroup out of memory: Kill process 310033 (java) score 1964 or sacrifice child
[16949523.973103] Killed process 182650 (java) total-vm:16705124kB, anon-rss:2079844kB, file-rss:22020kB

From the above information, you can see that the memory overflow is linux Killed java Application information ;

4、 But the memory usage of my application is not very high , adopt jinfo -flags <javapid> Find out java The pre application memory for application startup has reached 10G;

Launch parameters

5、 adopt free perhaps vmstat Check that the remaining memory size is only 10G 了 , The kernel has detected that the system is out of memory 、 The process of selecting and killing a process can refer to the kernel source code linux/mm/oom_kill.c, When the system is low on memory ,out_of_memory() Be triggered , And then call select_bad_process() Select a “bad” Process kill , How to judge and choose a “bad” What about the process , You can't choose at random ？ The selection process consists of oom_badness() decision , The algorithms and ideas selected are very simple and simple ： most bad The process that uses the most memory is the process that uses the most memory .

 6、oom_kill.c You can see in the code oom_badness() Rate each process , according to points To determine which process to kill , This points According to adj Adjust the ,root The process of permissions is often considered important , Should not be killed easily , So when scoring, you can get 3% Concession （adj -= 30; The lower the score, the less likely it is to be killed ）. We can manipulate each process in user space oom_adj Kernel parameters to determine which processes are not so easy to be OOM killer Select kill . such as , If not MySQL If the process is easily killed, you can find MySQL After the running process number , adjustment oom_score_adj by -15（ Be careful points The smaller it is, the less likely it is to be killed ）

7、oom_score_adj Set value range -1000 To 1000 Range interval , Setting example ： When an application memory request is used 1G when , Set up oom_score_adj=-500 when , actual linux You will think that this application only uses 500M, There will be a discount mechanism ; To ensure that some important applications are not accidentally kill fall ;