On December 8th, 2020, the memory of marketing MRC application suddenly increased, resulting in system oom

background

12.08 At noon on the th mrc Applications suddenly appear, memory continues to rise , From 67% Rise to 85% about （ Monitoring is as follows ）, Fortunately, the rising process is relatively slow , A decisive restart solved the problem . The process of solving and analyzing problems is as follows .

Problem solving process

mrc It's the bottom application of marketing , Main partial rule calculation , common 6 Taiwan machine （2 Next cluster , And cluster traffic is isolated from each other , Such as the upper layer hipc Cluster traffic will not be requested to k8s Cluster machines ）,6 At the same time, the memory keeps rising , Refer to sketch 1 .

Because it was a big promotion at noon that day , Considering that there are only 3 Taiwan machine , I'm afraid that in the process of restarting one , The other two can't stand the flow of big promotion , At first, I didn't dare to consider a single restart , After a short period of time, the decision was made taking into account the cpu Only 5% about , The worst worry is that memory can't take care of it all of a sudden , If frequent gc May affect the normal traffic access , So prepare for the worst ： Restart decisively （ Remove traffic before restart , meanwhile dump Memory for subsequent analysis ）, As a result, there was no problem , Refer to sketch 2 . The whole process is as follows ：

1. The target restarts the machine for traffic removal , Adjust to restart the machine dubbo The weight of 0 that will do , because dump Memory processes are memory consuming operations , Server may appear feign death phenomenon, affect normal call , So we need to remove the traffic .
2. Force the target machine to restart once full gc, The purpose is to reclaim the normal memory object occupation , To prevent the normal memory occupation and the influence of real memory leak objects , The impact analysis , You can use the following command ：
3. dump Next target machine memory , The order is as follows ：

 jmap -histo:live 13  （ Trigger full gc）
  or 
 jmap -dump:live,file=dump_001.bin 13  （ Trigger full gc, When triggered, put dump_001.bin File deletion ）
 or 
jcmd 13 GC.run  （ Trigger young gc)

4. Use IBMAnalyzer（ perhaps jdk Self contained jvisualvm Tools or mat Tools ） Yes dump File analysis is enough

 jmap -dump:format=b,file=dumpFile 13

After the event, the best plan is to add a new one to Tongyun maintenance mrc machine , And then restart each one , Refer to sketch 3 .

Post analysis

After the event dump Document analysis , As it involves specific business, I will not elaborate on it , Just describe the conclusion ： Because that day mrc Configuring the shadow library results in . The root cause is druid Threads that monitor shadow library configuration will not exit with the end of the pressure test , stay mrc After pressure testing, the thread creation is triggered without restart , Lead to mrc Application memory keeps rising .

Welcome to WeChat official account. ： Fang Chen's blog