Fault analysis - a case of excessive CPU load caused by a large number of short-time processes

author ： Ren Kun

Now living in Zhuhai , He has been a full-time Oracle and MySQL DBA, Now I'm mainly responsible for MySQL、mongoDB and Redis Maintenance work .

In this paper, the source ： Original contribution

* Produced by aikesheng open source community , Original content is not allowed to be used without authorization , For reprint, please contact the editor and indicate the source .

---

1、 background

The development environment of a project , A single virtual machine is equipped with 1 set mongo The cluster is used to test ,1 individual mongos + 3 node config + 1shard * 3 copy , A total of 7 individual mongo example . mongo edition 4.2.19,OS by centos 7.9.

After the test cpu The load is maintained at 50% about , And then mongo Of qps Has dropped to 0.

Only... Is installed on this machine mongo, Will all mongo Instance close ,cpu The load returns to normal immediately , then mongo Instance on , After a while cpu The load began to soar again . The scene can be reproduced , And confirm that it is with mongo The instance is related to .

2、 The diagnosis

perform top command ,cpu Of usr It's reached 40%, But the first few processes %cpu It doesn't add up enough .

see mongos Of qps, Indeed, the user's command has not been executed .

dstat View the overall load (vmstat Bad formatting , The last few columns are always out of alignment ).

except cpu Abnormal load , Other indicators are normal , Interruption and context switching are not high , It's unlikely that these two triggered .
perf record -ag – sleep 10 && perf report see cpu The implementation of .

There are a lot of mongo call , however API Naming is not intuitive , Unable to guess the corresponding execution logic .

thus , Confirmation is mongo Problems caused by instances , however mongo Your application connection is 0, Look at the call API The stack can't find useful information .

Back to the beginning of this article ,top Process cpu The combined utilization rate is far less than cpu Overall load , The probability is that there are frequent short-term processes that steal this part CPU resources , Lead to top The command has no time to capture Statistics .

sar -w 1 View the number of processes generated per second , Average number of new entries per second 80 Multiple processes , It should be it .

Find applications that frequently establish short-term processes , May adopt execsnoop, The tool passes through ftrace Monitor the progress in real time exec() Behavior , And output the basic information of short-time process , Including process PID/PPID、 Command line arguments .

# download execsnoop#
cd /usr/bin
wget https://raw.githubusercontent.com/brendangregg/perf‐tools/master/execsnoop
chmod 755 execsnoop 

Here's the output , It's all the monitoring system , Keep connecting mongo And execute... On the output result grep Filter , Each operation spawns a new thread / process ,10s Capture 了 400 Multiple records .

take zabixx Process shutdown ,cpu Return to normal immediately , Found the culprit .

Our other environments have also adopted zabbix monitor , But they haven't encountered similar problems .

This node is deployed 7 individual mongo example ,zabbix The default is for each mongo All instances are monitored , It is equivalent to the amplification of execution loss 7 times , There is only one machine 4 nucleus CPU Virtual machine .

When these factors are put together, problems will break out . This is a development environment , Temporarily closed zabbix monitor , Subsequently, the monitoring logic should be optimized , Minimize connections db The number of times and grep The length of the call chain .

3、 Summary

When the machine cpu The load keeps rising but can't grasp top Process time , May adopt execsnoop Grab short-term processes , Similar tools include iosnoop、opensnoop.