How to use different tools to analyze and optimize code performance when CPU utilization is high

$CPUsenduserate=1-(emptyLeisurewhen{between}_{new}-emptyLeisurewhen{between}_{old})/(totalCPUwhen{between}_{new}-totalCPUwhen{between}_{old})$

1. procc see CPU Usage rate

1.1 proc/stat Parameter details in

Linux adopt /proc Virtual file system , Provide system internal status information to user space , and /proc/stat What is provided is the system CPU And task Statistics , The specific analysis is as follows :

$ man proc
		/proc/stat
              kernel/system  statistics.   Varies  with  architecture.  Common
              entries include:

              cpu 10132153 290696 3084719 46828483 16683 0 25195 0 175628 0
              cpu0 1393280 32966 572056 13343292 6130 0 17875 0 23933 0
                     The  amount  of  time,  measured  in  units  of   USER_HZ
                     (1/100ths   of   a  second  on  most  architectures,  use
                     sysconf(_SC_CLK_TCK) to obtain the right value), that the
                     system  ("cpu"  line)  or  the specific CPU ("cpuN" line)
                     spent in various states:

                     user   (1) Time spent in user mode.

                     nice   (2) Time spent in  user  mode  with  low  priority
                            (nice).

                     system (3) Time spent in system mode.

                     idle   (4)  Time  spent  in  the  idle  task.  This value

/proc/stat #  Check by shortcut after entering 

According to the above proc/stat The relationship between each column is

1. user ( abbreviation us) User mode CPU Time , It doesn't contain nice Time , Contains only guest
2. nice ( abbreviation ni) Low priority user mode CPU Time , Process nice Adjusted to 1-19, The higher the priority, the lower
3. system ( abbreviation sys) Kernel mode CPU Time
4. idle( abbreviation id ) free time , Not including waiting IO Time iowait
5. iowait ( abbreviation wa) wait for IO Of CPU Time
6. irq ( abbreviation hi) Hard interrupt CPU Time
7. softirq ( abbreviation si) Soft interrupt CPU Time
8. steal ( abbreviation st) When the system is running on a virtual machine , Occupied by other virtual machines CPU Time
9. guest Virtualization running other operating system time , Running virtual machines CPU Time
10. guest_nice Indicates the time of running the virtual machine with low priority

1.2 utilize proc/stat see

If we only focus on CPU, You can see this with the following command

$ cat /proc/stat | grep ^cpu
cpu  2400 0 4299 20978 0 624 0 0 0 0
cpu0 234 0 675 2551 0 293 0 0 0 0
cpu1 150 0 100 3210 0 18 0 0 0 0
#  Column information :  User mode ,  Weak user state ,  Kernel mode ,  Space time , IO wait for   Hard interrupt  ...

2. top see CPU Usage rate

top Display the overall system CPU And memory usage , And the resource usage of each process

$ top #  The default for each 3s Refresh once 
top - 22:50:25 up 15 min,  0 users,  load average: 0.52, 0.58, 0.59
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
#  The following line is CPU Usage rate ,  The meaning of each column is 1.1 Has been given in ,  Just put CPU Change the time to CPU Usage rate ,  Such as 8.4 Indicates that the user status is occupied 8.4% Of CPU Usage rate 
%Cpu(s):  8.4 us, 12.4 sy,  0.0 ni, 77.9 id,  0.0 wa,  1.4 hi,  0.0 si,  0.0 st
KiB Mem :  8271548 total,  3149636 free,  4892560 used,   229352 buff/cache
KiB Swap: 25165824 total, 24378928 free,   786896 used.  3245256 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    8948    396    324 S   0.0  0.0   0:00.14 init
   10 root      20   0    9296    228    180 S   0.0  0.0   0:00.01 init
   11 zjq       20   0   17032   3644   3544 S   0.0  0.0   0:00.38 bash
   86 zjq       20   0   17644   2044   1492 R   0.0  0.0   0:00.03 top
   
#  Be careful  top The default display is CPU Average ,  Press down 1,  Switch to each CPU Usage rate 
top - 22:56:33 up 22 min,  0 users,  load average: 0.52, 0.58, 0.59
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.9 us, 12.3 sy,  0.0 ni, 80.4 id,  0.0 wa,  5.4 hi,  0.0 si,  0.0 st
%Cpu1  :  2.0 us,  1.7 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8271548 total,  3127012 free,  4915184 used,   229352 buff/cache
KiB Swap: 25165824 total, 24396948 free,   768876 used.  3222632 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    8948    396    324 S   0.0  0.0   0:00.14 init
   10 root      20   0    9296    228    180 S   0.0  0.0   0:00.01 init
   11 zjq       20   0   17032   3644   3548 S   0.0  0.0   0:00.39 bash
   87 zjq       20   0   17644   2044   1492 R   0.0  0.0   0:00.04 top

Above we find , There is a process %CPU Column , Representing a process CPU Usage rate , It is user mode and kernel mode CPU The sum of usage , Contains the process user space used CPU, Execute kernel space through system calls CPU, And the ready queue waiting to run CPU; In a virtualized environment , It also includes the CPU.

because top There is no segmentation of process user mode and kernel mode CPU, How to view the details of each process ?

3. pidstat Command view CPU Usage rate

install : sudo apt install sysstat

$ pidstat 1 5 #  Output a set of data per second ,  Output 5 Time 
23:03:04      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
23:03:05     1000       492    0.00    1.00    0.00    0.00    1.00     0  pidstat

The information in each column is :

1. %usr User mode CPU Usage rate
2. %system Kernel mode CPU Usage rate
3. %guest Running virtual machines CPU Usage rate
4. %wait wait for CPU Usage rate
5. %CPU The total CPU Usage rate

We can see it up here , The highest %CPU Occupied is pidstat process , That is to say PID=492 Number

4. pref location CPU High usage code location

Come here , Through the above three methods : proc, top, pidstat Can easily determine CPU High usage processes , Next , According to the process with high utilization , How to locate the occupancy code ?

pref yes Linux2.6.31 Later built-in performance analysis tools , Based on performance event sampling , It can not only analyze various events of the system and kernel performance , You can also analyze the performance problems of the specified application

install :

sudo apt install linux-tools-common linux-tools-5.4.0-120-generic  linux-cloud-tools-5.4.0-120-generic

4.1 perf Use a : perf top

As shown in the above figure , The first row is three data , Number of samples (samples), Event type (event), The total number of events (event count)

It should be noted that : When sampling is too little , For example, dozens of , It's meaningless

The output of the above figure is an example , We see , Occupy CPU Most of all perf The tool itself , There is only a 7.28%, That means the system doesn't have CPU Performance issues .

4.2 perf Use two : perf record and perf report

self-examination

5. application

nginx+PHP Of web Service as an example

5.1 ab Apache bench A common HTTP Service performance testing tool

ab -c 10 -n 10000 http://www.baidu.com #  Concurrent 10 Request tests nginx performance ,  A total of tests 1w Requests 

5.2 Find the occupation in the service CPU Highly utilized code functions

Service machine	The client
Provide Web service Such as www.baidu.com	Browser access www.baidu.com
Only per second 11.63 Responses to requests	ab -c 10 -n 10000 http://www.baidu.com
Start testing
top Press the number 1, Find out php-fpm Command to occupy %CPU The highest , And the corresponding process ID yes 22222
To analyze
perf top -g -p 22222 # -g Turn on call relationship analysis , -p Specify the process number
Replace the direction key with php-fpm, Press enter to expand the calling relationship , Find occupancy CPU Most of all sqrt and add_function Two functions
View the sqrt grep sqrt -r Source folder Found a place , Found a mistake

6. summary

1. user CPU and Nice CPU high , It indicates that user mode processes occupy a lot CPU, So focus on troubleshooting process performance problems
2. System CPU high , It shows that the kernel state takes up more CPU, Focus on the performance problems of kernel threads or system calls
3. IO wait for CPU high , Explain waiting IO Long time , Focus on System Storage IO problem
4. Soft interrupt and hard interrupt high , It indicates that the interrupt handler takes more CPU, We should focus on checking the interrupt service program in the kernel
5. Final : encounter CPU Increased usage , With the help of top pidstat, Confirm to trigger CPU Performance asked his source , Use perf Tools , Check the specific functions that cause performance problems , To optimize