A troubleshooting of CPU bottom falling

Come up in the morning , service cpu Down .

Log in to the machine , Troubleshooting methods ：

1. Carry out orders    top ( see cpu Processes that account for a large proportion pid),

2. perform top -Hp pid  ( View the corresponding process below cpu Accounting for a large proportion   Threads id( That's the corresponding pid))

3. use printf "%x\n" ( The first 2 Thread found in step id)    convert to 16 It's binary , Expected thread id There is a corresponding... In the stack 16 It's binary nid

4. perform jstack pid | grep 'nid( step 3 Medium nid)'   -A 20  View the stack information of the corresponding thread . 

Add jstack command ：

2. Network troubleshooting

netstat -antop      Show tcp Network connection information of

• -a or --all Show... In all connections Socket.
• -n or --numeric Use it directly IP Address , Instead of going through the domain name server .
• -t or --tcp Show TCP The connection status of the transport protocol .
• -o or --timers Display timer .
• -p or --programs Display in use Socket The program process pid And program name .

Two parameter information displayed recv-Q  send-Q , The two values should normally be 0, If not for 0 There may be a problem .packets There should be no stacking in either queue . It's not acceptable 0 situation

recv-Q Indicates network reception queue
Indicates that the received data has been buffered locally , But how much is not taken away by the process ,recv()
If the receive queue Recv-Q It's stuck , It could be a denial of service denial-of-service attack .

send-Q Indicates the network send queue
The other party didn't receive the data or didn't Ack Of , Or local buffer .
If the send queue Send-Q It can't be cleared quickly , It may be that some applications send packets out too fast , Or the other party is not fast enough to receive packets .

ss -ltn  Show tcp Full connection of   and Number of semi connections

-l   Shows the socket in listening state

-t  Show tcp network connections

-n  Do not resolve service name , Has been digitally displayed

<1>  When socket yes listen  state (eg: ss -lnt)
    Recv-Q:  The size of the full connection queue , That is, three handshakes have been completed and waiting for the server  accept()  Of  TCP  Connect 
    Send-Q:  Maximum queue length of full connection 
<2>  When socket  Right and wrong listen  state (eg: ss -nt)
    Recv-Q:  Number of bytes not read by the application process ;
    Send-Q:  Number of bytes sent but not acknowledged ;

3. Check thread deadlock .

jps  yes java The one provided shows all the current java process pid The order of .

1 adopt jps Find the process

2. jstack pid( Process number )  Then check it out In the stack log deadlock keyword

 

TCP The maximum value of the fully connected queue depends on somaxconn and backlog The minimum value between , That is to say min(somaxconn, backlog),  linux The default size is 128.

• somaxconn  yes Linux Kernel parameters , The default value is 128, Can pass  /proc/sys/net/core/somaxconn  To set its value ;
• backlog  yes  listen(int sockfd, int backlog)  Function backlog size ,Nginx The default value is 511, You can set its length by modifying the configuration file ;

tcp_abort_on_overflow There are two values that are 0 and 1, It means respectively that ：

• 0 ： If the full connection queue is full , that server Throw away client Sending a ack ;
• 1 ： If the full connection queue is full ,server Send a  reset  Give it to client, It means to discard the handshake and the connection ;

If there's a constant connection because TCP Full connection queue overflow discarded , You should turn it up backlog as well as somaxconn Parameters .

tcp Full connection overflow statistics command view ： netstat -s | grep "overflowed"

tcp Semi connection overflow statistics command ：netstat -s | grep "SYNs to LISTEN",

tcp Half connection size view ：