PHP server interacts with redis with a large number of close_ Wait analysis

Recently, it was found that there was a small number of visits （ Probably 1000 Time / God ） Server for , With the redis In the process of interaction , A lot of CLOSE_WAIT Network state , Lead to fd Resources are full . At first glance, it's counter intuitive , The number of visits is small and takes up a lot of resources , Finally, through analysis tcp agreement , And analysis php-fpm Model , Finally found and solved the problem .

 Inquire about linux in tcp The order of the situation ：
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}' 

 

A lot of close_wait state

1.CLOSE_WAIT principle

It's disconnected tcp When the connection , There will be one. 4 The process of waving . The party who actively disconnects will initiate a FIN package , The receiver will return a ACK Bao told the initiator that I knew you were going to disconnect , At this time, the status of the receiver changes to CLOSE_WAIT And deal with the closing work . Under normal circumstances , After the receiving party completes the closing work , Will send a FIN package , Then after receiving the reply from the developer ACK After the bag , Whole 4 The process of waving is over .
But under abnormal circumstances , The receiver does not send FIN package , Leading to its state has been CLOSE_WAIT state , If every connection is like this , It will occupy a lot of system resources , As a result, services cannot be provided externally .

4 Second wave

2. Such a small number of visits php The server , And redis Why does interaction occur CLOSE_WAIT state

php And redis Interactive analysis

stay php in , did not redis The concept of connection pool , So in order to avoid continuous connection 、 Disconnect from redis The connection of , We are php In general, the form of long connection is used with redis Interact .

This form creates the life cycle of the connection and php-fpm The life cycle of a process is consistent , So every time a new request comes , City reuse php-fpm Originally created redis Connect , This saves constant connections 、 disconnect redis The resulting network overhead .

php-fpm Life cycle of

php-fpm There is a configuration pm.max_requests Express ,php-fpm Process process pm.max_requests Requests will be destroyed and a new child process will be generated . This means that 1 individual fpm The process needs to go through max_requests Requests will be destroyed

pm.max_requests = 50240

Focus on the problem

from php And redis Interaction analysis of , We can draw ： One reids Long connection life cycle and php-fpm Child processes are consistent .
and php-fpm The life cycle of is experience pm.max_requests A request , The configuration of our server is 50240.
meanwhile , The number of server visits with problems is particularly small （1000 Time / God ）, That is to say a php-fpm The life cycle of is at least 50 God （50240/1000=50, There should be more , Because there is more than one process ）, that redis The life cycle of the connection is at least 50 God .
Here 50 In the day ,redis Will give php The server initiates a disconnect request ,php The server will return a ACK, Then change the status to CLOSE_WAIT, however php The server will not send another FIN Give it to redis, Because its life cycle is not over . So it will eventually produce a lot of CLOSE_WAIT.

3. Why does a large number of visits not appear CLOSE_WAIT

Large amount of visits , signify php-fpm Process max_requests Easy to reach ,php-fpm The life cycle of will be very short , Not yet redis Trigger time of server timeout disconnection .

4. Solution

Use a short connection ~

Reference resources ：

http://www.gosanye.com/post/10160.html

Reference resources ：http://t.zoukankan.com/xinfang520-p-8961129.html