During JMeter pressure measurement, time_ The number of requests does not go up due to many waits. The problem is solved

Background introduction

        In order to find out the performance of the project , Performance testing is required . After some research , Decided to use Tencent cloud TKE Distributed jmeter Carry out pressure test , The advantage is jmeter-suite You can use , It's convenient to build the environment ; Container deployment can easily increase pod To raise pressure .

        But in the actual pressure , Found that the amount of requests did not go up , Can't reach the pressure measurement effect . Found by location , Containers pod There is a lot of TIME_WAIT, The actual data transmission connection is far less than the set number of concurrent threads ：

netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

Why is there TIME_WAIT

        This is a TCP Connection released 4 The process of waving ：

1. The party that actively closes the connection , call close(); Protocol layer send FIN package
2. Received by the passive Closing Party FIN After package , Protocol layer reply ACK; Then the passive Closing Party , Get into CLOSE_WAIT state , The active Closing Party waits for the other party to close , entering FIN_WAIT_2 state ; here , Active Closing Party
3. wait for
4. Passively shut down one party's Application , call close operation
5. After the passive shutdown party completes all data transmission , call close() operation ; here , Protocol layer send FIN Package to active shutdown party , Waiting for each other's ACK, Passive Closing Party entry LAST_ACK state ;
6. Received by the active Closing Party FIN package , Protocol layer reply ACK; here , The party that actively closes the connection , Get into TIME_WAIT state ; And the passive Closing Party , Get into CLOSED state
7. wait for 2MSL Time , Active Closing Party , end TIME_WAIT, Get into CLOSED state

        This process can get the following information ：

• ESTABLISHED state , Indicates the connection that is sending the request , That is, the number of requests under pressure
• The party that actively closes the connection will eventually enter TIME_WAIT state
• TIME_WAIT Will wait by default 2MSL After time , Before finally entering CLOSED state ;
• In a connection not entered CLOSED Before status , This connection cannot be reused

What would have produced so much TIME_WAIT, How to deal with

The number of threads is really large , A lot of TIME_WAIT

        For example, there are tens of thousands of parallel threads , Generally, the pressure party will disconnect the connection , So a lot of TIME_WAIT. Suggested solution ：

• Distributed piezometers are recommended , Spread the number of threads across multiple machines , Here, you can use the cloud native pressure measurement platform to

jmeter The configuration of will affect TIME_WAIT The birth of

• It is recommended to enable this configuration , Use long connection , This will multiplex the connection to send the request

• Ramp-up Time （ second ）, This configuration indicates how long it takes to generate all threads , It needs to be configured according to the business situation , Avoid generating too many configurations at once , Directly bring down the pressure machine , Accumulate more TIME_WAIT
• Same user on each iteration, stay JMeter in ,user The thread , This option means that each iteration uses the same thread . Its effect is that a single thread iteratively uses the same thread multiple times , Because destroying and creating threads themselves will take up resources , May affect performance test results . Recommended Opening

Linux Recycling is not set for itself TIME_WAIT State connection

        As described in Section 2 ,TIME_WAIT State connection , need 2MSL Time before recycling the port to create a new connection , But the actual Linux The kernel configuration supports fast recycling TIME_WAIT State connection , Configuration can be viewed ：

cat /etc/sysctl.conf

• net.ipv4.tcp_tw_recycle, This configuration represents a fast recycle TIME_WAIT Connect , But in NAT Under the network , Will cause the connection to fail （ Just use NAT）, in addition Linux from 4.12 The kernel version began to be removed tcp_tw_recycle To configure , My machine here is 4.14, So comment out
• net.ipv4.tcp_tw_reuse, Indicates whether reapplication is allowed TIME-WAIT State of socket For the new TCP Connect ; that , When the connection is reused , Delayed or retransmitted packets arrive , How to judge the new connection , The arrived data belongs to the reused connection , Or the connection before reuse ？ This needs to rely on net.ipv4.tcp_timestamps Field . After reusing the connection , The time of this connection is updated to the current time , When the delayed data reaches , The delay time of data is less than the time of new connection , therefore , The kernel can tell by time , Delayed data can be safely discarded .
• net.ipv4.tcp_timestamps, In the case of reusing connections , This configuration can help the operating system to identify whether the new data is old or new

Under the experiment, the modified version has been modified

        Use Baidu to test pressure measurement , Measured under the same thread , Pressed out QPS A substantial increase .