Test work summary - performance test related issues

1. HTTP Communication principle

1. Browser based on url The domain name resolves url Address
2. be based on IP Address and server
3. Client construction HTTP request , Contains header information and communication data
4. be based on HTTP The network protocol transmits the request to the corresponding interface of the server
5. Interface generates response structure
6. Response results are based on HTTP Please Return by the Way You Came
7. Front end based rendering , Display the generated results

2. Performance test requirements

To develop , Provided by operation and maintenance team

1. Capacity planning capability
2. System risk identification
3. System bottleneck identification
4. Performance tuning guidance

3. Performance testing and analysis optimization （ Technology tree ）

1. Code tuning
2. operating system os
3. database
4. Storage
5. application server 、 middleware
6. The Internet
7. Pressure measurement
8. Monitoring analysis tools
9. cache
10. queue
11. web The server
12. Containerization 、 layout 、 The Internet
13. Big data technology

4. System performance problems

1. Storage problems Mysql、redis、Memcache
2. Middleware issues Tomcat、httpd
3. Message queuing problems kafka、ActiveMQ
4. Call link problem Internal dependence 、 Third party reliance
5. Details of the problem
   1. Redis The number of memory connections is full
   2. Nginx Connection refused
   3. Core dependency service exception
   4. Non core dependent service exception
   5. soft / Hardware degradation scheme failed
   6. Message queue backlog

99. Problem cases

1. A pressure measurement ,CPU Fill it up directly

1. Solution steps
   1. top Check that pid Occupy CPU Higher
   2. top -H -p pid Look at this. pid Information
   3. Pstack pid View stack information
   4. Trace -p pid View the corresponding stack information code , Specifically analyze which codes occupy CPU
2. Location problem
   1. Code logic problems
   Synchronous resolution interface , Match the returned content in a regular way , Because the returned content is too large , Lead to CPU soaring

2. A pressure measurement , System CPU And other indicators are normal , Occasionally, the request is very high

1. Log data ：
   [Full GC(Ergonomics)[PS Young Gen: 944K -> 890K(2048K)][ParOldGen: 7129K->7129K(7168K)] 8074K->8019K(9216K), [Metaspace: 3357K->3357K(1056768K)] 0.1213761secs][Times: user = 2sces, sys = 0.00, real = 2secs]
2. Location problem :
   1. JVM GC problem : Full GC Stop the world
3. resolvent
   1. Reduce FullGC Time , Decrease in the elderly generation

3. A pressure measurement , CPU/ Memory / The network is good , But the response takes a long time , Monitor disk view IO abnormal , The trace discovery log level is Debug

1. Location problem :
   A lot of log printing is a drag on performance
2. resolvent :
   Set the log as Info Level

4. A pressure measurement , CPU/ Memory / The Internet / Disk index is good , But the response takes a long time , see Nginx journal , Find out request_time( Full interaction time ) A long , but upstream_response_time(Nginx Create a request to the back end ) The actual time is short

1. Possible problems :
   1. The load of the generator is high , Unable to process connection to backend ;
   2. The network environment of the generating machine is quite different from that of the pressed machine ;
2. terms of settlement :
   1. Shut down other services you don't need , Reduce the pressure generating machine load ;
   2. Keep the network environment of the generating machine consistent with that of the pressed machine ;

5. A pressure measurement , The same number of concurrent paths , Good early performance , Post database CPU soaring

1. Possible problems :
   1. Generate large amounts of level data , Data growth leads to performance loss ;
   2. The pressure measurement data is unreasonable , This causes the same device to be associated with multiple users , There are no restrictions on the service in Inquire about ;
   3. Unreasonable paging , Number of pages not done limit, This leads to full query of new data in the database ;

6. A certain stability , A lot of concurrency , Good early performance , Shard caching simulates a large number of database penetrations after a single point of cache failure

1. Possible problems
   1. Cache unreasonable fragmentation strategy , Use the mode of division , Cause a large number of caches to expire at the same time
   2. Unreasonable load balancing algorithm
2. resolvent :
   1. Uniformity Hash Solve the cache problem

7. A certain stability , If HTTP The inlet flow is only 100% QPS, But downstream RPC Hang up

1. Problem location
   Medical list data , for Loop call downstream to solve , Resulting in a request hundreds of times larger
2. Solution :
   Use batch The interface reduces pressure ( It may bring hidden functional problems )