Web service performance monitoring scheme

One 、 Use of local testing tools

advantage ： 

1. Trigger multiple requests at once , And each parameter can change dynamically , It is helpful to make up for the insufficient coverage of single unit test scenarios of interfaces
2. It can complete the qualitative test of interface performance , Make preliminary selection for interface performance test .
3. Developers share a development environment database , It is convenient for you to have an intuitive and comprehensive understanding of all microservice interface tests

defects ：

1. Local machines vary in performance , And the number of concurrent threads cannot be controlled . As a result, the response time results of interface test fluctuate greatly , The result data of quantitative analysis on the degree of interface performance is not reliable .
2. It is impossible to make sequential link calls to test the overall performance of a module's interface .
3. When an interface parameter depends on the result of the previous interface , To ensure a desired interface hit rate through regular expressions , More troublesome .
4. It is impossible to measure the actual operation of the interface after it passes through the gateway .

Conclusion ：

      Local testing tools are only applicable to developing interfaces at interfaces , Complete the interface unit test and preliminary performance test locally , Want to complete the quantitative analysis of interface performance and the full link analysis of interface calls , We have to rely on other tools .

Two 、 Development environment interface performance testing and monitoring

        Due to the instability of the local environment and the short board of the local interface test tool , So we need a stable environment to deploy other tools to make up , This environment is preferably a development environment .

Request sending and monitoring tools jmeter

    The request sending tool of the development environment uses jmeter, It can make up for the shortage of local testing tools , Such as ：

1. It can generate a continuous and stable number of concurrent messages sent to an interface , Measured interface TPS Value in steady state .
2. It can realize the sequential link call of the interface , Test the whole module TPS value .
3. You can get the request parameters according to the file , Improve the hit rate of the interface .

Request link analysis tool skywalking

    In the course of the request , If an interface is found TQS Couldn't get on , You can use skywalking It takes time to analyze each link .

 

Prometheus+Grafana Monitor performance metrics

Jmeter monitor

      You can see it clearly Jmeter The total number of requests 、 Number of failures 、TPS Change trend and other information during interface debugging

 web Application monitoring

      Main monitoring jvm Usage and garbage collection    

Mysql monitor

    monitor mysql Connection number 、QPS、 Slow query, etc

 Redis monitor

      Main monitoring redis Memory 、 The Internet IO、 Of each database key Value usage .

 web Apply performance analysis tools Arthas

   Prometheus+Grafana It can be done to web Monitoring to jvm Usage and GC situation , But to analyze the specific reasons, we have to rely on Arthas.

  As per cpu Usage analysis thread state

 

 

  3、 ... and 、  Summary of interface test steps

Local testing ：

Complete the unit test through the local interface test tool , And the preliminary selection of performance test , If the interface test results cannot meet the query 300 Time /s, Additions and deletions 100 Time /s, Then debug the development environment .

Development environment debugging ：

1. Deploy in the development environment jmeter、skywalking、Prometheus、Grafana And other monitoring tools and configure relevant monitoring parameters .
2. Use jmeter Make concurrent requests for interfaces that need to be debugged in the development environment , Observe each middleware and web Service cpu usage , Find out the performance bottleneck of the link .