Talk about the establishment of performance testing environment

Yesterday, a classmate of knowledge planet community asked a question ： The performance test environment must be consistent with the production environment 1:1 Is the configuration consistent ？

This question is actually very interesting , Because the point of the problem begins to approach the most basic part of the work .

We often hear all kinds of problems related to performance testing , For example, high concurrency / performance optimization / Various performance indicators and pressure measuring tools .

But as the basis for performance testing activities ： Test environment , But few people mentioned .

This article , I want to talk about some practical experience and personal views on how to build a performance testing environment .

This article is also called ：

• Build a performance test environment , What should we pay attention to ？

• Performance test environment configuration , Must it be consistent with production ？

• Under what circumstances can the performance test environment be consistent with production ？

• How to build a performance testing environment , In order to maintain a balance between cost and value ？

The importance of an independent performance environment

Some students are worried about , Because the configuration of the performance test environment is inconsistent with that of the online environment , As a result, enough performance problems cannot be found in the performance results obtained in the offline environment , Unable to evaluate the capacity of online environment / Stability guarantee brings enough reference . The key points of concern are as follows ：

• Inconsistent configuration , The service capacity cannot be well converted ;

• Inconsistent configuration , As a result, too high concurrency cannot be simulated during pressure measurement ;

• Inconsistent amount of data , Some performance problems will be exposed after being measured ;

In fact, these worries are not unreasonable , From a technical point of view , Only keep consistent with production as far as possible , In order to ensure that the test results have sufficient reference value for the stability of production , It is also an important factor to ensure the delivery quality of performance testing .

but , The performance test environment is consistent with production , Is it really possible to avoid performance problems in the production environment ？ not necessarily ！

The environment configuration is a factor that determines the performance results , But not all factors . Test ahead of time 、 Expose problems in advance , The lower the cost of repair . Our common performance problems , such as JVM GC/ slow SQL/ Deadlock /for  loop / Cache life is not medium , Can be found in advance in the offline performance environment .

The discovery and repair of these problems are not directly related to the server configuration , It is a better choice to be able to solve the problem offline in advance at a lower cost . You can think about , The role of the test environment for functional testing , You can understand the role of independent performance testing environment .

We build an independent function testing environment , It is to fully verify various dimensions before the project goes online , Try to ensure the online delivery quality . But it is by no means that the functional test environment is ok , There will be no problem online . The test environment can be verified by functional testing / Find problems as much as possible by means of automated testing and regression , Production environments also need regression testing / Automated testing for routine inspection , Find and repair early .

In principle, bug As long as it doesn't affect users , It will not cause direct impact and loss . For online bug, What is needed is a more perfect monitoring and alarm mechanism and oncall Repair process , Fix before making a bigger impact .

Empathy , The function of performance testing environment is the same ：

Before publishing on the service line , Try to expose some existing performance problems and repair and verify . Online through full link voltage measurement / Routine inspection / Disaster recovery drill / Monitoring alarms and other means to ensure the stability of online services

.

If there is no offline performance test, it can be directly tested in production , Test for exceptions in performance 、 High availability testing may not be fully implemented ; meanwhile , Repair performance  bug  It also needs functional regression , These all increase the complexity of process management .

therefore , The importance of building an independent performance testing environment mainly includes the following points ：

• Perform performance verification in advance , Find and fix problems ;

• An independent performance testing environment can avoid the impact of other testing actions ;

• Conduct performance tests of various means in advance , Reduce the repair cost of online problems ;

Some suggestions for building an independent performance environment

In fact, performance testing is not a sufficient and necessary requirement in daily work , Don't be confused by the concepts of high concurrency and high availability of major manufacturers in the industry . Most companies , The peak online traffic may not reach 1w Concurrent , Equally spread to all service clusters , Maybe some core interfaces QPS No more than 10.

And most businesses are read-only , Few complex business scenarios are written or read before write , Ascending distribution + Capacity expansion + cache , It can solve most performance problems . Not in Dachang , Or enterprises with rapid business growth , There are few so-called high concurrency performance requirements . Pain points are sometimes assumed by technology , From the perspective of business and cost , It's not a pain point .

The so-called high concurrency , It's a false proposition , In fact, it's just what you personally think of as high concurrency , But it's not high .

Of course ,

Build an independent performance testing environment , stay

Both cost and demand are met

Under the circumstances , You can refer to the following suggestions

：

• Minimize proportionally to production , namely 1 A service 1 Taiwan machine ;

• If it's cloud services , Try to deploy in different zones ;

• The machine configuration type is consistent with the production (cpu type / How many cores and how many G);

• If it's cloud services , Choose to charge by volume , Instead of annual package ( cost reduction );

• Non core services ( The adjustment amount is not large / Low importance ), Multiple services can be deployed on the same machine ;

• The amount of data is reduced , Database isolation and configuration reduction (8C64G The database instance of is enough to meet the needs of daily pressure measurement );

• There should be link tracking / Basic monitoring / Technical facilities such as application monitoring must be deployed , Facilitate problem location and troubleshooting ;

At the end of the paper

The independent performance environment can support the service to expose the existing performance problems and repair and verify them as much as possible before online release .

The stability of online services is guaranteed through full link voltage testing / Routine inspection / Disaster recovery drill / Monitoring alarm and other means to ensure .

Build an independent performance testing environment , It is necessary to fully evaluate the necessity of requirements, hardware costs and management and maintenance costs .

Performance testing can also be layered , Offline environment solves offline problems , Online stability guarantee is solved by other means .

Early needs assessment / Technical proposal review is a good performance risk assessment window , The performance test can also test the left shift .