[performance test] full link voltage test

One 、 What is a full link pressure test

Based on the actual production business scenario 、 System environment , Simulation of massive user requests and data on the entire business chain for stress testing , And keep tuning the process .

For example, the financial system involves many departments , For example, the author once came into contact with a financial product and financial e-commerce project involving related systems when doing performance tests ：NDS（ Core trading system , The bank's money trading system ）,CII（ Public security network verification , Online verification of ID card information system ）,ESB（ E-commerce platform for financial business carried out by banks ）,ICS,BPH（ Other bank card transfer system ）,SMS（ SMS system ）,BLN（ Blacklist system ）, Tonglian （ Third party payment platform , Five elements verification ）, Bonsein （ Third party anti fraud platform verification ）, PENGYUAN （ Third party credit reporting platform ）,ANF（ East Asia anti fraud system ）, Shang Tang OCR（ Face recognition system ）,ICA（ UnionPay comprehensive certification , Four elements certification ）,BDP（ Big data platform inside the bank ）, Network cross platform , Clearing platform ,

There are so many related systems , If it is your own internal system , Can communicate with relevant departments to cooperate , Build a personalized stress testing environment , Conduct full link drill . Besides , If the external third-party platform , During the basic full link voltage test, a baffle server should be built （ Drop the third party api Set the return time randomly within the actual range ） Simulate a third party , Generally, third-party services do not necessarily cooperate with pressure testing , Even with , The pressure test server is also general , It cannot be used as a pressure measurement environment

Two 、 What problem does the full link voltage test solve

The business scenario is becoming more and more complex 、 The availability of the whole business system chain under the impact of massive data 、 The bottleneck of service capability , Let technology serve the business better , Create more value .

3、 ... and 、 Problems faced and solutions

1、 Business model combing

First of all, it should be clear ： Full link voltage measurement is aimed at modern more and more complex business scenarios and full link system dependence . Therefore, we should first split the core business and non core business , Confirm which business scenarios and modules are targeted by the traffic peak ,

Targeted capacity expansion preparation ,

2、 Data model construction

Data construction and preparation , These points should be considered ：

①、 Authenticity and availability of data

An equivalent data package can be completely ported from the production environment , As the basic data of pressure measurement , Then based on the basic data , By analyzing the growth trend of historical data , Estimate the current possible amount of data ;

②、 Data desensitization

Full link voltage measurement based on production environment , One thing to consider is that dirty data cannot be generated , So as not to affect production , Affect user experience, etc , Therefore, data desensitization is required during data preparation ;

③、 Data isolation

Again , In order to avoid dirty data writing , It can be considered to isolate and process the pressure measurement data , Fall into the shadow Library ,mock Objects and other means , To prevent data pollution ;

3、 Selection of pressure measuring tools

The full link voltage test responds to the impact of a large number of user requests , Distributed pressure measurement can be used to simulate user requests , At present, there are many open source tools that can provide distributed pressure measurement , such as jmeter、Ngrinder、locust etc. .

Secondary development can be carried out based on these pressure measuring tools . Considering the large pressure measurement, the return test results will be correct agent Itself causes certain resource occupation , Consider asynchronous uploading , Even the transaction compensation mechanism .

4、 Pressure measurement environment construction

The full link voltage measurement is based on the production environment , It solves the business model and data as well as the selection and development of pressure measurement tools , We need to consider system expansion and risk avoidance , For example, pressure measurement cannot affect the actual production business operation , There are also resource applications .

5、 System capacity planning

Business split and traffic estimation were mentioned earlier , In the system capacity planning stage , First, you should benchmark a single interface and a single service , Adjust configuration parameters , Get a baseline , Then deploy the distributed cluster , adopt nginx Load balancing .

As for expansion , Consider service expansion and DB Resource expansion , And the diminishing effect of service expansion .

As for the case of large flow impact , Consider queue waiting 、 Container lock 、 Long connection callback 、 Transaction degradation and other methods .

6、 Test cluster deployment

Business system capable of full link voltage measurement , Basically, they are distributed system architectures , Service cluster deployment and load balancing .

The problems that need to be solved are ：

①、 Communication problems between services

There are two general ways of communication ： Synchronous and asynchronous .

A synchronous invocation ：

REST（JAX-RS,Spring Boot）

RPC（Thrift, Dubbo）

Asynchronous call ：

(Kafka, Notify, MetaQ)

Strong consistency of synchronous calls , But consider the performance and the transaction processing of the call failure .

Asynchronous call , It can reduce the coupling between services , Improve the performance experience , But consistency needs to be solved （ Distributed architecture has a CAP theory , Interested parties can check relevant information ）.

②、 Load balancing problem

The large traffic impact needs to be distributed evenly to each machine in the cluster , At present, the better load balancing server is nginx, but nginx There seem to be some problems in the deployment of , Our company has encountered the problem of repeated orders before .

③、 The problem of disaster tolerance

One thing to make sure is ： When one or part of the service goes down , Service forwarding can be carried out in time , Instead, the service of the whole system link will not hang up under the chain reaction （ Please refer to my previous blog ： Disaster recovery test ）.

2、 Standards to be followed in disaster recovery testing

①、 Simulation extreme error occurs , Test the business recovery function and business continuity process ;

②、 Discover the potential hidden dangers of the platform , Ensure that the platform can operate normally in case of outgoing line emergencies ;

③、 Under the impact of extreme flow , Sacrifice a small part of non main business functions or a small part of user experience , Ensure the stability of the overall system and the normal operation of main functions （ shunt 、 service degradation ）;

④、 When testing , Need to synchronize analysis logs （ Confirm whether the currently displayed results are due to the effectiveness of disaster recovery test cases ）.

3、 Key points of disaster recovery test

①、 Core principles ： Based on business impact analysis , Improve in an all-round way IT The anti risk ability of the system ;

②、 Focus on two important indicators ：RTO（ recovery time ） and RPO（ Data loss ）;

③、 Do three things well ： The data transfer 、 Business switching 、 Disaster recovery drill and monitoring ;

④、 Implement the operating system 、 file 、 database 、 Apply four recoveries .

7、 Data collection monitoring

Pressure measurement data collection , Need by agent Machine return to Contorller machine , However, too much data will occupy certain resources , Consider asynchronously sending back test results .

As for monitoring , Now there are many excellent professional monitoring tools , such as Nmon、Zabbix, Full link monitoring tool Zipkin、PinPoint And Ctrip's open source full link monitoring tool CAT.

Or it can be tailored to the needs , Secondary development JVM Some self-contained monitoring tools , Achieve real-time and comprehensive monitoring .