Load test practice of pingcode performance test

This paper is written by  Pingcode  Wu Shichao shares

What is performance testing

Performance testing is a form of software testing , Focus on the performance of the system running the system under a specific load . This has nothing to do with finding software errors or defects . Different types of performance tests are measured according to benchmarks and Standards . Performance testing provides developers with the diagnostic information they need to eliminate bottlenecks .

The type of performance test

There are two main performance testing methods ： Load test and pressure test . however , There are other types of test methods that can be used to determine performance . Some examples are as follows ：

• The load test
  ： Usually, the purpose of load testing is to understand the behavior of the system under a specific expected load . This load can be the expected number of concurrent users on the application , The application executes a specific number of transactions within a set duration . This test will give the response time of all important business critical transactions . The database will also be monitored during the test 、 Application server, etc , This will help identify bottlenecks in the application software and the hardware where the software is installed

• Pressure test
  ： It is usually used to understand the capacity limit in the system . This test is conducted to determine the robustness of the system in terms of extreme loads , And help application administrators determine if the current load is much higher than the expected maximum , Whether the system can operate fully .

• Soak Test
  ： Also known as durability test , It is usually used to determine whether the system can withstand the continuous expected load . During the soak test , Memory utilization is monitored to detect potential leaks . Equally important but often overlooked is performance degradation , That is to ensure that the throughput and response time after a long period of continuous activity are as good or better as at the beginning of the test . It essentially involves putting a lot of load on the system over a long period of time . The goal is to find out how the system behaves under continuous use .

• Peak test
  ： By suddenly increasing or decreasing the load generated by a large number of users , And observe the behavior of the system to complete . The goal is to determine whether performance will be affected , The system will fail , Still able to deal with drastic changes in load .

• Breakpoint test
  ： Breakpoint testing is similar to stress testing . Apply incremental loads over time , At the same time, the predetermined fault conditions of the monitoring system . Breakpoint testing is sometimes called capacity testing , Because it can be said that it determines the maximum capacity that the system will perform according to its required specifications or service level agreements . The breakpoint analysis results applied to the fixed environment can be used to determine the best expansion strategy according to the required hardware or the conditions that should trigger the horizontal expansion event in the cloud environment .

• Configuration testing
  ： Instead of testing performance from a load perspective , Instead, create tests to determine the impact of configuration changes to system components on system performance and behavior . A common example is trying different load balancing methods .

• Isolation test
  ： Isolation testing is not unique to performance testing , It involves repeating tests that cause system problems . Such a test can usually isolate and confirm the fault domain .

And our test practice this time adopts load test .

Basic process of performance test







The test environment is to set up software 、 Hardware and network to perform performance tests . Use the test environment for performance testing , Developers can use the following seven steps ：

• Determining the test environment ： By identifying available hardware 、 Software 、 Network configuration and tools , The test team can design tests and identify performance test challenges as early as possible . Performance test environment options include ：

a.  Subset of the production system , There are fewer low specification servers

b.  A subset of production systems with fewer servers of the same specification

c.  Copy of production system

d.  Actual production system

• Determine performance indicators ： In addition to determining the response time 、 Throughput, constraints and other indicators , Also determine the success criteria for performance testing , According to the regulations of the team in the project .

• Plan and design performance tests ： Considering the variability of users 、 Performance test scenarios for testing data and target metrics , You can create one or two models .

• Configure test environment ： Prepare the test environment machines and tools needed to monitor resources .

• Develop test scripts

• Execute test plan ： Running performance tests , Monitor and capture the generated data .

• analysis 、 The report 、 To test ： Analyze the data and share the performance conclusions . Run the performance test again with the same parameters and different parameters .

In the follow-up practice, we will basically follow this basic process to develop .

Performance testing tools

IT  Teams can use various performance testing tools , It depends on their needs and preferences . Some examples of performance testing tools include ：

• JMeter
  ： It's a  Apache  Performance testing tools , It can be done to  Web  Load test with application services . JMeter  Plug ins provide flexibility in load testing , And covers graphics 、 Thread group 、 timer 、 Functions and logic controllers . JMeter  Support integrated development environment  (IDE), For browser or  Web  Test the application , And for load testing based on  Java  Command line mode of the operating system .

• LoadRunner
  ： from  Micro Focus  Development , Used to test and measure the performance of applications under load . LoadRunner  It can simulate thousands of end users , And record and analyze the load test . As part of the simulation , The software generates messages between application components and end-user actions , Similar to key click or mouse movement . LoadRunner  It also includes cloud oriented versions .

• NeoLoad
  ： from  Neotys  Development , by  Web  And mobile applications to provide load and stress testing , Designed to test applications before release to achieve  DevOps  And continuous delivery  IT  The team can use this program to monitor  Web、 Database and application server . NeoLoad  It can simulate millions of users , And perform tests internally or through the cloud .

This time  PingCode  The testing tool used in load testing is  Jmeter.

Performance test indicators

Many performance indicators or key performance indicators can help IT The mission evaluates the performance of the current system .  Performance indicators usually include ：

• throughput ： How many information units does the system process in a specified time .

• response time ： The amount of time elapsed between the request entered by the user and the system beginning to respond to the request .

• bandwidth ： The amount of data that can be moved between workloads per second , Usually refers to through the network .

• Per second  CPU  Interruption times ： The average value is taken . Refers to the number of hardware interrupts received and processed by the processor per second .

• Memory usage ： The amount of physical memory available to processes on the computer .

• Disk usage ： The amount of time the disk is busy executing read or write requests

These and other indicators can help IT The team performs many types of performance tests .

In this load test practice, our main concern is ： throughput 、 response time 、 Memory usage and disk usage .

A non systematic performance test practice

Test type

The load test ： Test a specific API At the maximum number of concurrent users , Continuous service 5Min, Get the TPS and RT value .

Test range

Because of our API quite a lot , The test cannot be completed in a short time , We take priority testing  「 Accessibility 」 high API.

The final decision will be API Near service 6 The number of visits per month is in reverse order 10 individual ：

• Sub products involved ： Project , Testhub , Wiki , Insight , Goals , Flow 6 Sub products .

• Basic services ：Typhon  service

Test purpose

As Pingcode The first performance test of the project , The purpose of this round of testing is defined as follows ：

• Ensure the reliability of the system in the case of multiple users ： By implementing different performance test scenarios , Evaluate and verify the reliability of the core functions of the system under high load .

• Provide a round of system performance test , Provide baseline data for the next round of production environment performance testing .

• Evaluate the rationality of system deployment resources ： Continuously collect and analyze performance test process data , Evaluate and make suggestions on the system resources required for the next system deployment .

• Optimize and solve system performance problems ： During the test , Optimize the parameters of the system and solve the system performance problems found .

Test index standard

• According to the number of online users provided by operation and maintenance , To calculate the users of the production environment  TPS, Pass the test of benchmark environment , To verify the real  TPS  Whether it meets the requirements of the production environment  TPS  demand , And whether our operation and maintenance structure and resource use are reasonable .

• Guaranteed at  CPU  The utilization rate is less than 80%, Memory less than 80%, And there is no mistake  Http  request .

Test practice

Testing tools

• use  Jmeter + Grafana + Influxdb  To complete the performance test .

• Jmeter GUI  GUI tools ： In order to facilitate development and debugging , Use the graphical interface to write the required test plan .

• Jmeter NO-GUI Pattern ： Support large load test scenarios 、 Improve the efficiency of script execution and facilitate the subsequent configuration of scripts to  Jenkins  To achieve continuous integration on , Make automated testing .

Environment building

• install  Java  Of  Jdk, Version follows  Jmeter  Version corresponding .

• install  Jmeter  Tools

Jmeter  The value of the number of passes in the center line is based on

In practice  Jmeter  Testing practice , We will simulate the multi-user operation mode through the number of threads , But what is the appropriate value for this thread number ？

We can estimate the number of outgoing lines through the following two points ：

• First of all, understand the number of users 、 Number of threads 、TPS

 Because the response time in the actual test will certainly not always be  100ms. So usually , None of the above ratios will remain unchanged , But as the number of concurrent threads increases , There will be a trend relationship .

Upper figure , We can according to online users 10000, Through the industry benchmark concurrency 5%, Calculate the ideal TPS by 500tps; Or given directly according to the product 500tps As an ideal TPS. And this ideal TPS, Will be calculated Jmeter The benchmark of the number of threads .

• adopt TPS Calculate the number of threads of the press according to the benchmark

The calculation formula is as follows ：

 actually , For pressure tools , As long as there is no error , We care about  TPS  And response time , because  TPS  What is reflected is the processing power corresponding to the server , At least the number of stress threads , Don't care .

We will do it later in the actual operation , According to the calculated number of threads, we can get  API  The actual request  TPS  and  RT, And the actual RT Situation and ideal TPS and RT comparing , To determine the current API Performance .

Jmeter  Project directory structure

 At present, we need to pay attention to   

bin

   In the catalog   

templates

   The folder and   

jmeter.properties

   File can .

templates  Folder

 among   

templates.xml

   The document is used to specify the list of test plan templates to be provided , Corresponding  Jmeter GUI  The interface is as follows ：



jmeter.properties  file

 among  template.files  Configuration items are used to configure  Jmeter  The path of the template to be read , But it must be ensured that  Jmeter  Under the installation root directory of , In the later development, this configuration item will be modified for convenience of storage .

Project preparation and configuration file modification

In order to facilitate the collaborative development of many people , We use code version management tools  Github  To manage the required templates and test plan files . For the convenience of the following , Unification is called   

pc-perf

   project

clone  Git  Warehouse

Get into  Jmeter  Of  bin  Under the table of contents

cd jmeter/bin
git clone [git Project address ]

perf-test  Catalog

• common Folder ： It is used to store the test plan template and template configuration file  templates.xml.

• Sub application directory ： And  common  The peer directory is the sub application directory , As the goal  Goals、 project  Project  etc. . This directory is used to store the corresponding  .jmx  Test plan file .

modify  Jmeter  Configuration file for  jmeter.properties

template.files = /bin/pc-perf/common/templates/templates.xml

JMeter  Template development

adopt  JMeter GUI  Create a new test plan

add to  setUp  The thread group component is used to obtain login information



stay  setUp  Add user parameter processor under the line layer group

 The user parameter processor is used to store some user level public data . for example ： Login name 、 Passwords and so on .

stay  setUp  Add under thread group  BeanShell  Preprocessor

 The  BeanShell  Preprocessors are used to globalize common attributes , Global variables can be called in multiple thread groups . for example ： Request protocol 、 Request port 、IP、Host  as well as  Influxdb  Links, etc .

stay setUp Add under thread group http Request sampler



Add Http Request the contents of the sampler

api  Information about , If the request header 、IP、Port、Host、URL  The information of the request body can be obtained from  api  Get... From the document . In addition, you can also use the browser's own  copy fetch  To obtain specific interface information . This time we use the second method to obtain  api  Information .

• Open browser developer tools

pingcode After logging in, the project will be redirected to the front page of the project ,sigin The request will flash away , So you need to check it in advance Preserve log( Keep logs ).

• obtain signin Information 

 We go through  Copy as fetch, You can get the following code ：

fetch("http://at.pingcode.fun/**/signin",
 {
 "credentials":"include",
 "headers":{
 "accept":"application/json, text/plain, */*",
 "accept-language":"zh-CN,zh;q=0.9",
 "content-type":"application/json"
 },
 "referrer":"http://at.pingcode.fun/signin",
 "referrerPolicy":"no-referrer-when-downgrade",
 "body":"{\"signin_name\":\"**\",\"password\":\"**\",\"captcha_code\":\"\"}",
 "method":"POST",
 "mode":"cors"
 });

From the above information to  Http  Request in sampler



stay  Http  Request to add  Http  Header manager component

• You can use the previous step in the developer tool of the browser  Copy request headers  Get the complete request header information

• It comes with components “ Add... From the clipboard ” function , One click configuration request header information 



stay  Http  Request to add  JSON  Extractor

This extractor is used to get the information returned by login .



 There are many ways to extract data from response messages , See ：

  Jmeter Extraction method of response message in . 



stay  Http  Request to add a post debug handler in the component

This component can be viewed during debugging  JSON  The information content obtained by the extractor .





stay  Http  Request to add  JSR223  Post Processors

This component is used to format the response result . For example, get the corresponding users 、 The team 、token Wait for information and put it into  Jmeter  In local variables of .

• Use Javascript Language to write scripts 

• Extract users 、 The team 、token Information

//  here loginResponse_1 It's from JSON The debugging results of the extractor can be obtained
var loginResponseString = vars.get("loginResponse_1")

var loginResponse = JSON.parse(loginResponseString)

log.info("loginResponseString:"+ loginResponseString)

//  Set variables in this thread group
vars.put("user", JSON.stringify(loginResponse.user))
vars.put("team", JSON.stringify(loginResponse.team))
vars.put("access_token", loginResponse.access_token)

stay  Http  Request to add  BeanShell  Post Processors

This element is used to put token And other information is set as a global variable for cross thread requests .

 Write code ：

//  Will the user 、 The team 、token Information is declared as global variables , Used for later thread groups
${__setProperty(user, ${user})}
${__setProperty(team, ${team})}
${__setProperty(access_token, ${access_token})}

stay  Http  Request to add a view result tree listener in the component

This component is used to view specific  Http  Request details .





Run test plan

here , Basically, a complete request is completed , Then we start the test plan , View the run results .



 You can also view  JSON  The value of extractor ：

JSON  The extractor will treat the response result as an array , because   

data.value

   It's a single object , So the processed value is   

loginResponse_1

  

Optimize parameter management

•  Modify user parameter processor

 Store the user name 、 password 、 Request mode 、 request host And request port , Used in other components or script processors ：

Local variable calling method ：
Use... In components ：${key}
Use in script ：vars.get(key)

• modify BeanShell Preprocessor

 Usage mode ：

How to call global variables ：
Use in components ：${__P(key)}
Use... In scripts ：props.get(key)

Create thread group A

At present, a test plan is stipulated as 1 Sub products , A thread group contains a specific sub product  API  request . For better statistical information , Can be  API  Name as thread group name and request name .

 Among them, we will use the above calculation formula to calculate an appropriate number of threads , For example, the number of threads is 240.

that Ramp-up How much time should I take ？ We can refer to

 ramp-up  Definition and value  

To make sure . such as ： With 240 Number of threads , ramp-up for 240.

In thread group A Add in http Request element



stay Http Add a request header manager in the request component

 In the request header  host  as well as  token  Can be dynamically introduced from global variables .

stay  Http  Request to add a view result tree component in the component

 Set the file path to write the result in the red box , At present, the result is written to jtl Format file , For post generation html Test report .

stay  Http  Request to add a back-end listener to the component

This component is used to send  Influxdb  To transmit data



stay  Http  Request to add an aggregate report listener in the component

This element can be used in Jmeter View aggregate results .

 Here followed by transmission to influxdb, The final presentation is grafana There is little difference between the data in .

Grafana Information

application  Is the sub app name ,transaction  by   Single  API  request .

Save the current test plan as a template

Save the template to the previous Git The warehouse common Of templates Under the table of contents

 adopt  VsCode  modify  templates.xm  file , Configure the newly saved test plan into the template list ：

name： Template name , Used to display in the template list
fileName:  The path of the test plan to be configured as a template
description： Template description
parameters： Template parameters , You can use the template UI Page to page test plan .jmx File dynamic transfer parameters .

modify  pingcode-perf-template.jmx  file , Receive parameters from the template

adopt [=key] Receive template parameters in the form of

Development sub application  API  test plan

By modifying the imported  JMeter  Template , To complete other  API  test .

Click the template toolbar , Select the template to use



Through the template interactive interface , Enter the corresponding test information



To write  API  test

Due to the different regulations of the same sub application at present api Under the same test plan , Therefore, ensure that the thread group execution mode of the test plan is serial ：

 According to the reality  API  Information modification thread group A Information , Create different thread groups in turn to complete other tasks of the sub application  API  test .

adopt  Non-GUI  Way to execute all test plans

Command line usage ：

jmeter -n -t <testfile> -l <logfile>

Example ：jmeter -n -t test.jmx -l test.jtl
Explain ： Run on the command line test.jmx Script and generate test.jtl Log report .

-h  Print usage information and exit
-n  Not GUI Pattern  ->  In Africa GUI Run in mode JMeter
-t  The test file < Parameters > -> To run jmeter test (.jmx) file .
-l  Log files < Parameters > -> Generated log files
-H  Set the... To use JMeter proxy server  
-P  Set the... To use JMeter Proxy port

The results are as follows ：

 After executing all test plans , adopt  Grafana  see  API  Request situation , according to  Influxdb  data , Statistics of all  API Reality  TPS  and  RT  data .

Come here , Our whole load test is finished . Next, we just need to put the  API  Reality  TPS  and  RT  Statistics come down , Then analyze the data , We can get these  API  Which ones need to be optimized , So as to achieve the purpose of our load test .

Information

• wikipedia ：
   https://en.wikipedia.org/wiki/Software_performance_testing 
  

• Jmeter：
   https://jmeter.apache.org/ 
  

• LoadRunner：
   https://en.wikipedia.org/wiki/LoadRunner 
  

• NeoLoad：
   https://www.tricentis.com/products/performance-testing-neoload/