Uncover the practice of Baidu intelligent testing in the field of automatic test execution

Last one , Introduces the test activity test input 、 The test execution 、 Test and analysis 、 Research and practice of intelligent test input in the five steps of test positioning and test evaluation , Including abnormal single test generation 、 Interface use case generation 、 Research and practice of action set generation . This chapter focuses on the intelligent practice of test execution . Test execution refers to the set of use cases generated by the test 、 Datasets run on these sets manually and automatically , In essence, test execution cannot improve the level of error detection , But how to execute the test set efficiently and stably is also the key to affect the test effect .

Test execution is intelligentized by integrating data 、 Algorithm 、 Organic combination of engineering and other related technologies , Generally include test case recommendations 、 Test traffic filtering 、 Test task scheduling 、 Intelligent construction 、 Perform self-healing , Excellent research and practice in academia and Industry . The methodology generally includes the selection algorithm based on coverage correlation 、 Data based modeling or a combination of the two . This chapter will be from a number of practical perspectives , Introduce the objectives in related fields 、 Ideas 、 Technical points involved 、 effect , I hope I can give you some reference .

01 Risk based manual use case recommendation

The test execution , Because the code changes 、 Environment and other factors , It is often not necessary to execute all use cases , How to find the use cases that are most likely to expose mistakes in the full set of use cases , It is the research field of this direction , It is also the field with the most performance research .

Manual test case recommendation mainly refers to the recommendation of associated manual test cases through code changes , A key goal is to select high coverage use case combinations , Find problems early . Use code coverage directly to correlate recommendations , Multiple use cases associated with common functions will be redundantly recommended , Inefficient execution , It is not conducive to the timely discovery of problems . Therefore, risk-based manual use case recommendation is introduced , Give priority to recommending use cases according to risk codes , More accurate . First , Digitized metric code , Abstract the code into a syntax tree , Extracted code loop 、 Branch letter, etc 21 An index that can reflect the complexity of design and the difficulty of program development ; secondly , Select the appropriate model for reasoning , Get flawed and flawless predictions for your code , Get the associated use case ; Last , Sort and de duplicate , Select the predicted defective results and the use cases with high importance in defect free as the recommended results . among , Bayesian classification can be used for defect reasoning 、SVM、KNN、 Algorithms such as logistic regression , It can also be transferred to deep learning , Combined with the long-term and short-term memory model （LSTM） And deep neural networks （DNN） Make defect prediction . After the implementation of the scheme , The use case recommendation ratio continues to decline 50% Compress to 20%, The regression days are determined by 3 Days down to 1 God , Recommended findings bug The proportion of use cases continues to increase .

02 A traffic filtering scheme based on parallel coverage

During the test , It is common to use online traffic to diff、 Performance and pressure test , How to select the most appropriate flow for testing from the flow of the line , It is the key of this research .

A traffic filtering scheme for parallel coverage , Is to be able to get from the massive online traffic , Find the smaller traffic set that covers the most test scenarios , So as to achieve the test coverage scenario , Reduce the leakage of problems , The purpose of ensuring the stability of the system . In a traditional setting , It is usually impossible to copy the data of full online traffic to offline traffic , Because the magnitude is too large , So there will be some screening schemes ; The most commonly used is based on the computer room 、 Random sampling of time and other attributes , Try to improve the coverage of sampling , But the randomness is too strong , Uncertain about the coverage of the final business and scenario . The traffic filtering scheme based on parallel coverage is mainly under the guidance of coverage , Minimize the magnitude of traffic , Improve business coverage . There are two main steps ： By analyzing the source logs , Conduct preliminary flow screening , Classic scenes have devices , regional , User properties, etc , Filtering can cover the smallest set of these scenarios , In the first step, you can screen out most meaningful traffic through the log . The second step is to analyze the coverage of generator pressure flow , Use greedy algorithms , Analyze the coverage of different traffic , Cover as many scenes as possible with as little traffic as possible . The above two points , The coverage of traffic filtering can be improved by combining the actual business scenarios and coverage . At present, it has been applied in many product lines and modules , The coverage is lossless , Even promotion 60% On the premise of , Traffic has been cut in half , On the premise of improving efficiency, it also ensures the coverage of traffic .

03 Intelligent construction

Intelligent building is dedicated to static CI Dynamically execute tasks in task arrangement , Realize task simplification 、 Task skip 、 Task to cancel 、 Result reuse 、 self-healing 、 Automatic annotation and other functions , Ensure the efficient and stable construction of test tasks . In daily changes , You may often encounter the following scenarios ：

1、 Changes only modify the log 、 Format or some unimportant functions , Whether it is necessary to return to the full test task ;

2、 Code iterations , Whether it is necessary to perform the same task more than once ;

3、 Whether it is necessary to run the same task repeatedly in the branch and trunk phases .

The traditional approach is to perform full-scale tasks , But it will lead to inefficient test building , High resource consumption .

With intelligent construction, you can dynamically adjust the construction tasks according to the change scenarios , Effectively shorten the construction time , Improve build efficiency . say concretely , Intelligent construction can be divided into analysis and decision-making , Analysis is aimed at the business code base , And this change ( Such as git diff), Use the tool to calculate the characteristic analysis of this change ( Change the number of lines of code 、 Change the specific function of the code 、 Change the call chain information of the code, etc ); Decide whether to execute the task / wait for / restart / Cancel, etc , Make the final decision , For example, business settings business-related white lists , If the analysis features all hit the white list, you can skip the specified task ; Intelligent construction is under the premise of ensuring the ability to expose errors in test cases , Use limited resources to meet the needs of users for timeliness . At present, Baidu has created a strategy development 、 Plug in Party 、 Intelligent building system including business lines , Among them, the strategy developer develops the strategy according to the build system interface specification , And register the policy into the build system , Build a system opening strategy for business parties to use ; The plug-in responds by building policies , For example, cancel the execution of invalid tasks 、 Self healing of accidental tasks 、 Interception of process control tasks ; The business side configures the strategy on the pipeline by building a system ; At present, intelligent construction has radiated 3000+ A module .

04 Algorithm scheduling based on task priority

This direction focuses on how to , Schedule test tasks according to stability and error detection capability .

In the test execution phase , It is difficult to schedule a large number of parallel test tasks with limited resources , Because the user perceived waiting time is the queuing time and execution time of all test tasks , Unreasonable test scheduling will lead to low test efficiency . Therefore, an algorithm scheduling strategy based on task priority is explored to solve the above problems .

In the mobile end test , Commit for a large number of parallel tasks , Create a task priority queue . According to the importance of the task , Waiting time , Resource requirements , A priority formula is generated to reduce the queuing time of key tasks . Based on the formula analysis results , Optimize test task scheduling . When the average queuing time is guaranteed , Reduce the average queue time for key tasks .

For a single case The efficiency of the task , Initially, we grouped tasks based on the duration of offline historical tasks , Unit time coverage and other data , Find the optimal convergence point to predict the optimal stop time of the task , Optimize test task execution duration . In effect , The core product line is covering , The problem is that the quantity is not degraded , The execution time is shortened 10%, However, there is still a problem that the real-time status of the task cannot be perceived , So on this basis , Establish a stop decision model , Through real-time monitoring of task execution , Based on execution time , Number of screenshots , Test control coverage change rate and other characteristics , Real time decision whether the task can reach the stop state , And identify tasks that perform poorly , Early termination , Reduce the time of invalid execution . On the optimization effect , Increase in test coverage 10% At the same time , Reduced execution time 12%.

05 UI Automatic self-healing

App Automate use cases during execution , The context will encounter various unexpected complexities , Such as ,App Trigger the upgrade popup 、 The page loads slowly and a white screen appears 、 page xpath Path change, etc . Existing automatic operation mechanism , Do not have the ability to handle these abnormal boundary conditions , Lead to Case Execution interrupted and failed . These automation stability issues , Increase the maintenance cost of automated use cases , It greatly reduces the cost of automation ROI. We analyze existing automated tasks top Reasons for failure , Use 3 Class generic Case Perform self-healing techniques , promote case Execution stability , Reduce maintenance costs .

1、 Exception pop-up processing , When the automation execution encounters a pop-up window and fails , Use the pop-up detection technology to remove the pop-up window and resume execution . Facing the difficulties of various pop-up windows , Use object detection techniques , Realize the recognition of generalized pop-up window and pop-up window closing control ; Facing the difficulties of inconsistent recommended actions in different contexts , Using the technology of pop-up window classification based on page copy , The pop-up window in different scenarios can be accurately understood , And recommend correct action to remove .

2、 Atomic waiting technology , The asynchronous loading and waiting technology in the automatic execution process is a recognized technical bottleneck in the field , We use vision UI Understand technology , stay Case In the process of execution , Use efficient video streaming for parallel acquisition , And image recognition algorithm for continuous picture analysis , Realize the intelligent judgment of stable and unstable state of the page , To guide the Case Real time intelligent delay waiting and intelligent shortening of waiting time , Guarantee Case Stable and efficient execution .

3、 Universal Case Self healing technology , Provide spatiotemporal context based Locator, Go pop up , Images and other recognition methods are recognized and self-healing in sequence , In particular , Execute history to the moment of success xpath,icon Record the pictures and other information , When there is a failure , Try to retry with the element types that have been successful in history .

Through the above Case Perform self-healing techniques , In our automated execution practices , Realized 51% Of Case Self healing effect , Effectively improve the stability of automated use case execution .

Recommended reading 【 Technical gas station 】 series ：

Uncover the exploration of Baidu intelligent test in the field of automatic test generation

【 Technical gas station 】 On the three stages of Baidu intelligent test

【 Technical gas station 】 Uncover the large-scale landing of Baidu intelligent test