The most ideal automated testing model, how to achieve layering of automated testing

Introduction to automated testing

automated testing (Automated Testing), It refers to the process of transforming human driven test behavior into machine execution . In fact, automated testing often through some testing tools or frameworks , Write automated test cases , To simulate the manual testing process . for instance , During project iteration , Continuous regression testing is a very boring and repetitive task , And the testers work repeatedly every day , There is no growth .

At this point, carrying out automated testing can help testers from repetition 、 Liberated from the boring manual test , Improve test efficiency , Shorten regression test time . Generally speaking , Automated testing is often associated with continuous integration systems （ such as Jenkins） In combination with .

But in the process of automation practice , It is often found that there is a big gap between ideal and reality . The disadvantages of automated testing , Mainly reflected in the following aspects ：

1 Relative to manual testing , Automated testing has relatively high requirements for testers ;

2 Test cases need to be updated according to version iterations , There is a certain maintenance cost ;

3 You can't expect automated testing to find more new BUG, Automated testing can find far fewer defects than manual testing ;

4 The output value of automated testing often lies in long-term regression testing , The role may not be obvious in the short term ;

 

Problems that we hope to solve with the help of automated processes

1 The test time is tight , Manual testing may not cover all , It is easy to miss some boundary conditions ;

2 When there is strong coupling between modules , When testing simply from the page , It is difficult to find problems in depth ;

3 When regression testing , Need to invest a lot of manpower / Working hours ;

4 Implement test tasks that cannot be achieved by manual testing ;

5 By writing test cases , Deepen the understanding of business / Cognition of data , It is helpful to find hidden problems in the next iteration ;

 

Prerequisites for introducing automated testing

Long project cycle , Demand changes infrequently The stability of test cases determines the maintenance cost of automated testing . If software requirements change too frequently , Testers need to update test cases and related test scripts according to changing requirements , Script maintenance itself is a process of code development , Need modification 、 debugging . If the cost is no less than the test cost saved by using it , So automated testing is a failure .

Some modules in the project are relatively stable , And some module requirements are very variable . We can test relatively stable modules automatically , And the big change is still manual testing .

Automated test scripts are reusable If you take great pains to develop a set of near perfect automated test scripts , But script reuse is very low , As a result, the cost consumed is greater than the economic value created , Automated testing is meaningless .

Manual testing of test tasks is difficult to achieve Like stress testing , Big data or a lot of duplicate data testing , Must be supported by automation tools .

 

The ability to do automated testing

• Have coding ability At least be familiar with automation tools / Framework code language , It's best to have a certain coding ability , At the same time, the code logic should be clear , Otherwise, not only the logic of the use case cannot be guaranteed 、 Business and robustness , There is no guarantee of efficiency ;

• Be familiar with the tested system ; Familiarity with the system under test is the minimum requirement for any tester ;

• Master an automated testing framework / Tools ; According to the mastered code , Learn a framework for automated testing , Such as Selenium/Appoum/Robot Framework/Nunit/TestNG etc. ;

• Keep on learning , Good at learning , Know what it is, know what it is ; “ To be behind is to be beaten .”

 

At what stage is the automation use case generally completed

Generally, it lags behind the manual testing stage of new functions , After the manual use case execution is completed or the function is online , Then add the use cases of Automation . Automation does not follow new requirements , It's about measuring the influence of changing things on unchanging things , Never do work that is automated for the sake of Automation .

 

Layered automated testing

Before understanding tiered Automation , Let's take a look at the classic test pyramid .

• UI layer ： Interface automation testing . It can be seen that its value is the smallest , It's closest to the user's real scene , It's also easy to find problems , But it's the most expensive to implement and too vulnerable to external dependence , It is easy to affect the success rate of script . On the whole , Proper interface automation testing is necessary , But there is no need to UI Too much investment ;

• Service layer ： Interface automation testing . Its value is in the middle , It is appropriate to cover most of the main interfaces . This layer requires testers to be very clear about the structure of the system and the scheduling between systems , At the same time, understand the logical relationship of the interface , Otherwise, the interface test code is easy to miss some abnormal scenarios ;

• Unit layer ： unit testing . The most valuable test , But the requirements for testers are relatively high , It's usually done by developers , Otherwise, only pair programming can be used . Generally speaking , Manual testing is the most basic , It can be close to 100%, And for automated testing , It's more like a " Body armor ", Used to protect the main parts of the body . Some people think that the automation rate has improved , You can save manpower , This is actually very one-sided , Because improve the automation rate , It means that more manpower needs to be invested in the cost of maintenance . Because the requirements of the system are constantly changing , Every change will lead to automatic test cases that need to be updated and adjusted .

therefore , What kind of automated testing is good , It should also be analyzed in combination with the above test pyramid . about UI Level of automated testing , Ensure a small amount of necessary mainstream process , Do not use automated testing at this level " Body armor " Become bloated " Spacesuit ";Service Interface automation test at level , Consider covering most of the processes ;Unit Single test at level , Achieve 100% It's the best , Even if there is a change in demand , In general, it rarely affects the existing use cases . Generally speaking , Unit tests can find 80% The defects of .

 

Principles for designing automated use cases

 

The basic principle of

• The scope of automated test cases must be relatively core business processes , That is, the core test points that cover the main functions and modules with high repetition rate ;

• When the test script and the tested code remain unchanged , The result of the test case should be stable , This is very important ;

• Unless it is necessary , Otherwise, any use case should avoid persistent operations , To ensure that the environment is always clean ;

• Once Written, Run Anytime as Desired ;

• Not all manual test cases can be implemented using automated testing , Automated testing cannot replace manual testing , The effective combination of the two is the key to ensure the quality of the project .

• Regression test scenario , The selection of test cases is generally positive , Supplemented by reverse ;

 

Use case design principles

keep Case Independence

Generally speaking , One Test Suite The following contains a group of similar or related Test Cases. And every one Test Case Only one scenario should be tested , according to case Complexity , Different scenes can also be big , It can be a unit test , It can also be end-to-end testing (E2E), Of course, there are also special writing methods, such as workflow testing and data-driven .

 

Case What are the concerns about the independence of ?

First Test Suite Internal Cases It should not affect each other in execution , It means that when we run one of them randomly Case Or run these in disorder Cases when , The test results should be accurate .Suite level and Directory level We should also pay attention to the issue of Independence . When the system is complex , There may be hundreds of thousands Cases Run together ,Robot It will not stipulate Case Order of execution , So in a way, the same level of Cases It's random .

It's typical , How to run and pass the test case in local debugging , Put it in Server On all the Cases When we run together, we will Fail, It could be accidental , In this case, it is most likely due to other Case It's affected by the traces of the world , It's often time consuming to find the root of a problem .

 

keep Case The portability of

Case There are three main considerations for the portability of : Case Dependence on execution environment ; Case Dependence on external devices ;Case Dependence on test objects .

Case Dependence on execution environment Minimize dependence on the execution environment . For example , You're local PC Upper use rf Framework preparation 、 After debugging the use case , Upload to Git, Then your leader may pull your use case and run it locally , Then it was deployed to the continuous integration server .

So when you write use cases, try to avoid using libraries of different platforms or shell command .

Another example , If you modify the source code of the test library because of business needs , At this time, whether it is others in the group or CI The server , It will certainly fail , How can we solve this situation ？

Here are two solutions ：

1 Make the modified library into a test library , Upload to Git perhaps Pypi, The other party can pass pip Install updates ;

2 Use robotremoteserver Make a shared library and put it on the remote host , For details, please refer to the article of the entomologist ;

 

Case Dependence on external devices Sometimes for business testing needs , We will introduce some external devices to assist the test , External devices may continue to be upgraded or replaced , When writing use cases, we need to consider how to use a set Case Better compatibility with these test equipment . For example, the operation of external devices can be separated from the test case , Encapsulated into test libraries or keywords ;

Case Dependence on test objects If the test object is a software platform , Software platforms usually need to adapt to a variety of devices , The hardware configuration of the device may be various ：CPU、 Memory 、 The performance and quantity of components may vary . The dependence on test objects should not only consider the enforceability on different devices , The key point is also to consider the test coverage . Due to the increase of device components, your use case may not be able to cover these components , Or fail to capture a certain personality bottleneck , In this way, the reliability of the test results is greatly reduced .

 

promote Case Execution efficiency

Different case The execution time is far apart , It can last for a few seconds or days . There is no comparability between simple function test cases that take seconds and stability test cases that take days . But I think when we look at a certain group or group case when , We need to pay attention to Case Efficiency of execution .

Whether agile process or sequel, Chengdu pays attention to rapid feedback , Developers can quickly get feedback on test results after submitting code , Testers can perform a wider range of test coverage in the shortest time , It can not only improve the work efficiency of the team , It can also enhance the confidence of the team .

To use the rf For example , When writing use cases, we can improve the execution efficiency of use cases through the following aspects .

1. If there is an inspection of the execution conditions , If the check fails , Exit the execution as soon as possible ; 2. Extract data preparation or environment cleaning into keywords and put them at a higher level ,, You may need to make some combinations when extracting , However, duplicate create and delete operations are not allowed ; 3. Use cases appear as little as possible sleep, Suggest using "wait until ..." Instead of ; 4. You can use the method of executing use cases concurrently to improve efficiency ;

 

Automation use case writing specification

 

Naming specification

Keyword name

The first word should start with a lowercase letter , The following words begin with a capital letter . Such as ：getProjectId, connectDB

 

Constant naming

Constant names should be in uppercase letters , The complete meaning of the constant is pointed out . If a constant name consists of multiple words , You should separate these words with underscores . Such as ：MAX_CHAR_LENGTH

 

Parameter named

The naming standard of parameters is the same as that of methods , Please make the parameter name as clear as possible under the condition that the parameter name is one word .

 

Such as ：{investorName}

Use Tags

RF Provided by Settings Set in tags To manage use cases .Tag The application of is very extensive and flexible , For example, it can be used for use case screening 、 version management 、 Statistical strategies, etc .

 

How to fight tag It looks more convenient ？

• You can type the folder name under each folder tag, In this way, it can be based on tag Run the use cases under this folder separately , It's better to check the test report ;

• Mark some important use cases tag, You can run key use cases alone ;

• If you don't want to execute some use cases , You can type tag, Set not to execute .

 

Give Way case Documented

In considering Coding Style We can set some fixed rules , Everyone just follow this rule , After several practice Coding Style Will tend to be unified . And consider Case Writing like a document requires more subjective initiative .

 

agile development (Agile Development) The development in China has become more and more perfect , With it comes agile testing (Agile Testing). Agile thinking emphasizes human centered , Throughout the development process , Write only the necessary documents or write as few documents as possible , This is also the difference between it and the traditional waterfall model .

 

In order not to cause misunderstanding , It is necessary to insert several features of agile testing ：

• Agile testing should be part of agile development ;

• Agile testing has distinct characteristics of agile development , Such as test driven development (TDD), Acceptance test driven development (ATDD). in other words , Unit testing is the foundation of agile testing , If there are not enough unit tests , Can't cope with the rapid iteration of future requirements , Nor can we achieve rapid and stable continuous delivery ;

• Excellent agile testing is based on automated testing ;

• Agile testing happens all the time , Everywhere .

Requirements design is constantly updated , And documents often can't be updated in time , In this case, how can testers quickly master the requirements and current status of a function or product ?

 

Clear and understandable use case name

In practical engineering , We may create a new directory to store test cases with similar test points . every last Case Each corresponds to a test point , The use case name should summarize the core content of the corresponding test point , So when we are browsing a set of use cases , You can get a general idea of the test content just by using the name of the use case , It is also convenient to find someone Case.

In practical engineering , We may create a new directory to store test cases with similar test points . every last Case Each corresponds to a test point , The use case name should summarize the core content of the corresponding test point , So when we are browsing a set of use cases , You can get a general idea of the test content just by using the name of the use case , It is also convenient to find someone Case.