Do you know the limitations of automated testing?

Catalog

Preface

1、 The benefits of automated testing

2、 Limitations of automated testing

3、 Testing a single page application

4、 Some combination of browser extensions .

5、 Reconcile developer and user expectations

6、 Get useful feedback

---

Preface

When selling software , Nothing is more important than the quality of the software itself . If your software doesn't perform what users expect , Then your business will be struggling . This means that it is essential , You must [1] Know what the user wants the software to do , And know that your software will do this .

1、 The benefits of automated testing

Before considering limitations , Let's first consider how automated testing    do   Help us meet user expectations . in many ways , Your development team is your first and most important user . Automated testing makes these “ user ” Can very clearly describe how they expect the software to work . in fact , When you can describe how the software should behave with such precision , The computer can check , You can check , The behavior of the software is described in    Fast   and    Over and over again . These are the huge benefits of automated testing , And these benefits have driven the commercial software industry to take automated testing seriously over the past few decades .

                        

2、 Limitations of automated testing

however , Even if you simply confirm that the software behaves as expected , Automated testing also has some limitations . Because expectations are encoded in the test suite , So there is the first limitation . in the majority of cases , The developer who implements this function is the one who writes the test . For developers , In addition to the scenarios he considered when building the functionality , It is difficult to consider other options . This means that even possible marginal situations may not be considered , Not to mention testing . 

There is another reality ： By definition , Any software of sufficient value will be paid for by others , For anyone's mind , Tracking each of its active parts is inherently too complex . therefore , It is impossible to adequately express this complexity in the form of automated test suites . Even with a powerful automated test suite , Users will also find errors , Because no development team can expect every possible combination of states or conditions that users may encounter . Besides , No development team can foresee a myriad of user expectations for the application （ It's usually different ） It is expected that the software will and should run .

3、 Testing a single page application

In the build “ Single page application ”（SPA） when , These restrictions will be even more serious . In traditional Web In the application , A large part of the computing takes place on the back-end server . And then the result （HTML,CSS and JavaScript form ） Returned to the user and processed by the browser . When the core complexity of the software is concentrated on the server , Can be in a highly similar environment （ for example , On some continuous integration servers ） Test the software . however , A single page application unloads the core computing into the user's browser . expect   The development team    Understand the details of their server environment with great accuracy , But it's hard    expect   Details of your various user browsers . They can use one of many different browsers , Running one of several browser engines on top of one of the few operating systems , And configured on one of a plurality of devices in a plurality of possible ways

4、 Some combination of browser extensions .

In spite of all Web Software City    With    Various possible conditions （ And other conditions ） A particular combination of , however SPA By calculating... In certain specific environments   The increased complexity introduced by the main logic out of millions of possibilities    It strongly limits the efficiency of automated testing . Your team can run automated tests in a specific environment , You can also try running tests in multiple environments . however , The latter option linearly increases the time cost of the test suite , It also increases the complexity of the test suite , Because now it is required to have enough versatility , Can run in a variety of environments of your choice .

                    

5、 Reconcile developer and user expectations

Final , If no actual users use your software and confirm that their behavior is in line with expectations , It is impossible to confirm whether your software meets the user's expectations . Of course , This is how you provide software to users . You give them the software , They can run the software in any specific environment they have , Then they either live up to their expectations , Or they didn't meet their expectations . 

exactly , In the simplest form , Mistakes are just unmet expectations . Sometimes , The development team developing the software will consider these expectations , But in some cases the implementation is wrong . But sometimes , The development team has never considered this expectation . however , From the customer's point of view , There is no difference between . They expect software in a way , Structure in a way , The act of providing certain information in certain places , In fact, there is no . 

The development team usually organizes this feedback based on original considerations . If the team initially wanted the software to run as intended , Then the problem reported by the customer is an error . If the team doesn't want the software to run as expected , Then the problem reported by the customer is the function request . however , In any case , Both developers and product managers need the same core information ：

• What do users expect  

• What is the actual situation  

• Why unmet expectations become a problem  

Of course , As every software development team knows , Users simply do not or cannot provide such clear feedback . Most users don't even report unmet expectations , And those who often provide little detail and do not pay attention to the point of view .

6、 Get useful feedback

There is no better feedback than the structured feedback provided by real people who actually use your software . and , There is no better way to collect this feedback than a population test . This is because it is not enough to provide the software only to real users who want to run the software in a real environment . These users must also report useful details to you , The way software behaves , Their expectations , reason , place , Conditions and other conditions . 

Automated tests can provide your development team with useful details when they fail , But they can never provide expectations . Your development teams will have to code themselves as tests . Your real users are always in the real world “ test ” Your software , But they rarely give you information about their expectations , Useful details about specific situations and how the software behaves . These alone are not enough .