Seven imperceptible truths in software testing

I hope this article will enlighten you .

1, When you are the test leader of a project , Have you ever questioned the project members why they didn't test out a specific bug, Or because someone didn't measure bug And directly blame him ？

2, When you improve the test coverage, do you find the product bug The quantity has not changed ？

3, Have you ever spent a lot of time testing before release and found nothing , And after the release bug But unexpectedly ？

4, Can developers test their own code ？

5,bug Is it true that the later it is discovered, the higher the repair cost ？

6, Have you ever tested software in a way that doesn't follow the routine , And call it " “ Exploratory testing ”？

7, Do you need to pass QA Activities to improve product quality ？

The truth 1： Tests don't find all bug

Unfortunately this is true , There is no test method that can guarantee to find out all bug.

Some testing activities are indeed less efficient than starting directly , So we can not pay so much attention to the type of test , Instead, what we need to do is to select the appropriate test type and use it comprehensively , So as to achieve better results with the least amount of work .（ such as ui If we want to make the automated testing of the system very complex, it will cost considerable development and maintenance costs , But there is no ui Automation , It's unrealistic to rely on human flesh for each round of testing , Therefore, it is more appropriate to use some stable core paths ui Automation to achieve , Balanced input-output ratio , Maximize efficiency with less effort ）

When someone complains about why the test doesn't find some problems, please remind them ： There is really no way to guarantee that certain defects will be found in the test .

We will often repeat the missed test results , Unfortunately , This is a backward idea , It's like saying that the trick is simple after the magic is revealed , It's easy to see through . It is not an effective analysis method to repeat the missed test afterwards .

Never blame a test engineer , They didn't write bug, contrary , They have been trying to find out the defects introduced in the development process . Nothing is perfect , While accepting the reality, the test students also need to remember not to stand flag, Because the test cannot find all bug.

The truth 2： There is little correlation between test coverage and test effectiveness

Yes , You're not mistaken . We have enough scientific evidence to show that , Increasing unit test coverage does not necessarily improve our test suite discovery bug The efficiency of ！ Maybe it's time to focus on the test - related content rather than the amount of code being tested .（ The author's original words here are We already have enough scientific evidence to say that increasing unit test coverage may not necessarily increase your test suite effectiveness in finding defects! The improvement of unit coverage will not improve the efficiency of the test suite in finding defects , Tell the truth , I think the unit test coverage is consistent with the test findings bug It doesn't matter how efficient you are , Coverage represents the degree to which the code is tested , And found bug Efficiency refers to the relationship between time and output , Find out bug High efficiency does not mean good product quality , vice versa . However, see the following description when quoting materials , We can see that the author's proof basically reveals a message , That is the unit test coverage and bug There was not much correlation before the quantity of , In other words, it is not that the higher the unit test coverage , The better the quality of the product , Because good product quality generally means observable bug Relatively few , and bug It has nothing to do with unit test coverage . Here for the sake of preciseness , I will not translate the author's quotation .）

1. *A. Mockus, N. Nagappan, and T.T. Dinh-Trong, “Test Coverage and Post-verification Defects: A Multiple Case Study,” Proc. 3rd Int’l Symp. Empirical Software Eng. and Measurement (ESEM 09), 2009, pp. 291–301.*The correlation between coverage and defects was none or very weak. Moreover, the effort required to increase the coverage from a certain level to 100% increased exponentially.
2. *M.R. Lyu, J. Horgan, and S. London, “A Coverage Analysis Tool for the Effectiveness of Software Testing,” IEEE Trans. Reliability, vol. 43, no. 4, 1994, pp. 527–535.*Qualitative analysis found no association between the defects and coverage.
3. *B. Smith and L.A. Williams, A Survey on Code Coverage as a Stopping Criterion for Unit Testing, tech. report TR-2008–22, Dept. of Computer Science, North Carolina State Univ., 2008, pp. 1–6.*The results did not support the hypothesis of a causal dependency between test coverage and the number of defects when testing intensity was controlled for.
4. *L. Briand and D. Pfahl, “Using Simulation for Assessing the Real Impact of Test Coverage on Defect Coverage,” Proc. 10th Int’l Symp. Software Reliability Eng., 1999, pp. 148–157.*The results did not support the hypothesis of a causal dependency between test coverage and the number of defects when testing intensity was controlled for.
5. *P.S. Kochhar, F. Thung, and D. Lo, “Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems,” Proc. IEEE 22nd Int’l Conf. Software Analysis, Evolution, and Reengineering (SANER 15), 2015, pp. 560–564.*A moderate to strong correlation was found between coverage and defects. However, the coverage was manipulated and calculated manually.
6. *L. Inozemtseva and R. Holmes, “Coverage Is Not Strongly Correlated with Test Suite Effectiveness,” Proc. 36th Int’l Conf. Software Eng. (ICSE 14), 2014, pp. 435–445.*A weak to moderate correlation was found between coverage and defects. The type of coverage did not have an impact on the results.
7. *X. Cai and M.R. Lyu, “The Effect of Code Coverage on Fault Detection under Different Testing Profiles,” ACM SIGSOFT Software Eng. Notes, vol. 30, no. 4, 2005, pp. 1–7.*A moderate correlation was found between coverage and defects, but the defects were artificially introduced. The correlation was different for different testing profiles.
8. *G. Gay et al., “The Risks of Coverage-Directed Test Case Generation,” IEEE Trans. Software Eng., vol. 41, no. 8, 2015, pp. 803–819.*Coverage measures were weak indicators for test suite adequacy. High coverage did not necessarily mean effective testing.

The truth 3： The test workload increases exponentially

Many sources point out that , The tester will find more defects at the beginning of the test activity , Fewer defects are found at the end . There are signs , To find the next flaw , The effort to increase coverage and perform tests is growing exponentially .

In the paper “Test Coverage and Post-verification Defects: A Multiple Case Study,” (A. Mockus, N. Nagappan, and T.T. Dinh-Trong, Proc. 3rd Int’l Symp. Empirical Software Eng. and Measurement (ESEM 09), 2009, pp. 291–301.) Disclosed ： Increase coverage from a certain level to 100% The effort required grows exponentially .

according to “Implementing automated software testing: How to save time and lower costs while raising quality.” (Dustin, E., Garrett, T., & Gauf, B. (2009). Pearson Education.), Description of ： The software reliability model shows that , As more time is invested in testing , The number of defects found per unit time decreases exponentially .

The truth 4： Developer bias

If a developer misunderstands a requirement directly in the development phase , Then the code he wrote was wrong , The same is true for testers .

If development students simply forget to deal with certain situations in the code , For example, specific condition verification , Then he probably won't remember to test this scenario .

To avoid that , Developers can test each other's code , But they'd better not test their code , This is called cross testing .

Without designing any test cases , Development students can still test their own code , In this way, we can avoid some preconceived deviations as much as possible .

Test driven development may reduce the probability that developers forget to do something , But it does not reduce the probability of misunderstanding certain requirements .

The truth 5： Defects found late may not cost more to repair

This is very counter intuitive , Because people may be used to seeing such pictures ：

The figures above may have moisture ,Laurent Bossavit Paris software consulting company CodeWorks Agile methodology experts and technical consultants , He was in github Articles on “Degrees of intellectual dishonesty” It is revealed that the information may have been fabricated .

In an article entitled “Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle” (Menzies, T., Nichols, W., Shull, F. et al. Empir Software Eng 22, 1903–1935 (2017) https://doi.org/10.1007/s10664-016-9469-x) The paper points out that ： There is no evidence that it takes longer to fix defects after the code is put into production .

The paper “What We Have Learned About Fighting Defects” (Forrest Shull, Vic Basili, Barry Boehm, et al., Proceedings of the 8th International Symposium on Software Metrics (METRICS ‘02). IEEE Computer Society, USA, 249. 2002.) in , Author points out ： Fix some non critical bug The cost of is almost constant throughout the life cycle （ Early project average 1.2 Hours , Average at the later stage of the project 1.5 Hours ）.

However, a lot of research has been done to measure positioning errors and repair them bug The amount of work , So what did they ignore ？

regression testing ： We need to do frequent regression before release , In order to verify some important bug It's been fixed , We need to do regression tests on the main process and even most of the logic again and again , This is obviously a huge workload ;
Opportunity cost ： When problems are found late in the project , Many people tend to think bug Schedule to the next version or project , This is likely to lead to late delivery of the project ;
The cost of business reputation ： Enterprises may be fined , Customers may lose money , The user experience is naturally not good .2020 year 12 month , game 《 Cyberpunk 2077》 Removed from Sony store due to many technical problems during sale . Sony offers a full refund . And then , developers CD Projekt Red Declare right PS4 and Xbox Players can refund . On the investor conference call ,CD Projekt Red Express “ Compared with restoring the company's reputation ,《 Cyberpunk 2077》 The cost of repair “ Be of no great importance ”. The company's shares have gone from 2020 year 12 Per share for the month 31 The dollar fell to 2021 year 6 Per share for the month 10 dollar .
bug Not introduced in the code . For example, there are defects in the technical design at the early stage of the project , Or the demand itself is bug, The later you find out about the disaster, the bigger it will be .
therefore , Although the workload of fixing code errors after release may not increase that much , But early repair bug Can save a lot of energy 、 Money and trouble .

The truth 6： Exploratory testing requires processes and documentation

Many people think that if they do some unpredictable operations on the product , For example, randomly input and submit in the form , So they are doing exploratory testing .

In fact, exploratory testing does not mean that you have to do something special about the system or product , It often means that we need to understand how the system really works , And it is carried out simultaneously with the ordinary function test .

In other words, exploratory testing can （ And better yet ） Supported by existing documentation , For example, requirements documents and design documents . The difference here is that the test cases for exploratory testing are not pre written .

Exploratory testing can be scripted , Once you find a problem, you can bug On record , Then the repeatable script can be used repeatedly in the later test process .（ For example, you can use the browser's own recording function when exploring tests , Find out bug Then save the recorded script , For later regression tests ,chrome Now it has its own recording capability ）.

Test cases should still use boundary value analysis 、 Equivalence class partition and other techniques to design . There is no reason to design random test cases , Because they may not be cost-effective or effective in detecting defects .

The truth 7： Improving non quality assurance activities in the process can improve product quality

（ I really don't understand the original statement here and can't find the appropriate data to support it , So simply paste the English version ）

A 2009 study in Brazil (in Portuguese) involving 135 software development organizations had their capacity to identify and fix defects increased by improving their processes. These companies were part of a Brazilian software process improvement program called “MPS.Br,” where they should adhere to a software process improvement model (the MPS Model).

This model has stages, and 58 of these companies were in the first stage, where they were required to improve their Project Management and Requirements Management processes.

While it’s unclear why this happened, we can reasonably expect that projects that identify the right people to participate in the team, training needs, and proper budget and schedule will likely have the people, the time, and other resources to improve quality.

Reward level （ Interesting facts ）： Bermuda plan

ok , It's an interesting story , But unexplained facts , It may not really work .

Bermuda plan is the strategic name for faster completion of the project . Send part of the team to Bermuda （ namely , Remove them from the project ）, The project will be completed faster .

It's thought to be the Brooks' law （Brooks’s law） The response of the （ An observation of software project management , According to the law ,“ Adding manpower to a delayed software project will make the project delivery slower ”）. that , If you remove people , Whether the project should proceed more quickly ？

According to my experience , Every new member of the team will take up an old man's working time 1/3 about , Until the new people gradually increase their productivity .

therefore , Kicking out recent recruits may actually improve productivity .

If there are too many conflicts in the team , Other reasons it can work are ： Removing people who are not aligned with the team's goals may help the team .

If there are too many people on the team , Then communication costs may be large enough to hinder productivity . under these circumstances , Splitting the team may work well （ This is technically different from removing people from the project ）.

otherwise , Removing people will only reduce the team's output in the short term .

in any case , I'm just sharing the Bermuda plan , Because it's always fun to talk about it .

---

Foundation consolidation

Introduction to software testing to project practice ,7 Hours of software testing quick start course from small white-collar to white-collar workers

Practical course of software testing 《 Learn to drive 》APP Actual software testing

First public - dark horse headline software test actual project Full version

The basics of software testing take you hand in hand Excel Implement the management interface use case

Fundamentals of software testing - Teach you how to understand the test environment project deployment

Software testing tutorial Charles Test practice of packet capture tool

Software test engineer must MySQL database ,mysql Elaborate on the system + Practice after class

Advanced course of software testing wechat applet testing practice — The whole net starts