Perfect Software and Other Illusions about Testing: Lessons Learned from the book by Gerald M. Weinberg

Recently I came across the book called Perfect Software and Other Illusions about Testing written by Gerald M. Weinberg. It was so interesting and meaningful that I’ve decided to share the most important lessons with you.

Lesson 1: The primary goal of testing is to gather information on what a program actually does but not to make decisions (for instance, whether the product should be shipped or not). There may be situations where managers manipulate the testers to give the answers they want, then blame them in case of failure or take the credit if the decision turns out to be right. To escape such cases it is necessary to remember that it is a manager’s job to make decisions. The testers’ job is to provide information (but not all the information) that informs that decision.

Lesson 2: Testing does not improve the product. When we come across a buggy application, sometimes we may conclude that testers are to blame that the given software doesn’t meet our needs or fails to perform some action. That’s not quite certain as testing is not fixing. Perhaps the software was well tested, but no one acted on the provided information. We should remember that quality is a product of the entire development process. Poor testing can lead to poor quality, but good testing won’t lead to good quality unless all other parts of the process are performed properly.

Lesson 3: Testing demonstrates the presence of the bugs but not their absence. No amount of testing will prove the program is correct. All a test can show is that something failed or didn’t fail under specific conditions. The truth is that today managers still think that it is possible to do exhaustive testing to ensure that the product is working correctly. There is an infinite number of tests that can be performed on a particular product candidate. Rather than asking for “all” tests to be performed, managers and testers must understand the risks by sampling. But there is a huge risk of wrong sampling. People may claim that if the software works fine for three users, obviously it will work the same for a hundred of them. This phenomenon is called The Composition Fallacy which assumes that if something is true of some part of the whole then it is true of the whole. But the reality is not so linear. Sometimes you can end up with an explosion when simply joining small systems together.

Lesson 4: People often view the information provided by testing as threatening. Testers may feel motivated to hide bugs just not to make the developers angry since the reported bugs make the developer look bad in the eyes of the manager. We develop an “immune system” to protect us by repressing (denying things that we deem to be unacceptable), rationalizing the unreasonable, projecting our qualities onto other people, displacing blame to absolve ourselves of responsibility, etc. As a result of such defensive actions, managers can often miss hearing the information they need to know to make their decisions.

Lesson 5: Testing is not just banging keys. You can perform some actions, show that all parts of the code have been touched by some tests, but you can’t say that those parts have been thoroughly tested unless analyzing the tests for relevance and comprehensiveness. So you can test by pressing no key but banging keys doesn’t yet mean that you are testing. We should not forget that the number one testing tool is the human brain. One of the testing activities that involve lots of brainpower is a technical review. It takes place when peers sit down together to analyze the good and bad qualities of some technical products, such as code, design, a specification. Technical reviews can be a great source of information about a product that could be used for some purpose.

Lesson 6: Data do not speak for themselves. They are meaningless until someone determines their meaning. Different people give different meanings to the same data. So it is important not to confuse the interpretation with the original intake. When you want some data and you are being presented with test interpretations, use The Data Question: What did you see or hear that led you to that interpretation? You have to keep repeating that question until you reach the level of meaning-free data. Never forget about The Rule of Three (thinking of at least three possible interpretations of the given data) should be applied to make meaning out of the intake. When you receive a bug count as a test result, actually it is useless as a test report unless you use it as a starting point for further investigation. Fancy and tidy reports are not enough. They should require reading, analyzing and interpreting to assign meaning.

Certainly, these are not all the lessons that can be drawn from this book. There are still many important points that you can learn from and use in your practice.

Thank you that you are still here! Feel free to share your thoughts in the comments.