How Much Testing is Enough?
This is a question that I have been asked when conducting my training. In a conference event about exploratory testing when Paul Holland was keynote speaker, I was talking to him, I also asked him “In Context-Driven Testing, how much testing is enough?”. His reply was “How much time you give me to test?”. It sounds like interesting. We acknowledge that “exhaustive testing is impossible”, this means it’s very hard to say that the testing is enough. There are two stories help me having a new look about “enough testing”, let consider both of them
The 1st Story:
Few years ago, one of the teams that I have known was testing a health-care application. This application has been tested over some years. For that version, they just enhanced their product core by altering a bit its workflows. The modifications were rather simple, so it was easy to identify all relevant test cases needed for the regression testing (the term, they used as a part of their testing process). The testing was carried out with only one day and was ready to go live in the day after. Unfortunately, after only few days, their clients blamed for a bug escaped. They said, “your testing did not cover this case, it is not enough”. Right after then, the cause was identified because a new alternative flow was not reached & tested when it had not been clearly mentioned in any notes or any requirement document changes.
The 2nd Story
Recently, another team I have also known performed the testing for an application which has been built by & used by the governors. The application was very big and complicated while the time was given to test is rather tight. One week prior to the release, the team made a report on their testing results. It indicated that the testing was not be able to cover as many angles of the application features as expected. In addition, there were many bugs still remaining. However, the key stakeholders still decided to let it go live. A few month later, the application was still evaluated well and has been fulfilling the needs.
Two these stories are opposite. One application was tested for many years. With only one bug escaped, however, we said that “the testing is not enough”. Another remains many bugs there, testing was not shining all product’s angles as expected, but it still has been accepted, so the testing is “good-enough”? And, what is “enough” should be understood here?
After researching and talking to some testing experts & practitioners, here is a list of opinions and comments on the term “enough”
- The testing is enough when it covers all requirements – I don’t like this. But it is good to have something to measure the testing in term of “enough”
- The testing is to make informed about the quality. As a tester, I will let you know about our testing strategy and what the product status. As a product owner, you decide if the testing is enough and the application is good to go live or not.
- How much time you give me? Through testing models and heuristics which are built (and used) in many ways, I will tell you what my testing covered, what if it is enough.
Each opinion has its sense. It is not important to argue which one is correct or incorrect.
How much time is enough for your exercise? Someone says 15’, 30’, 1 hour or 3 hours, … Some studies say “30’ exercise every day is good enough”. So, 30’ is enough? Of course, the answer is that it depends. It is not only depending on how long to do exercise, but it depends on a lot of other factors such as: what is your weight, your health status, your body, your location, your kind of exercises, your environmental conditions, … Enough here, in my opinion, is to answer two questions: “what do you have now? (tool, time, team, … or in general we call all of these as context)” and “what is your objective?”
Let’s look back to my two stories. We will consider the context given in these ones. With the first story, this application has been tested in many years. The bug escaped was belonged to one of the flows which was used by end-users. The application is a health-care product. With all these inputs, the client was not expecting any issue encountered in any flows which are used with “normal” conditions (or we usually call “happy cases”). On the contrary, in the second story, the application is a service, the end-users are only with a few appointed groups (or on-duty groups), the application was to simplify the process reduce time & cost for paperwork. More importantly, at that moment (release period), the application was as a MVP to confirm its vision & mission.
With all that said, it is very important to know what is the context & the goal of the product at the period of testing. If your product under test is a “PoC”, it is good enough if your testing is to confirm if the product is suitable to use when it can solve the primary given problem that is a part of “Acceptance criteria”. If the product is in the operation and serving critical missions (any issue will cause huge damages), your tests have to have a minimum set of scenarios to make sure the end-user will have no issue when they operate the product in right ways. If the product is going to be used by a large number of users (for example: your product is an online shopping and it is running in “Black Friday” Season), our tests must target to its performance while the UI tests will not be critical ones. We call our testing is enough if it helps the product to meet the goal and solve the problem in current given context
A framework which helps us initiate the test strategy is that “Agile Testing Quadrants” (defined by Lisa Crispin). At every period, we must identify what “Facing” is mandatory to design our tests. In this quadrant, it doesn’t imply any order. Our product doesn’t have to go through the quadrants from 1 to 4. The quadrants are merely a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it. When identified the facing to focus on, next is to decide the specific critical goals that the product at that moment and design tests (or charters) to cover them. For example: if we know the product now is in Quadrant 2 because when there are many new designs implemented for new features. The accuracy of functions is less important than how the user evaluate if the product is best-fit for their business. Then, you will need to know what key functions for their business and design your test charts to validate (1). how to fulfill the testing for these functions; (2). All data tracked/ stored and proceed by the application is sufficient and aligned with their business; (3). UI is easy enough to use. Then, completion of test execution for all these test charters will be called “enough” at this moment.
The importance of test items depends on the business context that results in making the decision for an “enough” testing.
Assume that we anticipate into the testing for a car trading system such as Kelley Blue Book (KBB). A portion of the search as the image below indicates that there are some charters need to be created & executed to cover its functional testing, while the advertisement will be something we put it as a lower priority. However, ignoring the business context will be a big mistake. Imagining that how much money the company (here is Toyota as a sponsor) may be lost if this advertisement isn’t loaded and rendered as expected.
How many test cases, test charters designed and executed is not very much important and don’t have any means if your product has issues which against its value. An issue can be very serious in this context, but much more minor with other context. It depends on the mission of the product at that moment when it is used. Knowing this will be useful to design your tests and to answer the question “How Much Testing Is Enough”.
By FlowerPecker (A Tester at MeU)
