WHY ISN’T YOUR REGRESSION TESTING GOOD ENOUGH?

New features are added, bugs are reported — so, the testers have to work on the testing – they will carry out the regression testing of course. Somehow the regression testing was failed, the question is, “WHY ISN’T YOUR REGRESSION TESTING GOOD ENOUGH?” that made the regression testing fail. Surely, if we want to be really cautious, we can test the entire application from the ground up, but that is time-consuming, expensive to the company or client. Therefore, we have to look for the answers to that question; find out what is its challenges and solutions.

First, what is regression testing?

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software. Changes may include software enhancements, patches, configuration changes, etc. Source: Wikipedia

Then, when we do regression testing?

Regression testing is performed when any of the following situations happens during the software development lifecycle:

• A new feature or function is added: This is the most common reason to run regression testing. The old and new code must be fully compatible. When developers introduce new code, they don’t fully concentrate on its compatibility with the existing code. It is up to regression testing to find possible issues.
• A bug is fixed: Bug fixing requires changing the source code, which in turn calls for re-testing and regression testing.
• A change in original requirements
• A new module, component or subsystem is developed and integrated to the existing modules or system

Let’s look at an example: Imagine that our product is a hotel rating system. It has been up and running for several years now, and the rating calculations are working as expected, 100% correctly. However, after a long time, the data was getting bigger and bigger, and it was affecting the performance of the calculation. We have to make some performance enhancements to the code: the calculated results should now display on the website ten times as fast as before.

That means we will have two situations that need to be tested. First, we would need to test the calculator’s performance to make sure the new code really is making results display ten times as fast. Secondly, even more importantly, we need to test the accuracy of the calculator, to make sure that the new code has not caused any unexpected calculation errors.  After all, to a customer, data accuracy is always more important than performance.

Since the calculations should yield the same answers as before the code change, any testing performed that validates calculation accuracy is considered regression testing.

Why is regression testing important?

• Regression testing increases our chances of detecting bugs caused by changes to software and application – either enhancements or defect fixes. Also, keep one thing in mind that we also do not give any guarantee that for performing regression testing there is never any side effects also.
• Regression testing also detects undesirable side always caused by changing the operating environment.
• Effective regression testing can save a company’s time and money. It should become a routine procedure while developing an application.

So, what is the cause of this problem?

– Lacking domain knowledge:  It happens when new testers join the team, they will have some basic knowledge about testing after a period of learning and research about the product. Also, when do Regression tests in an Agile project, they usually focus on the acceptance criteria in stories to make sure it is working as desired and do not care much about the other one due to time constraint. Hence, this knowledge might be sufficient for major functional testing, but insufficient for the regression testing. You cannot perform complete regression testing with partial knowledge.

– Lacking time and documentation for regression testing: The advent of Agile has brought a new challenge that tester has to overcome. Minimal documentation with changing project scope and frequent iterations – short of time. The pace at which Agile projects move leaves little time for manual testing. Consequently, it is becoming increasingly more challenging to ensure efficient test coverage of a project.

– Out of date test cases: As your project grows, some of the added functionality will start lacking sufficient test coverage. More test cases are added to the regression test suite, and some of the test cases may become obsolete or unnecessary. Especially, new testers are not very comfortable in gaining knowledge of what happened in the past; they do not want to dig into the old test suite that developed for a long time. For these reasons, they are easy to lose track of test cases, overlooking the new important test cases only which may miss out the importance testing of old test cases.

– Boring and time-consuming: You know If testers keep on creating regression tests all the time, doing something repeatedly, over and over the feeling of boring is unavoidable, they might get disappointed. That may affect to their performance (Bored -> lack of focus -> low quality of execution -> cannot find bugs -> escaped bugs). Also, what if your company release a new build every week, or every day -> Regression is a nightmare.

Some tips to make the Regression Tests more effective.

Ok, we have found out what regression testing is and what is the cause of this problem. Let’s proceed to some of the regression testing good practices. Let’s look at a couple of recommendations that can make the life of a tester simpler.

– Tip 1: Improve Manual and Automation Techniques

Firstly, we should improve our testing techniques in both manual and automation, understand clearly about each technique because of some techniques just applicable in a particular context, that will help our testing more effective. If automation is used, it saves a lot of our time & efforts for regression testing.

Besides that, we could use the tools to support our testing like mind-map, test case management, etc. That will help to have an overview of the whole system, easy to maintain the test cases up-to-date, track the revision of product and requirement version precisely, easy to manage the interaction between components in the system; this will bring great help when doing Regression tests.

Especially, Risk-based testing can be a good attendance when doing Regression Testing, , Risk-based testing can figure out which area can be a risk, from this pick-up test cases/test charters into the Regression Testing session. This technique will optimize our time /cost/risk when performing Regression Testing.

Automated tools also an assist in our regression testing. The most common areas that require automation in 99.99% of cases include:

• Running the same regression tests on different platforms. This includes different operating systems, browsers, and devices.
• Running tests after a bug fix or a minor update.
• Data-driven testing. For instance, testing how your application handles different input data.

– Tip 2: Model the system

Understand the product is the key to the success of every project. Therefore, you have to make a model of the system in the way that you can understand, remember and easy to manage. The more you understand about your product, the easier it is to determine what might be affected by changes in your products. We can create a model as a spider web; they have node and strings that represent functions, features or modules of the software as a node in the web and where the string represents any connection or integration between the features/modules/components/etc … The idea being that if you know, one area is going to implement code changes, example a node in the web, then you can see the integration and regression risks, with direct connections followed by secondary and third connections from that node. For example, change in the auth protocol may cause the login API to fail, fixing an error message could lead to failure in the reporting API.

To improve our knowledge, besides learning from the product we are testing, learn from the other similar product as well, sharing our expertise with the team as the ways to expand our knowledge about the system too.

– Tip 3: Communication and Collaboration

Should have more and more communication with your stakeholders to clarify all the requirements or which case you confused about the system. Make it all clear before start doing testing. If you are a newcomer to the team or when you will test the features/components that have no documents, how can you check it correctly and thoroughly? From this, you should work with the team: Dev, Client, etc. to make your assumption clarify, exploit for more information, save time and somehow somewhat we can build a positive association between testers and developers and client.

If your developers carry out code reviews before pushing their code to a shared repository, then try to get involved in those reviews. You don’t need to learn how to code or read the code, but listening and asking questions about what might be affected will help target your testing. Or if you is able to understand the code, then you can actively perform the code diffing to see what had changed in the code to help target your testing too.

Also, you could pair with your developers as they are programming and take notes or even with other testers. This is an excellent way to learn about changes to help you target your regression testing. The collaborative nature of an agile team means testers should be more informed of what has changed and be able to determine what regression testing to carry out. However, this requires the whole team to take responsibility for regression testing and not only the testers.

– Tip 4: Take note

If your team favors exploratory testing, then you may have a stored list of charters, test sessions, and notes that you refer to during regression testing. An exploratory testing session using previous session notes can guide testers enough. However, they require more skill and understanding of what has been tested so far to help determine what session to execute. Though we already have the defect-tracking tool to manage issues. Taking note is another effective way to improve our testing, in the testing session should write down those things that we concern, assumption or suspicious or the particular case need to be a concern when running test case(s)/test charter(s). From that when doing Regression tests, based on testing notes, the tests we performing will become more effective. Furthermore, it helps the debriefing or retrospective session extra impressive.

– Tip 5: Define good regression testing strategy

A good regression testing strategy encourages targeting your testing towards areas of the product that are more at risk of changing than others, instead of exhaustively testing every part of the product again and again. Moreover, test cases are the primary elements of a regression test plan that contributes the most to make it successful. Hence, it is inevitable to select the most appropriate test cases to get the best results. If your team favors exploratory testing, you can apply some heuristics to help analyses candidates for regression testing more effective. So here are a few ideas for you to ponder.

• Test case had frequent defects: By reviewing the weekly/monthly bug reports, it is easy for you to identify the areas leading to maximum no. of defects. First of all, you can add those defects to regression and then look for increasing the test coverage of that particular area.
• Test case that covers the core features of products: Find out the core areas of the product. So get to the requirements specs, review the product design documents and come up with the features that are most critical to the product. Consequently, you can proceed with the test case selection and cover the desired functionality.
• Test case that covers the recently modify features: Try focusing on the recent changes and add cases which can break the existing functionality.
• Test case that covers the functionality that more visible to the user, …

– Tip 6: Attitude

If you have to test the things many times with the same test cases/test charters, do you feel boring? – Absolutely “YES” – And this is one of the reasons make our Regression tests will not good. So “How to release these boring?” – Just take a deep breath and talk with the team to come up with a plan to change the situation. We can apply exploratory testing; this testing has a time limit. It encourages testers not to take too long investigating one issue, the tester is flexible and can execute beyond the test case scope (explore and play a little to find more problems). They can do different things; they need to explore more and to be challenged.

Moreover, another small tip for you, although we tested a feature a thousand times before. Just “Be stupid” like always think that issue always exist, being curious about system, always put questions, use question words (What/When/Why/Where/Who/Which/How) like “Why it should be like that?”, ”When it occurred?”, “Is there something wrong here?”, “How does it work?”, “Why it should be like that?”, etc. Hence, we can explore more about the system, help generate more test ideas, make our testing more effective.

Do not fall to the “Pesticide paradox” situation (If the same tests are repeated over and over again, eventually the same test cases will no longer find new bugs. This is what pesticide paradox is actually referring to).

In conclusion: Be Bright and Careful

The most important thing when doing Regression Testing is careful with everything. Keep skeptical about the product, always ask questions about the product, learn and search how to improve yourself, and your team, successful will come to you. The communication well between team can make a regression testing better and smoother.

What do you think?

We hope you liked this article. If you are reading this article until this point, then we would like to hear feedback from you. Your valuable feedback, question, or comments about this article are always welcome.