Effective Exploratory Testing: Empower Your Exploratory Testing With Smart Tools – A Case Study (2)
Part 1: Effective Exploratory Testing (Part 1): Some good practices
Part 2: Effective Exploratory Testing (Part 2): More Effective With Pair Exploratory Testing
Part 3: Effective Exploratory Testing (Part 3): Explore It, Automate It
Part 4: Effective Exploratory Testing (Part 4): A Practice of Pair Exploratory Testing
Part 5: Effective Exploratory Testing (Part 5): Risk-based Exploratory Testing
Special Series 1: Effective Exploratory Testing: Empower Your Exploratory Testing With Smart Tools (1)
Special Series 2: Effective Exploratory Testing: Empower Your Exploratory Testing With Smart Tools – A Case Study (2) (Reading)
Context-Driven Testing Framework: Effective Exploratory Testing: Context-Driven Testing Framework
Background
A part of the solution for effectively with graphing your testing has been implemented by MeU Solutions that integrated into a tool called One2Explore. Up to this point, One2Explore is installed at local tester’s machine. It captures all the test actions and generates them into a graph format. It provides clear, concise directions and useful information for analysis of exploratory testing efficiency. One2Explore is integrated into One2Test (a testing management & dashboard) and One2Automate (*) (an automation test framework) to offer a complete solution in software testing. One2Test is to store all relevant data including, but not limited to test actions, test orders, screenshots, time interacted with each screen, bugs associated with screens & test session, internal data and external data. The functionality of One2Test is to: (1). Build master graphs; (2). Analyze missing scenarios by Master Graph and current graph.
(*) One2Automate is a framework built by MeU Solutions that facilities a great capability to automate all exploratory testing by using Graph advances. The solution for automation with One2Automate and One2Explore wrote in this article: https://meu-solutions.com/effective-exploratory-testing-part-3/
A project of 6 months with a team of 4 testers. This is a Sale of Point (POS) System which targets medium to small size retailer markets. The system offers a range of features for Ordering, Purchasing, Inventory Management, User Management, Pre-Booking.
Challenges
There were critical challenges that we encountered:
- It was built from scratch with a low level of the requirement document.
- Time constraints to go live. Project Sponsors with high demands put pressure to have the first release to the market within 2 months.
Solutions
- Used Context-Driven Testing in a form of exploratory testing to handle changes during product implementation and reduced time for detailed test-cases instead developing test charters which were used to communicate with the key stakeholders on the requirements
- Used One2Explore and One2Test to manage test coverage, to review current test execution and to come up with better test ideas
Facts and Achievements
- Over 150 test charters were developed to cover 5 functional streams. 30+ charters executed at every test cycle (2-week length) with more than 100 sessions run within 200 ideal hours
Figure 1 – A fragment of test charters in a particular Test Cycle
- About 40,000 steps recorded in image format which were used in graphing test executions and comparing with other test executions. More than a half of total test executions used One2Test to debrief, review and build new test ideas. One third of the test charters generated by comparing two test executions or comparing current test execution with the Master one. Following are two examples used by One2Test and One2Explore to decide the test efficiency and generate new test ideas
Figure 2- A result of comparison between two test executions (Green Nodes are pages missed from a test execution and Yellow are the ones missed from another and grey are the ones which executed by both).
Figure 3 – A result of comparison between current execution and Master execution (White Nodes are pages missed from Master execution (all previous test executions)).
Figure 4 – A result of a comparison between current execution and previous executions (Master) (White Nodes are pages missed from current execution and Green are the ones executed)
- As benefits of the info visualization, test leads at MeU Solutions commented on One2Test and One2Explore as the most powerful tools which benefit their exploratory testing, while a client at MeU Solutions said: “By using your tools, we could partly see the insight of our test quality overruns. It can be said One2Test and One2Explore are making an informed decision for our testing”
One2Explore and proposed approach have been applied to six projects at MeU at different fields of web application such as eCommerce, Healthcare, Government an SME Management. We conducted a 360-degree feedbacks from testers, test leads, and clients who joined directly in the project. The results are presented in Table 1 and 2 as follow
Table 1: The 360-degree feedbacks information
Table 2: The result of 360-degree feedbacks from testers, test leads, and clients
Table 2 (number in parentheses) shows the proposed approach and One2Explore provide the value for web applications testing. We also receive positive feedback from clients, automation experts as well as testers, test leads on the benefits of One2Explore tool.
The Brutal Facts
Life is not like a dream. One2Explore has been encountering many challenges in the way of its perfection. Key challenges can be listed as follows:
- (1) It was built from scratch with a low level of requirements document;
- (2) Time constraints to go live. Projects with high demands put pressure to have the first release to the market within two months;
- (3) The more complex testing is getting, the more complex graph is becoming.
While the first two challenges are something we have to confront with by getting more talented people and external expertise in Data Science, the last one can be solved by a solution is to layer the information and display the graph in given conditions. Particularly, the tool allows categorizing two types of information to display: (1) Graph information including nodes and edges. This is a skeleton of the graph. (2) Test information is elements such as bugs associated with each individual screen (node), URLs, screenshot. The graph is displayed in many layers. Each one is used to portray important information that describes each of the individual elements. These layers can be hidden or shown that help the view better.
In the conclusion
We have demonstrated how visualization can bring about a difference in the adoption of good practices in Context-Driven Testing in software testing. We have proposed using graphs as a method to display test execution and relevant information that helps the exploratory testing productivity. We also built a Graph exploratory testing tool for testers as well as test leads and clients. The feedbacks from Test leads and testers have shown that the reports and the graphs generated by One2Explore are ideal for Test Managers with auditing. Test managers can easily see how testers have utilized their allocated times. For automation, the graphing can help identify the most commonly touched areas and test cases in the system and develop scripts accordingly.
One2Explore can integrate with Jira so that the testers do not spend much time reporting bugs and entering bug details. Abnormal behavior is logged and analyzed, therefore, the evidence is retained for further analysis by the development teams. For each iteration, the master graph helps test managers to review the test approach of the previous iteration and the current iteration so that the test manager can highlight which areas need more attention in testing.
