Effective Exploratory Testing (Part 5): Risk-based Exploratory Testing
Part 1: Effective Exploratory Testing (Part 1): Some good practices
Part 2: Effective Exploratory Testing (Part 2): More Effective With Pair Exploratory Testing
Part 3: Effective Exploratory Testing (Part 3): Explore It, Automate It
Part 4: Effective Exploratory Testing (Part 4): A Practice of Pair Exploratory Testing
Part 5: Effective Exploratory Testing (Part 5): Risk-based Exploratory Testing (Reading)
Special Series 1: Effective Exploratory Testing: Empower Your Exploratory Testing With Smart Tools (1)
Special Series 2: Effective Exploratory Testing: Empower Your Exploratory Testing With Smart Tools – A Case Study (2)
Context-Driven Testing Framework: Effective Exploratory Testing: Context-Driven Testing Framework
Risk is an uncertain event that if it occurs has negative or positive effect. In this definition, risk is as two sides of a coin. However, in software testing, we use risk to imply to negative events (or conditions) which will cause harms to our product quality or will impact on value to some persons or organizations if they happen. Risk implicitly or explicitly participles in all testing strategy. It is used widely in both scripted testing and exploratory testing
Scripted Approach is just like it sounds. A senior tester has to take the time to write out the scripts (or called test cases) for other testers. Scripted approach aims to predefine the expected result in detail. However, in practice, requirements, specifications, and thus test cases are seldom perfect in terms of comprehensiveness and accuracy. Scripted approach splits up planning, designing, and executing tests. First, the tester tries to brainstorm (or do mapping with requirements, specs,..) on the function of application. Then s/he will develop tests for each area with all possibilities (hopefully), and then these tests can be executed by other testers
Otherwise, with Exploratory testing, all that happens at once. This approach does not rely on the documentation of test cases prior to execution. Exploratory testing emphasizes the freedom and responsibility of each tester to continually optimize the value of his work by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project
Hybrid Approach – each one has its own advantages that having benefits with given context. Scripted Testing is written to meet functional requirements however it becomes less useful when the functionality is already developed, while exploratory helps identify serious bugs which are not fallen into scripted context given by software requirements. So, we establish blend of scripted and exploratory testing, then our testing approach generates more valuable information, better test results and more higher test coverage
Risk-based testing is a risk-focused investigation approach in where the effort of testing is distributed on items as results of risk assessment. Risk-based testing is more perceptive than calculative. The objective of risk-based testing is to minimize the risk associated with the product. Formally, a risk is going through a 5-phase process including Planning Risk Management, Identifying risk, Analyzing risk, Panning Risk Response and Monitoring and Controlling with a lot of effort participated by many parties in a project that results in a significant highly cost. The more risk identified and respond on it, the more cost we have to spend on. In addition, as a nature, we cannot afford to mitigate all risks which can be occurred in our project (or our product). Two types of risk that we have to deal in a project:
- Project Risk and
- Product Risk
Project Risk is the risks associated with the project activities which can endanger the project such as: tight schedule in delivery, unstable in team structure, lack of leadership and management,… Otherwise, product risk is known as the risks associated with the software or system, the possibility that software or system may fail to satisfy end user expectations such as: inaccuracy or ambiguity in requirements, complexity in product design and product deployment, technology employed in the product is new (or less supportive by the community),…
Only two these risk types come up with many risk sources being sampled as follows:
| Project Risk Sources | Product Risk Sources |
|
|
A balance between effort (cost) for risk assessment and Impacts
Needless to say, it is a huge effort (cost highly) to identify, plan and respond to all risks which are generated from these sources., Therefore, balancing between risk mitigation and its cost while still keep pursuing the project/product’s objective for given contexts is the most important. This is a heuristic to apply in software testing as well. For example: if you are testing for a healthcare product in where it is used by a few users (doctors, clinician,..) , it is more necessary and more important to make sure there is no issue with using it in normal conditions rather than trying to crack the product or trying to catch “nice-to-have” bugs from UI. In opposite, a landing administration application which holds and accesses to confidential data will be needed to validate it in term of security breach to see if there is data/ information disclosure.
Risk Management in Exploratory Testing
Exploratory testing combines with Risk-based testing will be more efficient. Presence of risk-based testing in your exploration will give a clear direction, remove wastes and stay focused on the project’s goal. Throughout a chain of activities including: modeling, learning, designing and performing the experiment, observing, and asking & logical reasoning, risk-based testing participates into this process as an efficient driver to make our testing to be more focus on optimizing the value of our testing.
Various external & internal sources such as: requirements, designs, development notes, similar products, … contribute to building test charters which after then are prioritized in term of how risks impact on the value of the product (or organization) as a result of risk identification & analysis. Identifying risks should be context-based. We must understand the mission of product at the moment it is used, all test charters and components of product associated with the mission are organized in the order to carry out while any risk which can cause the mission failed must be addressed.
As said, risk comes from various sources, it is impossible to keep the product stay away from risks. Therefore, we have to answer ourselves the question of “how much effort for risk assessment is enough for our exploratory testing?”. Of course, again, it depends on the context. However, we can use the following sample questions in seeking answers (this is not a complete list of questions, this list is used as a heuristic in risk-based exploratory testing. A “complete” question list will be posted in other article in series of Effective Exploratory Testing later)
- What are the things that end-users (or key stakeholders) think that they should never happen? For Example: it is critical if a transaction is lost from a banking system. Then all charters and risks associated with processing the banking transactions must be placed in high priority
- What if this/that function fails? For example: if the answer is that it is not allowed to fail to cause the whole business interruption. Although this answer is not very sensitive for testing, it helps to understand how important the function expected, then we can plan out for a significant effort of testing to mitigate risks in our exploration.
- What must the product comply with (standards, regulations,…)? For example: HIPPA requires an accuracy in protecting health information held by the covered entity or business associate. It is necessary to have a (or many) charters to conduct an accurate and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and availability of protected data
- Questions related to level of expectations for performance, reliability, usability, compatibility,… to evoke requirements different kinds of requirements. What if the requirements associated with any of these elements are not met?
When we have identified all critical risks by answering these questions (although there are some other methods to collect risks, questioning & interviewing is the simplest one), we can perform a risk analysis that results risk exposures which gauge the risk level. Risk analysis is beyond the scope of this article, but you can find more info from this link: https://pmbasics101.com/how-to-perform-qualitative-risk-analysis/ . High risks need to be addressed immediately and are basis to adjust effort of our exploratory testing. While lower risks (very little impact if they happen) can be placed in a watch list that we will need periodically to review and ask ourselves what our testing has revealed about those things.
An important thing to remember is that a risk changes its exposure over time. So, risk analysis must be integrated & repeated in the heuristic exploratory testing framework (will be described in another article in the series of effective exploratory testing) for accurate evaluation.
Make transparency about risk with our stakeholder or our client: It is necessary to make risk communicated transparently with key stakeholders in a project. They may not be aware of how important to manage risk in our testing, however, getting them involved into the process is a way to make awareness. Risk needs to explain to key stakeholders in a language of their knowledge-based. For an example: we cannot speak to a business owner risks of technical-related, but they can explain them how it impacts on their business value. A practice for making risk awareness is to organize our testing into risk-associated mindmaps to bring our risk assessment to our stakeholder at each their review. Following are practices used at MeU Solutions
Risk-based in Test Planning for Exploratory Testing
Using Session-based Testing Management is a good practice to employ risk assessment in our exploratory testing. This session-based approach brings a clear structure to loosely defined exploratory testing. It is an approach to planning, managing, and controlling ET in short (almost) fixed-length sessions. All test charters are marked with numbers which represent its importance in term of testing value which is assessed by many factors such as: business impacts, technical constraints, technology,… Higher number indicate more value of testing delivered to the product. The sample below shows that charters marked by 1 are highest priority for execution. By doing this, risk assessment communicated regularly with key stakeholders, it also translates test strategy into a visual map
Risk-based in Test Report for Exploratory Testing
As results from test execution, testers can decide level of risks which associated with each module/ function and placed them into the test report to convey the info for quality risk assessment that help key stakeholders to make decision for “Go” or “Not-To-Go”
In summary, risk is a nature in all aspects of a project. It can be good or bad. In exploratory testing, combining risk assessment into itself would help the testing more directive. It is hard to say about an “enough testing”, but risk-based exploratory testing is a good approach to gain “enough” testing at a given context when it directs the testing to product’s goals.
