To Build Your Test Ideas Effectively In Context-Driven Testing
Consider that you are testing a Login page with the “User Name”, Password” and “Remember Me” as elements of the page. Easily you can turn out a set of cases as follows:
- Combinations of characters, numbers, …
- Combinations of normal and abnormal values
- The maximum size for Input fields
- Empty Values
- Check spaces which are placed at beginning/ middle and end of each input
- Check & Uncheck “Remember Me”
- And so on….
However, with security aspects, you may want to check:
- SQL Injections
- XSS
- How “Password” encrypted and transferred & stored in the database
Sometimes, you also will look at the application cookies to see what happens there. All of these are called as “Test Ideas”. The more test ideas you have, the better testing you carry out.
Test Ideas can be built by various sources or heuristics or derived from suitable practices. In this article, I would like to introduce some good practices which may be helpful for your testing
- Keep watching log files
Log files are used for developers to diagnose the problems. However, if testers can access log files to view them in detail – line by line. It would be a great deal to generate new test ideas. The log file is a handy resource which contains clues for problems.
Besides, testers can compare the log file from this execution with the ones of previous executions for the same test to point out changes from themselves. Some of these changes can be the root of causes or turn into new test ideas. While some minor changes may not necessarily manifest immediately, but compound effects can be enormous. I have observed the trends of transactions in the log file from an application over different builds. As the result of the analysis, a new process has been triggered in latest builds to process for data replication, but it was not documented in any document.
To monitor the log files effectively, here are some practices:
- Look into its size and compare it with previous ones: For example, a log file with standard size is at 30KB, with an active file, it has dumped 1M, so it is, of course, the thing we have to take a look at
- Compare the contents of an active file and previous ones: If you run the same test, the same workflow, you expect that the files generated into different runs should be similar. If no, you must know why. There are a lot of free tools support this comparison and highlight differences such as winmerge, …
- Pay attention to date/time logged for major workflows in different runs
- Keep watching competitors or similar products
Most testers only focus on the zone that they control, on features their developers have built, or what they are asked to test. However, if you take your time to see what the competitor’s products are, how they are built, what their issues are, risks or known bugs. This helps you develop your test ideas better. Key scenarios probably translate well across the teams and products. For example: if you test Lazada’s payment function (online shopping website which is somewhat famous in Asia in general or in VN in particular), let’s look at how similar function works with Amazon, or security mechanism used in Amazon can be compared with Lazada.
All feedback, comments, bug reports from users who are using your competitor’s products will very valuable to your testing. These are feeding your thinking to get more insightful test ideas.
- Pay attention to failure modes
Any failures are good models to learn. These models help us to build more of test ideas when delivering the testing. By answering “what if” questions will trigger your thought for new test ideas. For example: Assume you are testing a Login function of a web application, in addition to “normal” cases, you can ask:
- What if an authenticated user tries to login the system?
- What if an authenticated user closes the application without logout and tries to log after then?
- What if a user tries to copy & paste into the password field?
- What if a user wants to show his password during typing?
- What if a user press and hold (many clicks) “Login” Button at once?
- What if a user operates this function with other web browsers?
- What if an attacker tries to perform dictionary attack?
- What if a user blocked from entering the wrong username or password more than five times tries to login with a correct account
- What if a user remembers his account info, then how its cookies managed?
- …
- User’s emotions
Happy cases, alternative cases, and exceptional cases are terms frequently used in designing test cases. However, if you stand at user’s view, you can see that user’s emotions can involve into generating test ideas. For example, there is not only happy case, but other emotions can also be considered such as: scary, angry, embarrassing, desolate, forgetful, greedy, stressful. Each emotion constructs a set of different activities which can harm your application or turn your application into a context which cannot be controlled. For example: when a user feels angry, s/he can operate the application in the context of out-of-control such as shutdown the application suddenly even when the application transaction is in-progress, or perform crazy clicks on the whole application.
A set of emotions is a heuristic that helps the team to design more tests during exploratory execution. To make this work, you can think about the happy path and look at it for alternatives. When going through the happy path, keep thinking & seeking for other paths that could be taken using this set of emotions.
- Functionality Testing is Good; Capability Testing is Better
User stories which are implemented into features are a solid starting point for coming up with the deterministic check. Usually, testing teams end up proving what they have seen in story descriptions.
A product is a solution, but if the problem cannot be solved, the product doesn’t work. Testing the capability is to verify the solution with given business problem. Besides, focusing on capability leads to deeper insights and prevents tunnel vision. To test the product capability, you have to put yourselves into a real-world context where the product can solve given problem. For example, A Helpdesk Feature allows users to contact technical teams for their troubles; Helpdesk creates a ticket to submit to our teams. Under the view of capability, the system is not only to function correctly to create tickets, but it has to offers an alternative channel to deal with situations when the network access unstable or our web services get shut down by any reason. For these situations, we expect an email prominently on the form.
