Prioritization of Test Cases

Better Testing, Testing Tips


Importance of prioritizing
There is no limit to how much testing we could do. We cannot test everything, exhaustive testing (testing all combinations of inputs and preconditions) is impractical, so we must prioritise. It is easy enough to identify far more test cases than we will ever have time to execute so we need an approach to selecting a subset of them. Selecting test cases at random is not an effective strategy. We need to use a more intelligent approach that helps identify which tests are most important. In short, we must prioritize our tests.

Prioritize tests so that whenever you stop testing you have done the best testing in the time available.

How to prioritize
The principle is that we must adopt a prioritization scheme for selecting some tests above others. As we start from highest priority and scan the tests in decreasing order of priority, there must be a point at which we reach the first test that is of too low a priority to be done. All tests of a lower priority still are de-scoped. There are many different criteria that can be used to prioritize tests and they can be used in combination. Possible ranking criteria include the following:

  • Ask the end-user to prioritize the requirements (and test those first)
  • Test first the areas of the software that have had most faults in the past
  • Test most those areas of the software that are most complex or critical.
  • Tests that would find the most severe failures.
  • Tests that would find the most visible failures.
  • Tests that would find the most likely faults.
     
Reference: http://goo.gl/HL40G (iseb notes)


Session Based Test Management (SBTM)

Better Testing


Session Based Test Management is a formalized approach that uses the concept of charters and the sessions for performing the Exploratory Testing.A session is not a test case or bug report. It is the reviewable product produced by chartered and uninterrupted test effort. A session can last from 60 to 90 minutes, but there is no hard and fast rule on the time spent for testing. If a session lasts closer to 45 minutes, we call it a short session. If it lasts closer to two hours, we call it a long session. Each session designed depends on the tester and the charter. After the session is completed, each session is debriefed. The primary objective in the debriefing is to understand and accept the session report. Another objective is to provide feedback and coaching to the tester. The debriefings would help the manager to plan the sessions in future and also to estimate the time required for testing the similar functionality.


From WikiPedia:

Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with Scenario testing. Session-based testing was developed in 2000 by Jonathan and James Bach.

The debriefing session is based on agenda called PROOF.

  • Past: What happened during the session?
  • Results: What was achieved during the session?
  • Outlook: What still needs to be done?
  • Obstacles: What got in the way of good testing?
  • Feeling: How does the tester feel about all this?

The time spent “on charter” and “on opportunity” is also noted. Opportunity testing is any testing that doesn’t fit the charter of the session. The tester is not restricted to his charter, and hence allowed to deviate from the goal specified if there is any scope of finding an error.
A session can be broadly classified into three tasks (namely the TBS metrics).

  • Session test up: Time required in setting up the application under test.
  • Test design and execution: Time required scanning the product and test.
  • Bug investigation and reporting: Time required finding the bugs and reporting to the concerned.

The entire session report consists of these sections:

  • Session charter (includes a mission statement, and areas to be tested)
  • Tester name(s)
  • Date and time started
  • Task breakdown (the TBS metrics)
  • Data files
  • Test notes
  • Issues
  • Bugs

For each session, a session sheet is made. The session sheet consist of the mission of testing, the tester details, duration of testing, the TBS metrics along with the data related to testing like the bugs, notes, issues etc. Data files if any used in the testing would also be enclosed. The data collected during different testing sessions are collected and exported to Excel or some database. All the sessions, the bugs reported etc can be tracked using the unique id associated with each. It is easy for the client as well to keep track. Thus this concept of testers testing in sessions and producing the required output which are traceable is called as Session based test management.
Tools available for Session Based Testing:

References – Software Testing Guide Book – Software Testing Research Lab – http://www.SofTReL.org

 


Better Requirements | Better Testability

Better Testing, Test Management, Testing Tips


“Good Manual Testers are hard to find..”

In order to do better testing, testers should understand the quality attributes. Testability is one important quality attribute.

Testability is the degree of difficulty of testing a system. This is determined by both aspects of the system under test and its development approach.

  • Higher testability: more better tests, same cost.
  • Lower testability:  Weaker tests, same cost.

When the degree of testability is lower, then there are chances of skipping bugs during the testing. Main causes for lower testability –


1. Functional/Non-functional Requirements – Poor/unambiguous requirements written by the author. Insufficient requirement reviews by testers.

2. Technical requirements/Architecture/Design of the system – Development teams concerns functionality for the customer, not for the testers. Testing Team does not get Management support and testing team not involved in the design meetings.

Higher testability can be achieved –
Note – Testability is not only testable, but also Easy to Test. The goal of increasing the testability of software is not just to detect defects but more importantly, to detect defects as soon as they are introduced.



1. If requirements are explicitly defined. The source for many improvements to the testability of the system will stem from the requirements. Business Analyst should keep the requirements language simple so that it should be easily understandable by testing and dev teams.
While reviewing the requirements/specifications – testers/test case designers should inspect requirements for testability. But how Testers can inspect the requirements for testability?
– James Bach has defined the Heuristics of Software Testability. Go thru this checklist. This is very useful while reviewing the spec documents.

– Go thru Guidelines and Checklist for Requirement Document Review.

2. Testers should be involved in the software design/architecture meetings, so that testers can understand how the software system is being developed. This is very required because it will help testers in understand the design, the complexity of the system and relations of one function/module with another function/module. This will help testers in getting the higher testability. Thru this testing team knows which programs are being touched in what scenario.
Sometimes, to cover the client’s requirements, development team develops the requirements in a complex way which makes the testing difficult. For example, developers might include some nightly jobs (db jobs) in the system which corrects the data. Now these things increase the complexity of the system which results into lower testability. These things should be avoided. This is one more reason that why testers should be involved in the software design/architecture meetings


References:


Which test cases to be Automated?

Better Testing, Regression Testing, Testing Basics, Testing Tips


The testing elapsed time can be shortened, therefore leading to a huge saving in terms of time and money. Generally, the ROI begins to appear in the third iteration of automated testing.

High Scoring Test Cases
  • Tests that need to be run for every build of the application (sanity check, regression)
  • Tests that use multiple data values for the same actions (data driven tests)
  • Complex and time consuming tests
  • Tests requiring a great deal of precision
  • Tests involving many simple, repetitive steps
  • Testing needed on multiple combinations of OS, DBMS & Browsers
  • Creation of Data & Test Beds
  • Data grooming
Low Scoring Test Cases
    • Usability testing – “How easy is the application to use?”
    • One-time testing
    • “ASAP” testing – “We need to test NOW!”
    • Ad hoc/random testing – based on intuition and knowledge of application
    • Device Interface testing
    • Batch program testing
    • Back-end testing
– by the author of “Quick 101 on Automation”


Software Tester’s Mental Life

Better Testing, Testing Tips | 1 comment


Boris Beizer, identified five phases of a tester’s mental life:

Phase 0: There’s no difference between testing and debugging. Other than in support of debugging, testing has no purpose.

Phase 0 Thinking: Testing = Debugging

Phase 1: The purpose of testing is to show that the software works.

Phase 1 Thinking: The S/W Works

Phase 2: The purpose of testing is to show that the software doesn’t work.

Phase 2 Thinking: The S/W Doesn’t Work

Phase 3: The purpose of testing is not to prove anything, but to reduce the perceived risk of the software not working to an acceptable value.

Phase 3 Thinking: Test for Risk Reduction

Phase 4: Testing is not an act. It is a mental discipline that results in low-risk software without much testing effort.

Phase 4 Thinking: A State of Mind

Now think – Which tester’s mental phase are you?


Why start testing Early?

Agile Testing, Better Testing, Testing Basics, Testing Tips | 1 comment


Introduction :
You probably heard and read in blogs “Testing should start early in the life cycle of development”. In this chapter, we will discuss Why start testing Early? very practically.Fact One
Let’s start with the regular software development life cycle:

When project is planned

 

 
  • First we’ve got a planning phase: needs are expressed, people are contacted, meetings are booked. Then the decision is made: we are going to do this project.
  • After that analysis will be done, followed by code build.
  • Now it’s your turn: you can start testing.

Do you think this is what is going to happen? Dream on.

This is what’s going to happen:

This is what actual happened when the project executes

 

  • Planning, analysis and code build will take more time then planned.
  • That would not be a problem if the total project time would pro-longer. Forget it; it is most likely that you are going to deal with the fact that you will have to perform the tests in a few days.
  • The deadline is not going to be moved at all: promises have been made to customers, project managers are going to lose their bonuses if they deliver later past deadline.

Fact Two
The earlier you find a bug, the cheaper it is to fix it.

Price of Buggy Code

If you are able to find the bug in the requirements determination, it is going to be 50 times cheaper
(!!) than when you find the same bug in testing.
It will even be 100 times cheaper (!!) than when you find the bug after going live.

Easy to understand: if you find the bug in the requirements definitions, all you have to do is change the text of the requirements. If you find the same bug in final testing, analysis and code build already took place. Much more effort is done to build something that nobody wanted.

Conclusion: start testing early!
This is what you should do:

Testing should be planned for each phase
  • Make testing part of each Phase in the software life cycle
  • Start test planning the moment the project starts
  • Start finding the bug the moment the requirements are defined
  • Keep on doing that during analysis and design phase
  • Make sure testing becomes part of the development process
  • And make sure all test preparation is done before you start final testing. If you have to start then, your testing is going to be crap!

Want to know how to do this?
Go to the Functional testing step by step page. (will be added later)