Guide to Effective Test Status Reporting and Metrics Collection – Part 1

Test Status Report
Test Status Reporting is a formalized reporting on the status of the project from a testing point of view. This is part of the project Communication plan. The Report shows quantitative information about the project.

Test Status Reporting is the component that informs key project stakeholders of the critical aspects of the project’s status. Good status reporting prevents surprises to project sponsors and stakeholders. Formal status reporting needs to be provided as part of the project steering committee meetings.

Purpose
The purpose of a Test Status Report is to provide an ongoing history of the project, which becomes very useful in terms of tracking progress, evaluation and review. Test Status Reports forms a part of the Project Review Process both during and after completion of the project.A Test Status Report should identify the key areas of importance that will assist the stakeholders of the project in determining the “state” of the software development and test efforts. It helps in answering one of the most asked questions of system testers … “Will the software be ready for release on the agreed upon date?”
The Test Lead should try to maintain a careful balance between timeliness, accuracy, and consistency when preparing these reports as the status of the defects keeps changing.

Owner(s)
The person responsible for managing all test activities (Test Manager or Test lead) should be the owner of the test status report.

Who should use it?
Test Status Report is a document that will be used by the Project Manager to understand the ongoing history of the project, which becomes very useful in terms of tracking progress, evaluation and review. Test Status Reports forms a part of the Project Review Process both during and after completion of the project.
The target audience for a Test Status Report could vary. Any one interested on project health can be the audience. This can be the Project Management team, Project Steering committee, Customer or other key stakeholders of the project. (Click on the below pic to see clear view)

A brief description of the different attributes of test status report (figure-1) presented below.
1. Project Name – The project name that you are reporting metrics on.
2. Duration – The reporting period
3. Report by – Owner / Author of the Report.
4. Report to – Target Audience.
5. Previous weeks Accomplishment – Activities performed after last reporting date.
6. Missed Accomplishment – Not able to work on any planned activity of the previous report.
7. Plans for this week – Future planning for the tasks to be performed during the current reporting period.
8. Issues –

The role of a Test Lead is to give approximate if not accurate reports on the status of the project. At the start of test execution, Test Lead presumes that all the risks to be addressed by this phase of testing still exist. As we progress through the test plan, one by one, risks are cleared as all the tests that address each risk are passed. Halfway through the test plan, the tester can say, “we have run some tests, these risks have been addressed, here are the outstanding risks of release.”

Suppose testing continues, but the testers run out of time before the test plan is completed. The go live date approaches, and management wants to judge whether the system is acceptable. Although the testing has not finished, the tester can say, “we have run some tests, these risks have been addressed, here are the outstanding risks of release.” The tester can present exactly the same message throughout the test phase, except the proportion of risks addressed to those outstanding increases over time. How does this help?

Throughout the test execution phase, management always has enough information to make the release decision. Either management will decide to release with known risks, or choose not to release until the known risks (the outstanding risks that are unacceptable) are addressed. Most of the time Coding gets 90% of the effort and testing is squeezed yet again. To avoid this, code and test activities should be defined as separate tasks in project plan.

Most managers like the risk-based approach to testing because it gives them more visibility of the test process. Risk-based testing gives management a big incentive to let you take out a few days early in the project to conduct the risk assessment. All testers have been arguing that they should be involved in projects earlier. If managers want to see risk-based test reporting they must let testers get involved earlier. This must be a good thing for testers and our projects.

9. Cumulative Issues– Issues from previous report, if not addressed should be listed here.
10. Test execution Details for previous week – Test details for the reporting period

1. 1. 1. Total effort spent on test execution
      2. Total number of functionality tested during the reporting period.

11. Test Summary

1. 1. 1. Total number of test cycles, number of defects found, and defect tracking details should be listed in this section.
      2. Test Coverage details in terms of functionalities tested and test cases executed should be detailed in this section.

12. Project Milestone – Important project schedule from a testing point of view should be listed here.

The Test Status report should also represent the status through charts and graphs for easy and better understanding.
The following are few charts that can be included in the test status report.

Functionality Coverage vs. Duration
Functionalities tested during different releases or during different reporting period can be shown in a graph to give an indication about the progress of test coverage.

(Figure-2 : Functionality Coverage chart)

Defects reported vs. closed
The above report shows the find and fixes counts for bugs reported against the reporting period. This gives Management an idea of the defect trends. A flattening cumulative open curve indicates stability, or at least the inability of the test team to find many more bugs with the current test system. A cumulative closed curve that converges with the open curve indicates quality, a resolution of the problems found by testing. Overall, this chart gives a snapshot of product quality as seen by Testing, as well as telling Management about the bug find and fix processes.

(Figure-3: Defect status)

Defect Location
This graph displays the defect occurrence in different modules. This helps the management in assessing the most problematic area in the project. Based upon this a root-cause analysis can be conducted and appropriate corrective measures can be taken to reduce the risk.

(Figure-4: Defect Location)

Defect Classification
These details help in conducting a root cause analysis and taking corrective measures to improve product quality.

(Figure-5: Defect Classification)

Test Case Progression Chart
This gives a clear indication of test progress against the test plan over a time.

(Figure-6: Test Progression)

Severity wise Defect Distribution
This gives a indication about the severity wise defect distribution.

(Figure-5: Defect Distribution)

What should be the ideal frequency?
The frequency of test status report can be decided based upon the project test plan.
In case of a small project where the releases to testing team happens every alternate day , status report should be prepared on a daily basis to keep the management informed about the test progress.
But in case of large projects where the application released on in a month for testing status report should be prepared once in a week.

Periodic meetings should be held to discuss the project status, either verbally or based on the Test Status Report. The meetings should be often enough that progress could be reported against a number of milestones since the last meeting.

Read 2nd Part: Guide to Metrics Collection in Software Testing (includes Benefits of implementing metrics in software testing)