The main reason of the reviewing test cases: increase test coverage and therefore product quality.

As we know testers are involved on the Requirements Specification review process to provide the SQA knowledge to the requirements written. As testers are involved on the process they become experts on the area and on the application functionality and many times their knowledge helps avoid introducing future defects into the functionality that later the developer will code (it’s the phase that we called: defect prevention).

Once the requirements are approved and baselined, testers start designing test cases whether on their mind, or writing them (test drafts). Once all the ideas or test case drafts are understood and prepared, the SQA tester will start developing test cases. When this happens, each test case written is based on a specific requirement, so with that we start assuring having traceability between requirement and test cases. This will help SQA team to manage the Requirements coverage of what is going to be tested.

Once the test cases are developed, SQA tester should share-distribute-discuss those with the same team that reviewed the requirements (SRS writer, Developers, SQA tester, Implementation team, etc). However, sometimes this is not possible, as perhaps when the Requirements are baselined, the person who is in charge of the SRS starts on another project and has not even more time to dedicate reviewing a set of test cases. The same happens with the Implementations team, as they are perhaps installing the product on a customer site. There are cases where SQA tester and developer start more or less at the same time with their work based on the Requirements. Developer starts developing code and Tester developing test cases. There are other times that SQ Tester starts thinking or having test case drafts even before the Developer starter coding. That means that developing code and test cases are and should be separate processes.

Of course that having a Requirements-Usability people reviewing test cases has a lot of value, also having the implementations team doing the same. The problem has been that these often did not happen due the lack of resources, so the test cases review would progress only with the developer involved on the same project and functionality. In any case the developer review test cases always would go in the direction of adding details, parameters or circumstances not included in the tester written test cases or well even adding new test cases but never modifying the sense of the test cases written by the tester.

This is the approach and the how the test cases defined by testers need to be reviewed by the developer. We should also notice that some times when the test cases writer is a beginner, not a senior tester, or well does not have so much knowledge about the functionality, then someone from the SQA team with more experience should check the test cases before sharing them with the developer for review.

Benefits of having test cases reviews for SQA test cases written, including on them the developers:

• Defect prevention while SRS review: SQA tester could advance during SRS reviews possible issues before any code starts

• Conceptual and Technical Coverage: Requirements- Usability ensures the coverage from the Concept point of view and Developer ensures the coverage from the Technical Point of view. The traceability coverage track is assumed by traceability tools (Quality Center)

• Defect prevention while test cases review: If the developer has the opportunity to check the test cases while implementing code, it is possible that this will help him to realize codification that may be a cause of a defect. This will help to coding in the way of potential defects.

• Developer Knowledge add to test cases: Developer has also a very good understanding of the requirement (SRS), explicit and implicit as well. Also has done a deep analysis of them since he had to accomplish the SRA. He can bring experience on understanding better details or well some cases not being considered.

After having the test cases reviewed, the SQ team receives all the feedback and decides, based on its experience and knowledge on SQA and also on the functionality if feedback is applied or not. When not applied, the reason should be explained and discussed with the developer since there should be a final agreement on the test cases written.

Author – Unknown (Let us know if you know the author – Published from our old forums and posts)

The Agile Manifesto is a statement of the principles that underpin agile software development:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The QA team may want to add one more principle to the Agile Manifesto

Craftsmanship over Execution: This is meant to focus software developers on creating good code vs. simply writing code that barely works. Both craftsmanship and execution are good things, but taking care to create good code is viewed as more important from testers and customers point of view.

Principles Behind the Agile Manifesto:

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness
change for the customer’s competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months,
with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support
they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a
development team is face-to-face conversation.

7. Working software is the primary measure of progress.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10. Simplicity — the art of maximizing the amount of work not done — is essential.

11. The best architectures, requirements, and designs emerge from self-organizing teams.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Source: http://www.agilealliance.org/home, http://agilemanifesto.org/

Difference between Rational Functional Tester & Rational Robot:

Rational Functional Tester is an automated functional testing and regression testing tool.
Provides testers with automated testing capabilities for functional testing, regression testing, GUI testing and data-driven testing.

– Enables testers to automate tests resilient to frequent application user interface changes with ScriptAssure™ technology
– Validates dynamic data with multiple wizards, verification points and support for regular expression patterns
– Automated wizard for data-driven testing increases test coverage by reusing individual tests with multiple sets of test data
– Allows the use of keywords to power portions of manual tests with automated testing
– Offers testers a choice of scripting language for test authoring and customization: Java in Eclipse® or Microsoft® Visual Basic .NET® in Visual Studio.NET
– Supports custom controls through proxy SDK (Java/.Net)
– Includes out of the box support for Web-based, .Net, Java, terminal emulator based applications such as 3270 (zSeries™) and 5250 (iSeries™), PowerBuilder, AJAX, Adobe Flex, Dojo Toolkit, Siebel, and SAP® applications
– Operating systems supported: Linux, Windows

Rational Robot is a test automation tool for functional testing of client/server applications.
Test automation tool for QA teams for testing client/server applications. Enables defect detection, includes test cases and test management, supports multiple UI technologies.

– Provides a general-purpose test automation tool for QA teams for functional testing of client/server applications
– Lowers learning curve for testers discovering the value of test automation processes
* Enables test-automation engineers to detect defects by extending test scripts and to define test cases
– Provides test cases for common objects and specialized test cases to development environment objects
– Includes built-in test management, integrates with IBM Rational Unified Process tools
– Aids in defect tracking, change management and requirements traceability
– Supports multiple UI technologies
– Operating systems supported: Windows

Regression testing is often seen as an area in which companies hesitate to allocate resources. We often hear statements such as: “The developer said the defect is fixed. Do we need to test it again?” And the answer should be: “Well, the developer probably said the product had no defects to begin with.” The truth of the matter is, in today’s world of extremely complex devices and software applications, the quantity and quality of regression testing performed on a product are directly proportional to the commitment vendors have to their customer base. This does not mean that the more regression testing, the better. It simply means that we must make sure that regression testing is done in the right amount and with the right approach.

The two main challenges presented by regression testing are:

1. What do we test?
2. When do we test it?

The purpose of this article is to outline a few techniques that will help us answer these questions. The first issue we should consider is the fact that it is not necessary to execute our regression at the end of our testing cycle. Much of the regression effort can be accomplished simultaneously to all other testing activities. The supporting assumption for this approach is:
“We do not wait until all testing is done to fix our defects.”

Therefore, much of the regression effort can be accomplished long before the end of the project, if the project is of reasonable length. If our testing effort will only last one week, the following techniques may have to be modified. However, it is not usual for a product to be tested in such a short period of time. Furthermore, as you study the techniques outlined below, you will see that as the project’s length increases, the benefits offered by these techniques also increase.

To answer the questions of what should we test and when, we will begin with a simple suite of ten tests. In the real world, this suite would obviously be much larger, and not necessarily static, meaning that the number of tests can increase or decrease as the need arises. After our first test run with the first beta (which we will call “Code Drop 1”) of our hypothetical software product, our matrix looks like this.

In the matrix above, we have cross-referenced the defects we found, with the tests that caused them. As you can see, defect number 1 was caused by test 2, but it also occurred on test 3. The remaining failures caused unique defects.

As we prepare to execute our second test run (Code Drop 2), we must decide what tests will be executed. The rules we will use only apply to our regression effort. There are rules we can apply to the subset of tests that have passed, in order to find out which ones we should re-execute. However, that will be the topic of another article.

The fundamental question we must now ask is: “Have any of the defects found been fixed?” Let us suppose that defects 1, 2, and 3 have, in fact, been reported as fixed by our developers. Let us also suppose that three more tests have been added to our test suite. After “Code Drop 2”, our matrix looks as follows:

A few key points to notice are:

Of the tests that previously failed, only the tests that were associated with defects that were supposedly fixed were executed. Test number 9, which caused defect number 4, was not executed on Code Drop 2, because defect number 4 is not fixed.

Defect number 1 is fixed, because tests 2 and 3 have finally passed.

Test number 7 still fails. Therefore, the defect remains.

Test number 13 is a new test, and it caused a new defect.

We chose not to execute tests that had passed on Code Drop 1. This may often not be the case, since turmoil in our code or the area’s importance (such as a new feature, an improvement to an old feature, or a feature as a key selling point of the product) may prompt us to re-execute these tests.
This simple, but efficient approach ensures that our matrix will never look like the matrix below (in order to more clearly show the problem, we will omit the Defect # column after each code drop). We will also consider Code Drop 5 to be our final regression pass.

We will address tests 2, 7, and 9 later, but here are a few key points to notice about this matrix:

Why were tests 1, 4, 5, 6, 10, 11, and 12 executed up to five times? They passed every single time.
Why were tests 3 and 8 executed up to five times? They first failed and were fixed. Did they need to be executed on every code drop after the failure?

If test 13 failed, was the testing team erroneously told it had been fixed on each code drop? If not, why was it executed four times with the same result? We can also ask the question: “Why isn’t it fixed?” But we will not concern ourselves with that issue, since we are only addressing the topic of regression.
In conclusion, we will list some general rules we can apply to our testing effort that will ensure our regression efforts are justified and accurate. These rules are:

1. A test that has passed twice should be considered as regressed, unless turmoil in the code (or other reasons previously stated, such as a feature’s importance) indicates otherwise. By this we mean that the only time a test should be executed more than twice is if changes to the code in the area the test exercises (or the importance of the particular feature) justify sufficient concerns about the test’s state or the feature’s condition.

2. A test that has failed once should not be re-executed unless the developer informs the test team that the defect has been fixed. This is the case for tests 7 and 9. They should not have been re-executed until Code Drops 4 and 5 respectively.

3. We must implement accurate algorithms to find out what tests that have already passed once should be re-executed, in order to be aware of situations such as the one of test number 2. This test passed twice after its initial failure and it failed again on Code Drop 4. Just as an additional note of caution: “When in doubt, execute.”

4. For tests that have already passed once, the second execution should be reserved for the final regression pass, unless turmoil in the code indicates otherwise, or unless we do not have enough tests to execute. However, we must be careful. Although it is true that this allows us to get some of the regression effort out of the way earlier in the project, it may limit our ability to find defects introduced later in the project.

5. The final regression pass should not consist of more than 30% to 40% of the total number of tests in our suite. This subset should be allocated using the following priorities:
a. All tests that have failed more than once. By this we mean the tests that failed, the developer reported them as fixed, and yet they failed again either immediately after they were fixed or some time during the remainder of the testing effort.
b. All tests that failed once and then passed, once they were reported as fixed.
c. All, or a carefully chosen subset of the tests that have passed only once.
d. If there is still room to execute more tests, execute any other tests that do not fit the criteria above but you feel should nevertheless be executed.

These common sense rules will ensure that regression testing is done smartly and in the right amount. In an ideal world, we would have the time and the resources to test our product completely. Nevertheless, today’s world is a world of tight deadlines and even tighter budgets. Wise resource expenditure today will ensure our ability to continue to develop reliable products tomorrow.
Author – unknown. [If you know the author, or if you are the author please contact us, we will update the  post – Article is published from our old blog and forum]