Click Here to read Function Testing Step by Step – Part 1

Component and component integration testing
– For now, let’s not go in to that. This is a chapter on functional testing. Component and component integration testing are a responsibility of the Development department.
System and System Integration Test Execution
– Once IT tells you can start testing: do some smoke tests:
    – Go through the application part you want to test
    – Check whether you can walk trough the application
    – Look for server errors, buttons that don’t work, navigation problems.
    – Make a decision on this: is the application good enough to start system tests. If not: back to Development Team.

– Test execution
    – Take the lists you made in system test preparation
    – Check every test and mark it ‘pass’ or ‘fail’
    – Leave things you can not test on status ‘to test’ (don’t put them all on ‘fail’!)

    – Make good bug reports for every ‘fail’:
        – make a link with your test case list
        – place this bug in status ‘reported’

    – Once the system tests pass reasonably well, start exploratory testing.
        – Just put away the analysis documents and think destructive while you perform the tests.
        – Behave like you are an end user. What do you want to do with this application?

– Talk about the bugs before programming starts
    – Meet with IT, business and analysts to talk about every bug:
        – Is it relevant? if not : remove the bug from your project view
        – Give priority to each bug (show stopper, major, minor, nice to have)
        – If you work with release, ask in which release this bug has to be resolved
        – Keep track of priority / release number, etc

– Once the bug is fixed, perform retest of the problem as mentioned in the bug report
    – When a test fails, talk about it with IT, business and -if necessary- analysts before programming starts.

Acceptance test execution
– Preparation
    – Book test rooms / invite testers long enough before testing
    – Remind them the day before test execution
    – Make short introduction demo on what you expect of the acceptance testers
    – Tell them how to report to you
    – Explain golden rules for bug reporting

– Let real end users perform the tests you made during acceptance test preparation
– Ask them to use real live data
– After they performed the prepared tests, ask them to perform some exploratory testing
– Make good bug reports for every ‘fail’
– Talk about the bug before start programming (IT, business, testing, analysts)

This is how your project might look like while executing testing:
Project_view_test_execution_example.xls (password: osst)

Introduction
Are you an aspiring Test Lead or Test Manager? Are you ready for transition from test engineer to lead or Lear to a Manager? Here we explain how to set up good testing step by step. You will not find a lot of theory here, only a practical introduction on how to perform good functional testing. You can consider it as a Checklist or Guidelines set. These are the points to take care while planning the test project.

Planning
We are now at the very beginning of the software project. Someone has decided this project has to be started, so why should you wait to get started? Because you’re a tester, you still have lots of time? If you start at the end of the software life cycle, you’ll be in trouble. There’s a lot you can do. Start now!

Method
Make a decision on what testing method you are going to use

Tools
– What test management tool?
– Which Defect tracking tool are you going to use?
– What performance tool?
– Do you need more study on these tools, contact people?
– How and when will performance testing be performed?
– Need to customize the tools?
– Authorizations for these tools?

Test room/ Materials
– Test room?
– PC’s available?
– What software is needed on the testers PC’s?

Test environment
– Is there a separate environment for testing purposes only?
– Who can give authorization on it for your testers?
– Are all parts of the production environment available in the test environment? If not, how to resolve?
– Set up a directory per project for testing and define who has read/write rights / read only rights

Prepare demo’s
– Explain the basics of software testing to project players and testers
– Explain Project View (helicopter view) and other test level views
– Explain why you want the testers to do test preparation sessions before development
– Explain the Golden Rules for Bug reporting
– Explain for every level of testing what you want from the testers
– Explain IT why you want them to perform good testing
– Explain how testers are going to report to you? (Word document? Bring in defects in the bug tracking tool – make sure they have authorization for this)

Testers
– Look for good testing profiles
– End users (super users with the right attitude?)
– System testers (in house? outsourcing?)
– Performance testing, who?
– Do you need consultancy?

Arrangements with IT department
– Make arrangements with IT department on how they are going to test
– Talk about exit and entry criteria (when is software good enough to deliver to test team)
– Ask about organization of the Component and Component integration testing
– How are the builds going to be done? Once a week? Every day?
– Will there be time for smoke tests between the build and the start of your testing?
– Propose a demo for the developers on testing
– Ask about the IT organization of automated component testing

Acceptance test preparation
Okay, the analysts start to talk to key users to define what they need.
Don’t sit and wait till they are finished. Get your acceptance test preparation started!

Organize test preparation sessions with end users (and i mean, do this at the very beginning of the project)
– Ask end users what they really want
– Ask them to give examples of real live cases
– Go with the end users trough the current application or manual method they use now to perform what the application will have to do later
– Ask for details and keep on asking
– Make a list of the cases they mention
– Use a tool, excel.

• These are the test cases you are going to use at test execution
• Make a second list of the questions they might have

– Hand over these lists to analyst team (helps to find the bug early)  

Create (and maintain) an helicopter view on your project
– Decide for each project part what is the

• ‘Impact’ of a bug when it goes wrong (Business van tell you that)
• ‘Change’ of anything going wrong (IT can tell you this)

System test preparation  
– Take each analysis document and break it down into tests that can have a ‘pass’ or ‘fail’ answer
– This will get you a certain number of test cases
    – make a list, one per project part analysis
    – Use a tool, excel.
    – These are the test cases you are going to use at test execution  

– Don’t put anything in these tests that is not in the analysis documents – You can structure the list by using these divisions :
    – check fields:
         – Is the field there?
         – correct type (checkbox, input field, dropdown)

     – check buttons/links:
         – Is the button/link there?
         – does it do what it has to do

     – validation of fields:
         – e.g. is website name correct / incorrect

     – cross check fields:
         – if one field has a certain value this might have implications on another field

     – Processes:
         – make tests on the working of background processes
         – on a given input, what is the expected output
         – here you might ask more input from specialists to get good test cases

     – Database actions:
         – make tests for database fields that will change but are not on screen
         – check IT on how to make these visible

     – Navigation

– If you have questions about the analysis, talk to the analysts! If you find a bug here, no harm is done (just changing the text will do).
 – Look for incomplete info in the analysis (not enough detail, e.g. no defaults, dropdowns without detail, processes that are not clearly described, errors in navigation plan, etc). Make sure to report this information to the analysts.
– When ‘impact’ and ‘chance’ are high for this part of your application, make sure to make a very detailed test preparation. If both are low, don’t spend too much time on it.  

This is how your project might look like at the end of the test preparation phase:
Project_view_after_preparation_example.xls (password of file: osst)

Click Here to read Function Testing Step by Step – Part 2

Article for newbie who want to know what is agile, how agile works, how agile testing methodology is different from traditional  methodologies and what are the challenges in agile.

 
Today’s trend in Software Development is racing towards achieving Targets, Quality and Customer Satisfaction within a limited time frame. This is mostly because the Business Scenario in today’s world is different from what it used to be, a few years ago. Most of the Product Development companies are now adopting a new age concept called “Agile Methodology” for the Software Development Life Cycle.
Software Testing has been a prime focus ever since the IT Industry has realized the importance of Quality of Deliverables, no matter what methodology we follow with the Software Development. However, since the Software Testing is a part of the Business Model, the Testing process also needs to change accordingly. Needless to say, we have so called “Agile Testing” as a result of such a Business Model.

Introduction to Agile Testing
While the given application under test is still evolving depending upon the customer needs, the mindset of the end user and the current market condition, it is highly impractical to go for the usual standard SDLC Models like Water Fall, V&V Model etc. Such models are most suitable for the Applications that are stable and non-volatile. The concept of “Time-To-Market” is the key word in today’s IT Business that compels the Software vendors to come up with new strategies to save the time, resources, cut down the cost involved and at the same time, deliver a reliable product that meets the user requirements. In this case, a reasonably good amount of end-to-end testing is carried out and the product could be acceptable with known issues/defects at the end of an intermediate release. These defects are harmless for the Application usability.
To adopt such a process in a systematic way, we have a new concept called Agile Methodology. This methodology continuously strives to overcome the issues of dynamically changing requirements while still trying to maintain a well-defined process.

The process is as follows:
1. The Customer prepares the Business Requirements and the Business Analyst or the Engineering team reviews it. Ideally, the Quality Assurance/Testing team is also involved in reviewing these requirements in order to be able to plan further stages accordingly.

2. During the Design and Implementation stages, the Engineering team writes User Stories and the analysis of issues at various stages. The Customer reviews these on regular basis and updates the Requirement specifications accordingly. The Testing team would follow up on regular basis at every stage until a consolidated documentation is prepared. This is to ensure that the Customer, the Engineering team and the Testing team are at the same page always and thus ensuring complete test coverage.

3. While the Engineering team starts the implementation, the Testing team starts with test planning, test strategies and test cases preparation. These would be properly documented and handed over to the Customer and the Engineering team for review. This is to ensure the complete test coverage and avoid unnecessary or redundant test cases.

4. As and when the Developer implements the code, the Testing team identifies if the application can be built using this code for a quick testing. This is to identify the defects at the early stage so that the developer can fix them in the next round on priority basis and continue with further development. This iteration continues until the end of the code implementation. Once the testing cycle starts, the Test team can now focus more on major test items such as Integration, Usability Testing and System Testing etc.

Scope of testing an application
The Testing team knows the complexity involved and it is accepted by the customer that the software development and/or the enhancements and hence the testing is a continuous process. Testing of the application at a black box level would suffice in order to identify the issues and raise the defects by the Testing team. The application continues to evolve until it reaches the stage of final acceptance. Hence the scope of testing would continue to evolve as per the Customer needs.

Process followed at various stages in the product life cycle
Every intermediate release of the product would be divided into two short cycles, usually of the duration of 40 days each. Each cycle would be executed in the following stages. The roles and responsibilities of every individual and the team are clearly defined for each stage.

– Design Specifications: The Testing team’s efforts would focus on performing any tool or process improvements and reviewing, understanding, and contributing to the nascent specifications.
– Implementation: While the Engineering/Development team is implementing the code, Testers would develop complete Testing Plan and Test Sets (set of test cases) for each of the features included in the cycle. Engineering features must be included; they would likely require some level of collaboration with the engineering feature developer. All Test Sets should be ready to execute by the end of implementation period of the respective cycle. After Test Set preparation, calculate the time estimation and prioritization for the Test Set execution based on the complexity and expected execution time for each test suite.
– While the test execution time estimation is notoriously difficult, this number should provide the Customer with a starting point for benchmarking.
– Testing/QA: Test Set execution, raising defects and follow up with the Engineering Team. End-to-end validation of the defects. Focus simultaneously on improving the quality of test cases. Watching out for and adding new cases as testing proceeds. Testing the software end-to-end to discover regressions and subtle systemic issues. Learning to focus more on using the time available to uncover the largest number of and most important bugs. Any deviation from the estimated time should be communicated across well in advance, so that the schedule can be worked upon depending upon the priority of the pending tasks. If there are certain issues or test cases blocking due to unknown errors, they would be differed until the beginning of next Testing/QA Cycle.
– Before acceptance: Follow up on ad-hoc requests/ changes in requirements on a regular basis, besides trying to complete the defined tasks.

Looking at various broad areas of testing of a complex application, the system structure, and the depth of functionality implemented and the level of complexity, the complete end-to-end Test Execution within a limited time frame would be next to impossible with any standard SDLC Model available. The Product/application involves a good deal of learning of how and when to use the application and requires that the end user know the functionality of various modules prior to constructing a User defined model (UDM) for his Business purpose or even for testing.

Reasons for Agile Testing methodology to test an Application

The testing wouldn’t be 100% complete in this case before the finished product reaches the hands of the end user. This is true especially when the target audience and its system structure are unknown. Different users have their own set of ideas and unique problems. Given this fact, it is hard to say that it is 100% bug free software when it reaches the customer. Hence, taking into account the constraints involved in using the standard SDLC Models, it is worth adopting an approach such as Agile Testing, which is more suitable for the dynamically changing requirements.

Context Driven Testing and Rapid Testing:
This type of testing is what usually the Agile Testers incorporate. Context Driven testing is the one where the test scenarios are not known before hand. It mostly comes from the context in which the application is being executed. In our case, it is constructing the UDMs based on a given use case from the targeted end user and test it for the scenarios that are explained by his system configuration. It also includes constructing the UDMs based on any trouble-shooting discussion arising out of defects pointed out during the Customer acceptance and testing.

Rapid Testing is a process of defining our own strategies to test the Software, not necessarily following any specific Process or a Model. It focuses mainly on identifying the defects as quickly as possible rather than focusing on the end user requirements.

Heuristic approach is used in such cases. The tester uses his common sense and previous work experience to test the application at various levels in order to figure out where the application stands.

References: http://www.testing.com/agile

Online Documentation for Agile Testing and Rapid Testing:
Agile Testing Challenges
Rapid Software Testing

Error Guessing
Why can one Tester find more errors than another Tester in the same piece of software?

More often than not this is down to a technique called ‘Error Guessing’. To be successful at Error Guessing, a certain level of knowledge and experience is required. A Tester can then make an educated guess at where potential problems may arise. This could be based on the Testers experience with a previous iteration of the software, or just a level of knowledge in that area of technology. This test case design technique can be very effective at pin-pointing potential problem areas in software. It is often be used by creating a list of potential problem areas/scenarios, then producing a set of test cases from it. This approach can often find errors that would otherwise be missed by a more structured testing approach.

An example of how to use the ‘Error Guessing’ method would be to imagine you had a software program that accepted a ten digit customer code. The software was
designed to only accept numerical data.

Here are some example test case ideas that could be considered as Error Guessing:
1. Input of a blank entry
2. Input of greater than ten digits
3. Input of mixture of numbers and letters
4. Input of identical customer codes

What we are effectively trying to do when designing Error Guessing test cases, is to think about what could have been missed during the software design.

This testing approach should only be used to compliment an existing formal test method, and should not be used on its own, as it cannot be considered a complete form of testing software.

Exploratory Testing
This type of testing is normally governed by time. It consists of using tests based on a test chapter that contains test objectives. It is most effective when there are little or no specifications available. It should only really be used to assist with, or compliment a more formal approach. It can basically ensure that major functionality is working as expected without fully testing it.

Ad-hoc Testing

This type of testing is considered to be the most informal, and by many it is considered to be the least effective. Ad-hoc testing is simply making up the tests as you go along. Often, it is used when there is only a very small amount of time to test something. A common mistake to make with Ad-hoc testing is not documenting the tests performed and the test results. Even if this information is included, more often than not additional information is not logged such as, software versions, dates, test environment details etc.
Ad-hoc testing should only be used as a last resort, but if careful consideration is given to its usage then it can prove to be beneficial. If you have a very small window in which to test something, then the following are points to consider:

1. Take some time to think about what you want to achieve
2. Prioritize functional areas to test if under a strict amount of testing time
3. Allocate time to each functional area when you want to test the whole item
4. Log as much detail as possible about the item under test and its environment
5. Log as much as possible about the tests and the results

Random Testing

A Tester normally selects test input data from what is termed an ‘input domain’ in a structured manner. Random Testing is simply when the Tester selects data from the input domain ‘randomly’. In order for random testing to be effective, there are some important open questions to be considered:
1. Is random data sufficient to prove the module meets its specification when tested?
2. Should random data only come from within the ‘input domain’?
3. How many values should be tested?

As you can tell, there is little structure involved in ‘Random Testing’. In order to avoid dealing with the above questions, a more structured Black-box Test Design could be implemented instead. However, using a random approach could save valuable time and resources if used in the right circumstances. There has been much debate over the effectiveness of using random testing techniques over some of the more structured techniques. Most experts agree that using random test data provides little chance of producing an effective test. There are many tools available today that are capable of selecting random test data from a specified data value range. This approach is especially useful when it comes to tests associated at the system level. You often find in the real world that ‘Random Testing’ is used in association with other structured techniques to provide a compromise between targeted testing and testing everything.