Levels and Techniques in Software Testing

Testing techniques can be defined in three ways: Preparation, Execution and Approach.Preparation: From preparation point of view there are two testing techniques: Formal Testing and Informal Testing.

Formal Testing: Testing performed with a plan, documented set of test cases, etc that outline the methodology and test objectives. Test documentation can be developed from requirements, design, equivalence partitioning, domain coverage, error guessing, etc. The level of formality and thoroughness of test cases will depend upon the needs of the project. Some projects can have rather informal ‘formal test cases’, while others will require a highly refined test process. Some projects will require light testing of nominal paths while others will need rigorous testing of exceptional cases.

Informal Testing: Ad hoc testing performed without a documented set of objectives or plans. Informal testing relies on the intuition and skills of the individual performing the testing. Experienced engineers can be productive in this mode by mentally performing test cases for the scenarios being exercised.

From the execution point of view, the two testings types are: Manual Testing and Automated Testing.

Manual Testing: Manual testing involves direct human interaction to exercise software functionality and note behavior and deviations from expected behavior.

Automated Testing: Testing that relies on a tool, built-in test harness, test framework, or other automatic mechanism to exercise software functionality, record output, and possibly detect deviations. The test cases performed by automated testing are usually defined as software code or script that drives the automatic execution.

From the testing approach point of view, the two testings types are: Structural Testing and Functional Testing.

Structural Testing: Structural testing depends upon knowledge of the internal structure of the software. Structural testing is also referred to as white-box testing.

Data-flow Coverage: Data-flow coverage tests paths from the definition of a variable to its use.


Control-flow Coverage

Statement Coverage: Statement coverage requires that every statement in the code under test has been executed.

Branch Coverage: Branch coverage requires that every point of entry and exit in the program has been executed at least once, and every decision in the program has taken all possible outcomes at least one.

Condition Coverage: Condition coverage is branch coverage with the additional requirement that “every condition in a decision in the program has taken all possible outcomes at least once.” Multiple condition coverage requires that all possible combinations of the possible outcomes of each condition have been tested. Modified condition coverage requires that each condition has been tested independently.

Functional Testing: Functional testing compares the behavior of the test item to its specification without knowledge of the item’s internal structure. Functional testing is also referred to as black box testing.

Requirements Coverage: Requirements coverage requires at least one test case for each specified requirement. A traceability matrix can be used to insure that requirements coverage has been satisfied.

Input Domain Coverage: Input domain coverage executes a function with a sufficient set of input values from the function’s input domain. The notion of a sufficient set is not completely definable, and complete coverage of the input domain is typically impossible. Therefore the input domain is broken into subsets, or equivalence classes, such that all values within a subset are likely to reveal the same defects. Any one value within an equivalence class can be used to represent the whole equivalence class. In addition to a generic representative, each extreme value within an equivalence class should be covered by a test case. Testing the extreme values of the equivalence classes is referred to as boundary value testing.

Output Domain Coverage: Output domain coverage executes a function in such a way that a sufficient set of output values from the function’s output domain is produced. Equivalence classes and boundary values are used to provide coverage of the output domain. A set of test cases that “reach” the boundary values and a typical value for each equivalence class is considered to have achieved output domain coverage.

Various Software Testing Levels

Although many testing levels tend to be combined with certain techniques, there are no hard and fast rules. Some types of testing imply certain lifecycle stages, software deliverables, or other project context. Other types of testing are general enough to be done almost any time on any part of the system. Some require a particular methodology. When appropriate common utilizations of a particular testing type will be described. The project’s test plan will normally define the types of testing that will be used on the project, when they will be used, and the strategies they will be used with. Test cases are then created for each testing type.

Unit Testing: A unit is an abstract term for the smallest thing that can be conveniently tested. This will vary based on the nature of a project and its technology but usually focuses at the subroutine level. Unit testing is the testing of these units. Unit testing is often automated and may require creation of a harness, stubs, or drivers.

Component Testing: A component is an aggregate of one or more components. Component testing expands unit testing to include called components and data types. Component testing is often automated and may require creation of harness, stubs, or drivers.

Single Step Testing: Single step testing is performed by stepping through new or modified statements of code with a debugger. Single step testing is normally manual and informal.

Bench Testing: Bench testing is functional testing of a component after the system has been built in a local environment. Bench testing is often manual and informal.

Developer Integration Testing: Developer integration testing is functional testing of a component after the component has been released and the system has been deployed in a standard testing environment. Special attention is given to the flow of data between the new component and the rest of the system.

Smoke Testing: Smoke testing determines whether the system is sufficiently stable and functional to warrant the cost of further, more rigorous testing. Smoke testing may also communicate the general disposition of the current code base to the project team. Specific standards for the scope or format of smoke test cases and for their success criteria may vary widely among projects.

Feature Testing: Feature testing is functional testing directed at a specific feature of the system. The feature is tested for correctness and proper integration into the system. Feature testing occurs after all components of a feature have been completed and released by development.

Integration Testing: Integration testing focuses on verifying the functionality and stability of the overall system when it is integrated with external systems, subsystems, third party components, or other external interfaces.

System Testing: System testing occurs when all necessary components have been released internally and the system has been deployed onto a standard environment. System testing is concerned with the behavior of the whole system. When appropriate, system testing encompasses all external software, hardware, operating environments, etc. that will make up the final system.

Release Testing: Release tests ensure that interim builds can successfully deployed by the customer. This includes product deployment, installation, and a pass through the primary functionality. This test is done immediately before releasing to the customer.

Beta Testing: Beta testing consists of deploying the system to many external users who have agreed to provide feedback about the system. Beta testing may also provide the opportunity to explore release and deployment issues.

Acceptance Testing: Acceptance testing compares the system to a predefined set of acceptance criteria. If the acceptance criteria are satisfied by the system, the customer will accept delivery of the system.

Regression Testing: Exercises functionality that has stabilized. Once high confidence has been established for certain parts of the system, it is generally wasted effort to continue rigorous, detailed testing of those parts. However, it is possible that continued evolution of the system will have negative effects on previously stable and reliable parts of the system. Regression testing offers a low-cost method of detecting such side effects. Regression testing is often automated and focused on critical functionality.

Performance Testing: Performance testing measures the efficiency with respect to time and hardware resources of the test item under typical usage. This assumes that a set of non-functional requirements regarding performance exist in the item’s specification.

Stress Testing: Stress testing evaluates the performance of the test item during extreme usage patterns. Typical examples of “extreme usage patterns” are large data sets, complex calculations, extended operation, limited system resources, etc.

Configuration Testing: Configuration testing evaluates the performance of the test item under a range of system configurations. Relevant configuration issues depend upon the particular product and may include peripherals, network patterns, operating systems, hardware devices and drivers, user settings.

Identifying Testing Types and Exit Criteria

Identifing Manual / Automated Test Types

The types of tests that need to be designed and executed depend totally on the objectives of the application, i.e., the measurable end state the organization strives to achieve. For example, if the application is a financial application used by a large number of individuals, special security and usability tests need to be performed. However, three types of tests which are nearly always required are: function, user interface, and regression testing. Function testing comprises the majority of the testing effort and is concerned with verifying that the functions work properly. It is a black-box-oriented activity in which the tester is completely unconcerned with the internal behavior and structure of the application. User interface testing, or GUI testing, checks the user’s interaction or functional window structure. It ensures that object state dependencies function properly and provide useful navigation through the functions. Regression testing tests the application in light of changes made during debugging, maintenance, or the development of a new release.

Other types of tests that need to be considered include system and acceptance testing. System testing is the highest level of testing which evaluates the functionality as a total system, its performance and overall fitness of use. Acceptance testing is an optional user-run test which demonstrates the ability of the application to meet the user’s requirements. This test may or may not be performed based on the formality of the project. Sometimes the system test suffices.

Finally, the tests that can be automated with a testing tool need to be identified. Automated tests provide three benefits: repeatability, leverage, and increased functionality. Repeatability enables automated tests to be executed more than once, consistently. Leverage comes from repeatability from tests previously captured and tests that can be programmed with the tool, which may not have been possible without automation. As applications evolve, more and more functionality is added. With automation, the functional coverage is maintained with the test library.

Identifing the Test Exit Criteria:

One of the most difficult and political problems is deciding when to stop testing, since it is impossible to know when all the defects have been detected. There are at least four criteria for exiting testing:

Scheduled testing time has expired: This criteria is very weak, since it has nothing to do with verifying the quality of the application. This does not take into account that there may be an inadequate number of test cases or the fact that there may not be any more defects that are easily detectable.

Some predefined number of defects discovered: The problems with this is knowing the number of errors to detect and also overestimating the number of defects. If the number of defects is underestimated, testing will be incomplete. Potential solutions include experience with similar applications developed by the same development team, predictive models, and industry-wide averages. If the number of defects is overestimated, the test may never be completed within a reasonable time frame. A possible solution is to estimate completion time, plotting defects detected per unit of time. If the rate of defect detection is decreasing dramatically, there may be “burnout,” an indication that a majority of the defects have been discovered.

All the formal tests execute without detecting any defects: A major problem with this is that the tester is not motivated to design destructive test cases that force the tested program to its design limits, e.g., the tester’s job is completed when the test program fields no more errors. The tester is motivated not to find errors and may subconsciously write test cases that show the program is error free. This criteria is only valid if there is a rigorous and totally comprehensive test case suite created which approaches 100% coverage. The problem with this is determining when there is a comprehensive suite of test cases. If it is felt that this is the case, a good strategy at this point is to continue with ad hoc testing. Ad hoc testing is a black-box testing technique in which the tester lets his or her mind run freely to enumerate as many test conditions as possible. Experience has shown that this technique can be a very powerful supplemental or add-on technique.

Combination of the above: Most testing projects utilize a combination of the above exit criteria. It is recommended that all the tests be executed, but any further ad hoc testing will be constrained by time.

How to Organizing a Test Team

The people component includes human resource allocations and the required skill sets. The test team should comprise the highest-caliber personnel possible. They are usually extremely busy because their talents put them in great demand, and it therefore becomes vital to build the best case possible for using these individuals for test purposes. A test team leader and test team need to have the right skills and experience, and be motivated to work on the project. Ideally, they should be professional quality assurance specialists but can represent the executive sponsor, users, technical operations, database administration, computer center, independent parties, etc. The latter is particularly useful during final system and acceptance testing. In any event, they should not represent the development team, for they may not be as unbiased as an outside party. This is not to say that developers shouldn’t test. For they should unit and function test their code extensively before handing it over to the test team.

There are two areas of responsibility in testing:

1. Testing the application, which is the responsibility of the test team
2. The overall testing processes, which is handled by the test manager.

The test manager directs one or more testers, is the interface between quality assurance and the development organization, and manages the overall testing effort. Responsibilities include:

• Setting up the test objectives
• Defining test resources
• Creating test procedures
• Developing and maintaining the test plan
• Designing test cases
• Designing and executing automated testing tool scripts
• Test case development
• Providing test status
• Writing reports
• Defining the roles of the team members
• Managing the test resources
• Defining standards and procedures
• Ensuring quality of the test process
• Training the team members
• Maintaining test statistics and metrics

The test team must be a set of team players and have the following responsibilities:

• Execute test cases according to the plan
• Evaluate the test results
• Report errors
• Design and execute automated testing tool scripts
• Recommend application improvements
• Record defects

The main function of a team member is to test the application and report defects to the development team by documenting them in a defect tracking system. Once the development team corrects the defects, the test team reexecutes the tests which discovered the original defects.

It should be pointed out that the roles of the test manager and team members are not mutually exclusive. Some of the team leader’s responsibilities are shared with the team member and visa versa.

The basis for allocating dedicated testing resources is the scope of the functionality and the development time frame, e.g., a medium development project will require more testing resources than a small one. If project A of medium complexity requires a testing team of 5, project B with twice the scope would require 10 testers (given the same resources).

Another rule of thumb is that the testing costs approach 25% of the total budget. Since the total project cost is known, the testing effort can be calculated and translated to tester headcount.

The best estimate is a combination of the project scope, test team skill levels, and project history. A good measure of required testing resources for a particular project is the histories of multiple projects, i.e., testing resource levels and performance compared to similar projects.

Build Verification Testing : An Introduction

Build Verification Testing or Smoke Testing is a set of tests that run on new build to verify that whether the build is testable or not. It is done prior to its release to test team for further testing. This testing is done for Build Validation and Build Acceptance.


The test cases of Build Verification Testing can include core functionality test cases that ensure software / application is stable and can be tested thoroughly. Some key points for this kind of Software Testing is:

• The Build Verification tests are subset of tests cases that verify main functionalities
• These tests typically run for each build. If any of the tests fail, the build is rejected
• It is done to save the efforts of a testing team to setup and test a build when major functionalities are having defects
• An ideal BVT should not run more than 30 - 60 minutes depending on the testing points in the application.

It is better is these tests can be automated. If any of the tests fails, then developers fix the issues and deploy these to testing server.
In Build Verification Testing, one needs to check for the integrity of various modules of the application. Checking the integration of various modules is important when different teams work on different modules.


Some Basic Checks:

• Check whether - all the new and modified files are included in release
• All file formats are correct
• Every file version and language
• Flags associated with each file

Below are some tips to select Build Verification tests:

• Include only critical test cases and they should be sufficient for application test coverage
• Add only stable test cases and all the test cases should have known expected results
• Do not include modules in BVT, which are not yet stable
• Set some standards and these standards shall be met only by analyzing major project features and scenarios
• BVT automation scripts needs to be maintained and modified time-to-time. Include test cases when there are new stable project modules available
• Try to automate this process as much as possible - automate everything
• Do not write BVT test cases scripts in hurry

Process for running the build verification tests:

• The results are sent to TL / PM
• Results are analyzed by TL / PM
• The person who runs the tests and TL / PM diagnoses the cause of failure (if any)
• If there is any defect, the relevant information is sent to respective developers
• Developer fixes the bug

Once the bug is fixed; BVT test suite is executed again. This process gets repeated for every new build.


Also, remember that some times tests fail because of the following reasons:

• Test case coding error
• Automation Tool error
• Infrastructure error
• Hardware / software failures etc.

So, see the root causes of failures, and then take proper action. Log as much detailed info as possible to diagnose the BVT pass or fail result.

Spiral Testing Approach : An overview

The purpose of software testing is to identify the differences between existing and expected conditions, i.e., to detect software defects. Testing identifies the requirements that have not been satisfied and the functions that have been impaired. The most commonly recognized test objective is to identify bugs, but this is a limited definition of the aim of testing. Not only must bugs be identified, but they must be put into a framework that enables testers to predict how the software will perform.

In the spiral and rapid application development testing environment there may be no final functional requirements for the system. They are probably informal and evolutionary. Also, the test plan may not be completed until the system is released for production. The relatively long lead time to create test plans based on a good set of requirement specifications may not be available. Testing is an ongoing improvement process that occurs frequently as the system changes. The product evolves over time and is not static.

The testing organization needs to get inside the development effort and work closely with development. Each new version needs to be tested as it becomes available. The approach is to first test the new enhancements or modified software to resolve defects reported in the previous spiral. If time permits, regression testing is then performed to assure that the rest of the system has not regressed.

In the spiral development environment, software testing is again described as a continuous improvement process that must be integrated into a rapid application development methodology. Testing as an integrated function prevents development from proceeding without testing. Deming’s continuous improvement process using the PDCA model will again be applied to the software testing process.
Before the continuous improvement process begins, the testing function needs to perform a series of information-gathering planning steps to understand the development project objectives, current status, project plans, function specification, and risks.

Once this is completed, the formal Plan step of the continuous improvement process commences. A major step is to develop a software test plan. The test plan is the basis for accomplishing testing and should be considered an ongoing document, i.e., as the system changes, so does the plan. The outline of a good test plan includes an introduction, the overall plan, testing requirements, test procedures, and test plan details. These are further broken down into business functions, test scenarios and scripts, function/test matrix, expected results, test case checklists, discrepancy reports, required software, hardware, data, personnel, test schedule, test entry criteria, exit criteria, and summary reports.

The more definitive a test plan is, the easier the plan step will be. If the system changes between development of the test plan and when the tests are to be executed, the test plan should be updated accordingly.

The Do step of the continuous improvement process consists of test case design test development and test execution. This step describes how to design test cases and execute the tests included in the test plan. Design includes the functional tests, GUI tests, and fragment system and acceptance tests. Once an overall test design is completed, test development starts. This includes building test scripts and procedures to provide test case details.

The test team is responsible for executing the tests and must ensure that they are executed according to the test design. The do step also includes test setup, regression testing of old and new tests, and recording any defects discovered.

The Check step of the continuous improvement process includes metric measurements and analysis. “Quality Through a Continuous Improvement Process,” crucial to the Deming method is the need to base decisions as much as possible on accurate and timely data. Metrics are key to verifying if the work effort and test schedule are on schedule, and to identify any new resource requirements.
During the check step it is important to publish intermediate test reports. This includes recording of the test results and relating them to the test plan and test objectives.

The Act step of the continuous improvement process involves preparation for the next spiral iteration. It entails refining the function/GUI tests, test suites, test cases, test scripts, and fragment system and acceptance tests, modifying the defect tracking system and the version and control system, if necessary. It also includes devising measures for appropriate actions relating to work that was not performed according to the plan or results that were not what was anticipated. Examples include a reevaluation of the test team, test procedures, and technology dimensions of testing. All the above is fed back to the test plan, which is updated.

Once several testing spirals have been completed and the application has been verified as functionally stable, full system and acceptance testing starts. These tests are often optional. Respective system and acceptance test plans are developed defining the test objects and the specific tests to be completed.

The final activity in the continuous improvement process is summarizing and reporting the spiral test results. A major test report should be written at the end of all testing. The process used for report writing is the same whether it is an interim or a final report, and, like other tasks in testing, report writing is also subject to quality control. However, the final test report should be much more comprehensive than interim test reports. For each type of test it should describe a record of defects discovered, data reduction techniques, root cause analysis, the development of findings, and follow-on recommendations for the current and/or future projects.

The methodology provides a framework for testing in this environment. The major steps include information gathering, test planning, test design, test development, test execution/evaluation, and preparing for the next spiral. It includes a set of tasks associated with each step or a checklist from which the testing organization can choose based on its needs. The spiral approach flushes out the system functionality. When this has been completed, it also provides for classical system testing, acceptance testing, and summary reports.

Need of Continuous Improvement Process in testing

Software life cycle testing means that testing occurs in parallel with the development cycle and is a continuous process. The software testing process should start early in the application life cycle, not just in the traditional validation testing phase after the coding phase has been completed. Testing should be integrated into application development. In order to do so, there needs to be a commitment on the part of the development organization and close communication with the quality assurance function.
A test plan is started during the requirements phase. It is an organization of testing work. It is a document describing the approach to be taken for the intended testing activities and includes the items to be tested, the types of tests to be performed, test schedules, human resources, reporting procedures, evaluation criteria, etc.

During logical, physical, and program unit design, the test plan is refined with more details. Test cases are also created. A test case is a specific set of test data and test scripts. A test script guides the tester through a test and ensures consistency among separate executions of the test. A test also includes the expected results to verify whether the test met the objective correctly. During the coding phase, test scripts and test data are generated. During application testing, the test scripts are executed and the results are analyzed.

The application development cycle proceeds from user requirements and design until the code is completed. During test design and development, the acceptance test criteria was established in a test plan. As more details are refined, the system, integration, and unit testing requirements are established. There may or may not be a separate test plan for each test type, or one plan may be used.

During test execution, the process is reversed and starts with unit testing. Integration tests are performed which combine individual, unit tested pieces of code. Once this is completed, the system is tested from a total system point of view. This is known as system testing. System testing is a multifaceted test to evaluate the functionality, performance, and usability of the system. The final test is the acceptance test, which is a user-run test that verifies the ability of the system to meet the original user objectives and requirements. In some cases the system test serves as the acceptance test.

If you will recall, the PDCA approach, i.e., plan, do, check, and act, is a control mechanism used to control, supervise, govern, regulate, or restrain a system. The approach first defines the objectives of a process, develops and carries out the plan to meet those objectives, and checks to determine if the anticipated results are achieved. If they are not achieved, the plan is modified to fulfill the objectives. The PDCA quality cycle can be applied to software testing.

The Plan step of the continuous improvement process, when applied to software testing, starts with a definition of the test objectives, e.g., what is to be accomplished as a result of testing. Testing criteria do more than simply ensure that the software performs according to specifications. Objectives ensure that all responsible individuals contribute to the definition of the test criteria to maximize quality.

A major deliverable of this step is a software test plan. A test plan is the basis for accomplishing testing. The test plan should be considered an ongoing document. As the system changes, so does the plan. The test plan also becomes part of the system maintenance documentation after the application is delivered to the user. The outline of a good test plan includes an introduction, the overall plan, and testing requirements. As more detail is available, the business functions, test logs, problem and summary reports, test software, hardware, data, personnel requirements, test schedule, test entry criteria, and exit criteria are added.

The Do step of the continuous improvement process when applied to software testing describes how to design and execute the tests included in the test plan. The test design includes test cases, test procedures and scripts, expected results, function/test case matrix, test logs, etc. The more definitive a test plan is, the easier the test design will be. If the system changes between development of the test plan and when the tests are to be executed, the test plan should be updated accordingly, i.e., whenever the system changes, the test plan should change.

The test team is responsible for the execution of the tests and must ensure that the test is executed according to the plan. Elements of the Do step include selecting test tools, defining the resource requirements, defining the test setup conditions and environment, test requirements, and the actual testing of the application.

The Check step of the continuous improvement process when applied to software testing includes the evaluation of how the testing process is progressing. The credo for statisticians, “In God we trust. All others must use data,” is crucial to the Deming method. It is important to base decisions as much as possible on accurate and timely data. Testing metrics such as the number and types of defects, the workload effort, and the schedule status are key.

It is also important to create test reports. Testing began with setting objectives, identifying functions, selecting tests to validate the test functions, creating test conditions, and executing the tests. To construct test reports, the test team must formally record the results and relate them to the test plan and system objectives. In this sense, the test report reverses all the previous testing tasks.

Summary and interim test reports should be written at the end of testing and at key testing checkpoints. The process used for report writing is the same whether it is an interim or a summary report, and, like other tasks in testing, report writing is also subject to quality control, i.e., it should be reviewed. A test report should at least include a record of defects discovered, data reduction techniques, root cause analysis, the development of findings, and recommendations to management to improve the testing process.

The Act step of the continuous improvement process when applied to software testing includes devising measures for appropriate actions relating to work that was not performed according to the plan or results that were not anticipated in the plan. This analysis is fed back to the plan. Examples include updating the test suites, test cases, test scripts, reevaluating the people, process, and technology dimensions of testing.

NAS (Network Attached Storage) Testing

All media appear to be going digital these days, including your photos, music, home movies, and even television. But where do you store your ever expanding media collection so that the rest of your family can access it seamlessly across your home network? Or how do you make files accessible to your networked coworkers in a small office without investing in an expensive file server? Enter the relatively new product category of network-attached storage (NAS). While storage capacity, functionality, and ease of use are all important variables to consider, performance should not be overlooked. Not all NAS devices are created equal, and you'll really feel the difference when you're waiting for those huge files to open. In order to deliver relevant performance evaluations of NAS devices, companies Labs puts them through their paces with the real-world tasks of reading and writing files.

Test environment: Network-attached storage devices are tested using a Netgear GS605 five-port Gigabit Ethernet switch on a closed network. File transfer testing takes place over the network between the NAS device and a desktop system with a 3.4GHz Pentium 4 550 processor, 1GB of DDR2 SDRAM running at 533MHz, an Nvidia GeForce 6600 PCI-Express graphics card with 256MB of memory, an integrated Yukon 88E8053 PCI Express Gigabit Ethernet Controller, a 74GB Western Digital WD740 Raptor hard drive, and Windows XP Professional SP2.

This NAS device is installed according to the manufacturer's documentation, including installing any necessary client software and updating the device's firmware if available. As most NAS devices come preformatted, we test the drive with its existing partition types and sizes as is. Tested devices have only test files placed on them; the operating system and all application files reside on the test bed's permanent hard drive.

Transfer-speed tests: The read and write transfer-speed tests are done using a folder containing approximately 5GB of data; which contains DOC, XLS, JPEG, GIF, HTML, TXT, MP3, AVI, and application installation files, ranging in size from 1KB to 700MB.

Some companies use a custom utility to simulate the drag-and-drop mode of file copying in Windows Explorer. The custom utility automatically times the file transfer tests, reporting how long it takes to complete the transfer in minutes and seconds as well as megabytes per second.

Before testing begins, the test bed's hard drive is defragmented using Windows' built-in defragmentation tool. The write test is conducted by timing how long it takes to copy the 5GB folder from the test bed's hard drive to the NAS device. The read test is conducted by timing how long it takes to copy the 5GB folder from the NAS device to the test bed's hard drive. All files copied to the NAS device stay on the device and are not deleted between test runs. This allows the device to increasingly fill up with data as testing continues. The test bed's drive, on the other hand, has all files copied to it deleted between test runs in order to minimize the impact the test bed's drive will have on the performance of the NAS device.

All tests are repeated a minimum of three times. Each reported score represents an average of three scores that are within 5 percent of each other; iterations that vary by more than 5 percent are thrown out. All scores are reported in minutes and seconds, so lower scores indicate faster performance.

Thread Based Integration Testing : Introduction

If the Project is large enough and we have to deliver it soon, then, in order to deliver the product within the projected schedule, parallel development and rapid integration should occurred over many related software functional areas. To facilitate the decomposition of design into manageable components, the concept of a "functional thread" as the elementary building block for integration can be chosen. In this context, a "functional thread" is defined as a logical execution sequence through a series of interfacing software components resulting from or ending in the receipt of a message, event or operator interaction.

Threads not only serve as the basis for integration, they also tend to drive the entire software development effort from scheduling to status reporting. Each thread itself represents a microcosm of the system in that each has a documented definition and general execution path, an internal design and an associated test. Thread definition intends to communicate functional background and execution details between developers and from developers to testers. More importantly, the desired independence of threads supports incremental integration and system testing while the corresponding thread definition substantiates the results. Finally, since all system development activity progresses in relation to threads, management has an accurate method of judging the status of individual tasks, functional areas and requirements.


Keeping the goals of iterative development and software testing in mind, each thread has its own lifecycle with autonomous states and a formal process for state transitions. Individual team leaders usually decompose general requirements into groups of threads at the beginning of formal, six month software builds and assign threads to developers. Developers maintain ownership of their threads and are responsible for documenting a scenario under which an integrator can verify the basic functionality, providing rudimentary definition to the thread. Following implementation and unit testing, the developer releases the corresponding software components to a daily integration build, at which point the thread enters a "testable" state. After verifying the functionality in the integration build, the developer marks the thread "ready" for an integrator who performs more extensive testing and eventually "integrates" the thread and corresponding software components into the system. At the end of each formal build, a team of key engineers in conjunction with quality assurance checks all threads against requirements as a regression test and "finalizes" those threads which pass.

While the development team originally tracked threads manually, we quickly developed a shared database application to serve as a central repository for thread development, maintenance and tracking. The database provides a formal mechanism for defining and documenting threads, changing thread status and reporting status to project management. Moreover, the database manages references between threads. Threads can serve as preconditions to other threads and developers may incorporate thread test steps from previous threads. Most importantly, the interface helps enforce the process by demonstrating the autonomy of thread status and establishing clearly defined responsibilities among developers and testers.

Thread Test Steps
Thread test steps and other background information from the database serve as a contract between developers and integrators. Integrators use thread test steps as a simple scenario to identify the scope of a thread rather than as a rigid test case that may only rubber-stamp a developer’s unit test. Consequently, the integrators are responsible for developing several execution scenarios within the boundaries of the thread and applying appropriate testing mechanisms such as known exceptional cases and boundary checking. Furthermore, the integration team often stresses exercising subsystem interfaces during integration testing, which was an area that thread steps often overlooked.

In addition to helping formalize the implementation process, the thread testing approach standardizes the integration testing process as well. As a result, the number of detected coding errors increased almost 250 percent over three formal builds after thread testing had been introduced. Although errors attributable to integration doubled during the first formal build during which our group used threads, that number has subsequently dropped to almost fifty percent below the level at which we started using threads.

While thread-based development certainly contributes greatly to the main goals of early, rapid integration and iterative development, we have also identified several potential areas of further process improvement. Perhaps most notably, developers and testers shared concerns that thread scope lacked uniformity among subsystems. At times, thread definitions were far too specific and a conscientious integrator could verify the basic functionality in fewer steps than the developer identified. Likewise, developers sometimes defined threads at too high a level, requiring the integrator to seek further information from the developer to ensure a meaningful test. A thread review process, perhaps as part of a design walk through, may answer this problem. Likewise, we recommend requiring completion of a code walk through as a prerequisite to thread completion due to the implications of walk through initiated design and code changes.

Thread Maintenance
A related area of improvement is thread maintenance. While the process encouraged (and the database supported) threads referencing other threads, maintaining consistency was not always an easy task. Furthermore, while software that composes a thread often changes after a thread has been integrated, there is no formal update process for the thread. The changes to process here are obvious and one could modify the tool to help enforce these concerns. For example, the tool would benefit from the ability to attach references to source code units so that changes to code might trigger the need for associated thread changes.

In large projects, with less time, the thread process focuse on the integration activities rather than the full development lifecycle. This is certainly the main difference between thread-based approach and use-case analysis. The thread database requires references to user interface specifications where applicable, but the process did not link the thread directly to the requirements database. Thus software testing and overall system testing were somewhat disjoint in that system testers found it difficult to use the thread database as a reference when creating test cases. Though it might be desirable to shift thread definition to the requirements analysis phases of the project, such analysis usually occurs at a higher level than what we had used for our threads and almost always span subsystem boundaries. Instead I suggest a more hierarchical approach to thread definition rooted in requirement-based parent threads. This would directly link the software thread repository to system requirements and better facilitate a similar iterative approach to system-wide testing. Finally, by linking threads directly to requirements, project management would have better insight about the status of entire requirements.

Since threads drove the software efforts and status, developers viewed threads as the most visible formal process in place. The simplicity of the process, accurate status and integration efficiency contributed to the development team’s acceptance of the process and enthusiasm to suggest improvements.

In addition, the empirical results suggest that the introduction of thread-based testing exposed design and coding errors earlier and attributed fewer errors to the integration process itself, probably due to the enhanced communication between developers and testers. In short, our method appears to have synchronized the notion of task completion among developers, testers and management.

Summary: Thread-based integration testing played a key role in the success of this software project. At the lowest level, it provided integrators with better knowledge of the scope of what to test, in effect a contract between developers and testers. At the highest level, it provided a unified status tracking method and facilitated an agreement between management and the developers as to what would be delivered during each formal build. Furthermore, instead of testing software components directly, it required integrators to focus on testing logical execution paths in the context of the entire system. Because of this, it strongly supported the goals of early, rapid integration coupled with an iterative development approach. In summary, the thread approach can result in tangible executable scenarios driving development and integration while the autonomous, well-defined thread states strengthened the use of threads as an accurate method of scheduling and tracking status.

Task-Based Software Testing : An Inside view

Mostly cutomers want systems that are:

- On-time
- Within budget
- That satisfy user requirements
- Reliable

Latter two concerns (out of above four) can be refined into two broad objectives for operational testing:

1. That a system’s performance satisfies its requirements as specified in the Operational Requirement Document and related documents.

2. To identify any serious deficiencies in the system design that need correction before full rate production.

Following the path from the system level to software two generic reasons for testing software are:

- Test for defects so they can be fixed - Debug Testing
- Test for confidence in the software - Operational Testing

Debug testing is usually conducted using a combination of functional test techniques and structural test techniques. The goal is to locate defects in the most cost-effective manner and correct the defects, ensuring the performance satisfies the user requirements.

Operational testing is based on the expected usage profile for a system. The goal is to estimate the confidence in a system, ensuring the system is reliable for its intended use.

Task-Based Testing is a variation on operational testing. The particular techniques are not new, rather it leverages commonly accepted techniques by placing them within the context of current operational and acquisition strategies.

Task-based testing, as the name implies, uses task analysis. This begins with a comprehensive framework for all of the tasks that the system will perform. Through a series of hierarchical task analyses, each unit within the service creates a Mission Essential Task List (Mission of System).

These lists only describe "what" needs to be done, not "how" or "who." Further task decomposition identifies the system and people required to carry out a mission essential task. Another level of decomposition results in the system tasks (i.e. functions) a system must provide. This is, naturally, the level in which developers and testers are most interested. From a tester’s perspective, this framework identifies the most important functions to test by correlating functions against the mission essential tasks a system is designed to support.

This is distinctly different from the typical functional testing or "test-to-spec" approach where each function or specification carries equal importance. Ideally, there should be no function or specification which does not contribute to a task, but in reality there are often requirements, specifications, and capabilities which do not or minimally support a mission essential task. Using task analysis, one identifies those functions impacting the successful completion of mission essential tasks and highlights them for testing.


Operational Profiles: The process of task analysis has great benefit in identifying what functions are the most important to test. However, the task analysis only identifies the mission essential tasks and functions, not their frequency of use. Greater utility can be gained by combining the mission essential tasks with an operational profile an estimate of the relative frequency of inputs that represent field use. This has several benefits:

1. Offers a basis for reliability assessment, so that the developer can have not only the assurance of having tried to improve the software, but also has an estimate of the reliability actually achieved.

2. Provides a common base for communicating with the developers about the intended use of the system and how it will be evaluated.

3. When software testing schedules and budgets are tightly constrained, this design yields the highest practical reliability because if failures are seen they would be the high frequency failures.


The first benefit has the advantage of applying statistical techniques in:

- The design of tests
- The analysis of resulting data

Software reliability estimation methods such as Task Analysis are available to estimate both the expected field reliability and the rate of growth in reliability. This directly supports an answer to the question about software’s impact on a system’s mission effectiveness.

Operational profiles are criticized as being difficult to develop. However, as part of its current operations and acquisition strategy, some organizations inherently develops an operational profile. At higher levels, this is reflected in the following documents:

- Analysis of Alternatives
- Operational Requirements Document (ORD)
- Operations Plans
- Concept of Operations (CONOPS) etc.

Closer to the tester’s realm is the interaction between the user and the developer which the current acquisition strategy encourages. The tester can act as a facilitator in helping the user refine his or her needs while providing insight to the developer on expected use. This highlights the second benefit above the communication between the user, developer, and tester.

Despite years of improvement in the software development process, one still sees systems which have gone through intensive debug testing (statement coverage, branch coverage, etc.) and "test-to-spec," but still fail to satisfy the customer’s concerns (that I mentioned above). By involving a customer early in the process to develop an operational profile, the most needed functions to support a task will be developed and tested first, increasing the likelihood of satisfying the customer’s four concerns. This third benefit is certainly of interest in today’s environment of shrinking budgets and manpower, shorter schedules (spiral acquisition), and greater demands on a system.


Task-Based Software Testing

Thus, Task-based software testing is the combination of a task analysis and an operational profile. The task analysis helps partition the input domain into mission essential tasks and the system functions which support them. Operational profiles, based on these tasks, are developed to further focus the testing effort.

Debug Testing

Debug testing is directed at finding as many bugs as possible, by either sampling all situations likely to produce failures using methods like code coverage & specification criteria etc, or concentrating on those that are considered most likely to produce failures like stress testing and boundary testing methods.
Survey of unit testing methods are examples of debug testing methods. These include such techniques as statement testing, branch testing, basis path testing, etc. Typically associated with these methods are some criteria based on coverage, thus they are sometimes referred to as coverage methods. Debug testing is based on a tester’s hypothesis of the likely types and locations of bugs.

Consequently, the effectiveness of this method depends heavily on whether the tester’s assumptions are correct.

If a developer and/or tester has a process in place to correctly identify the potential types and locations of bugs, then debug testing may be very effective at finding bugs. If a "standard" or "blind" approach is used, such as statement testing for its own sake, the testing effort may be ineffectual and wasted. A subtle hazard of debug testing is that it may uncover many failures, but in the process wastes test and repair effort without notably improving the software because the failures occur at a negligible rate during field use.

Integration of Test Methods

Historically, a system’s developer relied on debug testing (which includes functional or "test-to-spec" testing). Testing with the perspective of how the system would by employed was not seen until an operational test agency (OTA) became involved. Even on the occasions when developmental test took on an operational flavor, this is viewed as too late in the process. This historical approach to testing amplifies the weaknesses of both operational and debug testing. I propose that task-based software testing be accelerated to a much earlier point in the acquisition process. This has the potential of countering each respective method’s weaknesses with the other’s strengths.

Conclusion: Task-based Software Testing evaluation is a combination of demonstrated, existing methods (task analysis and operational testing). Its strength lies in matching well with the current operational strategy of mission essential tasks and the acquisition community’s goal to deliver operational capability quickly. By integrating task-based software testing with existing debug testing, the risk of meeting the customer’s four concerns (on-time, within budget, satisfies requirements, and is reliable) can be reduced.

Functional Testing Vs Non-Functional Testing : Have a Look

Functional Testing: Testing the application against business requirements. Functional testing is done using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team.

Functional Testing covers:

• Unit Testing
• Smoke testing / Sanity testing
• Integration Testing (Top Down,Bottom up Testing)
• Interface & Usability Testing
• System Testing
• Regression Testing
• Pre User Acceptance Testing(Alpha & Beta)
• User Acceptance Testing
• White Box & Black Box Testing
• Globalization & LocalizationTesting

Non-Functional Testing: Testing the application against client's and performance requirement. Non-Functioning testing is done based on the requirements and test scenarios defined by the client.

Non-Functional Testing covers:

• Load and Performance Testing
• Ergonomics Testing
• Stress & Volume Testing
• Compatibility & Migration Testing
• Data Conversion Testing
• Security / Penetration Testing
• Operational Readiness Testing
• Installation Testing
• Security Testing (ApplicationSecurity, Network Security, System Security)

Rapid Testing : An introduction

Rapid testing is the testing software faster than usual, without compromising on the standards of quality. It is the technique to test as thorough as reasonable within the constraints. This technique looks at testing as a process of heuristic inquiry and logically speaking it should be based on exploratory testing techniques.

Although most projects undergo continuous testing, it does not usually produce the information required to deal with the situations where it is necessary to make an instantaneous assessment of the product's quality at a particular moment. In most cases the testing is scheduled for just prior to launch and conventional testing techniques often cannot be applied to software that is incomplete or subject to constant change. At times like these Rapid Testing can be used.

The structure of rapid testing is built four components:

• People
• Integrated test process
• Static Testing
• Dynamic Testing

There is a need for people who can handle the pressure of tight schedules. They need to be productive contributors even through the early phases of the development life cycle with the core skill as the ability to think critically.

It should also be noted that dynamic testing lies at the heart of the software testing process, and the planning, design, development, and execution of dynamic tests should be performed well for any testing process to be efficient.

It would help us if we examine each phase of a development process very carefully to see how the efficiency, speed and quality of testing can be improved, bearing in mind the following factors:

• Actions that the test team can take to prevent defects from escaping. For example, practices like extreme programming and exploratory testing.
• Actions that the test team can take to manage risk to the development schedule.
• The information that can be obtained from each phase so that the test team can speed up the activities.

If a test process is designed around the answers to these questions, both the speed of testing and the quality of the final product should be enhanced.

Some of the aspects that can be used while rapid testing are given below:

• Test for link integrity
• Test for disabled accessibility
• Test the default settings
• Check the navigations
• Check for input constraints by injecting special characters at the sources of data
• Run Multiple instances
• Check for interdependencies and stress them
• Test for consistency of design
• Test for compatibility
• Test for usability
• Check for the possible variability’s and attack them
• Go for possible stress and load testsAnd our favorite – banging the keyboard

Regression Testing : An Introduction

Regression testing is the re-running of Test cases that a program has previously executed correctly, in order to detect failures spawned by changes or corrections made during software development and maintenance.

These failures arise from incomplete or incorrect changes and are often witnessed as (unexpected) side effects in apparently unrelated application areas. It is common in the IT industry, that one in six (or seventeen percent*) of correction attempts are themselves defective.

This high rate of introduced defects is exaggerated when developers maintain a large number of poorly documented, integrated systems where they of ten have little or no experience of these systems. Regression testing may then be used to great effect at detecting subtle side effects and unconsidered inter-relationships within these environments, thus reducing risk.

In regression testing standard actions in a test procedure are carried out and the expected responses are checked for correctness. Failure of the system to reproduce any of the expected responses imply that the system may have regressed (there may have been one or more introduced defects), or that the regression test itself may be out of date or incorrect.

If all responses are as expected, there may still have been introduced defects. In this case they escaped detection. Each defect that is reported from live or field use, having escaped detection during regression testing must be carefully analysed and the regression test suite(s) updated to catch these or similar defects in the future.

The main source of regression test cases is usually from re-use of unit, integration or system test cases. It is good practice to batch test cases into logically cohesive test suites, rather than have a single huge regression test. This allows different sub-sets of tests to be executed when there is time-pressure, or where there is confidence that only certain tests need to be run.

When first creating a regression testing suite, the choice of tests to use can be guided by the 80/20 principle. Twenty percent of system functions are likely to be used eighty percent of the time. Thus these highly used screens, transactions, menus, or fields ought to be the first candidates for regression tests. This is easy to understand if we consider one of these popular functions experiencing failure. The company call centre will be inundated with calls and the majority of users will be negatively affected. If, however, one of the less common functions has a problem, fewer users would have used it and thus discovered a lack.

Further tests to add to regression suites may be guided by risk considerations. Certain failures may not occur often, but should they occur, would result in a highly negative business impact. These higher risk business areas, modules, screens, or transactions should therefore be tested each and every time there are changes in the system and/or its environment.

Additional regression tests can be added to application areas that are known to be difficult to maintain, and have a history of high failure rates.

Regression testing can begin at the unit level, where unit tests may be adapted and rerun after changes to the unit have been effected. Regression testing should then continue through integration, system, user acceptance and operational software development life cycle phases.

As a minimum, regression tests should be run prior to build releases into the broader community and/or company live Environment. These tests will help detect major anomalies that could have serious cost, schedule, productivity and/or company image Implications.

Web systems, and other multi-user systems might have ongoing regression tests run at regular intervals. For example, one such test may check that all hyperlinks on a web site remain correct and reachable. Links to other sites may become outdated, or may even be corrupted by hackers in a security breach.

Regression testing at regular intervals can also answer production questions such as: "Is the performance of our major transactions within acceptable time limits?", "or", is some factor slowing our response times on an important transaction?"

Regression tests of non-functional application attributes such as performance, usability or security, is also very important. A very small change to the code or design may have a significant impact on system performance for example. Also please take note that debilitating changes may not even be within the application software. Changes known to have had dire consequences include an update of PC BIOS, operating system software, network cards, or updates of third party database versions for example.

Regression testing is by definition repetitive, and thus many of the tests are likely to be suited to test automation. Test automation can deliver reduced testing costs after a few test iterations when compared to labour-intensive manual testing processes.

Many companies who use regression testing conduct a very abbreviated check test (sometimes called a 'smoke' or 'sanity' test) on newly delivered code, prior to starting their formal regression tests. This often saves time as the abbreviated test commonly exposes obvious errors (for example: a whole form may not be displayed because it failed to compile against a changed database format). Removing this type of problem prior to running the time-consuming regression testing scenarios can result in getting earlier developer help, and prevent testers completing a significant portion of the regression testing before finding such problems.

It should be noted that the control and make-up of the test environment is as critical for regression testing as it is for other testing types within the same software development life cycle phase. Refer to the article on page eight that discusses creating a testing environment.

Regression test suites, be they manual or automated, are an important company asset. They therefore need to be backed up, configuration-managed, and kept current to deliver maximum benefit to their owners. Specific ownership of and responsibility for the regression test suites therefore needs to be clearly defined.

Security Testing :An overview

The security testing is performed to check whether there is any information leakage in the sense by encrypting the application or using wide range of software’s and hardware's and firewall etc.
Before planning for Security Testing, you will need to think about the following parameters:

• Authentication - Testing the authentication schema means understanding how the authentication process works and using that information to circumvent the authentication mechanism. Basically, it allows a receiver to have confidence that information it receives originated from a specific known source.
• Authorization - Determining that a requester is allowed to receive a service or perform an operation.
• Confidentiality - A security measure which protects the disclosure of data or information to parties other than the intended.
• Integrity – Whether the intended receiver receives the information or data which is not altered in transmission.
• Non-repudiation - Interchange of authentication information with some form of provable time stamp e.g. with session id etc.

Selenium 1.0 Java Client Driver Configuration

In General configuration of Selenium-RC with any java IDE would have following steps:

·         Download Selenium-RC from the SeleniumHQ downloads page

·         Start any java IDE

·         Create new project

·         Add “selenium-java-<version-number>.jar” to your project classpath

·         Record your test from Selenium-IDE and translate it to java code (Selenium IDE has automatic translation feature to generate tests in variety of languages)

·         Run selenium server from console

·         Run your test in the IDE

These points have been delineated below with reference to Eclipse and IntelliJ:

Configuring Selenium-RC With Eclipse¶

Eclipse is a multi-language software development platform comprising an IDE and a plug-in system to extend it. It is written primarily in Java and is used to develop applications in this language and, by means of the various plug-ins, in other languages as well as C/C++, Cobol, Python, Perl, PHP and more.

Following lines describes configuration of Selenium-RC with Eclipse - Version: 3.3.0. (Europa Release). It should not be too different for higher versions of Eclipse

·         Launch Eclipse.

·         Select File > New > Other.

·         Java > Java Project > Next

·         Provide Name to your project, Select JDK in ‘Use a project Specific JRE’ option (JDK 1.5 selected in this example) > click Next

·         Keep ‘JAVA Settings’ intact in next window. Project specific libraries can be added here. (This described in detail in later part of document.)

·         Click Finish > Click on Yes in Open Associated Perspective pop up window.

This would create Project Google in Package Explorer/Navigator pane.

·         Right click on src folder and click on New > Folder

Name this folder as com and click on Finish button.

·         This should get com package insider src folder.

·         Following the same steps create core folder inside com

SelTestCase class can be kept inside core package.

Create one more package inside src folder named testscripts. This is a place holder for test scripts.

Please notice this is about the organization of project and it entirely depends on individual’s choice / organization’s standards. Test scripts package can further be segregated depending upon the project requirements.

·         Create a folder called lib inside project Google. Right click on Project name > New > Folder. This is a place holder for jar files to project (i.e. Selenium client driver, selenium server etc)

This would create lib folder in Project directory.

·         Right click on lib folder > Build Path > Configure build Path

·         Under Library tab click on Add External Jars to navigate to directory where jar files are saved. Select the jar files which are to be added and click on Open button.

After having added jar files click on OK button.

Added libraries would appear in Package Explorer as following:

Configuring Selenium-RC With Intellij¶

IntelliJ IDEA is a commercial Java IDE by the company JetBrains. Intellij provides a set of integrated refactoring tools that allow programmers to quickly redesign their code. IntelliJ IDEA provides close integration with popular open source development tools such as CVS, Subversion, Apache Ant and JUnit.

Following lines describes configuration of Selenium-RC with IntelliJ 6.0 It should not be very different for higher version of intelliJ.

·         Open a New Project in IntelliJ IDEA.

·         Provide name and location to Project.

·         Click Next and provide compiler output path.

·         Click Next and select the JDK to be used.

·         Click Next and select Single Module Project.

·         Click Next and select Java module.

·         Click Next and provide Module name and Module content root.

·         Click Next and select Source directory.

·         At last click Finish. This will launch the Project Pan.

Adding Libraries to Project:

·         Click on Settings button in the Project Tool bar.

·         Click on Project Structure in Settings pan.

·         Select Module in Project Structure and browse to Dependencies tab.

·         Click on Add button followed by click on Module Library.

·         Browse to the Selenium directory and select selenium-java-client-driver.jar and selenium-server.jar. (Multiple Jars can be selected b holding down the control key.).

·         Select both jar files in project pan and click on Apply button.

·         Now click ok on Project Structure followed by click on Close on Project Settings pan. Added jars would appear in project Library as following.

·         Create the directory structure in src folder as following.

Note

This is not hard and fast convention and might very from project to project.

·         Herein core contains the SelTestCase class which is used to create Selenium object and fire up the browser. testscripts package contains the test classes which extend the SelTestCase class. Hence extended structure would look as following.