24 Software Testing – Implementation

Testing process includes Planning, design and performance of testing are carried out throughout the software development process. The testing process involves the following steps:

  • Determining the test methodology phase
  • Planning the tests
  • Test design
  • Test implementation

 

Fig 1: Software Testing Process

 

In Test Methodology Phase, Testing is done throughout the development process. Testing is divided into phases beginning in the design phase and ending at the customer’s site. The main issues that testing methodology addresses are the appropriate required software quality standard and the software testing strategy. Different standards are required for different software applications. The expected damage resulting from failed software impacts the standard of software quality.

 

Example 1:

  • A software package for a hospital patient bed monitor requires the highest software quality standard considering the possibly severe consequences of software failure.

    Example 2:

  • A package developed for handling feedback information for an organization’s internal employee training program could make do with a medium-level software quality standard, assuming that the cost of failure is relatively low.

Classification of Software Failure:

(a) Damages to Customers and Users:

  • Endangers the safety of human beings
  • Affects an essential organizational function with no system replacement capability available
  • Affects functioning of firmware, causing malfunction of an entire system
  • Affects an essential organizational function but a replacement is available
  • Affects proper functioning of software packages for business applications
  • Affects proper functioning of software packages for a private customer
  • Affects functioning of a firmware application but without affecting the entire system.
  • Inconveniences the user but does not prevent accomplishment of the system’s capabilities

(b) Damages to the Software Developer:

  • Financial losses
    • Damages paid for physical injuries
    • Purchase cost reimbursed to customers
    • Damages paid to organizations for malfunctioning of software
  • Non-quantitative damages
    • Expected to affect future sales
    • Substantially reduced current sales

The issues that have to be decided include: The testing strategy: (Big – bang or bottom-up or top-down?) Parts of the testing plan that should be performed according to the white box testing model. Parts of the testing plan that should be performed according to the automated testing model.

 

Developers need to consider the following issues before initiating a specific test plan:

  • What to test?
  • Which sources to use for test cases?
  • Who is to perform the tests?
  • Where to perform the tests?
  • When to terminate the tests?

An approach to perfect testing is to recommend a full and comprehensive software test plan that requires:

  • Performing unit tests for all the individual units.
  • Integration tests for all the unit integrations.
  • A system test to test the software package as a whole.

Factors to be considered for Unit and Integration Tests:

Which modules need to be unit tested?

 

Which integrations should be tested? Low priority applications tested in unit testing may not be needed or included in the system tests. Lots of planning is needed, as testing is a very expensive undertaking.

 

The methods for modules, integrations and applications to determine their priority in the testing plan are based on two factors:

Factor A: Damage severity level: The severity of results in case the module or application fails.

 

Factor B: Software risk level: The level of risk represents the probability of failure.

 

Who performs the tests? is  the next question to be answered.

 

Unit Testing – done by the programmer and/or development team.

 

Integration Testing – can be the development team or a testing unit.

 

System Testing – usually done by an independent testing team (internal or external (consultants) team. For small companies, another testing team from another development team can be used and swapped. Unit and integration testing are naturally carried out at the software developer’s site.

 

If system testing is to be performed by external testing consultants, a third option arises: the consultant’s site.

 

The stage at which software testing should be terminated is with respect to system tests. Five alternative routes are available, each chosen on the different criteria:

  • The completed implementation route.
  • The mathematical models application route
  • The error seeding route
  • The dual independent testing teams route
  • Termination after resources have petered out

The Mathematical Models Application Route are applied to estimate the percentage of undetected errors. Testing would be terminated once the error detection rate declines below the rate that corresponds to a predetermined level of undetected errors. The disadvantage of this route is that the mathematical model chosen may not fully represent the project’s characteristics. Error seeding route are seeded in the tested software prior to the outset of testing. The percentage of discovered seeded errors will correspond to the percentage of real errors detected. Testing will terminate once the residual percentage of undetected seeded errors reaches a predefined level considered acceptable for “passing” the system.

Test design is the planning stage of the software system tests is commonly documented in a “Software Test Plan” (STP). The test design is carried out on the basis of the software test plan as documented by STP. In software test plan the scope of the tests is considered as the document that provide the basis for the planned tests. In testing environment it considers the testing sites and required hardware and firmware configuration. It also considers the preparation and training required of the test team.

 

In software test description the test procedures and the test case database/file may be documented in a “Software Test Procedure” document and “Test Case File” document or in a single document called the “Software Test Description”. Testing implementation phase activities consist of a series of tests, corrections of detected errors and re-tests. The tests are carried out by running the test cases according to the test procedures. In manual software tests, only a portion of the original test procedure is re-tested to save testing resources and shorten re-testing duration.

 

Fig 2: Software Testing Implementation

A test case is a documented set of the data inputs and operating conditions required to run a test item together with the expected results of the run. The tester is expected to run the program for the test item according to the test case documentation. Actual Results are compared with the expected results noted in the documents. When some or all of the results do not agree with the expected results, a potential error is identified.

In code auditing the test performs automated qualification testing. The code auditor checks the compliance of code to specified standards and procedures of coding. The auditor’s report includes a list of the deviations from the standards and a statistical summary of the findings. In coverage monitoring Produce reports about the line coverage achieved when implementing a given test case file. The output includes the percentage of lines covered by the test cases as well as listings of uncovered lines. These features make coverage monitoring a vital tool for white-box tests.

 

In functional tests, automated regression tests performed for the whole program verify that the error corrections have been performed satisfactorily. An automated testing tool that supports functional tests, the output comparator, will help in the regression test stage. The automated comparison of outputs of successive tests enables testers to prepare an improved analysis of the regression test.

 

For load tests to be performed, the maximal load environment must be first created. Load tests are based on scenarios of the maximal load situations. Virtual users and virtual events are generated and operated in a hardware and communication environment defined for load tests. Automated test management software provides features applicable for manual as well as automated testing and for automated tests only. The inputs together with the software package’s capabilities, determine the application’s scope. Alpha site and beta site tests are employed to obtain comments about quality from the package’s potential users.

Alpha site tests are tests of a new software package that are performed at the developer’s site. The errors identified by alpha site tests are expected to include the errors that only use by a real user can reveal, and thus should be reported to the developer. Once an advanced version of the software package is available, the developer offers it free of charge to one or more potential users. The users install the package in their sites (usually called the “beta sites”), with the understanding that they will inform the developer of all the errors revealed during trials or regular usage.

 

Summary:

 

The main issues to be raised with respect to testing implementation are test effectiveness and efficiency. Process of planning and designing tests, sources for test cases are very significant. Alpha and beta site test implementation are carried out to make sure that quality of the software is verified by the actual users of the software.