8 Software Testing

SOFTWARE TESTING

Testing is the process of demonstrating that errors are not present. Software testing is an activity to check whether the actual results match the expected results and to ensure that the software system is defect free. It involves execution of a software component or system component to evaluate one or more properties of interest. Software testing also helps to identify errors, gaps or missing requirements in contrary to the actual requirements. It can be either done manually or using automated tools. Some prefer saying Software testing as a white box and black box testing. Some of the contributors defined the software testing as

Glen Myers

Testing is the process of executing a program with the intent of finding errors.

Paul Jorgensen

Testing is obviously concerned with errors, faults, failures and incidents. A test is the act of exercising software with test cases with an objective of Finding failure and Demonstrate correct execution

Most common myths about software testing are:

Testing is Too Expensive Testing is Time-Consuming

Only Fully Developed Products are Tested Complete Testing is Possible

A Tested Software is Bug-Free

Missed Defects are due to Testers

Testers are Responsible for Quality of Product

Test Automation should be used wherever possible to Reduce Time

Anyone can Test a Software Application

A Tester’s only Task is to Find Bugs

IMPORTANCE OF SOFTWARE TESTING

Testing is important because software bugs could be expensive or even dangerous. Software bugs can potentially cause monetary and human loss, history is full of such examples.

In April 2015, Bloomberg terminal in London crashed due to software glitch affected more than 300,000 traders on financial markets. It forced the government to postpone a 3bn pound debt sale.

Nissan cars have to recall over 1 million cars from the market due to software failure in the airbag sensory detectors. There has been reported two accident due to this software failure.

Starbucks was forced to close about 60 percent of stores in the U.S and Canada due to software failure in its POS system. At one point store served coffee for free as they unable to process the transaction.

Some of the Amazon’s third party retailers saw their product price is reduced to 1p due to a software glitch. They were left with heavy losses.

Vulnerability in Window 10. This bug enables users to escape from security sandboxes through a flaw in the win32k system.

In 2015 fighter plane F-35 fell victim to a software bug, making it unable to detect targets correctly.

China Airlines Airbus A300 crashed due to a software bug on April 26, 1994, killing 264 innocent live

In 1985, Canada’s Therac-25 radiation therapy machine malfunctioned due to software bug and delivered lethal radiation doses to patients, leaving 3 people dead and critically injuring 3 others.

In April of 1999, a software bug caused the failure of a $1.2 billion military satellite launch, the costliest accident in history

In may of 1996, a software bug caused the bank accounts of 823 customers of a major U.S. bank to be credited with 920 million US dollars.

TYPES OF SOFTWARE TESTING

Typically Testing is classified into three categories.

Functional Testing

Non-Functional Testing or Performance Testing Maintenance (Regression and Maintenance)

TESTING LIFE CYCLE

Figure 1 shows the testing life cycle, In figure 1 errors can be introduced into the testing life cycle at multiple points. An error that would be introduced during the requirements specification could be an error of omission, while errors in the design, coding and fault resolution stages are more likely to be errors of commission.

Figure 1: Testing life cycle

The terminologies of life cycle testing are error, fault, failure, incident, test case.

Error

Represents mistakes made by people

Fault

Is result of error. May be categorized as

Fault of Commission – we enter something into representation that is incorrect

Fault of Omission – Designer can make error of omission, the resulting fault is that something is missing that should have been present in the representation

Failure

Occurs when fault executes.

Incident

Behavior of fault. An incident is the symptom(s) associated with a failure that alerts user to the occurrence of a failure

Test case

Associated with program behavior. It carries set of input and list of expected output Verification and Validation

Verification

Process of determining whether output of one phase of development conforms to its previous phase.

Validation

Process of determining whether a fully developed system conforms to its SRS document

Verification versus Validation

Verification is concerned with phase containment of errors Validation is concerned about the final product to be error free

CLASSIFICATION OF TESTING TECHNIQUES

There are two levels of classification

One distinguishes at granularity level Unit level

A unit test is the smallest testable part of an application like functions, classes, procedures, interfaces. Unit testing is a method by which individual units of source code are tested to determine if they are fit for use.
System level testing is a level of the software testing where a complete and integrated software is tested. The process of testing an integrated system to verify that it meets specified
requirements.
Integration level is the testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.

Other classification (mostly for unit level) is based on methodologies Black box (Functional) Testing

Black box testing also known as Behavioral Testing, is a software testing method in which the internal structure/ design/ implementation of the item being tested is not known to the tester. These tests can be functional or non-functional, though usually functional.

White box (Structural) Testing

White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) is a method of testing software that tests internal structures or workings of an application, as opposed to its functionality (i.e. black-box testing).

Testing Methodologies

Functional (Black box) inspects specified behavior and the structural (White box) inspects programmed behavior. Moreover, Unit Testing is essential for modular design since unit testing inspects individual modules and locates error in smaller region. On the other hand, in an integrated system, it may not be easier to determine which module has caused fault and hence both types of testing are necessary to reduce debugging efforts.

NEED FOR DESIGNING TEST CASES

Almost every non-trivial system has an extremely large input data domain thereby making exhaustive testing impractical. If randomly selected then test case may loose significance since it may expose an already detected error by some other test case.

DESIGN OF TEST CASES

Number of test cases do not determine the effectiveness For Example, To detect error in following code if(x>y) max = x; else max = y; {(x=3, y=2); (x=2, y=3)} will suffice

{(x=3, y=2); (x=4, y=3); (x=5, y = 1)} will falter

Each test case should detect different errors. Hence, test cases are necessary.

TESTING TOOLS

Testing tools are used for finding errors effectively. It also used to reduce the time to fix an error.
Different tools are used for different types of software.

QUALITY THROUGH TESTING

Testing improves quality by reducing errors. It used to increase the reliability of software. Testing helps to reduce the software failures.
Testing is coordinated between testing and quality management teams.

SOFTWARE TESTING-ISO STANDARDS

ISO/IEC 9126

This standard deals with the following aspects to determine the quality of a software application: Quality model, External metrics, Internal metrics and Quality in use metrics. Moreover, this standard presents some set of quality attributes for any software such as Functionality, Reliability, Usability, Efficiency, Maintainability, and Portability. The above-mentioned quality attributes are further divided into sub-factors which are also specified in the standards for quality measurement.

ISO/IEC 9241-11

Part 11 of this standard deals with the extent to which a product can be used by specified users to achieve specified goals with Effectiveness, Efficiency and Satisfaction in a specified context of use. This standard proposed a framework that describes the usability components and the relationship between them. In this standard, the usability is considered in terms of user performance and satisfaction. According to ISO 9241-11, usability depends on context of use and the level of usability will change as the context changes.

ISO/IEC 25000:2005

ISO/IEC 25000:2005 is commonly known as the standard that provides the guidelines for Software Quality Requirements and Evaluation (SQuaRE). This standard helps in organizing and enhancing the process related to software quality requirements and their evaluations. In reality, ISO-25000 replaces the two old ISO standards, i.e. ISO-9126 and ISO-14598. SQuaRE is divided into sub-parts such as ISO 2500n for Quality Management Division, ISO 2501n for Quality Model Division, ISO 2502n for Quality Measurement Division, ISO 2503n for Quality Requirements Division, ISO 2504n for Quality Evaluation Division. The main contents of SQuaRE are Terms and definitions, Reference Models, General guide, Individual division guides, Standard related to Requirement Engineering (i.e. specification, planning, measurement and evaluation process).

ISO/IEC 12119

This standard deals with software packages delivered to the client. It does not focus or deal with the clients’ production process. The main contents are related to the following items are set of requirements for software packages, instructions for testing a delivered software package against the specified requirements.

CONCLUSIONS

Testing is performed to detect errors. Life cycle testing is useful to find errors early. Testing tools are used to identify errors. Test cases are minimized for optimal testing. Finally, software quality is achieved by using effective testing