June 2001
Volume 3 • Number 3

Contents

TESTING
Resource Path Testing: A Framework for Design of System Testing

by: Yasuharu Nishi, School of Engineering, The University of Tokyo

Unforeseen failures occur in the simultaneous use of multiple PCs connected through a local area network. Proper system testing can detect these failures before they are delivered to customers. This article presents a systematic strategy and framework for system testing called resource path testing (RPT). RPT aims at the "weak points" of the system/software. First, the author concentrates on the common resources that are the weak points, the resources that are needed for the system/software to work properly, and whose changes by other software/hardware modules cause failures. Second, the author classifies resources into several types to facilitate test design. Third, he presents procedures and tools for RPT. Finally, the author discusses the application of RPT to the whole of system testing, in particular to configuration testing and stress testing.

Key words: common resource, configuration testing, failure analysis, stress testing, test design

INTRODUCTION

With the spread of PC use, it is common for one to simultaneously use various applications and devices in a PC, as well as have two or more PCs connected through a local area network (LAN). This simultaneous use, however, sometimes causes unforeseen failures. For example, Microsoft Windows 95 with Internet Explorer 5.0 becomes unbootable after the installation of a certain Diamond Viper V550 driver (BugNet 1999). PCs cannot boot up with a certain Adaptec SCSI card and a certain ATI video card (Saito et al. 1999). Workstations such as SGI’s obstruct network traffic in a Macintosh-only network. It is generally difficult to detect failures such as these that depend on interactions of various data, hardware, and software.

System testing is a process for detecting these failures so they are not delivered to customers. It consists mainly of configuration testing and stress testing. Configuration testing detects failures caused by specific combinations of hardware and software (Myers 1979). Stress testing detects failures caused by overloading resources (Beizer 1996). System testing is an important process for assuring software quality in an environment of complex defect-prone components.

Existing research on system testing has resulted in a good body of concepts, such as Myers’ and Beizer’s, but there is no systematic design strategy and framework for system testing that has a high probability for detecting failures. Accordingly, procedures and tools for system testing have remained undeveloped.

This article presents a systematic strategy and framework for system testing called resource path testing (RPT). RPT consists of a framework, a procedure, and tools, all of which are addressed in this article.

RESOURCE PATH TESTING

Failure Mechanisms and Common Resources

The failures mentioned previously occur through the following mechanisms. The Diamond Viper V550 driver installation changes COMCTL32.DLL into a different version from that needed for Windows to boot up. This kind of failure is generally called "DLL Hell." The SCSI card’s BIOS overwrites the memory area needed for the video card’s BIOS to run. The workstations apply too high a load to the network needed for Macintosh computers to communicate with each other.

Focusing on the DLL, memory, or network problems, the failure mechanisms can be abstracted, as shown in Figure 1. First, software and/or hardware other than the system/software under test (SUT) is installed, or functions are executed under high load.

By installation or high load, a DLL, a memory area, or a network in the SUT is changed. Consequently, the SUT fails. Resources such as a DLL, memory area, or network can be the weak points of the SUT. Sharing a resource with other software and/or hardware can cause failure of the SUT. The author calls a resource that is shared with two or more software and/or hardware components a common resource. Common resources are the weak points of the SUT.

RPT aims at the weak points of the system/software. In RPT, test cases are designed to modify common resources by installing potential failure-causing software and/or hardware modules (PFCMs), and to execute functions of PFCMs under high load to detect failure.

Resource Types

In system testing, various operations are necessary for detecting failures. In configuration testing, operations such as software and/or hardware installation, and change of configuration (such as preferences or property) can produce failures. In stress testing, operations such as reading huge data files and heavily loaded use of the network can produce failures. Considering the various operations to be done in a system test, resources can be classified into four types: module, data, storage, and semaphore.

Each type of resource has the following specific attributes. A change of an attribute causes a change in the behavior of the SUT.

• Module resource: DLL and so on, which have attributes such as version, date, and so on
• Data resource: Such as registry and file, which have attributes that refer to content
• Storage resource: Buffer, memory, disk, and so on, which have attributes such as size, capacity, and so on
• Semaphore resource: Such as Ethernet and DB record, which have attributes such as collision, lock, and so on Figure 2 shows an example of each type of resource and its attributes.

It should be noted that a change in an attribute of a module resource such as "version" and "date" is not a change in the attribute itself, but a change in the attribute as a result of the replacement of the resource. The "date/time" of a DLL, for example, should not be changed by modifying the information in directory entries, but by replacing it with a DLL created on a different date and time.

Framework of Resource Path Testing

In RPT, test design consists of the following:

1. Select which software/hardware modules share a common resource with the SUT
2. Determine how each common resource is modified
3. Determine which functions should be executed to detect changes in behavior

For no. 1, the author creates a graph that represents dependency of resources, called the resource dependency graph (RDG). Using RDGs, common resources are identified as included in more than one RDG, and PFCMs are selected.

For no. 2, the author creates a resource change matrix (RCM) that represents changes to common resources and conditions causing these changes. The author calls these conditions potential failure conditions.

For no. 3, the author creates a list of functions that are executed through paths including the modified common resource, the resource path list (RPL). RPLs are prepared only for the SUT. The process of RPT using the previously mentioned tools has five phases, as shown in Figure 3.

TOOLS FOR RESOURCE PATH TESTING

Resource Dependency Graph

The RDG is a nondirectional graph that represents dependency of resources, as shown in Figure 4. RDGs represent architecture. Nodes in RDGs represent resources. Links represent intercommunication, such as calling another resource, using devices, accessing disk and/or memory area, and communicating with a network. The common resource can be identified as the node included in both the RDG of the SUT and of the PFCMs. Monitoring tools such as Apius and BugTrapper make it easy to draw RDGs.

Resource Change Matrix

The RCM is a matrix that represents changes of the common resource and the potential failure conditions that cause the changes as shown in Figure 5. Columns in RCMs refer to common resources and rows refer to potential failure conditions, such as installation and high load, which change common resources and their attributes. Cells refer to changes in common resources or their attributes. Changes in common resources or their attributes should be described as concretely as possible, ideally in quantitative form.

Resource Path List

The RPL is a list of functions that are executed through paths including the modified resource as shown in Figure 6 (as well as the corresponding RDG). The left three columns of the RPL show paths for use in executing test cases and detecting failures, resource paths. Resource paths consist of three kinds of resources: an activation resource, a common resource, and an observation resource. An activation resource is where functions of the SUT, that is, test cases, are executed. An observation resource is where results of the functions, such as failures, are observed. For example, in the case of a failure that causes an SMTP server to stop because of a buffer overflow due to a huge e-mail sent from an SMTP client, the activation resource is the client who sends the e-mail, and the observation resource is the server that stops and shows the blue screen.

The right column of the RPL shows functions that are executed on the resource path indicated in the same row. In some cases, one or more functions are executed on one resource path.

CONDUCT OF RESOURCE PATH TESTING

Procedure for Resource Path Testing

Resource path testing is managed using the procedure shown in Figure 7, which consists of the five phases listed in Figure 3.

Application to Configuration Testing and Stress Testing

RPT can be applied to both configuration testing and stress testing. In RPT, the procedure for these tests is the same. The differences are the potential failure conditions and the types of common resources that are identified in the process.

In configuration testing, the RPT strategy should focus on replacing common resources and rewriting contents of common resources. For this purpose, common resources are usually selected from module and data resources, and potential failure conditions are usually installation and uninstallation of PFCMs, change of configuration (for example, "preferences" and "property"), and so on.

It should be considered that installation is sometimes not "commutative." Some software installers put different sets of modules into the system even if the same installation option is selected depending on what has been installed. In other words, reproducibility of a failure that should be detected in configuration testing can depend on the order of test cases. Consequently, a failure may seem unreproducible to test engineers. If budget permits, the test environment should be reset or cleaned between the execution of test cases.

In stress testing, the RPT strategy should focus on filling or overflowing common resources (for example, memory leaks and buffer overflows), and creating race conditions on common resources. For this purpose, common resources are usually selected from storage and semaphore resources, and potential failure conditions are usually accesses of huge data files, stressing of network under high load, use of printer spools so many times, and so on.

It should be noted that high load could be applied not only to the filling or overflowing of resources but to the creation of race conditions. It is usually difficult to create race conditions because they can depend on timing on the order of a millisecond. Alternatively, it is easier to create race conditions by administering a high load. The probability that one function will be executed at the same time as the other function increases if the one function is incessantly executed under high load. The probability will also increase if one function is executed much slower by thrashing under high load.

In addition, it should be noted that race conditions can occur between functions of the same application in a multitasking operation system. This situation is called self-common. It is difficult to identify self-common resources, because self-common resources have to be selected from one RDG, though common resources are easily selected from more than one RDG. Alternatively, self-common resources can be selected from resource paths.

Whichever testing RPT is applied to, the performance of RPT depends on the knowledge about the attributes of each type of resource and their potential failure conditions. Therefore, it is important in practice to acquire and accumulate this knowledge. It is also important to develop processes and tools for acquiring and accumulating this knowledge.

Applicability of RPT to whole system testing

RPT is a comprehensive and extensive paradigm that covers other types of testing in system testing. It can be applied to recovery testing, in which recoverability from dangers such as a power failure or damage of hard disks is assessed. It can be applied to security testing, in which vulnerability to malicious attacks such as smurfing and DNS spoofing is assessed. In recovery testing, RPT provides the strategy to determine where dangers may occur and where recoverability should be assessed. As for security testing, RPT assists in identifying new security holes caused by buffer overflow and so on. For these assessments, it is important to establish criteria for recoverability and vulnerability.

CONCLUSION

The author has presented the concept of a common resource, which is the weak point of system/software, and the resource path testing method based on this concept. RPT is a systematic design strategy and framework for system testing that focuses on identifying the weak points. It can successfully detect failures such as DLL Hell and Buffer Overflow. The author has also presented the procedure and tools for RPT. RPT can be applied to both configuration testing and stress testing, as well as any other type of system testing. For effective test design, it is important to identify the type of resource (module, data, storage, and semaphore), and the changes in the attributes of resources caused by potential failure conditions. It is essential in practice to acquire and accumulate knowledge on each type of resource and its potential failure conditions, and it is also important to develop processes and tools for acquiring and accumulating this knowledge.

Acknowledgments

The author would like to thank Professor Yoshinori Iizuka from the University of Tokyo for the intensive discussions, and Kunihiro Kudoh, Toshihiko Suzuki, Hirokazu Ezawa, Shinsaku Tange, Atsushi Wada, Takashi Nakamoto, Toshio Sakuyama, Hiroyuki Azuma, and the other members of System Verification Department of CSK Corporation for their enthusiastic and interested discussions.

References

Beizer, Boris. 1996. Software system testing and quality assurance. Boston: International Thomson Publishing.
BugNet. 1999. BugNet Database. See URL http://www.BugNet.com/ .

Myers, Glenford J. 1979. The art of software testing. New York: John Wiley and Sons.

Sausville, Mark. 1997. Moving a computer at the lab. See URL http://www.svisions.com/ml/ .

Saito, Takayuki et al. 1999. The truth of conflict between hardware components. Nikkei Byte. (March).

BIOGRAPHY

Yasuharu Nishi is a consultant at SQC, Inc. in Japan. He is also a research fellow at the University of Tokyo. He has more than seven years of experience in software testing, quality management, process improvement, project management, and TQM. He also teaches seminars in software testing and quality management. Nishi has a doctorate in chemical system engineering from the University of Tokyo. He is the chairman of the Software Testing Engineer’s Forum in Japan. He is also one of the principal translators of Software Engineering: A Practitioner’s Approach by Dr. Roger Pressman. He can be reached by e-mail at nishi@tqm.t.u-tokyo.ac.jp.

If you liked this article, subscribe now.