System level combinatorial testing in practice - The concurrent maintenance case study

Abstract

Combinatorial test design (CTD) is an effective test design technique that reveals faults resulting from parameter interactions in a system. CTD requires a test space definition in the form of a set of parameters, their respective values, and restrictions on the value combinations. Though CTD is considered an industry best practice, there is only a small body of work on the practical application of CTD to industrial systems, and some key elements of the CTD application process are under-explored. Specifically, little consideration has been given to the process for identifying the parameters of the test space and their interrelations, how to validate the test space definition remains an open question, the application of CTD in reported work concentrates mostly on function or interface level testing and is hardly expanded to other levels of testing, and there is a significant lack of evaluation of the degree to which CTD helped improve the quality of the system under test. In this work, we analyze the continuous application of CTD in system test of two large industrial systems: IBM® POWER7® and IBM® System z®. For POWER7, CTD was used to design test cases for server concurrent maintenance. The application of CTD was in direct response to inconsistent reliability of those features on the prior POWER6® servers, and resulted in noteworthy quality improvements on POWER7. Success with POWER7 led to application of the methods to System z Enhanced Driver Maintenance testing, also featured in this work. To the best of our knowledge, this is the first comprehensive analysis of CTD usage in system test of industrial products, and the first analysis of long term use of CTD. We describe the methodology that we followed to define the combinatorial test space, while answering some unique challenges rising from the use of CTD to design functional test cases at system level rather than at interface or function level. We also describe our methodology for evaluating the test space definition and continuously improving it over time. In addition, we describe advanced CTD features that we found helpful for achieving an effective yet affordable test plan. Finally, we quantitatively and qualitatively evaluate the overall effectiveness of CTD usage, and show that it resulted in significantly improved server concurrent maintenance features. © 2014 IEEE.