Using testability measures for dependability assessment

Abstract

Program 'testability' is the probability that a fault in a program, if present, will cause the program to fail. Measures of testability can be used to draw inferences on program correctness from the observation of a series of failure-free test executions, a common need for software with 'ultra-high reliability' requirements. For a program that has passed a certain number of tests without failing, a high value of testability implies a high probability that the program is correct. We give a general descriptive model of program execution and testing, and propose a more precise definition of program testability than that given by other authors. We then study the use of testability in: i) providing, through testing, confidence in the absence of faults and ii) bounding the probability of failures, from the results of operational testing. We derive the probability of absence of faults through a Bayesian inference procedure, criticise previously proposed derivations of this probability, and study the relationship between the testability of a program and its failure probability in operation. We derive the conditions under which a high testability improves one's expectations about program reliability. Last, we discuss the potential of these methods in practical applications.