Choosing a testing method to deliver reliability

Abstract

Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The question considered is whether it is better to use tests that seek out failures (`debug testing') or to simulate usage and find failures along the way (`operational testing'). `Better' is measured by the delivered reliability obtained after all test failures, have been eliminated. This comparison extends previous work, where the measure was the probability of detecting a failure. The theoretical treatment of the paper is probabilistic and analytical. Revealing special cases are exhibited in which each kind of testing is superior.