Using perturbation analysis to measure variation in the information content of test sets

Abstract

We define the information content of test set T with respect to a program P to be the degree to which the behavior of P on T approximates the overall behavior of P. Informally, the higher the information content of a test set, the greater the likelihood an error in the data state of a program will be manifested under testing.Perturbation analysis injects errors into the data state of an executing program and traces the impact of those errors on the intervening states and the program's output. The injection is performed by perturbation functions that randomly change the program's data state. Using perturbation analysis we demonstrate that different test sets may satisfy the same testing criterion but have significantly different information content.We believe that "consistency of information content" is a crucial measure of the quality of a testing strategy. We show how perturbation analysis may be used to assess individual testing strategies and to compare different testing strategies.The "coupling effect" of mutation testing implies that there is little variation among mutation-adequate test sets for a program. This implication is investigated for two simple programs by analyzing the variation among several mutation-adequate test sets.