Mutation versus All-uses: An Empirical Evaluation of Cost, Strength and Effectiveness

Abstract

Although mutation has been empirically found to be effective in detecting faults, it often imposes unacceptable demands on computing and human resources because of the large number of mutants that need to be compiled and executed on one or more test cases. We designed a case study to explore two alternatives of mutation to reduce its cost without significantly deteriorating its strength and effectiveness. The alternatives are (1) constrained abs/ror mutation which examines only the abs and ror mutants and ignores the others, and (2) randomly selected 10% mutation which examines only 10% of the randomly selected mutants of each mutation operator in Mothra. Data collected during experimentation have shown that both alternatives provide a significant cost reduction in terms of the number of test cases required to satisfy an adequacy criterion and the number of mutants to be examined. Such gain is accompanied by a small strength loss in the ability to distinguish non-equivalent mutants and cover feasible all-uses. In terms of decreasing order of fault detection effectiveness the ranking is mutation, constrained abs/ror mutation, all-uses, and randomly selected 10% mutation. These data suggest that examining only a small percentage of the mutants may be a useful heuristic for evaluating and constructing test sets in practice.