Prioritizing Mutants to Guide Mutation Testing

Abstract

Mutation testing offers concrete test goals (mutants) and a rigorous test efficacy criterion, but it is expensive due to vast numbers of mutants, many of which are neither useful nor actionable. Prior work has focused on selecting representative and sufficient mutant subsets, measuring whether a test set that is mutation-adequate for the subset is equally adequate for the entire set. However, no known industrial application of mutation testing uses or even computes mutation adequacy, instead focusing on iteratively presenting very few mutants as concrete test goals for developers to write tests. This paper (1) articulates important differences between mutation analysis, where measuring mutation adequacy is of interest, and mutation testing, where mutants are of interest insofar as they serve as concrete test goals to elict effective tests; (2) introduces a new measure of mutant usefulness, called test completeness advancement probability (TCAP); (3) introduces an approach to prioritizing mutants by incrementally selecting mutants based on their predicted TCAP; and (4) presents simulations showing that TCAP-based prioritization of mutants advances test completeness more rapidly than prioritization with the previous state-of-the-art.