ExMAn: A Generic and Customizable Framework
for Experimental Mutation Analysis∗
Technical Report 2006-519
Jeremy S. Bradbury, James R. Cordy, Juergen Dingel
School of Computing, Queen’s University
Kingston, Ontario, Canada
{bradbury, cordy, dingel}@cs.queensu.ca
October 2006
Abstract
Current mutation analysis tools are primarily used to compare different test suites and are tied to a
particular programming language. In this paper we present the ExMAn experimental mutation analysis
framework – ExMAn is automated, general and flexible and allows for the comparison of different quality
assurance techniques such as testing, model checking, and static analysis. The goal of ExMAn is to allow
for automatic mutation analysis that can be reproduced by other researchers. After describing ExMAn, we
present a scenario of using ExMAn to compare testing with static analysis of temporal logic properties.
We also provide both the benefits and the current limitations of using our framework.
1 Introduction
Mutation [Ham77, DLS78] has been used in the testing community for over 25 years and is traditionally
used to evaluate the effectiveness of test suites. Moreover, mutation provides a comparitive technique for
assessing and improving multiple test suites. A number of empirical studies (e.g., [ABL05, DR05]) have
relied on using mutation as part of the experimental process.
Although mutation as a comparative technique has been used primarily within the testing community,
it does have application in the broader area of quality assurance and bug detection techniques. Our work
is based on the idea that mutation can be used to assess testing (e.g., random testing, concurrent testing
with tools such as IBM’s ConTest), static analysis (e.g., FindBugs, Jlint, PathInspector), model checking
(e.g., Java PathFinder, Bandera/Bogor), and dynamic analysis. For example, previously we proposed us-
ing mutation to compare sequential testing with property based static analysis using Path Inspector and to
compare concurrent testing using ConTest with model checking [BCD05]. The goal of comparing different
techniques using mutation is to better understand any complementary relationship that might exist and to
use the assessment to design improved hybrid techniques to detect bugs. A combined approach to verifying
concurrent Java recently used manual mutants to develop a hybrid analysis approach using code inspection,
static analysis (FindBugs, Jlint), dynamic analysis and testing [LDG+04].
We propose a generalized approach to experimental mutation analysis (see Figure 1) in which all of
the components and artifacts can be interchanged with other components and artifacts. This generalized
∗This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).