An Evaluation of Test Suite Minimization Techniques

Abstract

As a software project evolves over time, the associated test suite usually grows with it. If test suites are not carefully maintained, this can easily result in massive test execution duration, reducing the benefits of regression testing because faults are found later in development or even after release. Test suite minimization aims to combat long running test suites by removing redundant test cases. Previous work mainly evaluates test suite minimization techniques based on comparably small projects, which are less practically relevant. In this paper, we compare four test suite minimization techniques by applying them to several open source software projects and evaluate the results. We find that the size and execution time of all the test suites can be reduced by over 70% on average. However, there is a substantial loss in fault detection capability of, on average, around 12.5%, restricting the applicability of this form of test suite minimization.