Stress Testing SMT Solvers via Type-aware Mutation

Abstract

This paper introduces type-aware mutation, a simple, but effective methodology for stress testing Satisfiability Modulo Theories (SMT) solvers. The key idea is mutating the operators of the formula to generate test inputs for differential testing, while considering the types of the operators to ensure the mutants are still valid. The realization of type-aware mutation was evaluated on finding bugs in two state-of-the-art SMT solvers, Z3 and CVC4. During the three months of empirical evaluation, 101 unique, confirmed bugs were found by type-aware mutation, and 87 of them have been fixed. The testing efforts and bugs were well-appreciated by the developers.