A Scalable t-wise Coverage Estimator

Abstract

Owing to the pervasiveness of software in our modern lives, software systems have evolved to be highly configurable. Combinatorial testing has emerged as a dominant paradigm for testing highly configurable systems. Often constraints are employed to define the environments where a given system under test (SUT) is expected to work. Therefore, there has been a sustained interest in designing constraint-based test suite generation techniques. A significant goal of test suite generation techniques is to achieve t-wise coverage for higher values of t. Therefore, designing scalable techniques that can estimate t-wise coverage for a given set of tests and/or the estimation of maximum achievable t-wise coverage under a given set of constraints is of crucial importance. The existing estimation techniques face significant scalability hurdles. The primary scientific contribution of this work is the design of scalable algorithms with mathematical guarantees to estimate (i) t-wise coverage for a given set of tests, and (ii) maximum t-wise coverage for a given set of constraints. In particular, we design a scalable framework ApproxCov that takes in a test set mathcal{U}, a coverage parameter t, a tolerance parameter varepsilon, and a confidence parameter delta, and returns an estimate of the t-wise coverage of mathcal{U} that is guaranteed to be within (1pm varepsilon) -factor of the ground truth with probability at least 1-delta. We design a scalable framework ApproxMaxCov that, for a given formula mathsf{F}, a coverage parameter t, a tolerance parameter varepsilon, and a confidence parameter delta, outputs an approximation which is guaranteed to be within (1pmvarepsilon) factor of the maximum achievable t-wise coverage under mathsf{F}, with probability geq 1-delta. Our comprehensive evaluation demonstrates that ApproxCov and ApproxMaxCov can handle benchmarks that are beyond the reach of current state-of-the-art approaches. We believe that the availability of ApproxCov and ApproxMaxCov will enable test suite designers to evaluate the effectiveness of their generators and thereby significantly impact the development of combinatorial testing techniques.