Annotation-Based Input Modeling for Combinatorial Testing

Abstract

Combinatorial testing (CT) is an efficient and effective black-box testing technology, combinining mathematically guaranteed input space coverage with comparatively small test sets. However, it requires a current and complete model of all input parameters to a system under test (SUT), their respective value domains, and any constraints between parameters. This factor greatly hinders the adoption of CT in real-world development settings, as creating and maintaining an input parameter model requires signficant effort and is often not sufficiently integrated into relevant workflows in the face of software evolution. To alleviate this drawback, we propose an annotation-based method that aims to improve the locality of model information. It allows developers to define parameter values as well as constraints in immediate vicinity to function or method definitions, enabling them to incorporate modeling into their workflows with minimal overhead. Required oracles are implemented following a common structure and interface, permitting flexible evaluation of results while retaining low complexity for common cases. By incorporating the automated generation and execution of combinatorial test sets into continuous integration processes, our method streamlines the practical application of CT and thus aims to facilitate the industrial adoption of this high-assurance testing approach. A practical implementation targeting the Kotlin programming language serves as the basis for our evaluation, which verifies the applicability of our method when incorporated into an existing medium-sized codebase. At the same time, it offers directions for future work, including improvements regarding stateful testing in object-oriented languages.