Towards Continuous Integration for Combinatorial Testing

Abstract

Combinatorial testing is an efficient black-box approach that permits practitioners to pseudo-exhaustively cover the input space of a system under test. It offers mathematically guaranteed coverage up to a user-defined strength while requiring a small number of test cases. Despite these advantages, industrial uptake of this technique has been slow, not least because of the significant investment required to construct and maintain an accurate input parameter model, create reliable oracles and automate testing processes. This work introduces a hierarchy of embeddings of combinatorial testing into continuous integration and deployment pipelines for use in real-world software development workflows. It further describes the practical implementation of a combinatorial security testing pipeline, enabling automated detection of SQL injection vulnerabilities throughout software evolution. Finally, it details lessons learned throughout the design, deployment and utilization of the resulting processes.