HybridCISave: A Combined Build and Test Selection Approach in Continuous Integration

Abstract

Continuous Integration (CI) is a popular practice in modern software engineering. Unfortunately, it is also a high-cost practice - Google and Mozilla estimate their CI systems in millions of dollars. To reduce the computational cost in CI, researchers developed approaches to selectively execute builds or tests that are likely to fail (and skip those likely to pass). In this article, we present a novel hybrid technique (HybridCISave) to improve on the limitations of existing techniques: to provide higher cost savings and higher safety. To provide higher cost savings, HybridCISave combines techniques to predict and skip executions of both full builds that are predicted to pass and partial ones (only the tests in them predicted to pass). To provide higher safety, HybridCISave combines the predictions of multiple techniques to obtain stronger certainty before it decides to skip a build or test. We evaluated HybridCISave by comparing its effectiveness with the existing build selection techniques over 100 projects and found that it provided higher cost savings at the highest safety. We also evaluated each design decision in HybridCISave and found that skipping both full and partial builds increased its cost savings and that combining multiple test selection techniques made it safer.