Regression models for performance ranking of configurable systems: A comparative study

Abstract

Finding the best configurations for a highly configurable system is challenging. Existing studies learned regression models to predict the performance of potential configurations. Such learning suffers from the low accuracy and the high effort of examining the actual performance for data labeling. A recent approach uses an iterative strategy to sample a small number of configurations from the training pool to reduce the number of sampled ones. In this paper, we conducted a comparative study on the rank-based approach of configurable systems with four regression methods. These methods are compared on 21 evaluation scenarios of 16 real-world configurable systems. We designed three research questions to check the impacts of different methods on the rank-based approach. We find out that the decision tree method of Classification And Regression Tree (CART) and the ensemble learning method of Gradient Boosted Regression Trees (GBRT) can achieve better ranks among four regression methods under evaluation; the sampling strategy in the rank-based approach is useful to save the cost of sampling configurations; the measurement, i.e., rank difference correlates with the relative error in several evaluation scenarios.