Determining the best prediction accuracy of software maintainability models using auto-WEKA

Abstract

Highly accurate prediction of software maintainability models is a significant requirement to achieve software quality assurance. Development an accurate prediction model may involve on trying several types of machine learning models with different configurations that include tuning parameters and selected features. However, this is a difficult and very time-consuming task to implement. In this paper, we report on the experience of using a new, rapid automated tool to identify the best prediction accuracy of a software maintainability model, namely Auto-WEKA, applied to sets of different models with various configurations. Auto-WEKA is applied to five datasets collected from real-world open-source software systems. The mean magnitude relative error (MMRE) value is used to evaluate the accuracy of predictive models, along with ZeroR model to compare selected model performance with the baseline. The results obtained from this study provide empirical evidence of the ability of Auto-WEKA to identify the best model to predict software maintainability. Auto-WEKA selected the following as the best prediction models: SMOreg in Eclipse JDT Core dataset, RandomSubSpace in Eclipse PDE UI dataset, KStar in Equinox Framework dataset, RandomForet in Lucene and Mylyn datasets. These selected models achieved a low MMRE value and improved performance of the accuracy prediction over the baseline. The proposed tool was found useful in identifying the best model in predicting software maintainability.