Investigating the influence of data splitting on the predictive ability of QSAR/QSPR models

Abstract

The study was aimed at investigating how the method of splitting data into a training set and a test set influences the external predictivity of quantitative structure-activity and/or structure-property relationships (QSAR/QSPR) models. Six models of good quality were collected from the literature and then redeveloped and validated on the basis of five alternative splitting algorithms, namely: (i) a commonly used algorithm ('Z:1'), in which every zth (e.g. third) from the compounds sorted ascending (according to the response values, y) is selected into the test set; (ii-iv) three variations of the Kennard-Stone algorithm and (v) the duplex algorithm. The external validation statistics reported for each model served as a basis for the final comparison. We demonstrated that the splitting techniques utilizing the values of molecular descriptors alone (X) or in combination with the model response (y) always lead to the development of the models yielding better external predictivity in comparison with the models designed with methodologies based on the y values only. Moreover, we showed that the external validation coefficient (Q EXT2) is more sensitive to the splitting technique than the root-mean-square error of prediction (RMSEP). This difference becomes especially important when the test set is relatively small (between 5 and 10 compounds). In the case of the models trained/validated with a small number of compounds, it is strongly recommended that both statistics (Q EXT2 and RMSEP) be taken into account for the external predictivity evaluation. © 2011 The Author(s).