Receiver operating characteristic prediction for classification: Performances in cross-validation by example

Abstract

The stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluated as predictors for metastasis in patients with colorectal cancer, was investigated. Data belonging to patients admitted with the diagnosis of colorectal cancer from January 2014 until September 2019 in a single hospital were used. There were 1688 patients eligible for the study, 418 in the metastatic stage. All investigated inflammatory ratios proved to be significant classification models on both the full models and on cross-validations (AUCs > 0.05). High variability of the cut-off values was observed in the unrestricted and restricted split (full models: 4.255 for NLR, 2.745 for dNLR and 255.56 for PLR; random splits: cut-off from 3.215 to 5.905 for NLR, from 2.625 to 3.575 for dNLR and from 134.67 to 335.9 for PLR), but with no effect on the models characteristics or performances. The investigated biomarkes proved limited value as predictors for metastasis (AUCs < 0.8), with largely sensitivity and specificity (from 33.3% to 79.2% for the full model and 29.1% to 82.7% in the restricted splits). Our results showed that a simple random split of observations, weighting or not the patients with and whithout metastasis, in a ROC analysis assures the performances similar to the full model, if at least 70% of the available population is included in the study.