Decision curve analysis confirms higher clinical utility of multi-domain versus single-domain prediction models in patients with open abdomen treatment for peritonitis

Abstract

Background: Prediction modelling increasingly becomes an important risk assessment tool in perioperative systems approaches, e.g. in complex patients with open abdomen treatment for peritonitis. In this population, combining predictors from multiple medical domains (i.e. demographical, physiological and surgical variables) outperforms the prediction capabilities of single-domain prediction models. However, the benefit of these prediction models for clinical decision-making remains to be investigated. We therefore examined the clinical utility of mortality prediction models in patients suffering from peritonitis with a decision curve analysis. Methods: In this secondary analysis of a large dataset, a traditional logistic regression approach, three machine learning methods and a stacked ensemble were employed to examine the predictive capability of demographic, physiological and surgical variables in predicting mortality under open abdomen treatment for peritonitis. Calibration was examined with calibration belts and predictive performance was assessed with the area both under the receiver operating characteristic curve (AUROC) and under the precision recall curve (AUPRC) and with the Brier Score. Clinical utility of the prediction models was examined by means of a decision curve analysis (DCA) within a treatment threshold range of interest of 0–30%, where threshold probabilities are traditionally defined as the minimum probability of disease at which further intervention would be warranted. Results: Machine learning methods supported available evidence of a higher prediction performance of a multi- versus single-domain prediction models. Interestingly, their prediction performance was similar to a logistic regression model. The DCA demonstrated that the overall net benefit is largest for a multi-domain prediction model and that this benefit is larger compared to the default “treat all” strategy only for treatment threshold probabilities above about 10%. Importantly, the net benefit for low threshold probabilities is dominated by physiological predictors: surgical and demographics predictors provide only secondary decision-analytic benefit. Conclusions: DCA provides a valuable tool to compare single-domain and multi-domain prediction models and demonstrates overall higher decision-analytic value of the latter. Importantly, DCA provides a means to clinically differentiate the risks associated with each of these domains in more depth than with traditional performance metrics and highlighted the importance of physiological predictors for conservative intervention strategies for low treatment thresholds. Further, machine learning methods did not add significant benefit either in prediction performance or decision-analytic utility compared to logistic regression in these data.