The continuous net benefit: assessing the clinical utility of prediction models when informing a continuum of decisions

Abstract

The net benefit and decision curve analysis are increasingly being used to assess the clinical utility of prognostic models. This metric assesses the value added by a model’s predictions when individuals are treated differently according to whether they are over or under a chosen threshold. Although such ‘treat or not’ decisions are common, prognostic models are also often used to tailor and personalise the care of patients, which implicitly involves the consideration of multiple interventions at different risk thresholds. We aim to extend decision curve analysis to estimate the net benefit of a model over multiple thresholds. We take a weighted area under a rescaled version of the net benefit curve, deriving the continuous net benefit. In addition to the consideration of a continuum of interventions, we also show how the continuous net benefit can be used to evaluate single treatments in populations with a range of optimal thresholds, due to individual variations in expected treatment benefit or harm, highlighting limitations of current proposed methods that calculate the area under the decision curve. We propose this not as a substitute for decision curves, but as a complementary evaluation metric, in lieu of single-threshold point estimates. We showcase this metric through two examples of model validation in cardiovascular preventive care. The continuous net benefit brought additional insight over point estimates when comparing models over a range of decisions. The continuous net benefit informs those looking to validate clinical prediction models of their clinical utility, and helps decision makers understand their usefulness, improving their viability towards implementation.