The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis

Abstract

Importance: Risk prediction models for patients with suspected sepsis have been derived on and applied to various outcomes, including readily available outcomes such as hospital mortality and ICU admission as well as longer-Term mortality outcomes that may be more important to patients. It is unknown how selecting different outcomes influences model performance in patients at risk for sepsis. Objectives: Evaluate the impact of outcome selection on risk model performance and weighting of individual predictor variables. Design, Setting, and Participants: We retrospectively analyzed adults hospitalized with suspected infection from January 2014 to September 2017 at 12 hospitals. Main Outcomes and Measures: We used routinely collected clinical data to derive logistic regression models for four outcomes: hospital mortality, composite ICU length of stay greater than 72 hours or hospital mortality, 30-day mortality, and 90-day mortality. We compared the performance of the models using area under the receiver operating characteristic curve and calibration plots. Results: Among 52,184 admissions, 2,030 (4%) experienced hospital mortality, 6,659 (13%) experienced the composite of hospital mortality or ICU length of stay greater than 72 hours, 3,417 (7%) experienced 30-day mortality, and 5,655 (11%) experienced 90-day mortality. Area under the receiver operating characteristic curves decreased when hospital-based models were applied to predict 30-day (hospital mortality = 0.88-0.85;-0.03, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90-0.81;-0.09) and 90-day mortality (hospital mortality = 0.88-0.81;-0.07, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90-0.76;-0.14; all p < 0.01). Models were well calibrated for derived (root-mean-square error = 5-15) but not alternate outcomes (root-mean-square error = 8-35). Conclusions and Relevance: Risk models trained to predict readily available hospital-based outcomes in suspected sepsis show poorer discrimination and calibration when applied to 30-And 90-day mortality. Interpretation and application of risk models for patients at risk of sepsis should consider these findings.