Predicting mortality in patients treated differently: updating and external validation of a prediction model for nursing home residents with dementia and lower respiratory infections

Abstract

Objective: To evaluate whether a model that was previously developed to predict 14-day mortality for nursing home residents with dementia and lower respiratory tract infection who received antibiotics could be applied to residents who were not treated with antibiotics. Specifically, in this same data set, to update the model using recalibration methods; and subsequently examine the historical, geographical, methodological and spectrum transportability through external validation of the updated model. Design: 1 cohort study was used to develop the prediction model, and 4 cohort studies from 2 countries were used for the external validation of the model. Setting: Nursing homes in the Netherlands and the USA. Participants: 157 untreated residents were included in the development of the model; 239 untreated residents were included in the external validation cohorts. Outcome: Model performance was evaluated by assessing discrimination: area under the receiver operating characteristic curves; and calibration: Hosmer and Lemeshow goodness-of-fit statistics and calibration graphs. Further, reclassification tables allowed for a comparison of patient classifications between models. Results: The original prediction model applied to the untreated residents, who were sicker, showed excellent discrimination but poor calibration, underestimating mortality. Adjusting the intercept improved calibration. Recalibrating the slope did not substantially improve the performance of the model. Applying the updated model to the other 4 data sets resulted in acceptable discrimination. Calibration was inadequate only in one data set that differed substantially from the other data sets in case-mix. Adjusting the intercept for this population again improved calibration. Conclusions: The discriminative performance of the model seems robust for differences between settings. To improve calibration, we recommend adjusting the intercept when applying the model in settings where different mortality rates are expected. An impact study may evaluate the usefulness of the two prediction models for treated and untreated residents and whether it supports decision-making in clinical practice.