Man vs. Machine: Comparing Physician vs. Electronic Health Record–Based Model Predictions for 30-Day Hospital Readmissions

Abstract

Background: Electronic health record (EHR)-based readmission risk prediction models can be automated in real-time but have modest discrimination and may be missing important readmission risk factors. Clinician predictions of readmissions may incorporate information unavailable in the EHR, but the comparative usefulness is unknown. We sought to compare clinicians versus a validated EHR-based prediction model in predicting 30-day hospital readmissions. Methods: We conducted a prospective survey of internal medicine clinicians in an urban safety-net hospital. Clinicians prospectively predicted patients’ 30-day readmission risk on 5-point Likert scales, subsequently dichotomized into low- vs. high-risk. We compared human with machine predictions using discrimination, net reclassification, and diagnostic test characteristics. Observed readmissions were ascertained from a regional hospitalization database. We also developed and assessed a “human-plus-machine” logistic regression model incorporating both human and machine predictions. Results: We included 1183 hospitalizations from 106 clinicians, with a readmission rate of 20.8%. Both clinicians and the EHR model had similar discrimination (C-statistic 0.66 vs. 0.66, p = 0.91). Clinicians had higher specificity (79.0% vs. 48.9%, p < 0.001) but lower sensitivity (43.9 vs. 75.2%, p < 0.001) than EHR model predictions. Compared with machine, human was better at reclassifying non-readmissions (non-event NRI + 30.1%) but worse at reclassifying readmissions (event NRI − 31.3%). A human-plus-machine approach best optimized discrimination (C-statistic 0.70, 95% CI 0.67–0.74), sensitivity (65.5%), and specificity (66.7%). Conclusion: Clinicians had similar discrimination but higher specificity and lower sensitivity than EHR model predictions. Human-plus-machine was better than either alone. Readmission risk prediction strategies should incorporate clinician assessments to optimize the accuracy of readmission predictions.