0465 Performance of Revised Machine Learning Models for Prediction of Non-Diagnostic Home Sleep Apnea Tests

Abstract

Introduction: Home sleep apnea testing (HSAT) is a less expensive modality than in-lab polysomnography (PSG). Most patients with non-diagnostic HSAT studies (AHI < 5/hr) should undergo in-lab PSG due to the risk of false negatives. We previously developed machine learning models to predict which patients are at high risk of a non-diagnostic HSAT using validated patient questionnaires and anthropomorphic measurements collected during routine clinical practice as inputs. The current study extends this work, examining revised machine learning models that incorporate several additional questionnaire items. Methods: Data from 415 patients undergoing HSAT within the Greater Los Angeles VA were analyzed. Pre-HSAT surveys included the Epworth Sleepiness Scale, Insomnia Severity Index, Patient Health Questionnaire-2, STOPBANG and 13 De novo items (e.g. use of sleep aids and caffeine, sleep schedule, nap frequency, diagnosis of post-traumatic stress disorder [PTSD]). Data were split into 85% training and 15% testing sets. Missing data (2.3%) was imputed using a k-nearest neighbors algorithm. Random forest and gradient-boosted decision tree (GBDT) models were trained using the Partial Area Under the Precision-Recall Curve (pAUPRC) as the optimization metric. Repeated k-fold cross-validation was used in addition to validation using the heldout test set. Results: The GBDT model yielded the best result (pAUPRC on test dataset: 0.628 for GBDT versus 0.606 for random forest). Using the preferred calibration threshold, this model achieved a sensitivity of 0.43 and specificity of 0.96 on the test dataset. The following items significantly improved predictive model performance: 1) rank order representation of patient-reported bedtime and waketime, 2) binary representation of PTSD presence or absence, 3) frequency of sleep aid medication use, 4) number of caffeinated drinks per day, and 5) number of naps per week. These De novo items carried greater predictive value than all other model inputs with the exception of age, body mass index (and its components) and collar size. Conclusion: This study identifies several new questionnaire items that demonstrate promise for inclusion in machine learning models as important predictors of non-diagnostic HSAT results.