Identifying factors associated with user retention and outcomes of a digital intervention for substance use disorder: a retrospective analysis of real-world data

Abstract

Objectives: Successful delivery of digital health interventions is affected by multiple real-world factors. These factors may be identified in routinely collected, ecologically valid data from these interventions. We propose ideas for exploring these data, focusing on interventions targeting complex, comorbid conditions. Materials and Methods: This study retrospectively explores pre-post data collected between 2016 and 2019 from users of digital cognitive behavioral therapy (CBT)—containing psychoeducation and practical exercises—for substance use disorder (SUD) at UK addiction services. To identify factors associated with heterogenous user responses to the technology, we employed multivariable and multivariate regressions and random forest models of user-reported questionnaire data. Results: The dataset contained information from 14 078 individuals of which 12 529 reported complete data at baseline and 2925 did so again after engagement with the CBT. Ninety-three percent screened positive for dependence on 1 of 43 substances at baseline, and 73% screened positive for anxiety or depression. Despite pre-post improvements independent of user sociodemographics, women reported more frequent and persistent symptoms of SUD, anxiety, and depression. Retention—minimum 2 use events recorded—was associated more with deployment environment than user characteristics. Prediction accuracy of post-engagement outcomes was acceptable (Area Under Curve [AUC]: 0.74–0.79), depending non-trivially on user characteristics. Discussion: Traditionally, performance of digital health interventions is determined in controlled trials. Our analysis showcases multivariate models with which real-world data from these interventions can be explored and sources of user heterogeneity in retention and symptom reduction uncovered. Conclusion: Real-world data from digital health interventions contain information on natural user-technology interactions which could enrich results from controlled trials.