Creating a Learning Health System through Rapid-Cycle, Randomized Testing

Abstract

Last year at NYU Langone Health, we showed millions of best-practice alerts in the electronic health record system to prompt physicians to avoid adverse events and to promote guideline-based care. We called hundreds of patients to remind them that they were overdue for their annual physical examination. We made approximately 19,000 postdischarge telephone calls to patients in an attempt to reduce their risk of re-admission. We sent thousands of letters to remind patients of unmet preventive care needs. In addition , we started a community health worker program in the emergency department to connect hundreds of high-risk patients to outpatient care. Collectively, these programs alone cost our institution more than a million dollars and used resources that potentially could have been used in other ways to improve care and outcomes. Until recently, we had no real idea whether any of these efforts were working. Health care systems typically implement such interventions wholesale because they seem like good ideas. To our knowledge, they rarely formally evaluate the effectiveness of these interventions , let alone rigorously perform iterations of tests for improvement. At best, a hospital may track outcomes over time in the hope of seeing a benefit. However, such before-and-after analyses are typically limited by secular trends, selection biases, regression to the mean, loss to follow-up, lack of control groups, inconsistent implementation, different concurrently implemented interventions, and a host of other real-world challenges. Evaluations are rarely unbiased enough for personnel at health care systems to be confident that a program works; conversely, ineffective programs are routinely continued for years for lack of persuasive evidence that they are failing. In January 2018, with seed funding provided by a hospital trustee, we began to upend this status quo and turn NYU Langone Health into a learning health system through rapid-cycle, ran-domized tests of existing systems-level programs (i.e., randomized quality-improvement projects). A learning health system is characterized by "continual improvement and innovation" with "new knowledge captured as an integral by-product of the delivery experience." 1 We now know with confidence that changing the text of a provider-targeted prompt to give tobacco cessation counseling in an office produces a significant increase in rates of medication prescriptions and that changing just a few sentences in telephone outreach scripts can both shorten telephone calls and increase rates of appointments for annual examinations. We have also learned that our postdischarge telephone calls have made no difference in rates of readmission or patient-experience ratings, that our appointment reminder letters were completely ineffective, and that our community health worker program was inadvertently targeting patients who were unlikely to benefit (Table 1). Interestingly, the randomized quality-improvement projects have also uncovered unrecognized systems errors: for instance, the influenza vaccination alert was inappropriately being triggered in the operating room, and the algorithm used to identify patients with mental health disorders who were at high risk for visiting the emergency department included low-risk diagnoses such as nicotine dependence. In just 1 year, we have completed 10 random-ized quality-improvement projects, and the learning health system program has already shown that it can pay for itself through increased adoption of preventive services. The value of the program , however, lies beyond short-term quantifi-able return on investment. By learning that many of the interventions we had regarded as routine are not working, we can iteratively test until they become effective, or, if appropriate, we can reassign staff to perform different interventions that are more effective. We think of studies that show The New England Journal of Medicine is produced by NEJM Group, a division of the Massachusetts Medical Society.