Nimble Approaches to Curriculum Evaluation in Graduate Medical Education

Abstract

G old-standard approaches to curriculum evaluation in medical education are well established 1–4 and possess considerable legitimacy among educators. However, despite their apparent validity, these traditional methods are often resource intensive and time consuming and can require specialized training that faculty may lack. 5,6 Given today's resource-constrained climate, there is a need for speedier and more nimble approaches. In this issue of the Journal of Graduate Medical Education, Willett and colleagues 7 describe the use of the ecological momentary assessment (EMA) to evaluate the internal medicine ambulatory morning report during a period of almost 3 years at the University of Alabama at Birmingham. An evaluation methodology rooted in behavioral medicine, EMA is designed to assess ''complex and temporally dynamic psychological, behavioral, and physiological processes in the natural environment.'' 8(p35) The EMA involves repeated sampling of individuals in real time, thus providing immediate evaluation data and minimizing recall bias. 9 The study by Willett and colleagues 7 is one of the first reports of the use of EMA in graduate medical education. Willett et al 7 conducted a prospective study of 125 internal medicine residents attending ambulatory morning report during a 32-month period. 7 The authors created an 8-item EMA tool that assessed a resident's views of individual morning report sessions by including their opinion of session content, structure, and learning attained. This tool was administered immediately following each morning report session (3 times per week) and took residents less than 1 minute to complete. Assessments were anonymous and approximately 75% of residents responded, on average, across sessions. During the first 12 months of the EMA data collection, the investigators discovered that senior residents viewed morning reports as less educationally valuable compared with more-junior residents. In response to these data, teaching faculty implemented a new morning report format with content of a higher cognitive level; senior residents' EMA scores improved during the subsequent 6 months. Thus, this study demonstrates successful use of the EMA to direct curriculum evaluation and provides ''proof of concept'' for the use of this approach in graduate medical education. Important limitations of the study include the inability to account for clustering of assessments within residents (because of the complete anonymity of assessments), and EMA scores were globally high and subject to ceiling effects, as observed for many assessments in medical education.