Investigating Best Practices for Ecological Momentary Assessment: Nationwide Factorial Experiment

Abstract

Background: Ecological momentary assessment (EMA) is a measurement methodology that involves the repeated collection of real-time data on participants’ behavior and experience in their natural environment. While EMA allows researchers to gain valuable insights into dynamic behavioral processes, the need for frequent self-reporting can be burdensome and disruptive. Compliance with EMA protocols is important for accurate, unbiased sampling; yet, there is no “gold standard” for EMA study design to promote compliance. Objective: The purpose of this study was to use a factorial design to identify optimal study design factors, or combinations of factors, for achieving the highest completion rates for smartphone-based EMAs. Methods: Participants recruited from across the United States were randomized to 1 of 2 levels on each of 5 design factors in a 2×2×2×2×2 design (32 conditions): factor 1—number of questions per EMA survey (15 vs 25); factor 2—number of EMAs per day (2 vs 4); factor 3—EMA prompting schedule (random vs fixed times); factor 4—payment type (US $1 paid per EMA vs payment based on the percentage of EMAs completed); and factor 5—EMA response scale type (ie, slider-type response scale vs Likert-type response scale; this is the only within-person factor; each participant was randomized to complete slider- or Likert-type questions for the first 14 days or second 14 days of the study period). All participants were asked to complete prompted EMAs for 28 days. The effect of each factor on EMA completion was examined, as well as the effects of factor interactions on EMA completion. Finally, relations between demographic and socioenvironmental factors and EMA completion were examined. Results: Participants (N=411) were aged 48.4 (SD 12.1) years; 75.7% (311/411) were female, 72.5% (298/411) were White, 18.0% (74/411) were Black or African American, 2.7% (11/411) were Asian, 1.5% (6/411) were American Indian or Alaska Native, 5.4% (22/411) belonged to more than one race, and 9.6% (38/396) were Hispanic/Latino. On average, participants completed 83.8% (28,948/34,552) of scheduled EMAs, and 96.6% (397/411) of participants completed the follow-up survey. Results indicated that there were no significant main effects of the design factors on compliance and no significant interactions. Analyses also indicated that older adults, those without a history of substance use problems, and those without current depression tended to complete more EMAs than their counterparts. No other demographic or socioenvironmental factors were related to EMA completion rates. Finally, the app was well liked (ie, system usability scale score=82.7), and there was a statistically significant positive association between liking the app and EMA compliance. Conclusions: Study results have broad implications for developing best practices guidelines for future studies that use EMA methodologies.