A population-based investigation of participation rate and self-selection bias in momentary data capture and survey studies

Abstract

Participant selection bias is of concern to researchers conducting surveys of all types. For momentary data capture studies, such as Ecological Momentary Assessment, the level of burden associated with these techniques and the possibility of low uptake rates makes the concerns especially salient. This study invited 3,000 individuals to participate in a study of health and mood and recorded the uptake rates at various points in the process. Respondents expressing interest in participating in general were randomized into a one-time survey, a low-burden momentary study, or a high-burden momentary study. Overall, 85.9% of the sample did not respond to the study invitation (including confirming non-interest); 6.9% of the sample expressed interest in the study by completing a brief survey; 2.1% agreed to participate in the study when the protocol specifics were described (none of the study protocols were actually run). Whites were more likely to complete the survey. Of those completing the survey, individuals who reported higher income, a more “open” personality, better typing skills, better computer skills, who viewed the research topic as important, and who expressed interest in research on daily feelings more likely consented to being enrolled in the experiment. The number of prior surveys taken had an inverted-U shaped association with participation in this study. Finally, all individuals randomized to the one-time survey group agreed to participate compared to two-thirds of individuals in the momentary groups. These results suggest that participant selection bias may affect both one-time survey and momentary data capture studies, with the caveat that the degree of such bias will be related to a study’s hypotheses.