A Mixed-Method Investigation Into Measurement Reactivity to the Experience Sampling Method: The Role of Sampling Protocol and Individual Characteristics

Abstract

Since the introduction of the experience sampling method (ESM), there have been concerns that the repeated assessments typically related to this method may alter the behavior, thoughts, or feelings of participants. Previous studies have offered mixed results with some studies reporting reactive changes, while others failed to find such effects. Our aim was to investigate under which circumstances ESM induces reactive effects. Students (N = 151) were randomly assigned to receive a questionnaire containing 30 or 60 items three, six, or nine times per day for 14 days. A random sample of 50 participants took part in qualitative interviews after the end of the data collection. We investigated changes over time in the data, while taking into account the sampling protocol and characteristics of participants, and analyzed qualitative reports of measurement reactivity. Decreases in completion time, within-person variance of ratings and subjective reports of habituation point toward the existence of a habituation period. While participants reported increases in emotional awareness in interviews, ESM measures indicated a decrease in emotional awareness over time. Changes in behavior were rare in quantitative and qualitative reports. Positive affect was decreasing over time in the ESM data, and various changes in affect, emotion regulation, and thoughts were reported in interviews. Individual characteristics and sampling protocol had inconsistent effects on changes over time. The results suggest that ESM induces changes in within-person variability, completion times, affect, or emotional awareness over time. Further research is needed to explore whether observed changes affect the validity of ESM data.