Engagement and retention: Measuring breadth and depth of participant use of an online intervention

Abstract

Background: The Internet provides us with tools (user metrics or paradata) to evaluate how users interact with online interventions. Analysis of these paradata can lead to design improvements. Objective: The objective was to explore the qualities of online participant engagement in an online intervention. We analyzed the paradata in a randomized controlled trial of alternative versions of an online intervention designed to promote consumption of fruit and vegetables. Methods: Volunteers were randomized to 1 of 3 study arms involving several online sessions. We created 2 indirect measures of breadth and depth to measure different dimensions and dynamics of program engagement based on factor analysis of paradata measures of Web pages visited and time spent online with the intervention materials. Multiple regression was used to assess influence of engagement on retention and change in dietary intake. Results: Baseline surveys were completed by 2513 enrolled participants. Of these, 86.3% (n = 2168) completed the follow-up surveys at 3 months, 79.6% (n = 2027) at 6 months, and 79.4% (n = 1995) at 12 months. The 2 tailored intervention arms exhibited significantly more engagement than the untailored arm (P