Lessons learned from replicating a randomized control trial evaluation of an app-based sexual health program

Abstract

This study presents findings from a randomized control trial replication evaluation of Pulse, an app-based pregnancy prevention program implemented with Black and Latinx women aged 18–20, a population with high rates of unplanned pregnancy. We used social media advertisements to enroll 1013 women online across the U.S. and automatically randomized participants to either the Pulse reproductive health app or a general health control app, stratifying by age and race/Latinx ethnicity. Participants received reminder text messages to view the app as well as text messages with app-related content throughout the intervention. Linear probability models were conducted on the analytic sample of 871 participants who completed the six-week survey and 798 who completed the six-month survey and adjusted for permuted block randomization and multiple hypothesis testing. Compared to the control group, intervention group participants had higher contraceptive knowledge (p = 0.000), which replicates findings from an earlier evaluation. However, these impacts were not sustained at six-month follow-up (p = 0.162). We found no other significant program impacts. This contrasts with an earlier evaluation that found intervention participants were less likely to have had sex without a hormonal or long-acting reversible contraceptive (LARC) method and had greater self-confidence to use contraception consistently than the control group. Different demographic characteristics, lower app usage, and more negative attitudes about and usage of hormonal/LARC contraception in the current sample may help to explain fewer impacts than the earlier evaluation.