Assessing the External Validity of Using News Websites as Experimental Stimuli

Abstract

As news is increasingly distributed online, studies of how people behave on news websites have become important tools in understanding news exposure. Numerous studies have created websites as stimuli, asked experimental subjects to browse the website, then inferred that their behavior in the experimental context resembles their behavior in a natural setting. The external validity of these studies, or the degree to which online behavior in an experiment can generalize outside of the experiment, remains unclear. This study explores the external validity of two such online experiments by comparing the results to field tests. Across two different experiments on a major Canadian and a major U.S. news website redesign, we find that results from the online experiment offered data that were directionally the same as the field tests. Our results suggest that online experiments can offer a generally reliable picture of actual online behavior.