Treatment format and external validity in international relations experiments

Abstract

Political scientists use short vignettes or mock news stories to embed treatments in experimental survey designs. We investigate whether the choice of format entails a trade off between internal and external validity. On the one hand, short vignettes may improve internal validity by isolating key variables without overloading respondents with information, thereby mitigating satisficing and improving data quality. On the other hand, mock news stories may improve external validity by approximating the circumstances under which individuals consume political information, which may enhance the credibility of the information. We find no evidence, however, that short vignettes mitigate satisficing more than mock news stories. Nor do we find that mock news stories enhance individual perceptions of the credibility of the information. Instead, we find that short vignettes are susceptible to confounding, which mock news story mitigate. These findings have important implications for debates about the use of survey experiments in political science.