The Search for Solid Ground in Text as Data: A Systematic Review of Validation Practices and Practical Recommendations for Validation

Abstract

Communication research frequently applies computational text analysis methods (CTAM) to detect and measure social science constructs. However, the validity of these measures can be difficult to assess. In addition, there are hardly any established standards and little guidance for researchers on how to best validate CTAM. But how do these challenges affect current validation practices in applied research? And what practical recommendations for better validation of text-based measures can we derive? To answer these questions, we conducted a systematic review of current validation practices and qualitative expert interviews. We focused on political communication, a subfield that has arguably played a pioneering role in embracing the application of CTAM in communication research. Our results show that researchers apply a great variety of validation steps, which, however, are rarely selected based on a unified understanding of validity. The qualitative interviews further reinforce this notion, as interviewees bemoan a lack of established guidelines and frameworks for validation. Based on our empirical findings, we then derive practical recommendations to guide researchers on how to approach validation conceptually. Additionally, we provide insight into ongoing research that focuses on applied validation frameworks, designed to provide hands-on guidance to researchers involved in CTAM validation.