Coding quality in manual content analysis: An exploration of coder characteristics and category types for crowdworkers and student coders

Abstract

Although the quantitative content analysis is one of the most important methods in empirical social research, the coding process receives little attention. This is particularly concerning in light of current developments, such as the rise of the use of crowdworkers as coders. Therefore, this study aims to shed light on the coding process by examining a) how the coding quality in terms of reliability and validity differs between student coders, which are traditionally often used for coding, and increasingly used crowdworkers, b) how coder characteristics such as personality traits and sociodemographics influence the coding quality, and c) how manifest and latent category types, that are coded with varying levels of difficulty for coders, impact coding quality. To test these research questions, 300 tweets on the topic of abortion were coded by both students and crowdworkers. A descriptive comparison reveals that the validity in both coder groups is sufficient for manifest, i.e. simple, category types but inadequate for latent, i.e. difficult, category types. Regarding reliability, student coders outperform crowdworkers slightly, particularly when more stringent criteria such as Krippendorff's alpha are considered. The results show that the coder characteristics have only a minor impact on quality, while category types have a significant impact.